Bonded deficit round robin scheduling for multi-channel networks

Computer Networks 55 (2011) 3503–3516

Contents lists available at ScienceDirect

Computer Networks journal homepage: www.elsevier.com/locate/comnet

Bonded deficit round robin scheduling for multi-channel networks Dessislava Nikolova ⇑, Chris Blondia PATS Research Group, Dept. Math. and Computer Science, University of Antwerp, Middelheimlaan, 1, B-2020 Antwerp, Belgium Interdisciplinary Institute for BroadBand Technology (IBBT), Belgium

a r t i c l e

i n f o

Article history: Received 9 May 2010 Received in revised form 13 January 2011 Accepted 7 July 2011 Available online 22 July 2011 Keywords: Scheduling Multiple channels Access networks Wired networks DOCSIS3.0 Channel bonding

a b s t r a c t In order to increase the link capacity in telecommunication networks the bandwidth of multiple channels can be aggregated by transmitting on them simultaneously. The latest data-over-cable service interface specification (DOCSIS 3.0) for hybrid fiber coax networks defines a mechanism for channel bonding at the link layer. Thus, the scheduler at the cable modem termination system, which distributes the packets on the network, not only has to support per-flow queuing but also has to distribute the packets to one modem over possibly several channels. In this article we propose two downstream multi-channel packet scheduling algorithms designed to support scheduling amongst flows possibly using different numbers of channels. Both algorithms are based on the deficit round robin (DRR) scheduler. The bonded deficit round robin (BDRR) algorithm, has complexity dependent only on the number of the channels and requires only one queue per flow. It is shown that the algorithm is a latency-rate server and the latency is derived. Furthermore, BDRR bounds the packet reordering and the maximum bounds on the packet delay and the reorder buffer needed at the receiver are calculated. The paper explores also a second algorithm which has more similarities with load balancing algorithms. It uses fully independent channel schedulers thus avoiding the need for modification in the single channel DRR algorithm. The transmission channel for a packet is selected upon its arrival. However, the algorithm does not bound the latency and packet reorder for flows assigned to receive on multiple channels. Flows for which such bound is needed should be assigned on a single channel. Ó 2011 Elsevier B.V. All rights reserved.

1. Introduction One of the currently most deployed wired broadband access network solutions is the hybrid fiber coax (HFC) network. It uses the legacy community antenna television cables and is a point-to-multipoint network with tree topology. The terminal equipment placed at the root of a HFC network is referred to as a cable modem termination system (CMTS). It is connected via coaxial or both optical and coaxial cable to cable modems (CM) situated at the customer premises. There is a family of data-over-cable

⇑ Corresponding author at: London Center for Nanotechnology, Gordon Street 17-19, WC1H 0AH, London, UK. Tel.: +44 7980 589518. E-mail address: [email protected] (D. Nikolova). 1389-1286/$ - see front matter Ó 2011 Elsevier B.V. All rights reserved. doi:10.1016/j.comnet.2011.07.007

service interface specifications (DOCSIS) which standardize the physical and the medium access control (MAC) layers and also the QoS support. The speed of a DOCSIS-based network depends on the modulation used on the physical layer. For example a network based on the DOCSIS 2.0 standard [1] offers effective downstream bandwidth (from the CMTS to the CMs) of approximately 40 Mb/s provided 256QAM modulation is used. Increasing further the modulation would increase the speed but is technologically expensive. Another access network technology, the FTTx (fiberto-the-home, fiber-to-the-business or fiber-to-the-curb), uses optical fiber and has gained lots of momentum in the recent years. The most promising fiber network is considered the passive optical network (PON). There exist a number of different PON types like ATM PON (APON)

3504

D. Nikolova, C. Blondia / Computer Networks 55 (2011) 3503–3516

and Ethernet PON (EPON) to name some of them. They are also point-to-multipoint networks and offer very high speeds. For example the throughput of EPON approaches 1 Gb/s [2]. In order to be able to provide higher bandwidths to compete with the FTTx offerings a new technology was specified in the DOCSIS 3.0 specification for HFC networks [3]. Particularly DOCSIS 3.0 defines a mechanism of forwarding upstream and downstream packets between the CMTS and the CM by utilizing the bandwidth of multiple physical layer channels. In this way the throughput can be significantly increased without increasing the modulation and still use the legacy cable. The mechanism is termed channel bonding and is realized on MAC level. By balancing the CMs amongst the channels the full capacity of a channel bonded system can be utilized. However DOCSIS 3.0 significantly expands the downstream service offering by requiring the DOCSIS 3.0 CM to be capable of receiving and transmitting on multiple channels simultaneously. The individual CMs can have different capacities in terms of the number of channels they can receive simultaneously. The legacy CMs can receive on only one channel, while the DOCSIS 3.0 CMs on 2, 3 and more channels. An example of a system with 4 channels is given on Fig. 1. There are 4 CMs reached by these channels. However, the modems have different capacities in terms of the number of channels they can receive on simultaneously and are balanced amongst the channels. CM1 has 4 receivers and is assigned to all channels. CM2 and CM4 have each 2 receivers and are assigned correspondingly to the bonding groups (BG): BG1 consisting of frequencies f1 and f2 and BG2 consisting of f2 and f3. Finally, CM 3 has only one receiver and is assigned to f4. Thus a packet transmitted on channel f1 will be received by CM1 and CM2, while a packet transmitted on channel f2 by CM1, CM2 and CM4. Similarly a packet transmitted on channel f3 would be received by CM1 and CM4, and a packet transmitted on f4 will by CM1 and CM3. The CMTS may dynamically change the assignment of a CM to a different BG. However, this incurs some protocol overhead and delay, thus, does not happen on a short timescale. In this paper we consider only static channel assignment i.e. the CMTS does not dynamically reassign CMs. The scheduler at the CMTS has to distribute the packets over the set of the downstream (DS) channels for delivery

Fig. 1. Frequency assignment.

to a single CM. Each complete packet is transmitted on a single channel. The channels can have different modulation and thus different bit rates. The packets are tagged with a sequence number. In this way the proper packet sequencing is not lost if there are different latencies on the channels. The CM restores the original sequence before forwarding the packets to the user devices. DOCSIS provides quality-of-service (QoS) by mapping packets into service flows. A CM has at least one service flow for each direction. Thus, the scheduler at the CMTS should support per-flow queuing. It is also expected that it is work conserving and utilizes efficiently the available bandwidth on all channels. Existing load balancing (also called striping or load sharing) algorithms for distributing packets over multiple channels, like [4–8], do not schedule amongst flows. Existing multi-channel scheduling algorithms [9–13], for technologies like point to point networks, standardized in [14] and for Ethernet systems [15], presume that all flows can be transmitted on all channels. A schematic view of the system model is given on Fig. 2. These scheduling algorithms use a single scheduler, which selects the next packet to be transmitted according to the scheduling discipline and then sends it on any free channel. In networks where all flows can be transmitted on all channels they are work conserving and fully utilize the available capacity. However, when applied to a system, like the DOCSIS 3.0 HFC network, where the flows can not be distributed over all channels, such disciplines might lead to scheduler blocking. For example, when the single scheduler selects the next flow to transmit a packet, it is possible that none of the free channels is a channel on which this flow can transmit. Thus the scheduler will remain blocked until one of these channels becomes free. The contribution of this article is that it proposes and analyzes two multi-channel scheduling algorithms, which can schedule flows that can receive only on a selection of the available channels. Even though they both use deficit round robin (DRR) as the channel scheduler, they use different types of queuing which results in different performance. The output queuing – DRR (OutQ-DRR) is a simple extension to the DRR for multiple channels. The reason is that it does not require modification to the single channel DRR algorithm. The multi-channel scheduling is achieved by adding a packet distribution process preceding the single channels DRR. Thus the OutQ-DRR is potentially easy and cheap multi-channel scheduling solution. The bonded DRR (BDRR), on the other hand, is an algorithm which modifies the enqueue and dequeue processes of the single channel DRR and as such is more complex to implement. Our analysis shows that the BDRR bounds the latency and packet reordering. These bounds are derived. The packet reordering bound indicates the size of the buffer needed at the client side. The latency suggests the maximum delay a packet from a leaky-bucket shaped flow will experience, which is of importance for time-critical applications. We have also shown in our analysis that OutQ-DRR is not a latency-rate server for flows which can be transmitted on multiple channels. Thus, if the network needs to provide to a flow delay guarantee or limit the packet reordering, it has to be assigned on one channel,


3505

Fig. 2. System model of point-to-point multi-channel networks.

in which case the OutQ-DRR bounds the latency and packet reordering. Before describing the two possible queuing mechanisms in Section 3, the following section contains more details on QoS provisioning in DOCSIS 3.0 and reviews related work including the DRR algorithm. The parameters and variables necessary to be defined for this algorithm and their adaptation for use in multi-channel environment are discussed in Section 4. In Section 5 an algorithm for output queuing is described and in the following section the BDRR algorithm which combines DRR with input queuing of the packets is presented. Theoretical analysis of the BDRR algorithm, its latency and packet reorder are provided and derived in Section 7, followed by simulation results to support the analysis. In the last section the conclusions are drawn.

grant mechanism and the QoS parameters. Often different schedulers are used for the different service types. For example the main requirements for the best effort service scheduler is to have low complexity, guarantee fairness, isolation of the flows and a maximum reserved rate. Thus, any packet round robin scheduler combined with a traffic shaper can be used. The unsolicited grants service has very stringent delay and delay variation requirements and a sorted priority scheduler is more suitable for this service. Moreover, in the upstream concatenation and fragmentation is allowed. Different aspects of the upstream MAC layer and scheduling are studied in [16–19]. Due to the inherent differences in the MAC layer definition between the upstream and the DS transmission, the bonding mechanisms themselves are quite different in the two directions. In this article we concentrate solely on the downstream direction.

2. Background and related work 2.2. Related work 2.1. QoS in DOCSIS 3.0 DOCSIS 3.0 inherits the Quality-of-Service (QoS) support from the older specifications. The QoS support in HFC networks is realized via mapping the packets to service flows (SF) and scheduling these SFs according to a set of QoS parameters. These parameters may include traffic priority, token bucket rate shaping/limiting, reserved (guaranteed) data rate, latency and jitter guarantees. The downstream packets are classified into downstream SFs based on their Ethernet header and/or the TCP/IP/UDP headers as configured by the service provider. Thus, a SF is a unidirectional flow of packets that is provided a particular Quality of Service. A CM has at least one SF for each direction referred to as primary SF and can have many different SF. The scheduler at the CMTS arbitrates the allocation of the DS bandwidth amongst the different SFs. Thus a DOCISIS 3.0 scheduler should be able to support per flow queuing. The MAC layer differs significantly for the downstream (DS) and the upstream (US). In the US the CMTS arbitrates the transmissions from the CM via grants. The standard allows for five different scheduling service types ranging from best effort, which uses request via contention channel to unsolicited grants service, where the CMTS allocates bandwidth on regular intervals. They differ on the request

There are a significant number of per flow queuing algorithms for single channel reported in the literature. Sorted priority scheduling algorithms like WF2Q+ [20], and STFQ [21], have complexity depending on the number of flows/ users to be scheduled. Round robin algorithms like DRR [22], LBFS DRR [23], Stratified RR[24], EBRR [25], Pre-order DRR [26], ERR [27], have low complexity which makes them preferred choice for high-speed networks. From them DRR is the most studied and deployed in some high-speed routers [28]. According to the DRR algorithm each flow contending for resources is assigned a quantum and a deficit counter (DC). The quantum indicates the portion of the resources in a round robin cycle a flow should get. The DC tracks the amount of service the flow can still receive in a round. The flows are serviced in a round robin order. Each round a flow is visited once. Upon a visit to a flow, its DC is increased with its quantum. A packet is transmitted if its size is not more than the DC. After a packet is sent the DC is decreased with the size of the packet. In this way if parts of the deficit counter remained unused in a round it will still be available for the next one. Only backlogged flows are serviced. To realize this, the DRR scheduler maintains a list of all backlogged flows. When a flow is no longer backlogged it is removed from the list and its DC is set to 0.

3506


When a flow becomes backlogged it is added at the tail of the list. The DRR is a fair scheduler. It provides rate guarantees when the quantums are assigned proportionally to the flow’s required rate. Link aggregation is a common technique to increase the available capacity. For point to point networks, it is standardized in [14] and for Ethernet systems in [15]. It is analogous to multi-server systems, which arise also in multiprocessor architectures, multi-path storage I/O and cluster-based Web servers, to name a few. Load balancing algorithms, also referred to as load sharing or stripping algorithms, are used to share in a fair manner the load amongst the servers or channels. There is a plethora of load balancing algorithms proposed, ranging from simple static policies, such as random distribution or round-robin policy [29], to ones using hashing [6], incorporating prediction schemes [5] or resulting in minimal [8] or completely ordered output [7]. These load balancing algorithms however, do not perform flow scheduling. For example, for the striping algorithm proposed in [29], the surplus round robin (SRR) scheduler [4] is used. For each channel it keeps a surplus counter, which tracks the amount of bytes dispatched on the channel in a round. Note that it does not schedule amongst flows but amongst channels. Algorithms supporting per-flow queuing for multi-link systems are designed to schedule for point-to-point systems, multiple processor and for Ethernet based networks. A multi server DRR has been proposed in [13]. It works in the same manner as a single channel DRR but just transmits the packet on the first available channel. In networks where all flows can be transmitted on all channels it is work conserving and fully utilizes the available capacity. However, as pointed out in the Section 1, such a straightforward extension of the DRR to the multi-channel system with flows, which can be transmitted only on a selection of the available channels might result in an inefficient utilization. When the next flow is selected for service, the channels on which it can transmit packets might not be free. Thus, the scheduler will be blocked until one of these channels becomes free. In [11] the theoretical framework for the evaluation of the sorted-priority multichannel scheduling algorithms is set. The authors consider first multi-channel scheduling which emulates single channel. However, as it is discussed in the same reference such a scheduler is not work conserving. They propose and analyze also a sorted priority scheduler, which is work conserving, presuming that all flows can send packets on all available channels. The problem of provisioning QoS among competing flows over a system of links was addressed in [9]. The scheduling algorithm, multi server fair queueing (MSFQ) is based on the generalized processor sharing (GPS) [30] system, an idealized service discipline, that is also representative of a perfectly fair system. MSFQ simply selects the next packet that would leave in the associated GPS system and sends it on the next available link. The complexity of the algorithm is determined from the complexity of the Packet GPS which is O (logN), where N is the number of the backlogged flows. In [10] the MSFQ algorithm is further extended by dividing the scheduling process into two steps. Each time

a link becomes free and there has been a change in the set of backlogged flows firstly the partitioning step is executed. It assigns flows to channels and corresponding weights for each flow for each channel it is assigned to. In the second step packet GPS is used to schedule amongst the flows. This algorithm could be easily extended to be applicable to systems where the flows can be distributed only over a selection of the available channels. However, it has very high complexity, namely O(MlogM + NlogN), where M designates the number of channels and N the number of flows. In [12] a packet scheduling algorithm for parallel processing of packets in network nodes that can handle multiple flows is proposed. It uses a combination of sorted priority algorithm with surplus and deficit round robin. Again it presumes that all flows can be scheduled on all processors and there is one scheduler to do the task. To ensure strict ordered processing of the packets the scheduler keeps a sorted list of the processors in decreasing order of the number of bytes to be scheduled on each processor and a second sorted list in which the flows are ordered decreasingly in the amount of bytes they can send in the round. It is shown through simulations that the algorithm will result in a minimal packet reordering. However, no theoretical framework is provided to show whether the algorithm is a latency-rate server or the amount of reordering incurred with this scheduling mechanism. Furthermore the algorithm presumes that all packets can be scheduled on all processors and is not readily extended to a system where this is not allowed. The next section describes two possible queuing mechanisms applicable for multi channel systems and discusses their advantages and disadvantages.

3. Per-flow packet queuing in multi-channel networks In order to ensure QoS in networks per-flow queuing is necessary. In multi-channel architecture the packets to a bonded flow should be distributed amongst the channels of its BG. In this respect the scheduler might keep separate queues per channel per bonded flow or only one queue per flow as for single channel schedulers. Further in this section we will discuss the two possibilities. Fig. 3 shows a schematic view of a compound scheduling architecture, further on referred to as output queuing. It consists of a packet distributing process and a separate queue per flow for each channel. The distributing process

Packets

Packets Distributor

Fig. 3. Output queuing.


decides upon a packet arrival, to which channel the packet should be transmitted. The channel should belong, of course, to the bonding group of the flow the packet belongs to. Once the channel is determined, the packet becomes the responsibility of the channel scheduler, which can be any per flow queuing algorithm. Thus the packet distributor acts as a stripping algorithm. The rule by which the packets are distributed to the channels will also largely determine the packet delay. A disadvantage of output queuing is that it determines the channel, on which a packet is transmitted, at the time of the arrival of the packet in the system. It bases its decision on the information available at this moment or some estimation for the future load on the channels. However, the load on the channels can change when the packets are not transmitted in FIFO order. Thus, a situation might arise where there are many packets waiting to be transmitted on one channel while other channels, which reach the same flows, have no packets queued for transmission. Another major disadvantage of output queuing is that with each packet stored in the queue its sequence number also has to be stored. The advantage of output queuing is that the different channels operate completely independently, they have separate memory and existing scheduling algorithms can be applied without the need for modification. Fig. 4 shows a schematic drawing of the distributed architecture for input queuing. In this scenario the scheduler keeps only one queue per flow where the packets are queued immediately upon arrival. The channel, on which a packet will be transmitted, is determined just before the transmission itself starts. Each channel has a separate per flow-queuing scheduler. However, these channel schedulers are not independent. When a channel becomes free its scheduler selects the next flow based on the available information. Once the flow is selected the common memory pool of the packet queues is accessed. When a scheduler on one channel leaves a queue empty, in order to avoid unnecessary checks by the other channel schedulers, they should be informed. In this sense the channel schedulers are not fully independent. On the figure this dependence is indicated by sketching a central control

unit, which stores and updates the variables, common to all channels and manages the queues. It is also responsible for any concurrency issues. The advantage of input queuing is that the decision for the channel on which a packet will be transmitted is made at the moment of packet transmission. Thus, it is based on accurate up-to-date information about the state of all channels. Another advantage is that the packet order is preserved until transmission thus there is no need for extra buffer space to store the packet sequence number. Further on two algorithms are proposed, which use as channel schedulers the deficit round robin (DRR) algorithm. Before describing them, preliminary discussion on the quantum selection for each channel is given. 4. Rate partitioning The DRR scheduling is determined by a quantum per flow. It indicates the portion of the resources in a round robin cycle a flow should get and hence is related with the rate at which the flow can send packets. Consider that each flow has a pre-assigned weight wi and corresponding rate ri. For the common DRR algorithm these rates would be sufficient to determine the quantum. However for the bonded version we need to introduce flow’s channel weights in order to determine the flow’s quantum on a channel. In a single channel DRR scheduler the flow’s quantum is proportional to the flow’s weight and written for the flow’s quantums for the DRR scheduler on channel m is m m Qm i ¼ wi Q min :

SF identifiers

ð1Þ

The flow’s weights on the channels, wm i , are such that their sum should result in the flow’s weight in the system, which would correspond to the weight the flow would have if it was served by a single channel scheduler

wi ¼

X

wm i ;

ð2Þ

m2Mi

where Mi is the set of channels of the flow’s bonding group and Mi is the number of channels in the flow’s bonding group. The weights determine a rate by m m rm i ¼ wi r min :

Packet queues

3507

ð3Þ

The minimum rate corresponds to the minimum weight that can be assigned to a flow on a channel. A natural choice is that the minimum weight is 1 and is the same for all channels. In this way any flow with the minimum weight and a single channel in its bonding group can be assigned on any channel. Thus r m min ¼ r min . From Eq. (2) the rates satisfy

ri ¼

X

rm i :

ð4Þ

m2Mi

Packets

Fig. 4. Intput queuing.

The minimum quantum in DRR is typically the minimum possible quantum for which the scheduler has O(1) complexity, which is the maximum packet size on the network Lmax. The minimum quantums on the channels are naturally selected as the minimum possible quantum for which the single channel DRR scheduler has O(1) complexity,

3508


m which is the same for all channels Q m min ¼ Q min . A frame f is one round robin cycle amongst the backlogged flows on channel m. The sum of the quantums determines the frame size and taking into account Eqs. (1) and (3) is expressed as

fm ¼

Nm X

Qm i ¼

i¼1

Nm X rm i Q min ; r i¼1 min

ð5Þ

where N is the number of backlogged flows assigned to bonding groups which include channel m. 5. DRR for channel bonded system with output queuing (OutQ-DRR) An output queuing algorithm, as discussed in Section 3, is determined by the type of the scheduler (s) on the channels and by the packet distributing rule. The packet distributor acts as a load sharing algorithm. Typically in the existing load sharing algorithms there is no channel scheduler but FIFO queuing is used. The simplest algorithm distributes the packet to the channel with the smallest FIFO queue relative to the channel rate. This is roughly equivalent to the packet being distributed to the channel where it will be scheduled with the shortest delay. In the proposed OutQ-DRR algorithm the scheduler on each channel is DRR. The quantum for each channel is given with Eq. (1). The distribution rule aims at selecting the channel on which the packet will have the least waiting time. The waiting time on a channel depends on the amount of packets in the queue for this flow and the bandwidth share the flow will receive. At the moment of arrival of a packet for flow i the fair share of the bandwidth ~rm i for this flow on channel m is

wm rm i Rm ¼ P i m Rm ; m mw j2N j2N m r j j

ð6Þ

where Nm is the number of flows which have backlogged queues on channel m. Thus the waiting time for a packet for flow i on channel m provided there is no change in the backlogged flows for this channel is given with m

~dm ¼ qi ¼ i ~r m i

qm i

P

rm i

m j2Nm r j m

R

;

ð7Þ

where qm i denotes the size of the flow’s queue on channel m if the packet is added to this queue. The distribution rule proposed in this article is that the packet distributor forwards the packets to the channel with the least estimated waiting time ~ dm i at the moment of packet arrival

m ¼ argmin ~dm : i

1. i = p.Flow (); 2. Mi ¼ i:BondedChannelsSetðÞ 3. for each m 2 Mi 4. IF(i NOT IN BacloggedFlowsListm) m m i Þ ~m ¼ qi ðmqm þr 5. d i r Rm i

m

~r m P i ¼

Table 1 Packets distributing process.

6.

ELSE qm qm i ~ 7. dm i ¼ rm Rm i dm 8. m ¼ minM ~ i 9. IF(i NOT IN BacloggedFlowsListm) 10. qm þ ¼ rm i ; 11.p.stamp(SequenceNumber) 12.SequenceNumber + + ;

of channels Mi from its bonding group. For each channel m it keeps a variable qm indicating the total reserved rate of the flows having packets to be transmitted on this channel. Upon arrival, a packet is classified to a service flow, say i. The classification is based on the destination address or other parameters from the packets headers. The waiting times on each channel are subsequently estimated. As already discussed they depend on the queue size at the channel, qm i , and on the sum qm of the reserved rates of the flows backlogged on the channel. The process knows the set of channels in the bonding group of the flow and from it the channel with lowest ~ dm ~ is selected. If there are several channels with the same dm one is selected randomly. This is important feature in order to achieve spreading of a burst of packets amongst equally loaded channels from the flows’ bonding group. Provided that the flow is not backlogged on the selected channel its channel reserved rate, r m i , is added to qm. Before the packet is forwarded to a channel it has to be stamped with the sequence number. The packet distributing process is the one in the system that can keep track of the order of packet arrival. When a flow i is no longer backlogged on a channel m, the packet distributing process is informed such that it can reduce the qm variable with r m i . As can be seen from the pseudo code the process complexity depends on the number of channels in the bonding group of a flow thus the complexity is O(M). Here M indicates the number of channels on the system. For the DRR algorithms on the channels perform in constant time, thus the whole OutQ-DRR algorithm has O(M) complexity.

6. DRR for channels bonded systems with input queueing

ð8Þ

~ dm i is an estimated time because it can differ from the actual delay dm i if the set of the backlogged flows for this channel changes. Thus, if the set of the backlogged flows on the channels does not change the packet will have as an actual delay ~ dm i . Moreover they will be delivered in order because the size of the packet is taken into account in the estimated delay. In Table 1 the pseudo code for the packet distributing process is given. For each flow the process keeps the set

As was discussed in Section 3 in order to avoid unnecessary checks a distributed algorithm in input queuing architecture can not have independent schedulers. They are bonded by using a common memory pool for the packet queues and, as will be disclosed further, other common variables. To emphasize this fact the algorithm is named bonded deficit round robin (BDRR). It is described hereafter. As an input queuing algorithm, BDRR keeps one packets queue per flow. For each channel m there is a separate DRR


scheduling algorithm with BackloggedFlowsm list and per flow a quantum given by Eq. (1) and a deficit counter. Consider flow i which requires a minimum rate ri and is assigned to a bonding group consisting of Mi channels. When a packet arrives for a flow with an empty queue, a channel from the flow’s bonding group is selected randomly and the scheduler for this channel is notified. When a channel is notified the flow’s id, i, is added to the BackloggedFlows list of the channel’s DRR scheduling algorithm. If a second packet arrives before the first one is served and Mi > 1, a second channel from the bonding group is notified. With the further increase of the number of packets in the flow’s queue more channels are notified until all the channels from the flow’s bonding group are notified. Thus if there are pi packets in a flow’s queue the number of notified channels ni is

ni ¼ minðpi ; M i Þ:

ð9Þ

This condition guarantees that when a flow is selected for service from a channel scheduler, it will have at least one packet in its queue to be transmitted. The channels’ schedulers have to ensure that after a packet is transmitted the condition given by Eq. (9) is satisfied. For example if the transmission of a packet from the flow’s queue will cause the number of packets in the queue pi to become less than the number of notified channels minus one ni 1, the scheduler should no longer schedule packets from this flow. The flow should be removed from the BackloggedFlows list of the channel, thus the number of notified channels will be reduced by one and will become equal to pi. A flow can be selected for service on more than one channel at the same time. It is presumed possible to transmit different packets from the same queue on several channels simultaneously. This can be realized when the two packets are stored in different memory blocks which can be read simultaneously. Next, the algorithm is further clarified by means of its pseudo code but first the variables used are introduced. They are listed in Table 2. For each channel the algorithm keeps a separate BackloggedFlowsm list where the flows, which have packets to be scheduled on the channel, are kept. For each flow there is only one queue Queuei, where the packets classified for the flow are inserted. Per flow there are also the DRR variables, namely the flow’s quantum Q m i and deficit counter DC m i , assigned as discussed in Section 4. The BDRR does not notify all channels from the flow’s bonding group unless there are sufficient packets in the flow’s queue to have

3509

at least one packet transmitted per channel. Each flow keeps an indicator UnusedChannelsi of the channels of its bonding group which are not notified yet, i.e., the flow’s id is not added in their BackloggedFlows lists. A channel is removed from the set when the flow has notified the scheduler of this channel that it has backlogged packets. When the flow has no more packets for the specified channel it is inserted back into the set. The UnusedChannels set can be realized for example as a list to which channel IDs are added or removed or as a bitmap where a bit is set or reset when a channel is used or not. A variable is kept per flow ni indicating how many channels at the moment are notified that the flow has packets to transmit, i.e., the flow’s id is in their corresponding BackloggedFlows lists. Thus the sum of ni and the number of channels in the set UnusedChannelsi results in the number of channels in the bonding group of flow i. Finally there is the counter sequencenumber which is used to tag the packets so their order can be recovered at the destination. 6.1. Enqueue process The pseudo code of the enqueue process is given in Table 3. This function is invoked upon a packet arrival at the BDRR scheduler. When a packet arrives at the system it is classified into a flow, say i, and inserted in its corresponding Queuei. If there are channels on which the flow is not considered backlogged, one is selected and notified that the flow has packets to transmit. Consequently, the flow’s id is added it to the BackloggedFlows list of the channel and the variables as shown on lines 6 and 7 of the enqueue process are updated. The selectchannel function searches in the UnusedChannels set for a channel that has no scheduled transmission at the moment, which in the worst case results in O(M) operations. If such a channel is not present, one is selected randomly. Thus the complexity of the herein implemented enqueue process is O(M). The process can be optimized in a future version in a way that the channel selection is based on some indicators like load or number of backlogged flows. However there is no straightforward way to select the channel on which the flow will receive the fastest service without keeping extra variables. The selectchannel function can also be implemented as a simple FIFO which would result in O(1) complexity but some efficiency will be lost as it can happen that the selected channel to be informed is currently busy. 6.2. Dequeue process

Table 2 Bonded DRR variables. Per channel: BackloggedFlows lists – lists with backlogged flows per channels Per channel per flow: Queuei – pointer to the packets queue; m Qm i ; DC i – DRR variables for channel m; Per flow: queue - the packets queue; UnusedChannelsi – set, identifying the ids of the unused channels ni – the number of notified channels; SequenceNumberi – a counter to track the packets sequence;

The Dequeue process is called independently at each channel scheduler at the events of a flow being added to Table 3 BDRR enqueue process. 1. i = Packet Flow (); 2. Queuei Insert(packet); 3. IF(UnusedChannels > 0) 4. m = selectChannel(UnusedChannels) 5. BackloggedFlows[m] pushBack(i); 6. ni + + ; m 7. DC m i ¼ Qi ;

3510


Table 4 BDRR Dequeue process for channel m. 1. WHILE (Active list for channel m NOT empty) 2. i = BackloggedFlowsLists[m] popFront (); 3. while(Queuei NOT empty AND nextPacketLength = ni) 4. packet = Queuei popHead(); 5. packet stamp(SequenceNumberi) 6. SequenceNumberi + + 7. send (packet); m 8. DC m i ¼ DC i packet LengthðÞ; 9. IF(Queuei NOT empty AND Size(Queuei) > = ni) 10. BackLoggedFlowsList[m] pushBack(i); m 11. DC m i þ ¼ Qi ; 12. ELSE 13. addChannel(m, UnusedChannels); 14. ni ; 15. DC m i ¼ 0;

a channel’s empty BackloggedFlows list or when a packet finishes transmission on the channel. The process remains active as long as the list is not empty. The pseudo code is given in Table 4. After selecting the next flow i from the BackloggedFlows list its packets are processed until three conditions are met. The first, obvious one is that there are packets in the queue of the flow. The second one is that the length of the next packet is less than or equal to the deficit counter. These conditions are the same as for the DRR scheduler. The third condition is that there are at least the same number of packets in the queue as the number of channels on which the flow has scheduled transmission, given by ni. This ensures that when the same flow is selected for service on a different channel there will be at least one packet to be transmitted. As long as the conditions are met packets are popped from the flow’s queue stamped with the sequence number and sent on the channel. The deficit counter for this flow for the channel is reduced by the size of the transmitted packet and the sequence number counter is increased by one. When any of the conditions no longer true holds the process moves on to update the variables. If the only condition which is not met is the next packet not fitting in the deficit counter, then the flow is inserted at the back of the channels BackloggedFlows list for another round of service and the deficit counter is increased with the quantum for this channel. Otherwise the flow does not require any more service from the channel. This case covers the situation where either there are no more packets in the queue or the number of packets is less than the channels on which the flow is considered backlogged. In both cases the flow should no longer be considered backlogged for this channel thus its deficit counter is set to 0, the channel is added to the UnusedChannels set and the counter of notified channels is decreased. Note that, as there are common variables read and updated by the different channel schedulers, concurrency issues must be accounted for in order to avoid deadlocks [31]. To ensure the correct working when a scheduler checks the queue size for flow i on line 3 it should be able to continue to line 4 without any other scheduler accessing

the same Queuei. The same should hold for lines 5 and 6. Also the operations between lines 12(9) and 14 are performed without other channel schedulers accessing the ni variable. The longest operation is the sending of the packet on line 7. Thus the scheduler has to ensure that this operation can be done in parallel, i.e., that when a packet is transmitted on one channel the dequeue process on the other channels can be executed simultaneously. All the other operations perform in constant time thus any locks on lines 3–6 and 12–14 will pose very low overhead in comparison with the packet transmission. The reported algorithm for the dequeue process guarantees that the condition given by Eq. (9) is satisfied after it finishes. The algorithm has O(1) complexity. Thus the total complexity of the BDRR is O(M). 7. Theoretical analysis For flows, which can receive packets only on a single channel, both algorithms behave like single channel DRR. In order to be able to guarantee the rate of these flows the following inequality must hold for each channel m Nm X

m rm i 6 R ;

ð10Þ

i¼1

where Rm is the channel’s link rate, Nm is the number of flows which have a reserved rate on the channel as given by Eq. (4). The frame size from Eq. (5) is bounded by

fm 6

Rm Q ¼ Fm; rmin min

ð11Þ

where Fm denotes the maximum frame size. These are the same relations between the parameters as those for DRR scheduling on single channels. Thus, selecting the rate partitions as given by Eq. (10) would guarantee the reserved rates for the flows assigned on only one channel under both algorithms. Note that with OutQ-DRR the packets from one flow are not transmitted in order. BDRR, on the other hand, transmits the packets on the DS link in the order of their arrival. Still some reordering at the receiver is possible due to the different channel rates and packet sizes. That is, while packet k with length Li,k is being transmitted on a channel, packets from flow i with index greater than k could be transmitted. For a given channel, for these packets to arrive before Li,k, their transmission must take less time than the transmission of Li,k. In the worst case, Li,k is transmitted over the slowest channel of capacity Rmin. Thus, it takes Li,k/Rmin seconds to transmit this packet. During this time, L on channel m can be transmitted a total of R i;k Rm bytes of min packets whose index is greater than k. Hence, the number of bytes bi,k of the packets with index greater than k that arrive before Li,k can be at most

bi;k 6

Mi Li;k X R Rm 6 Lmax 1 Rmin Rmin m¼2

ð12Þ

P m bytes. Here R ¼ M is the total capacity of all chanm¼1 R nels. This bound will be achieved when only packets of the considered flow are transmitted on the network. If, as

3511


typicall in practical applications, the maximum rate at which a flow can transmit is restricted by some Rmax, bi,k will be bound by Lmax RRmax 1 . min Following the same consideration and not accounting for the processing delays at both sides, a packet of size Li,k will be delayed in the buffer of the CM due to reordering by at most

dreorder 6

Lmax Li;k : Rmin Rmax

ð13Þ

The second term is the time it takes for the full packet k to be received and the first term is the maximum time it takes a packet with index less than k to arrive at the receiver. Further on, we investigate the delay of the BDRR scheduler and more specifically we calculate the latency and maximum delay bound. In [32] a general model, called latency-rate server (LR-server), for analysis of scheduling algorithms was defined. The theory of LR-servers further detailed in [33] provides means to describe the worst-case behavior of a broad class of scheduling algorithms in a simple and elegant manner. The latency hSi of a scheduling algorithm S is defined in [33] as the minimum non-negative constant that satisfies

W i ðt 0 ; tÞ P max 0; r i t t0 hSi

ð14Þ

where t0 is the start of a busy period, t is any time instance within this period and Wi(t0, t) the amount of service received by session i in the time interval (t0, t). A busy period is defined as the maximal interval of time (s1,s2) such that for any time t 2 (s1, s2), packets arrive for flow i with rate greater or equal than its guaranteed rate ri, or,

Ai ðt 0 ; tÞ P ri ðt t0 Þ:

ð15Þ

With other words the latency of a scheduler is the time that a flow has to wait until it begins receiving service at its guaranteed rate. The LR-server theory has been used to derive the latencies of many scheduling algorithms for example for DRR in [34], for surplus round robin in [35], for virtual clock and self-clocked fair queuing in [33], for Pre-order DRR in [36], for elastic round robin in [27]. From the latency the maximum packet delay Dmax, for a leaky-bucket shaped traffic flow with parameters (ri, ri) transmitted over multiple channels is obtained by

Dmax 6

ri ri

þ hSi þ dreorder :

ð16Þ

For the multi channel BDRR the latency is given from the following theorem Theorem 7.1. Bonded DRR is a latency-rate server with latency determined by

hBondedDRR ¼ max 2 Mi hm;DRR þ i i m

ðM i 1ÞLmax ; ri

where Mi is the number of channels flow i is assigned to, ri is its guaranteed rate, Lmax is the maximum packet size on the network and hm;DRR is the latency of the DRR scheduler on i channel m.

Proof. Consider a channel bonded system with M channels, total bandwidth capacity R and a BDRR scheduler. Let the bandwidth of channel m be Rm and thus P m R¼ M m¼1 R . Suppose (t0,t1) is flow i’s busy period coinciding with the start of a backlogged period at t0, i.e. qi(t0) = 0 and t is some time instant within this busy period t0 < t 6 t1. The bonding group of flow i consists of Mi channels and its reserved rate ri is distributed over these channels resulting in channel rates according to Eq. (4), which are bounded by Eq. (10). The number of channels ni on which the flow is assigned at each moment is given by Eq. (9). Depending on the number of assigned channels ni at time t two cases can be distinguished. Case 1: ni 6 Mi 1. From Eq. (9) follows that there are at most Mi 1 packets in the queue. Thus for the queue state expressed in bits at time t we can write

qi ðtÞ 6 ðM i 1ÞLmax :

ð17Þ

On the other hand the queue state depends on the arrivals and departures by the relation

qi ðtÞ ¼ Ai ðt 0 ; tÞ W i ðt 0 ; tÞ þ qi ðt 0 Þ;

ð18Þ

where Ai(t0, t) is the number of bits that arrived for flow i during the interval (t0, t), Wi(t0, t) is the number of bits that were transmitted, i.e., the amount of service scheduled to flow i in this period and qi(t0) is the queue state at time t0. Expressing Wi from Eq. (18) and taking into account that the queue state at time t0 is 0, the service received by flow i in the interval (t0, t) can be bounded by

W i ðt 0 ; tÞ ¼ Ai ðt 0 ; tÞ qi ðtÞ þ qi ðt 0 Þ P Ai ðt 0 ; tÞ ðM i 1ÞLmax P r i ðt t 0 Þ ðM i 1ÞLmax :

ð19Þ

The first inequality comes from Eq. (17) and qi P 0. The last inequality comes from the definition of a busy period (see Eq. (15)). Case 2: ni = Mi. In this case the queue state at time t can not be bounded. In the period (t0,t) there can be a number of intervals which satisfy this condition. Without loss of generality let tMi be the time instant when the number of packets pi in the queue of flow i becomes Mi and remains greater than or equal to Mi until t. The service received by flow i in the interval (t0,t) can be split with respect to tMi as

W i ðt 0 ; tÞ ¼

Mi X m¼1

Wm i ðt 0 ; t M i Þ þ

Mi X

Wm i ðt Mi ; tÞ:

ð20Þ

m¼1

At time t M i the number of packets in the queue is nqi 6 Mi 1. The same is valid for the number of channels on which the flow is subscribed which corresponds to case 1. Thus the work done in the interval ðt0 ; tMi Þ; W i ðt 0 ; t Mi Þ, is bounded as Eq. (19). Accounting for the considered interval this gives a lower bound on the first term of Eq. (20) in the form

3512


Fig. 5. Illustration of the latency bound.

Mi X m¼1

Mi X

Wm i ðt 0 ; t M i Þ ¼ W i ðt 0 ; t M i Þ ðM i 1ÞLmax P r i t Mi t 0 : ri

m¼1

ð24Þ

ð21Þ

In order to find a bound on the second term of Eq. (20) we have to take into account that the intervals (tm, t) are backlogged intervals for the corresponding channels. Thus the service on each of the channels can be bounded from the bounds for the scheduler on a single channel if such exists. In the proposed Bonded DRR algorithm the scheduler on each channel is DRR. The latency hDRR was derived in [33,34] i based on a backlogged period. Thus the service received on each channel m in the backlogged period ðt Mi ; tÞ is bounded by the DRR latency

m Wm t t Mi hDRR;m : i ðt M i ; tÞ P max 0; r i i

DRR;k Wm : i ðt Mi ; tÞ P max 0; r i t t M i maxk2M i hi

ð22Þ

The service received on channel m in a backlogged period ðtMi ; tÞ is related to the latencies of all channels by the bound

DRR;m m Wm Þ i ðt M i ; tÞ P max 0; r i ðt t Mi hi DRR;k P max 0; r m i mink2Mi t t Mi hi P max 0; r m t t Mi maxk2Mi hDRR;k : i i ð23Þ Summing Eq. (23) over all assigned channels m 2 Mi gives

Replacing the bounds given by Eqs. (24) and (21) in Eq. (20) the lower bound on the service for a period (t0, t) is obtained to be

ðMi 1ÞLmax W i ðt 0 ; tÞ Pmax 0; r i t t 0 ri : maxm2Mi hm i

ð25Þ

From the definition of a LR server (Eq. (14)) the latency is expressed from Eq. (25) as the one stated in the theorem. To conclude the proof and show that this latency bound is tight, an example where it is actually achieved is given. Consider a system with M channels and a flow i assigned to Mi channels. Let all the channels in the set i Mi guarantee the same latency for flow ih1i ¼ h2i ¼ ¼ hM i except channel m which guarantees higher latency, i.e., 1 hm i > hi . Fig. 5 shows an illustration of the considered example for the latency bound of the BDRR algorithm. At time t0 a packet arrives for flow i. Channel 1 is notified and the flow is inserted in its BackloggedFlows1 list. Suppose that in the interval (t0, tm), Mi 1 more packets arrive and the flow receives no service. As a result at time tm all channels from the DCS of the flow are notified with the last being m at time tm. The time instant t 0 þ h1i marks the time when the flow starts to receive service on channel 1 at its guaranteed rate for the channel r 1i . The time instant

3513


t m þ hm i marks the worst case time instant that the flow starts to receive service at its reserved rate rm i for channel it starts to receive its m. In the interval t 0 þ h1i ; t m þ hm i reserved rate also on all the other channels. This follows m from hm i being the maximum latency. Thus t m þ hi is the time instant where the flow starts to receive service at its guaranteed rate ri. For the time interval (t0, tm), Mi 1 packets arrived, which in the worst case is (Mi 1)Lmax bits. As this interval is part of a busy period for the flow the minimum arrival rate in this interval is ri and thus in the worst case

ðM i 1ÞLmax tm t0 6 : ri Thus when this is added to the latency of the m-th channel, which is the channel with maximum latency on the system, it can be easily verified that the latency bound is exactly met. This concludes the proof. h Suppose the maximum channel latency is achieved on channel m, where the DRR parameters are m m m Fm; Q m i ; N ; R ; r i . Replacing the latency of the DRR scheduler derived in [34,18] in the latency expression for BDRR given by Theorem 7.1 results in

hBDRR ¼ i

m Fm Q m Lmax 1 i þ ðN 2ÞðLmax 1Þ þ rm Rm i

þ

ðMi 1ÞLmax : ri

ð26Þ

A number of conclusions can be drawn from the latency bound of the BDRR. Firstly, the latency of the bonded system depends on the latency of the schedulers on the different channels. Thus to minimize the latency of the bonded algorithm the latency on the single channels has to be minimized taking into account the frame length, the channel rate, the number of flows assigned on the channel and the partitioning of the flows’ rates on the channels. Secondly, it indicates that the higher the number of channels a flow is assigned to, the higher the latency is. Thus for a flow with reserved rate of say 4 Mb/s with respect to delay performance it will be better to assign it to only 1 channel than to spread it on 4 channels with reserved rate on each of them of 1 Mb/s. However, if assigned on only one channel the peak rate the flow can get will be restricted. Practically flows from time critical applications like voice or gaming are better assigned on a single channel. Flows from applications with highly variable traffic are better assigned on multiple channels. For flows which can receive on only one channel OutQDRR is a latency rate server as the service received by the flow on the channel, say m, can be bounded by

DRR W i ðt 0 ; tÞ P max 0; r m : i t t 0 hi

ð27Þ

However, for flows which can receive on more than one channel OutQ-DRR is not a latency rate server. Even though the DRR scheduler guarantees a minimum rate on each separate channel a flow is subscribed, the OutQ-DRR does not guarantee that the packets of a flow will be distributed on all channels of its bonding group. As a result the minimum rate allocated for the flow might be less than the

sum of the minimum reserved rates on all assigned channels. Consider a flow with reserved rate ri, with leaky bucket shaped traffic in the backlogged period (t t0)

Ai ðt 0 ; tÞ 6 ri þ r i ðt t0 Þ:

ð28Þ

k di

Let is the moment packet k from flow i is served and aki is the moment of its arrival. Then the amount served in the k period t 0 ; di is equal to the amount of traffic that arrived for flow i in the period t 0 ; aki i.e.

k Ai t 0 ; aki ¼ W i t 0 ; di :

ð29Þ

In the worst case the packets of a backlogged flow will be transmitted on only one channel. Thus, replacing the bounds from Eq. (27) and Eq. (28) results in

: ri þ ri aki t0 P rmi dki t0 hDRR i

ð30Þ

After some simple calculations one obtains for the maximum packet delay for such traffic k

DOutQDRR ¼ di aki 6 max

ri

þ hDRR þ i m

ri

k ri 1 ai t 0 : rm i

ð31Þ

The bound increases with the length of the backlogged per iod aki t0 . Thus time critical flows served under OutQDRR should be assigned to a single channel. The queue length at the time instant t in the worst case will be

qi ðtÞ ¼ Ai ðt 0 ; tÞ W i ðt 0 ; tÞ m DRR 6 ri þ ri rm i ðt t 0 Þ þ r i hi

ð32Þ

and also accumulates with the length of the backlogged period. The latency of the MS-DRR scheduler proposed in [13], discussed in Section 2.2, does not increase with the number of channels a flow can be transmitted. Thus for flows which can use all channels the MS-DRR scheduler provides lower latency than BDRR or OutQ-DRR. 8. Simulation results A variety of simulation results clarifying and comparing the performance of the two algorithms are presented in this section. A HFC network with 4 DS channels and DOCSIS 3.0 MAC protocol is simulated. The simulation program is implemented using the OMNET++ [37] event simulator. The simulated system has 4 channels, each having the same DS bandwidth of Rm = 40 Mb/s resulting in total DS system capacity of R = 160 Mb/s. Further on with the term ‘‘k-channel flow’’ is referred a SF that can transmit simultaneously on k channels i.e. its bonding group is of k channels. Specifically in this study a straightforward rule to assign the flows’ channel rates is selected. Namely, for flows assigned to a bonded group of k channels, the reserved rate on any one of the channels is determined from rm i ¼ r i =k. It is straightforward to see that the rule from Eq. (4) is satisfied. There are 100 active flows from which 8 are 4-channel flows with reserved rate ri = 4 Mb/s distributed equally amongst the 4 channels i.e. r m i ¼ 1 Mb/s. There are also 12 1-channels flows with ri = 4 Mb/s and 80 1-channels

3514


flows with ri = 1 Mb/s divided equally amongst the channels. Thus on each channel there Nm = 31 flows assigned PNm m and the total channel reserved rate is i r i ¼ 40 Mb/s. The minimum quantum Qmin on all channels is set to the maximum packet size which on DOCSIS network is Lmax = 1518 bytes without the overhead. The minimum channel reserved rate rmin = 1 Mb/s. From Eq. (5) we can calculate the frame size of a round for each channel with R = rm to be 12.144 ms. The packet delay is measured from the moment a packet enters the scheduler i.e. after the traffic shaper, if such is present, until it is ready to be transmitted from the CM to its client i.e. after the packet order is restored at the CM. Thus any transmission and reordering delays are also taken into account. For each flow the delay is measured over more than a million packets. Each point on the graphics is obtained by averaging out the packet delay from all flows in the corresponding group. The traffic generation process for each flow, unless otherwise stated, is ON/OFF with exponentially distributed ON times with mean 0.01 s and exponentially distributed OFF times with mean 0.99 s. The inter-arrival time of the packets is exponentially distributed with values depending on the desired load. The rate of the flows is increased correspondingly in order to obtain the desired total load on the channels and is varied in the range [0.33, 0.9] times the flow’s reserved rate. In the first simulated scenario four 4-channel flows have leaky bucket traffic source with maximum burst size r = 4 Mb and generation rate of 4 Mb/s, thus equal to the flows’ reserved rate. For this traffic we can calculate the maximum delay from Eq. (16) taking into account the latency from Eq. (26) and the reorder delay from Eq. (13). For the maximum delay of a leaky bucket shaped 4-channel traffic flow with parameters (1 Mb, 4 Mb/s) and reserved rate 4 Mb/s one obtains Dmax = 1.04 s. The leaky bucket shaped 4-channel flows 0 become active some time after the other flows, and all flows assigned on channel 1 become active 0.01 s after them. On Fig. 6 the average and the maximum packet delay of the 4 leaky bucket shaped 4-channel flows under the BDRR and the OutQDRR scheduling disciplines is shown. As we can see the maximum delay under the BDRR algorithm remains under the calculated maximum limit. There is no limit on the maximum delay under the OutQDRR algorithm and at higher load it becomes higher than the limit for BDRR. To provide better insights of the scheduling mechanisms, which give rise to these delays for the two algorithms Fig. 7 gives the throughput of flow 0, which is from the leaky bucket shaped flows in the first 4 s of its active period. The traffic arrives in bursts of 1 Mb. In the first 0.01 s of flow 0’s active period the 1-channel flows assigned on channel 1 are not yet active. They start their active periods at 1.02 s. Thus in the first 0.01 s of flow 0’s active period only it can transmit on channel 1. Hence the peak in the throughput (see Fig. 7a). The packet distributor of the OutQ-DRR algorithm has to distribute all the packets in the burst to the channels. Correspondingly, the majority of the packets are queued on channel 1 because there are very few other flows with backlogged packets. At time 1.3 s all the packets from flow 0 queued on

Fig. 6. Packet delay of a leaky bucket shaped traffic for 4-channels flow.

Fig. 7. The achieved throughput for one 4-channel flow vs. time under (a) OutQ-DRR and (b) BDRR scheduling.

channels 2, 3, and 4 are transmitted. If the load on channel 1 has not changed the packets on channel 1 would also have been transmitted. However, all other flows with reserved rates on channels 1 became active at time 1.02 s, hence, the rate with which the packets of flow 0 are scheduled on channel 1 after 1.02 s is less than the one estimated at


the time of the arrival of the packets and their delay is higher than the estimated. The BDRR algorithm serves flow 0 on all channels with minimum rate of 1 Mb/s. Consequently by 1.85 s the flow’s backlog is cleared under BDRR (see Fig. 7b), in difference with the OutQ-DRR. This scenario demonstrates that OutQ-DRR fails to utilize the full system capacity when the traffic is bursty. BDRR on the other hand distributes the packets of the 4-channel flows on all 4 channels when there are at least 4 backlogged packets, thus utilizing all the available resources in these periods. When the next burst arrives there will be some remaining packets to be transmitted on channel 1 under OutQ-DRR scheduling, thus more packets in comparison with the previous burst will be scheduled on channels 2, 3 and 4. As can be seen from Fig. 7a, as the time advances it takes longer for the bursts scheduled on these channels to be transmitted. With the further advance of the time the average load on all channels is equalized, moreover it has less variation. In such conditions the OutQ-DRR estimates the expected waiting time well and provides average throughput similar to the throughput under BDRR. This is further confirmed from the results for the packet delay for the traffic from bursty not shaped flows shown on Fig. 8. For both algorithms the average and maximum delays are similar though for the 4-channel flows scheduled under OutQ-DRR the average delay is higher than the one when scheduled under BDRR due to the different way the two algorithms treat bursts. The delay for the one channel flows is of course the same for scheduling under the two algorithms as they both use DRR as the channel scheduler. The low delay for the OutQ-DRR algorithm, similar to the BDRR one, however, is on the expense of large amount of packets transmitted out of order. The size of the reorder buffer at the client side i.e. at the CM is shown on Fig. 9. The size of the buffer under OutQ-DRR scheduling is in the order of a 1 Mbyte for average loads and is dependent on the traffic rate of the flow. The maximum size of the

Fig. 8. Packet delay vs. total load for 4-channel flows with weight wi = 4.

3515

Fig. 9. Reorder buffer size at the receiver.

buffer when BDRR scheduler is used remains less than one maximum packet size.

9. Conclusions The paper proposed two scheduling algorithms derived from the deficit round robin scheduler, applicable for packet scheduling of flows over multiple channels in a point-to-multipoint network. Unlike existing algorithms which presume that all channels can be transmitted on all channels, the proposed algorithms are designed to schedule flows which use different numbers of channels. The OutQ-DRR algorithm, which uses output queuing, requires one queue per flow per channel. The transmission channel is selected upon packet arrival by estimating the delay the packet will endue on each channel and subsequently choosing the one with the lowest value. The packets are not transmitted in order thus, the sequence number must be stored with them. We have shown in our analysis that OutQ-DRR is not a latency-rate server for flows which can be transmitted on multiple channels. Thus if the network needs to provide to a flow delay guarantee or limit the packet reordering it has to be assigned on one channel, in which case the OutQ-DRR bounds the latency and packet reordering. The preferred solution, bonded DRR, uses input queuing hence, it needs only one queue per flow. The packets are transmitted in order and are stamped with the sequence number just before transmission. It is proven that BDRR bounds the latency and packet reorder and the corresponding bounds are derived. The performed simulations show that both algorithms provide low average packet delay. For the OutQ-DRR algorithm, however, this is on the expense of a large amount of packets transmitted out of order resulting in a big reorder buffer at the client side. In the case of multi-channel flows with traffic with large bursts the OutQ-DRR can not guarantee equal distribution of the packets among the channels the flow can receive on. This results in long periods where the full system capacity is not utilized. BDRR on the other

3516


hand distributes the packets from multi-channel flows on all the channels they can use regardless of the traffic burstiness. References [1] Data-over-cable service interface specifications DOCSIS 2.0 radio frequency interface specification (2001). URL . [2] D. Nikolova, B.V. Houdt, C. Blondia, Dynamic bandwidth allocation for EPON with threshold reporting, Telecommunication Systems 28 (1) (2005) 31–52. [3] Data-over-cable service interface specifications DOCSIS 3.0 MAC and upper layer protocols interface specification (2007). . [4] H. Adisshu, G. Parulkar, G. Varghese, A reliable and scalable striping protocol, in: Proceedings ACM SIGCOMM, 1996. [5] J. Guo, L.N. Bhuyan, Load balancing in a cluster-based web server for multimedia applications, IEEE/ACM Transactions on Parallel and Distributed Systems 17 (11) (2006) 1321–1334. [6] W. Shi, M.H. MacGregor, P. Gburzynski, Load balancing for parallel forwarding, IEEE/ACM Transactions on Networking 13 (4) (2005) 790–801. [7] W. Shi, L. Kencl, Sequence-preserving adaptive load balancers, in: Proceedings of the 2006 ACM/IEEE Symposium on Architecture for Networking and Communications Systems, ANCS 2006, 2006. [8] J. Yao, J. Guo, L. Bhuyan, Fair link striping with FIFO delivery on heterogeneous channels, Computer Communications 31 (2008) 3427–3437. [9] Josep M. Blanquer, Banu Ozden, Fair queueing for aggregated multiple links, in: Proceedings of the ACM SIGCOMM, 2001. [10] Satya R. Mohanty, Laxmi N. Bhuyan, On fair scheduling in heterogeneous link aggregated services, in: Proceedings of the ICCCN, 2005. [11] J. Cobb, M. Lin, A theory of multi-channel schedulers for quality of service, Journal of High Speed Networks 12 (12) (2002) 61–86. [12] J. Guo, J. Yao, L. Bhuyan, An efficient packet scheduling algorithm in network processors, in: Proceedings of INFOCOM, 2005. [13] H. Xiao, Y. Jiang, Analysis of multi-server round Robin scheduling disciplines, IEICE Transactions on Communications E87-B (12) (2004) 3593–3602. [14] RFC 1990 – The PPP multilink protocol (MP) August 1996. [15] Carrier sense multiple access with collision detection (CSMA/CD) access method and physical layer specifications 2000. [16] S.K. Wen-Kuang Kuo, C.-C.J. Kuo, Improved priority access, bandwidth allocation and traffic scheduling for DOCSIS cable networks, IEEE Transactions on Broadcasting 49 (4) (2003) 371–382. [17] W. Liao, L.H. jiun Ju, Adaptive slot allocation in DOCSIS-based CATV networks, IEEE Transactions on Multimedia 6 (3) (2004) 479–488. [18] D. Nikolova, Design and performance analysis of scheduling algorithms for cable and optical access networks, Ph.D thesis, University of Antwerp, Belgium January 2010. [19] J.-Y. Jung, J.-M. Ahn, Novel bandwidth scheduling algorithm for docsis 3.0 based multiple upstream channels, in: Proceedings of AccessNets, 2009. [20] J. Bennett, H. Zhang, Hierarchical packet fair queueing algorithms, IEEE/ACM Transactions on Networking 5 (5) (1997) 675–689. [21] P. Goyal, H. Vin, H. Cheng, Start-time fair queuing: a scheduling algorithm for integrated services packet switching networks, IEEE/ ACM Transactions on Networking 5 (5) (1997) 690–704. [22] M. Shreedhar, G. Varghese, Efficient fair queueing using deficit round-Robin, IEEE/ACM Transactions on Networking 4 (3) (1996) 375–385. [23] D. Nikolova, C. Blondia, Last-backlogged first-served deficit round robin (LBFS-DRR) packet scheduling algorithm, in: Proceedings of the 15th International Conference on Networks (ICON2007), 2007. [24] S. Ramabhadran, J. Pasquale, The stratified round robin scheduler: design, analysis and implementation, IEEE/ACM Transactions on Networking 16 (6) (2006) 1362–1373. [25] L. Lenzini, E. Mingozzi, G. Stea, Eligibility-based round robin for fair and efficient packet scheduling in wormhole switching networks, IEEE Transactions Parallel Distribution System 15 (3) (2004) 244– 256. [26] S.-C. Tsao, Y.-D.J. Lin, Pre-order deficit round robin: a new scheduling algorithm for packet-switched networks, Computer Networks 35 (2-3) (2001) 287–305.

[27] S.S. Kanhere, H. Sethu, A.B. Parekh, Fair and efficient packet scheduling using elastic round robin, IEEE Transactions Parallel Distribution System 13 (3) (2002) 324–336. [28] CISCO 12000 series routers. . [29] A. Hari, G. Varghese, G. Parulkar, An architecture for packet-striping protocols, ACM Transactions on Computer Systems 17 (4) (1999) 249–287. [30] A. Parekh, R.G. Gallager, A generalized processor sharing approach to flow control in integrated service networks: the single-node case, IEEE/ACM Transactions on Networking 1 (3) (1993) 344–357. [31] Programming in C, lecture notes by Dave Marshall. . [32] D. Stiliadis, A. Varma, Latency-rate servers: a general model for analysis of traffic scheduling algorithms, IEEE/ACM Transactions on Networking 6 (5) (1998) 611–624. [33] D. Stiliadis, Traffic scheduling in packet-switched networks: Analysis, design, and implementation, Ph.D thesis, University of California at Santa Cruz, USA, June 1996. [34] S. Kanhere, H. Sethu, On the latency bound of deficit round robin’, in: Proceedings ICCCN, Miami, Florida, USA, 2002. [35] D. Nikolova, C. Blondia, Evaluation of surplus round robin scheduling algorithm, in: Proceedings SPECTS’06, Calgary, Canada, 2006. [36] S. Kanhere, H. Sethu, On the latency bound of pre-order deficit round robin’, in: Proceeding of the 27th IEEE Conference on local computer networks (LCN 2002), Tampa, Florida, USA, 2002. [37] OMNET++ simulator. .

Dessislava Nikolova received Master in Physics degree from the University of Sofia, Bulgaria in 1999. She joined Alcatel Bell in 2000 as research engineer on optical access networks. Since 2002 she is a research fellow at the Department of Mathematics and Computer Science at the University of Antwerp, Belgium, where in 2010 she obtained PhD in Computer Science. Her thesis was on design and performance analysis of scheduling algorithms for optical and cable access networks. She has one patent for a medium access protocol for passive optical networks. Her current research interests are Nanonetworks, Plasmonics and new technologies for Optical and Cable Networks.

Chris Blondia obtained his Master in Science and Ph.D. in Mathematics, both from the University of Ghent (Belgium) in 1977 and 1982 respectively. In 1983 he joined Philips Belgium, where he was a researcher between 1986 and 1991 in the Philips Research Laboratory Belgium (PRLB) in the group Computer and Communication Systems. Between August 1991 and end 1994 he was an Associate Professor in the Computer Science Department of the University of Nijmegen (The Netherlands). In 1995 he joined the Department of Mathematics and Computer Science of the University of Antwerp, where he is currently a Professor and head of the research group ‘‘Performance Analysis of Telecommunication Systems’’ (PATS). He has been member of many program committees of international conferences. He has been and is currently involved in many National and European Research Programs. He is member of IFIP W.G. 6.3 on ’’Performance of Computer Networks’’ and editor of the Journal of Network and Computer Applications. His main research interests are related to mathematical models for performance evaluation of computer and communication systems and the impact of the performance on the architecture of these systems. The systems that are studied are related to traffic management in multi-service networks, mobility management in IP networks, Content Distribution Networks, access control in access networks, both wired and wireless, and optical networks. He has published a substantial number of papers in international journals and conferences on these research areas.