CS 498: Network Systems Laboratory Laboratory Assignment 4: Implementation of AQM as Kernel Modules in Linux Jennifer C. Hou and Honghai Zhang Department of Computer Science University of Illinois at Urbana-Champaign jhou,
[email protected] Distribution date: October 4, 2005; Due date: October 21, 2005
I. Introduction The objective of this lab assignment is two-fold: (i) understanding of active buffer management; and (ii) learning how to implement kernel modules and write applications that interact with the kernel (modules). You will implement two of the active queue management schemes (AQM) you select as kernel modules and revise the tc source code to give the statistics of each of the AQM schemes implemented. TCP RED is already implemented in /usr/src/linux-(version)/net/sched. In this project, you are required to implement other AQM schemes such as Adaptive RED, SRED, BLUE, REM and AVQ. The information on adaptive RED can be found at http://www.icir.org/floyd/red.html.The information on BLUE can be found at http://www.thefengs.com/wuchang/work/blue/. The information on REM can be found at http://netlab.caltech.edu/netlab-pub/cbef.pdf. The information on AVQ can be found at www.seas.upenn.edu/~kunniyur/papers/avq.pdf.
2. Step-by-Step Instruction on Adding Module NewRED In what follows, we give step-by-step instructions on how to implement NEW RED as a new kernel module. All the paths below are relative to /usr/src/linux-(version): 1. Configure the kernel (e.g. using make menuconfig) to enable QoS support and QoS and/or fair queue. 2. In net/sched/Config.in, find the line contain RED and add a line below it as follows: tristate ' NEW RED queue' CONFIG_NET_SCH_NEWRED 3. In net/sched/sch_api.c, find the following lines #ifdef CONFIG_NET_SCH_RED INIT_QDISC(red);
#endif. Now add the following lines below the above lines: #ifdef CONFIG_NET_SCH_NEWRED INIT_QDISC(newred); #endif. Actually these lines can be add anywhere (outside other #ifdef directive) in the file. Put them behind the directive for RED is simply to make the code consistent. 4. Add the following line in /net/sched/Makefile. (Again it is better to put it after the line containing RED), obj-$(CONFIG_NET_SCH_RED) += sch_newred.o 5. In the directory /net/sched/, do “cp sch_red.c sch_newred.c” and in sch_newred.c, change the line struct Qdisc_ops red_qdisc_ops = into struct Qdisc_ops newred_qdisc_ops = line return register_qdisc(&red_qdisc_ops); into return register_qdisc(&newred_qdisc_ops); and line unregister_qdisc(&red_qdisc_ops); into unregister_qdisc(&newred_qdisc_ops); 6. Now reconfigure the kernel, in the networking options | QoS and/or fair queuing>, you will see a new item called NEW RED queue (NEW), select it as a module and recompile the kernel, and reboot the computer. The above are basic steps in adding a new AQM scheme. In order to implement different AQM schemes, you will need to change the content of sch_newred.c. You will need to define a variable that has the data structure struct Qdisc_ops and implement the functions in that data structure.
3. Modifying tc After adding two kernel modules, you will need to use tc to start, configure and stop them in the kernel. Both the source and the detailed information about tc can be found at http://snafu.freedom.org/linux2.2/iproute-notes.html. First you will learn to use tc to
configure RED parameters. Next you will need to add code in tc in order to set the parameters in the AVQ schemes you implemented, display the parameters (by tc –s qdisc) and show the queue statistics (also by tc –s qdisc). The steps for modifying tc are: To add a dummy parameter, you need to add member dummy in the data structure tc_newred_qopt in both the kernel include file (in /usr/src/linux-(version)/include/linux) and the user include file (in /usr/include/linux) pkt_sched.h, and in struct newred_sched_data in the kernel sch_newred.c file. To add a dummy statistic varible, you need to add member dummy in the structure tc_newred_xstats in both the kernel include file and the user include file pkt_sched.h. To add a tc statistics empty_arrive (the number of packets arrives when the queue is empty): 1. in enqueue: sch->stats.empty_arrive ++; 2. add empty_arrive member in both the kernel include file and the user include file pkt_sched.h
4. Experiments and Demo Now you are ready to carry out experiments to compare the performance of different variants of RED. To carry out the experiments, you need to establish a 2-hop connection, make the second hop a bottleneck and put the RED variants at the intermediate node. You need to use certain traffic measurement tools (such as iperf) to establish a large number of connections and record the throughput attained by the connections under a specific AQM scheme. In the demo shown to the TA, you will need to show: 1. How to add AQM (e.g., RED, REM, AVQ) to the network device using tc. 2. How to display the parameters and queue statistics using tc. At least one parameter and one queue statistics must be different from that in RED. 3. Establish TCP connections and measure the throughput attained under each RED variant. In the report, you will need to include: 1. Overview of your implementation approach. 2. Well-commented source code (with the modified part highlighted, together with sufficient comments to explain why and how the modification is made). 3. Example command or scripts used in your experiments. 4. Plots to compare the performance of the two AQM schemes you implemented, with the parameters you use.
5. Debugging experience. 6. Comments and suggests on improving this lab assignment. Record roughly how much time you spend on this project. Itemize the time spent on each part of the lab if possible.
5. References 1. RED, adaptive RED: http://www.icir.org/floyd/red.html 2. BLUE : http://www.thefengs.com/wuchang/work/blue/. 3. REM: http://netlab.caltech.edu/netlab-pub/cbef.pdf 4. AVQ: www.seas.upenn.edu/~kunniyur/papers/avq.pdf 5. Iproute2+tc: http://snafu.freedom.org/linux2.2/iproute-notes.html
Other Complementary information: 1. Flow of functions for queuing disciplines (based on the kernel version 2.4.24). When a packet is enqueued on an interface, dev_queue_xmit in net/core/dev.c is activated. In this function, it invokes the enqueue function of the device’s queuing discipline (field qdisc of struct net_device in include/linux/netdevice.h). Afterwards, dev_queue_xmit calls qdisc_run in include/net/pkt_sched.h to try sending the packet that was just enqueued. qdisc_run immediately calls qdisc_restart in net/sched/sch_generic.c, which is the main function to poll queuing disciplines and to send packets. qdisc_restart first obtain a packet from the queuing discipline using dequeue, and if it succeeds, it invokes dev_queue_xmit_nit to prepare the packet, and the device’s hard_start_xmit function to actually send the packet. If a queuing discipline is compiled into the kernel, it should be registered by pktsched_init in net/sched/sch_api.c dev_queue_xmit( in net/core/dev.c) q->enqueue qdisc_run qdisc_restart q->dequeue dev_queue_xmit_nit hard_start_xmit
2. Flow of functions in tc when one calls tc qdisc add dev eth0 root newred… When the first parameter matches qdisc, tc calls do_qdisc in tc.c. If the second parameter matches add, do_qdisc calls tc_qdisc_modify. tc_qdisc_modify parses the parameters dev eth0 root and newred. After it finds out the qdisc type, tc_qdisc_modify invokes q->parse_qopt (which must be implemented by each queue discipline). Afterwards, tc_qdisc_modify establishes a connection with the kernel and sets the queuing parameters in the kernel (by rtnl_open and rtnl_talk). 3. Flow of functions in tc when one calls tc –s qdisc : It still first calls do_qdisc in tc.c, which further calls qdisc_list. qdisc_list calls rt_open, rtnl_dump_request and rtnl_dump_filter. rtnl_dump_filter has a function parameter print_qdisc which calls the corresponding queue discipline’s print_qopt to print out the qdisc statistics, calls print_tcstats to print out tc statistics, and calls q->print_xstat. 4. Options data structure rtattr and macros: struct rtattr { unsigned short rta_len; unsigned short rta_type; }; /* Macros to handle rtattributes */ #define RTA_ALIGNTO 4 #define RTA_ALIGN(len) ( ((len)+RTA_ALIGNTO-1) & ~(RTA_ALIGNTO-1) ) #define RTA_OK(rta,len) ((len) > 0 && (rta)->rta_len >= sizeof(struct rtattr) && \ (rta)->rta_len rta_len), \ (struct rtattr*)(((char*)(rta)) + RTA_ALIGN((rt a)->rta_len))) #define RTA_LENGTH(len) (RTA_ALIGN(sizeof(struct rtattr)) + (len)) #define RTA_SPACE(len) RTA_ALIGN(RTA_LENGTH(len)) #define RTA_DATA(rta) ((void*)(((char*)(rta)) + RTA_LENGTH(0))) #define RTA_PAYLOAD(rta) ((int)((rta)->rta_len) - RTA_LENGTH(0))
References Werner Almesberger, “Linux Network Traffic Control --- Implementation Overview.” (This is based on a slightly older Linux implementation; nevertheless, it is worthwhile to read.)
