Developing a VM-based hybrid DNS system with high availability

The 2013 International Conference on Advanced Technologies for Communications (ATC'13)

Developing a VM-based Hybrid DNS System with High Availability Chang-Sheng Chen*,a, Chun-Hsien Sub Information Technology Service Center National Chiao Tung University Hsinchu, Taiwan a [email protected] (*corresponding author), [email protected] we had designed two testing cases to help implement and/or evaluate the overall system. Moreover, we had developed a two-phase DNS health checking algorithm and run the scheduled checking process periodically to avoid forwarding queries to any failed DNS backend.

Abstract—In this paper, we proposed to develop a set of lowcost, virtual machine-based hybrid DNS scheme (i.e., a frontend + multiple backend systems) with a scalable capability to provide DNS services with high availability. In this study, we had built two use case diagrams, namely the frontend (i.e., load-balancer) and the backend (i.e., DNS resolving) subsystems, for modeling the overall system. Moreover, we had developed a two-phase DNS health checking algorithm and designed two testing cases to help implement and evaluate the system. For validating the hybrid scheme, we had deployed and run the proposed DNS system for more than 2 years on our campus network. Based on the collected statistics, we had ensured that the proposed DNS systems could offer the expected DNS services with load-sharing capability and high availability for serving daily DNS queries on a regular basis and run smoothly. Keywords—DNS; high availability; scalable; use case; virtual machine

I.

INTRODUCTION

In this paper, using the virtual machine (abbreviated as VM) [1] technology, we proposed to develop a hybrid scheme (i.e., a frontend + multiple backend systems) of building scalable DNS [2] systems to provide DNS services with high availability. As we all know, many vendors could provide hardware appliance-based systems (i.e., namely layer4 and/or layer7 switches) with high availability and load balance functions to facilitate common network services. However, due to the higher operation costs (as compared to software approaches), we would like to develop a flexible, low-cost scheme for building a virtual machine-based (or VM-based) DNS system with high availability and load-sharing capability.

Fig. 1. An ideal framework of our proposed DNS scheme

For validating the proposed scheme, we had deployed and run the hybrid DNS systems on our campus network for more than 2 years. Based on the collected statistics, we had ensured that the proposed DNS systems could offer the expected DNS services with load-sharing capability and high availability for serving daily DNS queries. The rest of the paper is organized as follows. Section 2 describes the preliminaries of this study. Section 3 describes how we designed and built the proposed system. In Section 4, we describe experimental observations and issues concerning actual operations of the deployed DNS systems. Finally, Section 5 gives the concluding remarks.

Fig. 1 shows an ideal framework of our proposed DNS system with high availability. Ideally, we would like to have all DNS queries, received by the frontend (i.e., the Load Balancer), to be evenly dispatched (in round-robin mode) to a set of jointly working backend DNS servers for resolving and each backend system (i.e., as a traditional DNS resolving server) would serve the queries got and give corresponding responses to the original users. Thus, the integrated system could improve the overall DNS service quality. In this study, we had built two use case diagrams, namely the frontend (i.e., load-balancer) and the backend (i.e., DNS resolving) subsystems, for modeling the overall system. Next,

978-1-4799-1089-2/13/$31.00 ©2013 IEEE

II.

PRELIMINARIES

A. DNS Basics The Domain Name System [2] is responsible for translating between hostnames and the corresponding IP addresses needed by software. The mapping of data is stored in a tree-structured distributed database where each name server is authoritative (responsible) for a portion of the naming hierarchy tree. The

231

The 2013 International Conference on Advanced Technologies for Communications (ATC'13)

client side query process typically starts with an application program on the end user’s workstation, which contacts a local name server via a resolver library. As shown in Fig. 2, the client side of the DNS name server queries the related DNS servers recursively for the name in question until it gets an answer or is told there is none.

no IPv6-related example shown in this study). Finally, in the lower part of Fig. 3, we could find the use cases concerning other DNS system configuration tasks and the DNS response. C. Related Work – Network Resource Allocation In [3], the authors proposed to implement of a DNS without single-point-of-failure using virtual machine technology. In [4], authors addressed the problem of dealing with competition and cooperation relationships among nodes in a network with a CSMA (Carrier Sense Multiple Access) protocol. In [5], as cloud computing is becoming more popular, the authors proposed a framework to build a proper mobile service selection system and ensure the QoS. However, all these efforts are distributed piece by piece. In this paper, we proposed to develop a hybrid system (i.e., a frontend + multiple back-ends) of building a scalable DNS system with a simple round-robin scheme to provide DNS services with high availability. D. Overview of the System Platform As shown in Fig. 4 and Table I below, we had built the proposed systems using the same VM platforms, open source DNS package (i.e., BIND [2]) and on the same type of hardware platforms (i.e., Dell R710s). On the frontend host, we used the pf firewall (Packet Filter) [6] to implement the NAT server + port forwarding subsystem (i.e., explained in Section III).

Fig. 2. DNS working paradigm.

B. Use Case Modeling of DNS Resolving Systems Fig. 3 depicts a use case diagram concerning the basic operations of a typical DNS resolving server. As shown, in the upper part, the generalized DNS queries could be divided into two categories, namely, traditional DNS queries and DNS queries with DNSSEC [2] enabled. In essence, both categories of DNS queries could be implemented as UDP and TCP packets. On the other hand, the DNS resolving system might be required to conduct recursive queries whenever the required answer is missing and there is no evidence that the required answer does not exist.

Fig. 4. VM hypervisor architecture in this study

Table I shows the system inventory of both the frontend and backend subsystems. Table I indicates that the frontend system had only 512 MB of RAM installed, and each backend DNS system had 4096 MB of RAM installed. This is because the frontend only worked as a DNS forwarding gateway rather than a real DNS resolving system, which could utilize more memory space for implementing the required DNS cache to improve the overall system performance. TABLE I.

Fig. 3. Use case modeling of a DNS resolving system.

Next, each category above could be further divided into two sub-types, namely the forward and reverse queries. For example, a client program would issue a forward DNS query (e.g., “www.it.nctu.edu.tw”) to get the related IP address, 140.113.40.88. On the other hand, if any user got an IP address and wished to know its corresponding DNS name, she/he would need to conduct the reverse DNS query (i.e., 88.40.113.140.in-addr.arpa, in the above case). Moreover, DNS packets of both forward and reverse types could be further divided into two sub-types, namely, IPv4 and IPv6 packets (i.e.,

232

SYSTEM INVENTORY: FRONTEND AND BACKEND SYSTEMS

Items

Frontend subsystem

Backend subsystem(s)

CPU

Intel(R) Xeon(R), E5520@ 2.27GHz

Intel(R) Xeon(R), E5520@ 2.27GHz

RAM

512 MB

4096 MB

NIC

Intel(R) Pro/1000

Intel(R) Pro/1000

VM Hypervisor

VMware vSphere 5.1

VMware vSphere 5.1

OS

FreeBSD 9.0-Stable

FreeBSD 9.0-Stable

BIND (DNS)

N/A

9.8.5-P1

Port forwarding gateway

Pf + NAT (on FreeBSD)

N/A

The 2013 International Conference on Advanced Technologies for Communications (ATC'13)

Third, by keeping a list of healthy backend DNS servers, the frontend system could dispatch and redirect the received DNS queries to each healthy backend DNS server in roundrobin mode and the backend DNS servers work together to jointly serve the DNS query stream. As a result, each backend subsystem returns the processed responses (corresponding to each incoming query) to the frontend subsystem, which in turn forwards the results to the original users. In this way, we could easily implement a scalable VM-based DNS scheme.

In fact, we had installed a VM cluster (consisting of 8 physical machines) to provide DNS-related, mail-related and other general UNIX-like systems for supporting various university network services. In principle, for each physical host, we might deploy up to 20 VM hosts in average; however, for performance considerations, we had deployed only 50 virtual machines (i.e., average CPU loading =30%, memory usage=50%, currently), including 7 DNS-related servers. E. The Experimental Group vs the Control Group In this study, we rebuilt our major DNS systems with VMware for studying the validity and fully gaining the benefits on using VM technology: rapid deployment, highly scalable capability (i.e. CPU processing power, RAM size), etc. For studies and comparing purposes, we had deliberately built two independent DNS groups, all deployed using the same VM technology. In principle, the experimental DNS group consisted of a frontend (i.e., public IP address 140.113.1.1) and multiple backend subsystems (i.e., private IP addresses192.168.1.1, 192.168.1.2, 192.168.1.3), working jointly as a unified system. On the other hand, the control DNS group consisted of two independent VM-based DNS systems (i.e., public IP addresses – 140.113.6.2, 140.113.250.135), working as traditional monolithic DNS systems. The details will be given in later sections.

Fig. 5. System architecture of the proposed DNS scheme.

B. Use Case Modeling of the DNS Frontend Subsystem Fig. 6 depicts a use case diagram modeling of the DNS frontend subsystem. First, in the upper part, we could find the three use cases (and the related actors) corresponding to ordinary DNS user queries: namely, (1) the normal DNS queries, (2) the DNS forwarding subsystem (i.e., NAT + port forwarding), and (3) the expected DNS responses. As shown, with the and functions defined, we could find that the DNS frontend subsystem (as a forwarding gateway) worked in between the ordinary users (i.e., programs) and the DNS backend resolving servers (i.e., implemented as traditional DNS resolving servers with private IP addresses).

F. Virtual Machine Support - HA and DRS [1] In essence, both HA (High Availability) and DRS (Distributed Resource Scheduler) of VMware could facilitate the maintanence of the mentioned VM cluster in this study. For example, once a physical machine had shown symptoms of being malfunctioning (e.g., through VM health check), the VMs on the machine could be automatically moved to other healthy machines by the HA mechanism. On the other hand, if there is any resource management problem (e.g., CPU overloading, memory usage) identified on a VM, DRS could use the live migration mechanism (i.e., vMotion) to move the VM to other healthy physical host automatically and keep providing normal operations as usual. III.

BUILDING THE SCALABLE VM-BASED DNS SYSTEMS

A. System Architecture of the DNS Frontend Subsystem Fig. 5 indicated the system architecture of our proposed DNS scheme on FreeBSD systems (i.e., both the frontend and backend subsystems). The details would be outlined as follows: First, we adopted the PF firewall program (i.e., having the NAT and Port Forwarding functions) to implement the DNS frontend load-balance server. Using the mentioned tools (i.e., VMware + FreeBSD + PF), we could build a DMZ-like zone (DMZ: Demilitarized Zone) where the DNS backend subsystems are hosted.

Fig. 6. Use case modeling of the DNS frontend subsystem.

Second, in this study, we had implemented a pool of three backend DNS systems. If necessary (e.g., for performance considerations), with VM technology, we could quickly reproduce another new backend and add it into the working pool of backend DNS subsystems (i.e., by changing the setting of frontend system, too) for load sharing.

Next, in the lower part, we could find the group of system administrating use cases, which identified three major types of system tasks: namely, (1) rebuilding the system kernel, (2) reconfiguring the required system files (i.e., /etc/rc.conf and /etc/pf.conf ), (3) scheduling the DNS health check (i.e.,

233

The 2013 International Conference on Advanced Technologies for Communications (ATC'13)

/etc/crontab). Moreover, the configuration steps and the related files are listed as follows: x

x Input: A Candidate List of DNS backend system. x Output: A Valid List of DNS backend systems. x Process: Conducting the health checks on the frontend system every 10 seconds.

Step1: edit the kernel config file to enable PF device pf // Enable Packet Filter // Enable pflog0 NIC(in bpf mode) device pflog device pfsync // Enable pfsync0 to monitor the status options ALTQ // Enable ALTQ options ALTQ_CBQ // Enable Class Bases Queuing

x

Step 1. Copy the candidate list to the valid list of and set the next_IP index to 1. Step 2. Conduct the scheduled ICMP and DNS checks. Step 2.1 Select next IP for testing from the candidate list. Step 2.2. Conduct the ICMP check and DNS check. // Step 2.2.1 Conduct the ICMP check. if icmp_test is ok { //Step 2.2.1.1 Conduct the DNS check. if nsping_test ok { echo "DNS Server is alive" add Server IP to pf's } // Step 2.2.1.2 Remove the IP from the valid list else { echo " icmp ok, but dns query failed" delete Server IP from pf's } // end-of-nsping-check } // end-of-icmp-check // Step 2.2.2 Remove the IP from the valid list if icmp_test is failed { echo " DNS Server icmp failed" delete Server IP from pf's } Step 2.3 Increment the next_IP index by one. Step 3. Goto Step 2 if next_ip $ext_ip // Enable Port (53) Forwarding to backend DNS Servers rdr on $ext_if proto udp from to $ext_ip port 53 -> port 53

x

Step4. FreeBSD commands to enable and disable PF // with default config file /etc/pf.conf // Enable PF #pfctl –e #pfctl -f /etc/pf.conf // Reload pf.conf file #pfctl –d // Shutdown PF

Fig. 7. The two-phase DNS health check algorithm.

C. DNS Health Check Algorithm In this study, we had designed and implemented a set of two-phase health-checking script on the DNS frontend subsystem to keep monitoring the proper operation of the DNS backend systems. To validate if any backend DNS resolving server worked as expected, the frontend system performed two types of health checks (outlined briefly below), namely, ICMP check and DNS query test. x Phase 1: ICMP check (i.e., connection probing)

IV.

A. Observations on the Frontend Server - Two Test Cases In the following, we would give detailed descriptions of the testing cases in this study. 1) Test Case 1: We conducted tests on checking if the frontend subsystem could successfully do the DNS port forwarding to the corresponding DNS backend subsystems. In this case, we had two packages for testing. One is the HAproxy [5] program and the other is the pf program. Table II shows the details of the test cases. In the TCP packet forwarding tests, both worked as expected. However, in the DNS UDP packet cases, only the latter case (i.e., pf) passed the check.

// If no packet lost, then print ok ping -z 10 -c 1 -t 1 -i0.2 2>&1 | awk ‘if($7=="0.0%") { print "ok"; exit}

x

EXPERIMENTAL RESULTS AND DISCUSSIONS

Phase 2: DNS check (i.e., DNS query probing) [7]

timelimit -t 10 -T 20 nsping -c 3 -h www.google.com $ip | awk '{ if(NF==18){if($9 >= "2"){print "ok"} else{print "bad"}} }'

TABLE II. Test Scenario PreCondition

In principle, the frontend system inspected the status of backend systems by first conducting ICMP checks and next conducting the DNS probing checks. If either test failed while performing a check on some backend subsystem, the failure would be recorded on the system log and the corresponding backend system would be removed from the valid list. The details of the two-phase DNS health check algorithm is outlined as follows:

TEST CASE 1 : FRONTEND PORT-FORWARDING TESTS Checking the DNS port-forwarding capability on the frontend. 1) 2) 3) 4)

234

VM Software Installed (i.e., vSphere [1]) Guest OS (FreeBSD) on both frontend and backend systems installed BIND 9.8.5-P1 (DNS) installed on the backend system. Prepare one user PC, running Windows (e.g., Win7). z Configure the IP address of the PC (e.g., 192.168.5.6). z DNS resolver of the PC is configured to the DNS

The 2013 International Conference on Advanced Technologies for Communications (ATC'13)

Test Steps

Test Data Expected Results Actual Results PASS/FAIL

frontend system (i.e., IP address 140.113.1.1). 1) Start the system of user PC (i.e., running Win 7). 2) Launch the nslookup utility on Win7. 3) Issue some forward queries (e.g., www.facebook.com). 4) Issue some reverse queries (e.g., 1.82.13.31.in-addr.arpa). Some sample forward and reverse DNS queries (e.g., www.facebook.com, 1.82.13.31.in-addr.arpa). Must be able to conduct sample forward and reverse DNS queries successfully. 1) Using HAproxy [8] -> Failed (only support TCP packets). 2) Using pf + NAT + port forwarding -> Successful, on both UDP and TCP packets cases. z FAIL (using HAproxy) z PASS (using pf on FreeBSD)

2) Test Case 2: We conducted the automatic failover checks on the backends to verify if the dispatching function (i.e., for implementing load-sharing) of the frontend had been correctly implemented. In practice, since we would not kown when any DNS backend might fail, we had conducted this test by using simulation. As shown in Table III, the proposed system successfully passed the check. TABLE III. Test Scenario Pre-Condition

Test Steps

Test Data Expected Results Actual Results PASS/FAIL

Fig. 8. Simulation of automatic failover check on DNS backend.

Moreover, we had conducted another reverse test for validating the DNS health check scheme. Soon after we reenabled the once shutdown DNS backend (i.e., 192.168.1.1), the frontend quickly identified that the 192.168.1.1 host was up again and added it into the working lists of available DNS server (i.e., Stage 3 of Fig. 8).

TEST CASE 2 : AUTOMATIC FAILOVER CHECKS ON BACKENDS Checking the load-sharing capability of the frontend by manually shutting down one backend DNS subsystem. 1) VM Software Installed (i.e., vSphere) 2) Guest OS (FreeBSD) on both frontend and backend systems installed. 3) Pf + NAT +port-forwarding installed on the frontend. 4) BIND 9.8.5-P1 (DNS) installed on the backend system. 1) Show the valid list of DNS backend on the frontend by issue the command "pfctl -t dns_servers -T show”. z Result: 192.168.1.1, 192.168.1.2, 192.168.1.3. 2) Shut down one specific DNS backend (i.e., IP address 192.168.1.1) by issuing the command, "pfctl -t dns_servers -T show". 3) Inspect the status of the valid list of DNS backend. z Check if the above 192.168.1.1 IP had been removed from the valid list. A valid list (i.e., IP addresses) of DNS backend subsystem 192.168.1.2, 192.168.1.3

B. Observations on tne Backend DNS Servers In essence, when the frontend system forwarded the received DNS queries to some backend DNS server, the selected backend system would take over the responsibility for resolving the DNS queries and returned the expected responses to the original users through the frontend. Fig. 9 indicated an excerpt of DNS logs of the first backend subsystem 192.168.1.1 (i.e., similar results could be got from the other backend subsystems, too). For privacy concerns, the real IP addresses (i.e., IPv4) of the query sources were marked.

192.168.1.2, 192.168.1.3 PASS Fig. 9. DNS query logs from one backend DNS server (i.e., 192.168.1.1)

First, as shown in the upper part of Fig. 8 (i.e., Stage 1), we had established a valid list of three backend DNS subsystems, namely, {192.168.1.1, 192.168.1.2, 192.168.1.3}. In this test case, we first deliberly shut down one DNS backend (e.g., IP 192.168.1.1) subsystem manually through the VM management console. Next, we further check if the frontend could successfully identify the shutdown host (i.e., through the proposed two-phase DNS health check process) and automatically update the valid list to reflect the change. As shown in Stage 2 of Fig. 8, the proposed scheme had been correctly implemented.

C. Sample DNS Query Traffic Distributions In this study, Fig. 10 indicates sample DNS query traffic distributions between the time intervals: Aug 22-Sep 4, 2013. As shown, while the figures of the host 140.113.1.1 indicate the distribution of the experimental DNS group, the other two sets of figures indicate the corresponding distributions of the control DNS group (i.e., 140.113.6.2, 140.113.250.135). According to the collected statistics in Fig. 10, the query ratio (i.e., numbers of queries served) between the upper two DNS systems (i.e., 140.113.1.1 vs. 140.113.6.2) ranges from 1.107 (i.e., Sep 1, Sunday) to 1.367 (i.e., Sep 3, Tuesday),

235

The 2013 International Conference on Advanced Technologies for Communications (ATC'13)

might assume that there exists a DNS system with N (i.e., N=3, in this study) times in computing power, as calculated in (1).

with an average ratio 1.314 during the specified 2-week time interval. For example, on Aug 27 (Tuesday), there were 24696091 DNS queries on the former host and 19188151 DNS queries on the latter host. The ratio is about 1.287.

x Expected CPU loading of the experiment DNS group N CPU loading of a typical backend DNS  According to the metrics on both Fig. 11 and Fig. 12, the average CPU loading ratio between the two (140.113.1.1 vs. 140.113.6.2) could be calculated as follows: 3 * 10.88 / 23.43 = 1.393. As shown, the average CPU loading ratio is approximately the same to the average query ratio (i.e., 1.314 mentioned above) plus minor overheads. Therefore, we could ensure that the proposed DNS systems worked as expected (i.e., helping reduce the CPU loading of a monolithic system). F. SPOF on the Frontend System Needs Fixing In essence, there is still a minor Single-Point-of-Failure (SPOF) drawback on the proposed scheme. Currently, if the frontend system fails, the overall DNS service would fail completely. In the future, we will keep refining the system and rebuild the frontend with a failover mechanism (i.e., running another backup system).

Fig. 10. Sample DNS query traffic distributions

V.

D. Observations on the CPU Loadings of the Systems While Fig. 11 depicts the CPU loading distribution of one DNS backend (i.e., 192.168.1.1- one of three servers in the experiment group) in the past year (i.e., from Sep 2012 to Aug 2013), Fig. 12 shows the CPU loading distribution of another monolithic DNS 140.113.6.2 (i.e., one of the control group) during the same time interval. From these diagrams, we could ensure that both types of systems performed well because the CPU loading of both worked under 40% most of the time.

CONCLUDING REMARKS

In this paper, we had built a set of virtual machine-based, load-sharing DNS systems with high availability on FreeBSD systems. Moreover, we had developed a two-phase DNS health check algorithm to help keep the smooth operation of the proposed system. Based on the collected statistics, we had ensured that the proposed DNS systems could offer the expected DNS services with load-sharing capability and high availability for serving daily DNS queries. ACKNOWLEDGMENT This paper (work) was partially supported both by National Science Council (under Grant No. NSC 100-2218-E-009-019) and by the "Aiming for the Top University Program" of National Chiao Tung University and Ministry of Education, Taiwan, R.O.C. REFERENCES [1] [2]

Fig. 11. CPU loading distribution on one backend subsystem

[3]

[4]

[5]

[6] [7] Fig. 12. CPU loading distribution on 140.113.6.2

[8]

E. Expected CPU loadings of the experiment DNS group As mentioned, the experiment group contained multiple backend systems working in round-robin mode. That is, we

236

WMware on http://www.vmware.com. C. Liu, and P. Albitz, DNS and BIND, 5th edition, O'Reilly Media. June 2006. Y.D. Wang and C.S. Chen, “A study on the design and implementation of a DNS without single-point-of-failure using virtual machine technology,” TANet 2010 Conference, Tainan, Taiwan (Oct 2010). Zhefu Shi, Cory Beard, Ken Mitchell, “Tunable Traffic Control for Multi-Hop CSMA Networks(Link)”, Military Communications Conference (MILCOM 2008). IEEE, November 2008 Zhefu Shi, Ruirui Gu, “A Framework For Mobile Cloud Computing Selective Service System”, Wireless Telecommunications Symposium (WTS), April 2013. The Packet Filter and ALTQ (FreeBSD) on http://www.freebsd.org/doc/handbook/firewalls-pf.html Nsping & timelimit (FreeBSD Ports Archive) on http://www.freebsdsoftware.org/dns/nsping.html HA Proxy: Reliable, High Performance TCP/HTTP Load Balancer, on http://haproxy.1wt.eu/