Performance Evaluation of SNMP, NETCONF and ...

Performance Evaluation of SNMP, NETCONF and CWMP Management Protocols in Wireless Network Mariusz Słabicki and Krzysztof Grochla Institute of Theoretical and Applied Informatics of the Polish Academy of Sciences ul.Baltycka 5, 44-100 Gliwice, Poland Email: {mslabicki, kgrochla}@iitis.pl

Abstract—This paper compares the performance of three popular network management protocols: SNMP, Netconf and CWMP (TR069) applied to automate the configuration change and monitoring of wireless network equipment. The OMNeT++ discrete event simulator was used, with models of those three protocols. Two use cases were used for the evaluation: execution of a simple transaction (e.g. configuration change) and statistic collection on devices connected through a wireless network with different packet loss ratio. The results show that SNMP has smallest time required to perform typical management operations, because of the smallest transmission overhead. Among the three protocols analyzed the CWMP requires largest amount of time on average and has highest standard deviation, what shows that the time required variates significantly. The reconfiguration of the devices for SNMP protocol required more time than the monitoring of the devices statistics, but in NETCONF and CWMP it is heavily influenced by the link speed.

I. I NTRODUCTION The number of devices used to build a network constantly grows. The increasing size and complexity of the network makes the management of the network more complex and demand more time and effort. A few network management protocols have been proposed to automate remote management of network devices. They are used by the network management systems to communicate with the devices and perform operations related to reconfiguration, monitoring, fault detection, accounting and security. It allows to decrease the operational expenses of the network operator, by decreasing the human work related to the every-day management tasks of the network administrator. The complexity of the management of a network also grows as more devices are behind wireless channels. Currently the administrators need to control not only infrastructure devices, connected through a wired link to the network operation center, but also routers and bridges connected through IEEE 802.16 or IEEE 802.11 links. Aim of our research was to evaluate, how popular management protocols behave in typical operations executed over wireless network. Several methods to achieve this aim can be used. In [1] authors measured performence of the SNMP protocol on real devices, and the impact of a protocol implementation on different hardware. Authors in [2] proposed framework to emulate network devices. We decided to simulate network and all devices. In our research we analyzed only influence of the network features (delay, packet loss etc). We chose three widely used management protocols: SNMP

[3], Netconf [4] and CWMP (also known as TR.069) [5]. The expected outcome of the experiments were statistics of the time necessary to execute request realized by the management protocol. In our research we used tree topology of the network, and two management procedures: to get data from network devices, and to set value in network devices. To evaluate these protocols we provided models of protocols and determined the test environment which contained several different scenarios. To the best of our knowledge, so far there is no comparison of the three selected protocols in wireless network scenario. Authors in [6] evaluated SNMP protocol used to manage Wireless Sensor Network. In the paper [7] authors performed a quantitative analysis of the performance of the SNMP and Netconf protocols. They developed a simple test environment in which they investigated basic requests of protocols. Their results show that NETCONF is a reasonable alternative for recent complex networks. However they did not take into the consideration delays related to transmission messages through the network, which could be important. On the other hand, paper [8] considers usage of resources in SNMP and Netconf protocols implemented on embedded devices, but there is no evaluation of the influence of the network conditions on the transmission efficiency. There is quite a few papers analyzing a single protocol performance over wireless channel, such as [9] or [10] for SNMP, but they do not provide the comparison to other network management protocols. The rest of the paper is organized as follows: the simulation model of the network, proposed scenario and the model of protocols are described in Section II. Results are shown in Section III. The work is summarized in Section IV. II. M ODEL D ESCRIPTION In our research we decided to use OMNeT++ [11]. It is a well known discrete events simulator, designated to simulate networking protocols. To achieve reasonable model of network behavior we used INET Framework, which adds to OMNeT++ support for simulating network based on TCP/IP. A. Network description Simulated network had a tree topology. The schema of the simulation scenario is shown in Fig.1. The central element of the network was a network manager. It was connected through Ethernet switch to user modules. Between the manager and

TABLE I D ELAY PROBABILITY IN MAC LAYER

Delay [ms] Probability [%]

0 31.5

5 18

10 12.5

15 9.8

20 8.2

25 6.7

30 5.7

35 4.5

TABLE II PARAMETERS OF THE SNMP SIMULATION

GET scenario Request packet size Response packet size Timeout to resend Time to next procedure Request processing time

82 B 100 B 2s 10 s 0.1 s

SET scenario Request packet size Response packet size Timeout to resend Time to next procedure Request processing time

140 B 155 B 2s 10 s 0.2 s

Fig. 1. Network schema

Ethernet switch we inserted a module ”Delayer”. Its aim was to simulate a delay of the connection between end-user and backbone (ADSL or Ethernet). In the Ethernet connection we assumed that datarate is symmetric, and is equal 1 Gbps. In ADSL case the throughput to devices was equal 8 M bps, and to manager was equal 0.5 M bps. The delay in ”Delayer” module was calculated as a function of throughput of the channel and packet length. Moreover we added delay of the core network. The delay of the core network was constant, and was equal 10 ms. Because in our considerations was performance of the network management protocols cooperation with equipment located in user private networks, we assumed that the user equipment is installed in private home (for example behind a router with WLAN). The features of the WLAN were added by module called ”GE Hub”. Wireless channel model in WLAN was based on two separated models. To model packets loss in wireless channel we implemented a Gilbert-Elliot model [12], [13]. In our simulations each of the end user devices had their own Gilbert-Elliot module. Separate Gilbert-Elliot modules generated to each device independent periods of error-free communication. The implemented model used data from [14]. We used parameters related with two WLAN throughput: 5.5 M bps any 11 M bps; and frames with length of 1000 bytes. These parameters were marked respectively as a GEI and GEII. Our implementation gave similar Packet Error Rate as an authors of [14], respectively 1.35 % and 7.98 %. We compared these GEI and GEII results with simpler model based on percentage packet loss. They are labelled PERI and PER II. The packet loss level was the same as in the GEI and GEII. To model the propagation delay in the WLAN we developed our new model. The main idea of this model was that the biggest impact on the delay has the backoff time involved with collisions in the MAC layer. In the paper [15] authors investigated distribution of the delay time. We used their data in our model. Probability of the delays in WLAN channel is

shown in Table I. Values in table were rounded. To sum up we can expect the delay of the simple packet sent from manager to user device should be sum of the elementary delays as follows: • constant delay of the backbone rate – 10 ms, • delay of the connection between end-user and backbone (depends on link datarate set in Delayer and the packet length), • delay related to backoff time in WLAN MAC layer – values and probabilities shown in Table I • delay related to datarate in WLAN channel (we assumed that efficient datarate of this channel is 3 M bps or 6 M bps respectively if theoretical throughput is 5.5 M bps or 11 M bps). B. SNMP protocol – model SNMP [3] is a management protocol which uses UDP as a transport layer. Typical communication looks as follows: • Manager sends a UDP packet with GET/SET request, • Device sends a UDP packet with GET/SET response. In our consideration we do not define the exact version of SNMP protocol. There is no significant difference between protocol versions in simple requests evaluated in this research. Packet sizes used in simulations were determined by capture via Wireshark example packets of SNMP protocol and averaged sizes from series of the same packets. Simulation parameters are shown in Table II. C. Netconf protocol – model Netconf [4] is a management protocol proposed by IETF. It is based on remote procedure call (RPC) paradigm. The network manager creates a RPC request, code it in the XML and sends to a network device using a secure, connection-oriented session. The network device sends response in a similar way. The connection between devices is set and finished by the network manager. In our simulation we used raw TCP sockets,

40 3.2

TABLE III PARAMETERS OF THE N ETCONF SIMULATION

GET scenario Request packet size Response packet size Waiting time after failure Time to next procedure Request processing time

477 B 506 B 2s 10 s 0.1 s

SET scenario Request packet size Response packet size Waiting time after failure Time to next procedure Request processing time

TABLE IV PARAMETERS OF THE TR-069 SIMULATION

509 B 337 B 2s 10 s 0.2 s

but it is possible to use more sophisticated protocols e.g. SSH. The message trace in our implementation looked as follows: • Manager begins a TCP connection, • Manager sends a GET/SET request, • Device sends a GET/SET response, • Manager closes a TCP connection. Packet sizes used in simulations were determined by capture via Wireshark example packets of Netconf protocol and averaged sizes from series of the same packets. Parameters of the simulation are shown in Table III. D. TR-069 protocol – model CWMP, which is commonly called TR-069 [5], is a protocol described by the Broadband Forum. The protocol give availability to secure auto-configuration, management of the firmware and software, monitoring and diagnostics of the network devices. The main difference between TR-069 and previous protocols is that the connection is beginning by the user device. In many cases the user devices are hidden e.g. behind NAT or firewall, and it is impossible to begin connection to the user device from the network manager. TR-069 messages are send as a HTTP requests and responses. HTTP server are ran on the network manager. In our simulations we used raw TCP sockets and size of all requests were increased by the size of the HTTP overhead. Typical sequence in our TR-069 implementation looked as follows: • Device begins a TCP connection, • Device sends Inform request (HTTP Post), • Manager sends Inform reply (HTTP Response), • Device sends empty message (HTTP Post), • Manager sends GET/SET request (HTTP Response), • Device sends GET/SET response (HTTP Post), • Manager sends empty message (HTTP Response), • Device closes a TCP connection. Packet sizes used in simulations were determined by capture via Wireshark example packets of TR069 protocol and averaged sizes from series of the same packets. Parameters used in simulation are shown in Table IV. E. Monitored data In our simulation we assumed that network manager intend to execute request for two simple command: GET (to receive value of a variable), and SET (to set value of a variable). In SNMP and Netconf scenario the procedure is started by

GET scenario Inform request size Inform response size Empty request size Get request size Get response size Empty request size Inform proc. time Empty packet proc. time GET message proc. time Time to next procedure Waiting time after failure

2508 B 831 B 196 B 500 B 1023 B 196 B 0.1 s 0.1 s 0.1 s 10 s 3s

SET scenario Inform request size Inform response size Empty request size Set request size Set response size Empty request size Inform proc. time Empty packet proc. time SET message proc. time Time to next procedure Waiting time after failure

2508 B 831 B 196 B 700 B 400 B 196 B 0.1 s 0.1 s 0.2 s 10 s 3s

manager, in TR069 communication is started by the user’s device. Both requests were simulated in separate scenarios. We wanted to check the distribution of the time that is necessary to perform the whole request. To have comparable results we defined the way to gather data. In SNMP the time of the request was measured from sending the request to receiving the response. If response was lost, the time expands on the request resend. In Netconf and TR-069 the time of the request was measured from the start of the TCP initializing to receive a Response packet. The new procedure were not started before finish of the last one or before the procedure was interrupted by failure event (in TCP) or timeout (in UDP). All cases were repeated 50 times and results were averaged. F. Evaluated scenarios We decided to evaluate parameters as follows: • backbone datarate (ADSL or Ethernet), • different WLAN channel (5 M bps or 11 M bps), • two management procedures (GET or SET). III. R ESULTS We present results from network with manager and 200 network elements connected via ADSL or Ethernet. GE model used in simulation pretend connection via WLAN with 5.5 M bps (GEI and PERI) or 11 M bps (GEII and PERII). The network manager send 1000 GET (or SET) request to each network element. After finish all requests, we repeated the simulation (50 repetitions). A. Request time distribution Fig.2 and 3 shows the distribution of GET request time in different protocols. Respectively Fig.4 and 5 shows the distribution in SET request. Time achieved in network with ADSL is generally longer than in network with Ethernet. In all cases SNMP achieved the smallest results and TR069 the highest one. The loss rate (in both loss models) affects achieved results – higher level gave longer mean time. Our results show that TR069 is sensitive for delays and packet losses.

●

15000

10000

●

5000 ● ● ● ● ●

●

● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

Request time [ms]

10000

5000 ● ● ●

●

TR069

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

GE II

GE I

PER I

Netconf

PER II

GE I

GE II

PER I

PER II

GE I

GE II

SNMP

Fig. 2. GET request time for 200 nodes with ADSL connection

TR069

Fig. 4. SET request time for 200 nodes with ADSL connection 8000

8000

● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ●

4000

● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ●

4000

2000

2000 ● ● ●

SNMP

●

Netconf

● ●

● ●

● ●

● ●

● ● ●

● ● ●

● ●

● ●

● ●

● ●

● ●

● ●

TR069

Fig. 3. GET request time for 200 nodes with Ethernet connection

SNMP

Netconf

GE II

GE I

PER I

PER II

GE I

PER I

PER II

GE II

GE I

GE II

GE I

PER I

0 PER II

GE I

● ●

GE II

PER I

● ●

PER II

● ●

GE II

● ●

GE I

● ● ●

PER II

PER I

● ●

● ● ●

PER II

● ●

● ● ●

PER I

● ●

● ● ● ● ● ● ● ● ● ● ● ●

6000 Request time [ms]

● ● ● ● ● ● ● ● ● ● ● ●

6000 Request time [ms]

PER II

PER I

GE II

GE I

PER I

Netconf

PER II

GE I

GE II

PER I

PER II

GE I

GE II

PER II

SNMP

0

● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

0 PER I

0

●

GE II

Request time [ms]

15000

TR069

Fig. 5. SET request time for 200 nodes with Ethernet connection

B. Histograms

of peaks can be also observed, but they are less visible.

To show the time of requests distribution in all protocols, we summarized the histograms in all nodes for each simulation. Fig.6 shows a histogram of the GET requests in network with an ADSL links. It can be noticed, that in SNMP protocol requests are finished in turns – they are shown as distinct peaks. It is due to simplicity of the SNMP protocol. It contains only request and reply packets. If the packet is lost the next try is started after timeout. That is why the peaks are clear. Next protocols are more complex. They use TCP (so there is retransmission in the lower layer) and they use more packets to establish connection and send a request. Because of that fact, the histogram does not have clear peaks. There is the significant difference between Netconf and TR069 – Netconf needs less packets to finish whole request. That is why the Netconf had wider peaks, and TR069 do not have peaks. This effects can be also observed in Fig.7. This graph show the same value as the Fig.6 but with Ethernet backbone. Effects

C. Numerical results In Table V we show aggregated numerical results from simulations. Output of one simulation is a mean time needed to finish a requests on all network devices. Table contains a mean values and standard deviations of these outputs (averaged from 50 simulation repetitions). In SNMP protocol the GET procedure is on average faster than SET procedure. In Netconf and TR069 it depends on major factor – if more important is link speed (in ADSL scenario, GET gives longer values) or execution time (in Ethernet scenario, GET gives smaller results). IV. S UMMARY Research show that evaluated management protocols have different protocol overhead. This overhead influences the expected time necessary to finish request. Although the Netconf and TR069 protocols are more sophisticated (they use TCP as

SNMP

30000

TABLE V AGGREGATED REQUEST EXECUTION TIME [ IN MILISECONDS ]

20000 10000 0 Netconf

30000

SNMP ADSL

20000 10000 0 TR069

30000

SNMP Ethernet

20000 10000 0

0

5000

10000

15000

20000

25000

30000

Fig. 6. Histogram of requests time (ms) in all nodes (sum of node histograms) [ADSL, 200 nodes in network] 150000

Netconf ADSL

Netconf Ethernet

SNMP

100000 50000 0 150000

TR069 ADSL

Netconf

100000 50000

TR069 Ethernet

0 150000

TR069

GE I GE II PER I PER II GE I GE II PER I PER II GE I GE II PER I PER II GE I GE II PER I PER II GE I GE II PER I PER II GE I GE II PER I PER II

mean 1608 1676 1608 1673 212 521 214 521 4513 4633 4508 4590 371 1274 388 1352 11109 12381 11157 12627 1403 4968 1508 5471

GET std. dev. 1371 1524 1372 1522 329 921 334 920 3719 4463 3707 4444 686 2234 726 2313 8815 15540 8766 15319 1200 6321 1311 6627

mean 1948 2023 1948 2020 316 625 318 626 4174 4195 4168 4116 471 1373 487 1453 10194 11894 10285 12125 1601 5169 1711 5714

SET std. dev 1432 1620 1432 1618 328 920 334 920 3748 4344 3734 4257 688 2232 727 2315 7308 14257 7330 14167 1201 6425 1317 6662

100000 50000 0

0

5000

10000

15000

20000

25000

30000

Fig. 7. Histogram of requests time (ms) in all nodes (sum of node histograms) [Ethernet, 200 nodes in network]

a transport layer), the SNMP protocol achieved the smallest average time needed for transactions in all assumed scenarios. It is related to the small protocol overhead. Among the three protocols analyzed the TR069 requires largest amount of time on average and has highest standard deviation, what shows that the time required variates significantly. SNMP protocol requires more time for reconfiguration than for the monitoring of the devices statistics. In NETCONF and TR069 it depends on the link speed. Acknowledgments.: This work was funded by the Polish National Center for Research and Development, grant no LIDER/10/194/L-3/11. R EFERENCES [1] P. Carlsson, M. Fiedler, K. Tutschku, S. Chevul, and A. A. Nilsson, “Obtaining reliable bit rate measurements in SNMP-managed networks,” in Proceedings of the 15th ITC Specialists Seminar on Traffic Engineering and Traffic Management, 2002, pp. 114–123. [2] K. Grochla and L. Naruszewicz, “Testing and Scalability Analysis of Network Management Systems Using Device Emulation,” in Computer Networks. Springer, 2012, pp. 91–100. [3] W. Stallings, SNMP, SNMPv2, SNMPv3, and RMON 1 and 2. AddisonWesley Longman Publishing Co., Inc., 1998. [4] R. Enns, M. Bjorklund, and J. Schoenwaelder, “Netconf configuration protocol,” Network, 2011.

[5] D. W. Group et al., “TR-069 CPE WAN Mangement Protocol Issue: 1 Amendment 5 version 1.4,” 2013. [6] W. Chen, Y. Zhou, Y. Wu, X. F. Liao, and X. R. Ding, “Wireless Sensor Network Management Based on SNMP Protocol,” Applied Mechanics and Materials, vol. 303, pp. 292–296, 2013. [7] B. Hedstrom, A. Watwe, and S. Sakthidharan, “Protocol Efficiencies of NETCONF versus SNMP for Configuration Management Functions,” Ph.D. dissertation, PhD thesis, Masters thesis, University of Colorado, 2011. [8] A. Sehgal, V. Perelman, S. Kuryla, and J. Schonwalder, “Management of resource constrained devices in the Internet of Things,” Communications Magazine, IEEE, vol. 50, no. 12, pp. 144–149, 2012. [9] J. Kantorovitch and P. Mahonen, “Case studies and experiments of SNMP in wireless networks,” in IP Operations and Management, 2002 IEEE Workshop on. IEEE, 2002, pp. 179–183. [10] R. B. Johnson, Evaluating the use of SNMP as a wireless network monitoring tool for IEEE 802.11 wireless networks. ProQuest, 2009. [11] A. Varga et al., “The OMNeT++ discrete event simulation system,” in Proceedings of the European Simulation Multiconference (ESM2001), vol. 9. sn, 2001, p. 185. [12] E. N. Gilbert, “Capacity of a Burst-Noise Channel,” Bell system technical journal, vol. 39, no. 5, pp. 1253–1265, 1960. [13] E. Elliott, “Estimates of Error Rates for Codes on Burst-Noise Channels,” Bell system technical journal, vol. 42, no. 5, pp. 1977–1997, 1963. [14] J. Ar´auz and P. Krishnamurthy, “Markov modeling of 802.11 channels,” in Vehicular Technology Conference, 2003. VTC 2003-Fall. 2003 IEEE 58th, vol. 2. IEEE, 2003, pp. 771–775. [15] T. Issariyakul, D. Niyato, E. Hossain, and A. S. Alfa, “Exact Distribution of access delay in IEEE 802.11 DCF MAC,” in Global Telecommunications Conference, 2005. GLOBECOM’05. IEEE, vol. 5. IEEE, 2005, pp. 5–pp.