or reception of a packet, sensor being turned on or a timer firing. However .... however includes connecting to NetServ from a script written in Perl or Python and.
On Testing Wireless Sensor Networks Tomasz Surmacz, Bartosz Wojciechowski, Maciej Nikodem, and Mariusz Słabicki
Abstract Testing wireless sensor networks (WSNs) is not a trivial task, due to the massively-parallel communication between many independently running processors. Distributed way of operation does not allow step-by-step execution and typical debugging, so other techniques have to be used, such as detailed packet logging and analysis. We provide a 2-way WSN-to-TCP proxy architecture which extends the typical BaseStation software with packet sniffing and packet sending capabilities. This allows writing and executing WSN test scenarios and automatic test assessment by using typical client-server applications written in any programming or scripting languages. An example of such protocol testing is also shown.
1 Introduction The dominating application of Wireless Sensor Networks is environment monitoring. A typical setup used for monitoring consists of a WSN network, a data sink and possibly an off-site server that is used to collect, analyse and provide the data to involved parties. The WSN network in turn consists of a large number of nodes that are resource-constrained, i.e. have low performance, small storage capacity, and are usually battery-powered. The nodes in the network serve two purposes simultaneously – they collect data from sensors that they are equipped with and take part in routing the messages to the data-sink or Base Station (BS). Since nodes are equipped with low-power radio transceivers and small gain antennas (between 0 and 5 dBi), their effective communication range is usually very short. Therefore they usually communicate in a multi-hop fashion. Many layers of WSN-based monitoring systems make their behavior hard to understand, develop and test. It usually requires deep understanding of all levels, from sensor hardware and node software Institute of Computer Engineering, Control and Robotics, Wroclaw University of Technology, e-mail:
1
2
Tomasz Surmacz, Bartosz Wojciechowski, Maciej Nikodem, and Mariusz Słabicki
through network algorithms used in communication protocols stack, up to the system and application level. Even bridging the gap between network engineers and the experts in a given application requires a lot of effort [1]. In real conditions (out-of the lab) various problems may occur: hardware malfunctions, programming bugs or software incompatibility, when implicit rules are not clear from the API of a module, or a layer of the software stack [5]. This necessitates strict testing of components of WSN networks on every level and performing tests of whole systems that are extensive both in time and network size. E.g. proper network behavior means also that the power consumption in different scenarios is bound by some limits and it takes time to test this. Due to presence of many physical effects on radio transmission, such as like fading, double- and multi-path reflections and interference, operating conditions of a WSN are hard to describe and simulate or recreate in laboratory. Therefore, creating realistic, controllable and repeatable tests is difficult [4]. Testing embedded software (e.g. for sensor nodes) poses another type of problems due to the lack of input/output devices or the possibilities of error logging. One option is to use JTAG interface to debug a single device, but such an approach is not an option when debugging distributed network algorithms. Debugging protocols is even harder, and non-scalable, as many devices would have to be run in testing mode simultaneously. Also, nodes usually communicate in event driven or asynchronous manner and any test should recreate and check time-dependencies. There is no standard suite of benchmark applications for WSN-based systems, even though there were attempts at creating one. In [2] authors advocate creation of a standardized benchmark suite for TinyOS-compatible WSN nodes. They present sample benchmark results regarding performance and power consumption of different hardware components and call for further work in this area. Such standardized benchmark suite could be used to aid development of future hardware. Nazhandali et al. introduce new metrics in [7] that can be used to evaluate and compare wireless sensor systems as well as a set of applications representative of typical workload in WSNs. Predictions on how a large-scale WSN deployment will work can be made through simulations. This should provide an opportunity to eliminate some errors in early stages of implementation. However, in some cases even results of detailed simulations may stand in contradiction to real physical deployments [4] as it is difficult to simulate wave propagation and hence exact network behavior in sufficient detail. In this work we describe our experiences with testing applications using TelosB nodes operating with TinyOS operating system [6]. Each node is equipped with 3 LEDs that can be used in simple debugging scenarios. E.g. to indicate sending or reception of a packet, sensor being turned on or a timer firing. However, these methods are useless for monitoring the whole network. Moving from simple experiments that test the properties of point-to-point communication [8] towards complete networks, results in increased software complexity. This increased complexity of a complete WSN testbed was previously described in literature. For example, the authors of [5] point to software complexity as one of the key reasons of not reach-
On Testing Wireless Sensor Networks
3
ing the goals of their experiment. They also advocate applying rigorous software engineering approach to manage the growth in software complexity. Operation of a WSN network is real-time in the sense that communication is usually asynchronous with time dependencies affecting the behavior of the whole network. Therefore it is not easy to test the proper operation of the whole network. One approach to this problem is to use traces of network communication that were gathered with a packet sniffer to make test scenarios. Then this recorded communication is injected into the network and new logs are analysed. This should result in much more accurate testing conditions. To summarize, WSN testing can be done at different levels. At each level a different aspect is stressed: 1. node hardware – operation correctness in a range of physical environment conditions (e.g. temperature, humidity), performance, energy consumption with standardized benchmarks (including sleep modes), 2. node software – proper operation in response to any message (including erroneous and/or damaged packets), 3. network software – proper operation of the network protocol stack subject to proper and erroneous messages and varying amounts of communication, 4. system level – network and supporting infrastructure, including the BaseStation resiliency and correctness of the data analysis software.
2 Testing environment architecture For the testing setup we have developed a sniffer-basestation architecture where an embedded system with a Linux operating system with a Wireless Sensor Network mote attached through a USB port acts as both the BaseStation and the network observation point. It can be a laptop computer running any available Linux distribution, or an embedded system, such as BeagleBoard or Raspberry Pi with its native Linux version. The core of our architecture are two programs – TelosBaseStation and NetServ (Fig. 1). The first one is a TinyOS application running on CC2420-equipped motes, such as TelosB or XM1000 architectures. The BaseStation receives all the packets from the radio on a specified channel in a so-called promiscuous mode (i.e. sniffing all the packets that can be heard on the WSN network, not only those which were broadcast or addressed to the BaseStation). The received packets are timestamped and queued, then resent through the USB port. On the other end of this USB connection, the NetServ program receives these packets and makes them available in various ways, by storing them locally (acting as the packet sink) but also acting as a TCP proxy, by offering them to clients in many formats. Multiple clients can connect simultaneously to get these messages either in binary form (i.e. through a transparent USB-TCP proxy), a PCAP-compatible [3] timestamped packet stream (which can be used by Wireshark or other packet monitoring tools), or an ASCII
4
Tomasz Surmacz, Bartosz Wojciechowski, Maciej Nikodem, and Mariusz Słabicki
stream in hexadecimal form. All these formats allow both remote monitoring and remote data logging which supplements local storage facilities. Running the BaseStation software in promiscuous mode gives a better view on what is happening in the network. For the BaseStation-only way of operation unicast transmissions are usually used – packets addressed to the BaseStation node can be found in the incoming sniffer stream and for the logging purposes all the other packets can be filtered out. However, these extra packets can provide information about how messages are routed in the network or a very detailed data about with what transmission power must a message be sent in order to reach other nodes (the BaseStation or the next hop towards the BaseStation). For even better insight into the network operation, more than one sniffing station can be installed. As all the received packets are timestamped and marked with a tag unique to the sniffing station, all these packets can be later analyzed together. The NetServ program can connect simultaneously to USB and TCP sources of packets (i.e. the local sniffing node and the remote NetServ available through TCP) so all these sniffing points can be made available in just one place in real time. Additionally, the BaseStation software provides a reverse channel through which clients can send WSN messages to the BaseStation. These messages can instruct the TelosBaseStation software to change its communication parameters (e.g. change the communication channel or transmit power) or can be sent to the WSN network. In this function the BaseStation software again acts as a proxy, allowing sending any packet, without checking or analyzing its contents – the only modification done is calculating the correct CRC at the end of the packet (which simplifies testing when packets are created manually from client connections by typing them in hex form or using copy and paste functions from some predefined examples). The typical usage however includes connecting to NetServ from a script written in Perl or Python and sending messages, then observing network response in the same script. This allows
WSN
g in m m B US
a gr
o pr
tes tin g TC P
logging monitoring
B US
TCP
TCP
sniffing mote (TelosBaseStation)
NetServ
Fig. 1: Testing setup
On Testing Wireless Sensor Networks
5 nesC header files (structures)
XML packet description
XSLT processing
nesC TinyOS program source
data parser
Wireshark packet dissector
TinyOS application
Wireshark plugin
Fig. 2: Workflow of a WSN protocol development using formalized description
implementing automated test suites based on “send this packet”, “expect such response packet(s)” scenarios. In a similar manner a packet replay capability can be implemented. An external script can log a realistic communication scenario that is sniffed with a TelosBaseStation program and then recreate this communication many times to provide realistic and repeatable communication stress workload.
3 Formalized methods of protocol description Various tools can be used for observing the WSN traffic, starting with showing a raw packet stream, parsing this stream with custom-made interpretation scripts, or using specialized packet-inspection tools, such as tshark or Wireshark. In order to facilitate fast development of communication and control protocols we have adopted a formalized way of specifying the data sent over the WSN. The format of all the packets is specified in form of an XML file, describing the fields and variants of messages to be sent. From this “master” file various resulting formats may be generated automatically (Fig. 2), including header files for nesC TinyOS programs, packet interpretation scripts, and Wireshark dissector code. This simplifies investigating the data captured from the air, allows for throughout analysis of how the network behaves, and also, prevents many bugs that stem from software incompatibilities. Based on the XML packet description it is also possible to generate test cases for network testing or simplify the needed tools. So far, we have used mostly Python scripts for interacting with Netserv program in order to send messages to the WSN network and observe the network response. Examples of such tests are shown in the next section. The WSN traffic is made available by NetServ in different forms – simple byte-stream, PCAP-timestamped binary data or ASCII data in hexadecimal form. Testing scripts may use these data streams to allow automated verification of the tests (compatibility with our XML-based data description as one of the criteria). The PCAP-formatted stream may be connected to a Wireshark network analyzer allowing extended analysis of captured or live data.
6
Tomasz Surmacz, Bartosz Wojciechowski, Maciej Nikodem, and Mariusz Słabicki
4 Testcases For implementing the BaseStation and serving the captured data with NetServ we have used Raspberry Pi embedded systems. Even though a typical laptop or a desktop computer with Linux operating system could be used for that purpose, a self-contained small system like BeagleBooard or Raspberry Pi is much better suited in a real WSN deployment, where it acts as a typical packet sink. As it does not need any keyboard or a display for functioning, it can be just plugged in to the mains supply and work, collecting the packets. When equipped with a WiFi dongle, it can immediately serve the collected data online. This is also useful in network testing scenarios, where various observation points can be created by placing several Raspberry Pi computers at various locations to observe the WSN radio traffic (Fig. 1). Using this architecture we have performed numerous tests for our protocols. The most important ones were the tests of network self-organization, in which the network was switched back and forth between non-organized broadcast-mode operation and the organized mode with routing enabled. Two kinds of control messages were used for this purpose: • RESET – Upon receiving a RESET message, a node sets an internal timer for a time period specified in the message payload. After this delay the node reboots. The delay is added to prevent multiple reboots if the node received the same RESET message several times, as it was retransmitted over the whole network. To test the proper functioning of this routine, we created a scenario where RESET messages were injected into the network with delay parameters varying from 1 ms to 60 s. A misbehaviour of the code would manifest itself in the form of double
repeat { send HELO message collect traffic for n seconds send HELO message again collect more traffic if ( broadcast packets found ) , abort if (no. of hops > threshold ) , abort // routing loops found if (bi-birectional paths ) , abort // routing loops found send RESET message collect traffic for n seconds if ( unicast packets found ) , abort if ( duplicate transmissions found ) , report multiple resets and abort }
Fig. 3: Testing algorithm for HELO/RESET scenario
On Testing Wireless Sensor Networks
7
resets, that is – two or more messages with the same source address and sequence numbers would be seen if the reset was performed multiple times. Also, no unicast packets should be observed after the reset. • HELO – a packet of type HELO is used to let other nodes know about their neighbours. This is necessary to construct a routing tree in each node. Each node is supposed to retransmit every HELO packet it receives, but only once (sequence numbers are used to detect duplicates) and after increasing the number of hops parameter. After exchanging a number of HELO packets, a routing tree should be created. Therefore, the test consists of sending a HELO packet from the BS and analysing communication after a set period of time. All the messages should from thereon i ) be sent as unicasts, ii ) be targeted at the BS and iii ) there should be no routing loops. The automated tests have been performed using algorithm specified in Fig. 3. Sample packets received in these tests are shown in Fig. 4. This example shows network behaviour for two consecutive RESET-HELO-HELO tests. Line 1 shows the RESET packet sent from the BaseStation running on node 47. Therefore the following packets from nodes 7 and 33 (lines 3-5) were sent as broadcasts. That is why the packet with seq no 1 was received twice – directly from the originating node and retransmitted. However, the first packet from node 34 contained sequence number equal 3 (line 7), which means that packets with sequence numbers 1 and 2 were lost. After the HELO packet from the Base Station (line 11) the network started setting the routing tree. Lines 13-15 show packets which should be routed to the Base Station. Unfortunately there is a routing loop between nodes 7 and 33. If not for sniffing, such messages would never reach the Base Station. After another HELO packet in line 17 both nodes 7 and 33 correct their routes to the Base Station. When the RESET-HELO-HELO test scenario is repeated (lines 23-39) nodes 7 and 33 properly establish the routing table, but node 34 apparently did not hear the first HELO packet and still sends broadcasts. After another HELLO all nodes send messages directly to the Base Station. Although the above description shows the detailed “manual” analysis of network behaviour, the automatic assessment can also detect protocol violations, as described in the testing algorithm (routing loops, wrong type of packets at a certain stage of the protocol, assertions on particular field values, etc.). Such automated tests can be run several times to test whether the communication protocols are resistant to errors such as lost or duplicate packets, packet collisions or unfortunate timings.
5 Future work From the testing perspective, our toolchain has functionality to: • setup a testing network, • capture WSN messages on a PC and analyze them,
8
Tomasz Surmacz, Bartosz Wojciechowski, Maciej Nikodem, and Mariusz Słabicki
• inject prepared messages into the WSN network through the gateway, • perform automated tests by executing appropriate test scenarios. Unfortunately, it may still be not enough to accurately observe the whole network, because the wireless links are unreliable. If we do not receive the expected packet, we can not be sure why it happened. Software and protocol errors or a high level of radio noise may be the reasons. Therefore it is necessary to develop methods which can log all activities on each node and then analyze the behaviour of the whole network after the test.
1 15:28:12.668 SEND RESET from =47 , to =* , seq =11 , delay =768 ms ,(7 e 44 55 00 ff ff 00 00 0 e 00 0 c 01 05 ff ff 00 2 f 00 0 b 00 00 00 00 03 00 39 2 d 7 e ) 2 # 2 messages from node 33 , one of them through node 7 3 15:29:02.702 from =33 , to =* , via =7 , seq =1 , hops =2 , photo =8293.15 , path =[7 ,33] 4 02.720 from =33 , to =* , via =33 , seq =1 , hops =1 , photo =8293.15 , path =[33] 5 02.813 from =7 , to =* , via =7 , seq =1 , hops =1 , photo =8426.67 , path =[7] 6 # first packet coming from node 34 7 06.597 from =34 , to =* , via =7 , seq =3 , hops =2 , path =[7 ,34] , photo =255.58 8 # some strange paths 9 16.716 from =34 , to =* , via =7 , seq =8 , hops =3 , path =[7 ,33 ,34] , photo =137.33 10 16.736 from =7 , to =* , via =33 , seq =8 , hops =2 , path =[33 ,7] , photo =8934.02 11 17.008 SEND HELO from =47 , to =* , seq =12 , ( bin : 7 e 44 55 00 ff ff 00 00 0 e 00 0 c 01 05 ff ff 00 2 f 00 0 c 00 00 00 00 03 00 7 d 5 d 34 7 e ) 12 # unicast messages , but routing tree is not ok . 13 18.398 from =34 , to =47 , via =33 , seq =9 , hops =2 , path =[33 ,34] , photo =244.14 14 18.416 from =33 , to =47 , via =7 , seq =9 , hops =3 , path =[7 ,34 ,33] , photo =8319.85 15 18.474 from =7 , to =47 , via =33 , seq =9 , hops =2 , path =[33 ,7] , photo =8888.24 16 # another HELLO 17 12.020 SEND HELO from =47 , to =* , seq =13 , (7 e 44 55 00 ff ff 00 00 0 e 00 0 c 01 05 ff ff 00 2 f 00 0 d 00 00 00 00 03 00 1 c 8 c 7 e ) 18 12.955 from =33 , to =47 , via =33 , seq =37 , hops =1 , path =[33] , photo =8316.04 19 13.001 from =7 , to =47 , via =7 , seq =37 , hops =1 , path =[7] , photo =8937.84 20 14.920 from =33 , to =47 , via =33 , seq =38 , hops =1 , path =[33] , photo =8319.85 21 15.077 from =7 , to =47 , via =7 , seq =38 , hops =1 , path =[7] , photo =8899.69 22 # and 34 is somehow gone ? No sign of it until RESET 23 43.439 SEND RESET from =47 , to =* , seq =14 , delay =1000 ms , (7 e 44 55 00 ff ff 00 00 0 e 00 0 c 01 08 ff ff 00 2 f 00 0 e 00 00 00 00 03 e8 2 c 63 7 e ) 24 # broadcast 25 45.257 from =7 , to =* , via =33 , seq =1 , hops =2 , photo =8361.82 , path =[33 ,7] 26 45.284 from =33 , to =* , via =33 , seq =1 , hops =1 , photo =7667.54 , path =[33] 27 49.172 from =34 , to =* , via =7 , seq =3 , hops =2 , path =[7 ,34] , photo =251.77 28 51.596 SEND HELO from =47 , to =* , seq =15 , (7 e 44 55 00 ff ff 00 00 0 e 00 0 c 01 05 ff ff 00 2 f 00 0 f 00 00 00 00 03 00 ff ec 7 e ) 29 # 34 did not get the HELO message 30 53.007 from =34 , to =* , via =33 , seq =5 , hops =2 , path =[33 ,34] , photo =205.99 31 54.955 from =33 , to =47 , via =33 , seq =6 , hops =1 , path =[33] , photo =8251.19 32 54.977 from =7 , to =47 , via =33 , seq =6 , hops =2 , path =[33 ,7] , photo =8857.73 33 ... 34 # another HELO 35 15:30:32.947 SEND HELO from =47 , to =* , seq =16 ,(7 e 44 55 00 ff ff 00 00 0 e 00 0 c 01 05 ff ff 00 2 f 00 10 00 00 00 00 03 00 6 d 51 7 e ) 36 # and each node sends directly to BS (47) 37 34.065 from =33 , to =47 , via =33 , seq =26 , hops =1 , path =[33] , photo =8285.52 38 34.083 from =7 , to =47 , via =7 , seq =26 , hops =1 , path =[7] , photo =8831.02 39 34.097 from =34 , to =47 , via =34 , seq =26 , hops =1 , path =[34] , photo =469.21
Fig. 4: Sample packets observed by the testing script
On Testing Wireless Sensor Networks
9
We intend to develop two solutions for this problem: 1. prepare a meta-protocol which can be used for sending debugging information, i.e. routing tables, packet reception rates, RSSI values, etc. 2. gather all debug information in node’s Flash memory and then, after the experiment, collect the data via serial or radio connection. Both methods have some disadvantages. The first one may interfere with the normal network operation and the protocols being tested, as the same underlying communications stack must be used for sending and receiving radio messages. The debugging messages may also be lost, and implementing a special reliable protocol with no impact on other messages sent through the network seems hard, if possible at all. The second method can be implemented only on a Flash-equipped WSN motes and requires direct access to each node after the test, if serial/USB transmission is to be used. However, the log transfers are intended to happen after the testing has ended, so the radio transmission can also be used with reliable protocols, using acknowledgements and retransmissions while asking nodes one by one to dump their debugging logs over the radio. On the other hand, some nodes may still need manual intervention if during the test they have exhausted their power supplies.
6 Conclusions When developing communication protocols for WSNs it is hard to effectively test the proper operation of the whole network. The system that we have developed extends the functions of a typical BaseStation node by allowing us to send arbitrary messages from the BS and observing the network response through capturing and analyzing the packets sent between WSN nodes. Test scenarios can be programmed in Python (or any other scripting/programming language) and use TCP client-server model for interaction with the network. Automated assessment of repeated test results is possible and the packet trail of network operation can be used for detailed postfactum analysis of the behaviour of network nodes. To make this trail complete, we intend to extend our system to provide local logging of node operation in node’s Flash memories. After the experiment has ended, this data can be downloaded through a reliable communications link. The BaseStation-Netserv architecture of our system allows both logging such trails and providing a real-time access to WSN messages through TCP client-server connections. Wireshark packet analyzer can be used as a powerful tool for online or post-factum analysis of captured packets. Acknowledgement: This work was supported by National Science Centre grant no. N 516 483740.
10
Tomasz Surmacz, Bartosz Wojciechowski, Maciej Nikodem, and Mariusz Słabicki
References [1] Berezowski K (2012) The landscape of wireless sensing in greenhouse monitoring and control. International Journal of Wireless & Mobile Networks (IJWMN) 4(4):141–154 [2] Hempstead M, Welsh M, Brooks D (2004) TinyBench: the case for a standardized benchmark suite for TinyOS based wireless sensor network devices. In: Local Computer Networks, 2004. 29th Annual IEEE International Conference on, pp 585–586, DOI 10.1109/LCN.2004.129 [3] Jacobson V, Lere C, McCanne S (2009) libpcap: Packet capture library. Lawrence Berkeley Laboratory, Berkeley, CA [4] Langendoen K (2006) Apples, Oranges, and Testbeds. In: Mobile Adhoc and Sensor Systems (MASS), 2006 IEEE International Conference on, pp 387–396, DOI 10.1109/MOBHOC.2006.278578 [5] Langendoen K, Baggio A, Visser O (2006) Murphy loves potatoes: experiences from a pilot sensor network deployment in precision agriculture. In: Parallel and Distributed Processing Symposium, 2006. IPDPS 2006. 20th International, DOI 10.1109/IPDPS.2006.1639412 [6] Levis P, Madden S, Polastre J, Szewczyk R, Woo A, Gay D, Hill J, Welsh M, Brewer E, Culler D (2004) TinyOS: An operating system for sensor networks. In: Ambient Intelligence, Springer Verlag [7] Nazhandali L, Minuth M, Austin T (2005) SenseBench: toward an accurate evaluation of sensor network processors. In: Workload Characterization Symposium, 2005. Proceedings of the IEEE International, pp 197–203, DOI 10.1109/IISWC.2005.1526017 [8] Słabicki M, Wojciechowski B, Surmacz T (2012) Realistic model of radio communication in wireless sensor networks. In: Computer Networks, Communications in Computer and Information Science, vol 291, Springer Berlin Heidelberg, pp 334–343
