TCP/IP Over SpaceWire S. J. Mills and S. M. Parkes University of Dundee, Applied Computing, Dundee, DD1 4HN, Scotland, UK Phone: +44 1382 345194, +44 1382 344753 Fax: +44 1382 345509 Email:
[email protected],
[email protected]
Abstract The SpaceWire standard defines a network designed for handling payload data and control information onboard a spacecraft. Among the goals of SpaceWire are re-usability and reliability. The use of network protocols on top of SpaceWire is expected to enhance the already rich re-usability and reliability characteristics of SpaceWire. The Space Systems Research Group at the University of Dundee have developed software to allow data to be sent over SpaceWire using standard network protocols such as TCP/IP, as part of a program of work looking at network protocols for SpaceWire. This paper describes network protocols and their relationship to SpaceWire, introduces software written to allow network protocols to be layered on to SpaceWire, describes the research behind the development of this software, and gives results of some tests and observations made using the software.
1. Introduction SpaceWire is a network designed for handling payload and control information onboard a spacecraft [1], [2], [3], [4], [5]. One of the goals of SpaceWire is to improve re-usability. This re-usability can be further enhanced through the use of standard network protocols on top of SpaceWire, so that the entire communications stack is re-usable. Another goal of SpaceWire is to provide high reliability. Network protocols such as TCP/IP include services to cope with any errors which may not be dealt with by the hardware. These protocols are being considered for wider adoption onboard spacecraft by the Consultative Committee for Space Data Systems (CCSDS) Panel 1K - Spacecraft Onboard Interfaces (SOIF). The CCSDS was formed by the world’s major space agencies to discuss common problems in the development and operation of space data systems. Since its formation it has issued a number of recommendations used by space agencies and companies throughout the world. The SOIF group intends to issue CCSDS recommendations for the use of networking protocols onboard spacecraft. Among the SpaceWire-related research and development the Space Systems Research Group at the University of Dundee has performed is the development of a driver to allow delivery of data over SpaceWire using protocols such as TCP/IP. This paper starts by briefly describing some of the networking protocols considered for data transfer over SpaceWire, concentrating on the Internet protocol suite, TCP/IP, and the CCSDS Space Communications Protocol Specification (SCPS). It then goes on to
describe a driver written for the purpose of transmitting data over SpaceWire using these protocols. Finally some of the results obtained through testing with the driver are provided.
2. Networks and Networking Protocols In order to send data across a network reliably, be it a small network or the Internet, the use of common networking protocols is required so that nodes on the network can successfully talk to one another. Most networking protocols rely on a layered model, such as the Open Systems Interconnect (OSI) reference model [6], shown in Figure 1.
Application Layer
Application Layer
Presentation Layer
Presentation Layer
Session Layer
Session Layer
Transport Layer
Transport Layer
Network Layer
Network Layer
Data Link Layer
Data Link Layer
Physical Layer
Physical Layer
Figure 1: Communication between layers in the OSI stack
Each layer is responsible for different parts of the communication. For example, the transport layer manages the end-to-end communication, performing such operations as flow control and transmission error management, while the network layer handles the routing of data at each of the points between the two communicating nodes. A layer provides services to the layer above and requests services from the layer below. Although only the Physical Layer provides a physical connection between the nodes, each of the layers in the stack communicates with its corresponding layer on other nodes. For example, the Network Layer on one node will communicate with the Network Layer on another node. Nodes on the Internet communicate using the TCP/IP suite, which is similar in form to the OSI stack. The main difference is that the Application, Presentation and Session layers in
the OSI stack are represented by a single Application layer in TCP/IP. TCP/IP may operate over a number of different physical layers (network media). In order to allow TCP/IP to be layered on top of a particular network medium, an interface must be provided in the Data Link Layer, known as the Hardware Interface. The Network Layer element of the TCP/IP stack is provided primarily by the Internet Protocol (IP), an unreliable, connectionless protocol [7]. IP sends encapsulated chunks of data, called datagrams, from one node to another. Transmitted IP datagrams are not guaranteed to reach their destination, and connection state information is not held from one datagram to the next. At the transport layer, the Transmission Control Protocol (TCP) provides a connectionoriented, reliable, full duplex, byte stream service [8]. TCP is connection-oriented in that two applications using TCP, normally a client and a server, must first establish a TCP connection before they can exchange data. A connectionless service is also provided at the transport layer by the User Datagram Protocol (UDP). Many extensions and improvements have been made to TCP/IP over the years to cope with unusual operating conditions, faster transmission speeds, and new requirements. Among these developments is the Space Communications Protocol Specification (SCPS), a protocol suite designed for space communications between ground stations and spacecraft, which are typically subject to long delays and greater corruption than is experienced between two hosts communicating over a terrestrial link [9]. SCPS includes modifications and replacements to the TCP/IP suite to cope with these and other unusual operating conditions. Although work has been done on defining the SCPS standard for communication between ground and spacecraft, there are currently no standards for the use of network protocols onboard spacecraft. In many cases onboard communication protocols are re-written for each mission. The Consultative Committee for Space Data Systems (CCSDS) Panel 1K Spacecraft Onboard Interfaces Working Group (or SOIF) has therefore been set-up to address this problem. One of the decisions made so far by the SOIF group is that both TCP/IP and SCPS should be considered and investigated for use onboard spacecraft [10]. The SOIF Draft Network Layer Green Book [11] recommends the investigation of IP or the SCPS Network Protocol (SCPS-NP) for the Network Layer protocol onboard spacecraft while the Draft Transport Layer Green Book [12] recommends the investigation of UDP and TCP or the SCPS Transport Protocol (SCPS-TP). Note that CCSDS has been restructured recently and the work on Network and Transport Layer protocols is now being done in the Time Critical Network Services working group. The SpaceWire standard does not define a Transport Layer or full Network Layer. End to end flow control and retry mechanisms are not detailed in the standard. The University of Dundee are investigating suitable network and transport protocols for SpaceWire. The results of initial research considering TCP/IP and other protocols will now be described.
3. SpaceWire-PCI Network Driver Overview A PCI board has been developed by the University of Dundee and 4Links to allow PCs to send data over up to three SpaceWire links. This board is known as the SpaceWire-PCI board. A Linux network driver was written for the SpaceWire-PCI board to provide the Hardware Interface between networking protocols such as TCP/IP and the Physical Layer, in this case SpaceWire. It allows the SpaceWire-PCI board to be used in a similar way to Ethernet cards for connecting PCs to a SpaceWire local area network. In order to use network protocols over network hardware, a method of sending data must be defined for that particular hardware. This involves encapsulating the data to be transmitted in a packet with a format recognised by the network hardware. Data encapsulation specified in the SpaceWire standard only provides the destination address in a packet header. To support the simultaneous transfer of several different protocols over a SpaceWire network a protocol identifier is needed in the header. A source identifier and data length were also added to give the encapsulation format shown in Figure 2. The source and destination addresses are both SpaceWire logical addresses.
Dest. Src. Addr. Addr.
Type
1 byte1 byte 2 bytes
Data Length
IP datagram
4 bytes
Figure 2: SpaceWire Encapsulation Format
In addition to the network interface to the SpaceWire-PCI Driver, a low-overhead interface is provided to support a recommendation of the SOIF group. This interface allows data to be sent directly over SpaceWire without the overhead of protocols such as TCP/IP, but obviously without the benefits provided by these protocols. This means that devices using the driver can communicate with “dumb” devices - those that do not have support for network protocols. This is achieved through use of a “Type” field in the encapsulation header which distinguishes between different network protocol packets and low-overhead interface packets. The Maximum Transmission Unit (MTU) of a network is the largest size of datagram that can be transmitted over that network. At first thought it may seem that the best value for the MTU is the largest possible value that can be supported, however this is normally not the case. Use of a large MTU can result in large packets “hogging” the network while they are being transported, causing other packets to wait. Large packets can significantly increase the latency for packets that have to wait. The memory on the SpaceWire-PCI board limits the maximum packet size that can be sent to 21844 bytes. Eight bytes are used for encapsulation reducing the maximum MTU to 21836 bytes. Tests were
performed with standard Internet applications and custom-built data communications applications with MTUs of between 256 and the maximum of 21836 to determine the effect of changing the MTU for SpaceWire.
4. Performance of Transport Protocols over SpaceWire
A number of tests were carried out to evaluate the performance of TCP/IP over SpaceWire. The SpaceWire links here operated at 100 Mbits/s (although higher data rates are possible) so that the test results could be directly compared to 100 Mbits/s Ethernet. The result of one of the tests, measurement of the average time to transfer a 10 MByte file, is shown in Figure 3.
10 MByte Transfer 1.4 1.235
1.228
1.217
1.2
1.12 1.073 1.021
1
Time (seconds)
1
0.8 0.666 0.62 0.6
0.4
0.2
0 Ethernet (MTU = 1500)
SpaceWire (1 link, MTU = 1500)
SpaceWire (2 links, MTU = 1500)
SpaceWire (3 links, MTU = 1500)
SpaceWire (via router, MTU = 1500)
SpaceWire (1 link, MTU = 21836)
SpaceWire (2 links, MTU = 21836)
SpaceWire (3 links, MTU = 21836)
SpaceWire (via router, MTU = 21836)
Figure 3: Comparison of average transfer times of a 10 MByte file
As can be seen in Figure 3, the average time to transfer the 10 MByte file over Ethernet, with the default MTU of 1500 bytes, is 1.235 seconds (~65 Mbits/s). This provides a good baseline for comparison with SpaceWire. With the same MTU, the average time to transfer the file over one SpaceWire link is slightly quicker. This can be attributed to the smaller 8 byte header used with SpaceWire, compared to the 14 byte header used by Ethernet (in addition to a 4 byte CRC). Since the SpaceWire-PCI board has three SpaceWire links, tests were also performed using two and three links to transfer data. If two SpaceWire links are used the transfer
time is not halved as may be expected. When the SpaceWire driver receives a packet it will normally send it out the first link. If that link is in use, however, it will send it out the second link, or out of the third link if the second link is in use. If the operating system does not pass the packets to the driver quickly enough, only one of the links will be used. This is likely to be the case when sending very small packets, as the transmission time will be very small. For this reason, although there is a significant improvement when using two SpaceWire links, it is not twice as quick as using one link. Similarly when using three links there is only a slight improvement over two links. A SpaceWire router [13] was added to the network for one test, to measure the effect on performance. Using the router to transfer the file with an MTU of 1500 gives a slightly better average transfer time than when using one SpaceWire link without a router. The reason for this is that the router used transmitted data at 200 Mbits/s. Although the SpaceWire-PCI board sends data at 100 Mbits/s the receiving end receives the data at 200 Mbits/s, accounting for the small increase in performance. When the MTU is increased to the maximum possible value that may be used with the SpaceWire-PCI driver, the benefits of using more than one link with SpaceWire can be seen. The average transfer times when using one link with an MTU of 21836 bytes are slightly faster than when using the same configuration with an MTU of 1500 bytes. This improvement of around 10% is due mainly to the reduced number of bytes occupied by headers, as fewer packets are needed to send the same amount of data. When using two links, however, the driver gets a chance to make use of both links more often, as the larger packets take longer to send than when using an MTU of 1500. For this reason the average transfer time is around 40% quicker. Using three links there is yet another improvement, although not as significant since the third link is rarely called into action. Similar results are obtained when using a router in the test configuration. The SpaceWire-PCI driver was also used to demonstrate that SCPS could be transferred over SpaceWire, using the same methods used to transfer TCP/IP. Through use of both TCP and SCPS-TP it was also shown that a reliable protocol could be used to recover following a SpaceWire disconnect error. Using both these protocols at the same time, along with the low-overhead interface, demonstrated that multiple protocols may be sent over a SpaceWire network at the same time. Many of these tests were performed using standard Internet applications such as the Ping program with no modifications being required for them to work over SpaceWire. This not only demonstrates that once SpaceWire can be used to carry TCP/IP packets, a number of previously written applications may be used over it, but also that applications may be tested over other networking hardware such as Ethernet, before being tested over SpaceWire without any changes being made.
5. Disadvantages of TCP/IP
Although SpaceWire and TCP/IP have been shown to perform well with MTUs of 1500 or greater, as the MTU drops below this value the overhead of TCP/IP has a far greater effect
on performance, as shown in Figure 4. The TCP/IP header has a total minimum length of 40 bytes, 20 bytes for TCP and 20 Bytes for IP.
Percentage of MTU occupied by headers 25
256
20
Percentage
15
10
1024 1280
5
2048 3072 4096 5120
8192
16384
20480 21836
0 0
5000
10000
15000
20000
25000
MTU (Bytes)
Figure 4: Overhead caused by headers with various MTUs
As shown in Figure 4, an MTU of 256 results in almost 20% of the data sent consisting of header information. Onboard a spacecraft the MTU may be lower than 256 to allow all nodes reasonable access to the available bandwidth, decreasing latency. Some forms of data, for example control information, are naturally less than 256 bytes, so small packets will be sent regardless of the MTU specified. Although the headers used by SCPS are smaller and can be compressed, the overheads may still be relatively large compared to the total packet size in such situations. This means that although the use of TCP/IP or SCPS provides many benefits, the standard implementation may not be suitable for use with SpaceWire onboard spacecraft because of the high overhead it imposes on small packets. The University of Dundee is investigating the use of an alternative reliable link layer protocol over SpaceWire because of the higher overhead of TCP/IP for small packets. This will provide a low overhead mechanism for flow control and packet retry. It will also provide a method for supporting TCP/IP and similar transport and network protocols, and will provide a mechanism for using TCP/IP and similar protocols without the large header overhead currently limiting their use.
6. Conclusion
It has been demonstrated that transport and network protocols such as TCP/IP can be efficiently layered over SpaceWire. In comparison with Ethernet, both running at 100 Mbits/s, SpaceWire gave slightly better performance. It has also been shown that reducing the overhead of protocols such as TCP/IP may be necessary before they can be widely used onboard spacecraft. For this reason the Space Systems Research Group at the University of Dundee is continuing research into the development of lighter transport and network protocols, with less overhead.
7. Acknowledgements
The Authors would like to acknowledge the support of ESA for the DABUTUS project (ESA Contract No. 15128/01.NL/JA) on which this paper is based. The stimulating discussions on network and transport layer protocols for spacecraft onboard applications with members of the CCSDS SOIF working group have been valuable in driving the research reported here.
8. References
[1] European Space Agency, “SpaceWire Web Page”, http://www.estec.esa.nl/tech/spacewire/. [2] European Space Agency, “SpaceWire, Links, Nodes, Routers and Networks", Draft 3, ECSS-E-50-12A, European Cooperation for Space Data Standardization, November 2002. [3] Parkes S. M., McClements C., “SpaceWire Networks”, DASIA 2002, ISSN 1609 042X, ISBN 92-9092-819-0. [4] Parkes S. M., Rosello J., “SpaceWire – Links, Nodes, Routers and Networks”, DASIA 2001, ISSN 1609 042X, ISBN 92-9092-773-9. [5] Parkes S. M., “SpaceWire: The Standard”, DASIA ‘99, ISBN 92-9092-788-7. [6] International Telecommunication Union, “Open Systems Interconnection - Basic Reference Model: The basic model”, Recommendation X.200, July 1994. [7] Postel J., Ed, “Internet Protocol - DARPA Internet Program Protocol Specification”, Request for Comments: 791, Information Sciences Institute, University of Southern California/Defense Advanced Research Projects Agency, September 1981. [8] Postel J., Ed, “Transmission Control Protocol - DARPA Internet Program Protocol Specification”, Request for Comments: 793, Information Sciences Institute, University of Southern California/Defense Advanced Research Projects Agency, September 1981.
[9] Consultative Committee for Space Data Systems, “Space Communications Protocol Specification (SCPS) – Rationale, Requirements, and Application Notes”, Draft Green Book, CCSDS 710.0-G-0.3, Consultative Committee for Space Data Systems, April 1997. [10] Consultative Committee for Space Data Systems, “Spacecraft Onboard Interfaces – Concepts and Rationale”, Draft Green Book, Consultative Committee for Space Data Systems, August 2002. [11] Consultative Committee for Space Data Systems, “Spacecraft Onboard Interfaces – Network Layer”, Draft Green Book, Consultative Committee for Space Data Systems, October 2002. [12] Consultative Committee for Space Data Systems, “Spacecraft Onboard Interfaces – Transport Layer”, Draft Green Book, Consultative Committee for Space Data Systems, September 2002. [13] Parkes S.M., McClements C., Mills S. and Martin I., “SpaceWire: IP, Components, Development Support and Test Equipment”, DASIA 2003, to be published.
