Edge Server Software Architecture For Sensor ...

Edge Server Software Architecture For Sensor Applications Sean Rooney, Daniel Bauer, Paolo Scotton IBM Research, Zurich Research Laboratory S¨aumerstrasse 4, 8803 R¨ uschlikon, Switzerland E-mail: {sro,dnb,psc}@zurich.ibm.com Abstract Edge servers perform computation on data in addition to that performed by the application server. Typically edge servers are deployed just in front of the application server and perform simple generic functions such as load balancing. With the increasing integration of sensors into Internet based applications we foresee the need for the deployment of edge servers at the other end of the network, i.e. at the boundary with the external sensor network. These edge servers reduce the amount of data that has to be sent to the application server by filtering, summarizing and aggregating the sensor data. They are distinct from existing servers in both the location they run and the application-specific functions they perform. Both these differences influence their design. Here we describe our implementation of such an edge server.

1. Introduction The continuing reduction in the cost of both computation and network bandwidth is enabling applications which use large numbers of cheap sensors that monitor the dynamically changing state of some underlying system and forward measurement data to a central server site. The central server site typically employs an application server that processes measurement data, stores it in a database, and provides an interface to the application user. Example applications are flood warning applications which monitor the water levels in a countries water ways, inventory management applications which track the location and number of goods using radio frequency identification and traffic jam warning schemes which monitor the behavior of vehicles in order to detect and predict congestion. An important change in state in the system is

called an event 1 . Sensors inform the application server about the occurrence of events through notification messages. The required service rate of the application server is determined by the arrival process of events and the maximum acceptable propagation delay of notification messages. Buffering event information in the sensors allows the message arrival process at the application server to be smoother than the aggregated event arrival process of the sensors. If the event arrival process is very bursty this may significantly reduce the maximum required service rate at the application server. Moreover, buffered event information may be combined without information loss or superfluous data can be identified as such and dropped, further reducing the message rate. If sensors within some range can communicate among themselves then information can be combined across multiple sensors, e.g. sending an average reading. Many sensors do not execute in an environment in which they can be externally powered. They depend on internal batteries and have to be very simple to reduce their power consumption. They have small buffers that often are only sufficient to store information about a single event and their execution logic does not allow collaboration with other sensors, they simply broadcast or transmit messages to a preconfigured destination. For example, the minimal requirements for a smart transducer given in [9] has a 8 bit processor with 4 Kbytes of ROM and 64 bytes of RAM. In those applications where either there is a small maximum acceptable delay between event occurrence and message reception or where the sensors are too simple to buffer messages, the message rate 1

The terminology in the literature is somewhat confused; a distinction is often not made between the message that reports an event and the event itself. We shall use the same terminology consistently even when this means referring to a “CORBA message” when within CORBA [8] it would be referred to as a “CORBA event”

at the application server is mainly determined by the sum of the peak rates of all the sensors. Potentially, this can be very large. Load Balancing Edge Server

Clients PCs

Application Servers (1) E-Commerce type Application Sensor Networks Aggegating Edge Servers

Application Servers

messages are received and to which messages are forwarded and the channel controller which tracks to where messages should be published and from where they can be subscribed.

2.1. Application Boosters In order to distinguish between the software running on the application server and the applicationspecific software running on the edge server we term the latter a booster. The name emphasizes that the edge servers are there to enhance performances. There may be many different boosters running on the same edge server. Boosters receive messages from one or more input message channels, process them and forward them to one or more other output message channels. The technologies that support the input and output message channels may be distinct.

(2) Sensor application Completions

Figure 1. Integrating sensor networks with enterprise applications

In order to reduce the load on the application server an edge server may be placed at the boundary between the sensor network and the Internet. This edge server performs application specific processing on the data before forwarding it to the application server. This processing reduces the amount of data the application server has to receive, for example by filtering or aggregating the raw sensor data. Figure 1 shows the distinction between these edge servers and more traditional ones. As the edge servers are numerous and remotely distributed it is important that they be self managing and resilient to failure. As the logic they execute is applicationspecific it is important that they be easily extensible and modifiable. In what follows we describe our current edge server software implementation which achieves these twin design goals.

2. Edge Server Software Architecture The edge server runs on a single processor either a cabled PC or a lighter weight battery powered platform such as a PDA. Our software runs on a Java virtual machine either standard or micro edition. The three major components within the architecture are: the application-specific code that processes the messages, message channels from which

handleResult(AsyncResult res) { ... }

handleResult(AsyncResult res) { ... }

Proactor Thread

AsyncResult

Result Queue

Thread Pool

Threads running asyncronous operations

Figure 2. Proactor model for supporting boosters

The edge server software architecture is built around a proactor [10]. There is one proactor instance per booster. Boosters add asynchronous operations to the proactor which allocates them to threads taken from a thread pool. The asynchronous operations deliver the result from the operation to the proactor which then dispatches them to booster defined handlers. From the application programmers view the booster is defined by a list of asynchronous operations and the handlers that should be used to treat the results from those operations. The environment offers a set of stan-

dard asynchronous operations, these include those required to register and receive or transmit messages from different messaging systems such as the TAO/JacORB [2] of the CORBA notification service, IBM Research’s TSpaces [6] implementation of a tuplespace, IBM’s commercial MQ messaging system [5] and the SCADA sensor protocol MQTT [4] 2 . In addition, there are non messaging asynchronous operations of which the most useful are timers. Application programmers may add new asynchronous operations. Figure 2 shows the interaction between handlers, the proactors and the asynchronous operations. Example: an application programmer wishes to process a set of messages received from a MQTT broker before transmitting them to a application server using MQ. A maximum of 100 MQTT messages are mapped to 1 MQ message, but no message is held by the edge server longer than 30 seconds. The application programmers uses three asynchronous operations: • MQTTConsumerOp passing the address of the broker service and the type of the message to consume; • JMSMQProducerOp passing the address of the MQ broker and the type of the message to produce; • TimerOp passing the expiry period. The message types are simply represented as strings, which the MQTTConsumerOp maps to a filter expression at subscription to the MQTT broker and the JMSMQProducerOp maps to a JMS topic name. The application also writes the handler that does the application-specific processing, this handler takes JMSMQProducerOp as argument and is associated with MQTTConsumerOp and TimerOp. Every time an MQTT message is received it is dispatched to the handler, when either a 100 such messages have been received or the timer has elapsed the result is processed and the result placed in JMSMQProducerOp. This asynchronously formats the result as a JMS message and transmits it to the broker. The approach is similar to that outlined in [11] but unlike that one, it is independent of the service discovery mechanism, and publish/subscribe mechanism, i.e. Jini and JavaSpaces respectively. The OSGi [12] specify a service platform for networked appliances. Code is installed on this platform in the form of a bundle, a bundle is just a jar 2

Note that MQTT was original known as MQIsdp.

file with additional meta information describing the resources that the bundle requires, the services that the bundle imports/exports and the means to activate/deactivate the bundle. Clients can request the installation of new bundles from a bundle server or the bundle server can push bundles to the clients. The bundle server keeps track of the dependencies between bundles and negotiates with the clients to ensure that an update will not render them inconsistent. The OSGi also specifies a set of standard bundles offering useful services for the platform. We use the IBM implementation of the OSGi platform – the Service Management Framework (SMF) – for the configuration management of the boosters. Each booster is implemented as an active bundle, starting the bundle creates the appropriate asynchronous operations and the environment in which they run, stopping it remove them. The generic edge server code: proactor, thread pool, standard asynchronous operations, etc, is implemented as a library bundle.

2.2. Message Channels There is no single universal messaging format and so our edge server is designed such that it can handle many different ones and be easily extended as new ones appear, for example the emerging Web Service notification system [3]. The software that supports the various channel technologies, are themselves run as loadable bundles within the edge server. An important distinction between messaging systems is whether a message is retained within the channel until it is explicitly removed by a consumer — consume semantics — or whether it is dispatched to all subscribed entities and automatically removed from the system — multicast semantics. The former is most useful when a message should be consumed by exactly one consumer, while the latter is useful for the dissemination of the same message to many consumers. The consume semantics allows producer and consumer to be detached in time, i.e. the consumer may not even have been activated when the producer places the messages in the channel. IBM’s WebSphere MQ [5] is an example of a messaging system based on queues that provides consume semantics. The CORBA Notification Service [8] is an example of a system that offers multicast semantics. We have found with our trial applications that typically consume semantics are more useful for the treatment of the sensor data, as the systems that we have implemented are hierarchical in nature with the sensors as leaves of a tree and the application

server as root. An edge server is responsible for some region of interest, it subscribes to all input channels containing messages for that region of interest. Many edge servers may have the same region of interest if the volume of message is high or for reasons of fault tolerance, but each message is consumed by exactly one edge server. For example, in a system developed for the monitoring of GSM signaling messages to extract the location and velocity of vehicles on roads, an edge server was assigned to each base station controller. The edge server receives the handover messages between GSM cells. It identifies if the handover was for a vehicle on a motorway based on how far the handset has traveled since its last handover and if so on which stretch of motorway it is by correlating cells to a map. This enriched information is forwarded to the application server that then uses it to determine how vehicles are circulating on that stretch of motorway and combined with historic information attempts to predict traffic jams. The standard approach to the connection of sensors and activators (collectively called transducers) within a control network is through the use of a field bus, for example (CAN) [1]. Several approaches for integrating sensors into larger enterprise networks have been developed. These differ in the capacities they assume of the transducers. One approach is Arcom and IBM’s MQ Telemetry Transport (MQTT) [4]. It is a TCP/IP based protocol that has been designed to connect SCADA (Supervisory Control and Data Acquisition) devices to a pub/sub messaging system. MQTT minimizes the required bandwidth by using a very small header of two to three bytes. It offers three levels of delivery assurance. “At most once” involves no MQTT acknowledgment, “at least once” involves a receipt from the pub/sub system and “exactly once” uses a receipt and receipt acknowledge. Furthermore, MQTT actively monitors connectivity using heartbeat messages in order to detect failures, which is important in a wireless environment. It supports the notion of a “last will” message that is published by the pub/sub system once the connection to a client is lost. MQTT implicitly assumes that sensors are capable of supporting TCP/IP, which is inappropriate for low-end sensors. The IEEE 1451 [7] standard supports very lowend sensor. Its objective is to utilize existing control networking technology and develop standardized connection methods for smart transducers to these control networks. Transducers are connected through a transducer independent interface to a net-

work capable application processor (NCAP). This interface is based on synchronous serial communication that tightly attaches the sensors to the NCAP. The NCAP implements an object model that includes functional blocks from the application level as well as a concise description of network and transducer environment. The network communication model supports both point-to-point as well as publish/subscribe semantics for the communication between NCAPs. As a consequence, an implementation of the NCAP itself is a rather heavy-weight component, perhaps explaining why, to the best of our knowledge, it has not been deployed. The tight coupling of application dependent code within the NCAP to the sensors in the IEEE 1451 standards make this approach less suited for an environment where components need to be dynamically added. The OMG Smart Transducer Interface (STI) [9] proposes a means by which transducers can be clustered together via a field bus controlled by a master transducer. The transducers use the bus to communicate with a simple file system in which each file contains a maximum of 256 4-byte records. Files are assigned to transducers which can read and write data to them. In addition, there are special files containing control information such as the time slots that are allocated to the transducers on the bus. The file system is the means by which the transducer clusters can communicate both with one another and with external gateways. As this is an OMG standard, a CORBA gateway is proposed but nothing in [9] seems to require CORBA. MQTT is designed to be a minimal IP based messaging protocol. It would be natural to use MQTT as the protocol of choice to attach gateways to the enterprise network, thereby minimizing the requirements on the gateways themselves. However, MQTT has been designed for message dissemination rather than message consumption and therefore it is not applicable in our model. While sensors require a very light-weight protocol for producing messages, edge servers are able to handle a more complex protocol that allows for more functionality. In order to cover the different requirements of sensors and edge servers, we have developed an asymmetric message queuing system (AMQ) that uses a very light-weight protocol for senders and a more complex protocol that includes flow-control mechanisms for receivers. A typical usage scenario is depicted in Figure 3, where sensors enqueue messages using UDP/IP and edge servers receive these messages using a channel protocol over TCP/IP. While we have used UDP/IP to attach the sensors, other

light-weight protocols might be used.

UDP/IP

Channel protocol over TCP/IP

Sensor

Edge Server Sensor Asymmetric Queue Sensor

Sensor

Edge Server

GW Channel Controller

Figure 4. Range of Operation Figure 3. Use of an Asymmetric Message Queue

This approach allows us to move data away from the sensors as quickly as possible, such that the sensors do not have to buffer messages. Typical sensor data is redundant in nature and some loss can be tolerated, in which case UDP’s best effort semantic is adequate. Alternatively, sensors might use TCP to transport messages reliably. However, message loss might still occur in case the queue overflows. In order to prevent such a situation, the AMQ sends warnings to its associated channel controller as soon as the queue size crosses a threshold. In this case, administrative counter-measures can be taken such as installing additional edge servers to drain the queue faster, or redirecting sensors to another, less loaded queue. On the receiver side, a channel protocol with a credit-based flow control mechanism on top of TCP is used. When subscribing to the AMQ, each receiver reports it initial capacity, i.e. configures the initial credit. The AMQ drains this credit each time it dispatches a message to a receiver. Receivers, on the other hand, renew their credits after they have processed a given number of messages. This scheme allows to implement a load-balancing mechanism within the AMQ. In our current implementation, we have implemented a priority-based scheduler that dispatches messages to the receiver with the largest outstanding credit. Since credit updates reflect the relative processing capabilities of the edge servers, such a system achieves a good load-balancing. Figure 4 shows AMQ’s range of operation on a standard PC (Intel P4-2.5 Ghz). For the purposes

of measurement we used TCP for the transport of data both to and from the queue as this allows us to compare the performance of the AMQ with other messaging systems. The range of operation depends on the cumulative message rate as well as the number of sensors that are simultaneously connected to the queue. The white area shows the range where the system is stable, i.e. the message output rate matches the message input rate. In the darker areas, the output rate does not match the input rate and the queue is continuously growing; the system is unstable. The corner points of the operation’s range are 2800 sensors with a cumulative rate of no more than 200 Msgs/s or 200 sensors with a cumulative rate of 3000 Msgs/s.

2.3. Channel Control Parties interested in producing or subscribing data messages first subscribe to a meta-channel. Producers learn about the channels that are available for producing messages of their type, using some algorithm for choosing between them if there are many, e.g. the least loaded channel with the highest number of existing consumers for that type. Consumers use the meta-channel to determine the locations of the channel where messages of the type they are interested in are being produced. There is nothing special about the meta-channel other than the information it carries and it can be implemented using any broker. In our current implementation we have used both the CORBA Notification system and an MQ broker. The ChannelDescription information is published on the meta-channel by channel con-

trollers. Each channel within the system is associated with a channel controller which may reside within the same process as the channel or within an entirely different process. Note a given process may contain controllers for many different channels, however controllers must always reside on the same machine as the channel which they control in order to reduce the possibility of inconsistency within the system. The channel controllers periodically publish the ChannelDescriptions on the meta-channel, all receivers of ChannelDescriptions must treat them as softstate, i.e. information which must be refreshed regularly or expired. The ChannelDescriptions contains the following information: • the type of messaging system e.g. Corba Notification Service or WebSphere MQ; • the information needed to connect to the channel, e.g. address, port;

Receivers may make a local decision to ignore the alleged failure of a channel due to non reception of a ChannelDescription if they are successfully sending messages to or receiving messages from that channel. However, they must remove all stored state about that channel and not try to reuse it after their current sessions are terminated. As part of the initialization of the edge sever platform a controller booster is started which creates an asynchronous operation capable of consuming messages from an appropriate meta-channel. This booster receives the ChannelDescriptions and places them in a store. Information in this store is timed out at a rate proportional to the refresh period carried in the ChannelDescriptions. Other boosters on the same edge server can register with this store to be notified about the arrival of ChannelDescriptions containing information about message types. Booster

• the message types that the channel accepts; • the message types that are currently being published and subscribed; • information about the load, in particular the number of producers and consumers and the memory consumption as a percentage of the maximum calculated over a time window; • the time period in between publication of ChannelDescriptions in seconds, i.e. the refresh rate. This information is sent in the form of an XML document, meaning the same format can be used regardless of the broker supporting the meta-channel. Additional information may be added to the message in order to aid filtering in specific technologies. For example, adding the message types supported by the channel into the variable header part of the CORBA event or into property field with a JMS message. This allows consumers to specify that they are only interested in information about channel carrying messages of a given type. Failure of a channel may be inferred from nonreception of a channel description in a time period proportional to the time period stated in the last ChannelDescription, for example twice the announced refresh period. It is recommended that refresh periods should be at least in the order of many seconds in order to reduce the effect of queuing and propagation delays. The more frequently the refresh rate the more reactive the system is in response to change but the higher the control overhead.

Controller handleResult(AsyncResult res) { Read as JMS TextMessage Parse XML Place/Refresh in Store }

{ Create Auto Tracker for "X" }

Store

(1) Subscribe for "X" Message Type

(4)Notify Channels offering"X"

Channel Type

(3) Description

"X"

"CORBA"

C1

"X"

"MQTT"

C2

"Y"

"TupleSpace"

c3

(5) Create CORBAConsumerOP C1

(6) Recv "X" Message From CORBA Channel C1

JMSMQTTConsumerOp C2

(6) Recv "X" Message From MQTT Channel C2

(2) Recv ChannelDescriptions From Meta-Channel

Figure 5. Automatic Channel Tracking

As an example, we implement an AutoTrackingConsumerOp which takes a message type but no channel addressing information as argument. The AutoTrackingConsumerOp allows an application to be unaware of exactly which channels the messages of interest to it are being published. Channels may appear and disappear without the application needing to be aware and without it having to be stopped and restarted. The AutoTrackingConsumerOp registers with the store for information about the appropriate message type and gets notified each time a new channel offering that message type is found. The AutoTrackingConsumerOp creates other consumer operations of the type corresponding to

2.4. Example Application We used our edge servers to develop a system for tracking the ’in-flight time’ of items within a supermarket. Every time an item is picked up within the system a pick-up message is generated, every time an item is payed for a payed message is generated. The edge server run a queue manager which has a pay and pick-up queue for each item type, e.g. razors, and an autotracking operations which searches for queue managers, subscribes to them with an application-specific asynchronous operation. This operation matches pick-up and pay messages from the queue manager to which it is subscribed and sends summaries to an application server, identifying the average time item types are ’in flight’ . In addition, the server is alerted about individual items for which a long time has passed since they were picked-up and which potentially may have been stolen. The operation will only empty a queue if it contains more than a certain threshold number of items. The threshold is increased if more items than the current threshold are found in the queue. If no appropriate queue is found all thresholds are decreased by a factor of two and the operation sleeps for a period. By starting a new edge server we add two types of additional resources to the system: a new queue manager to which messages can be published and a set of subscribers to all queue managers to process the results. Graph 6 shows how the number of messages per second varies as we add additional edge servers into the system. The system reaches stability at six edge servers. The more than linear increase in the performance is due to the fact that we used a Tu-

3000

Transmitting Rate (Raw messages) Receiving Rate (Raw messages) Receiving Rate (Aggregated messages)

2500

Messages per second

the description in the channel record and adds them into the proactor associated with its handler. It removes them from the proactor if the connection channel fails or no message has been received for some time. The handler receives a stream of AsyncResults coming from one or more distinct channels. Note that these channels may even use entirely different technologies, but the handler needs to be able to interpret the result appropriately, e.g. as a CORBA structured event or a JMS Message. Figure 5 shows the general sequence of operations involved in autotracking, from the initialization of the auto tracker (1) to the creations of the consumer operations to the channels and the reception of message from them (6).

2000

1500

1000

500

0 1

2

3

4

5

6

Number of Edge Servers

Figure 6. Performance of edge servers pleSpace for supporting the queue manager and that the time to write a tuple is independent of the number of tuples in the space, but reading a specified tuple is linear with the number of tuples in the space. Hence, there is a double gain in adding a new edge server: there are more readers, and the average queue length is shorter. The bottom lines shows the receive rate of aggregated messages at the application server. The ratio of this to the receiving rate of raw messages is the edge sever gain. At stability the gain is about 10, but arbitrarily high gains can be achieved at the cost of increased delay.

3. Conclusion We have described a edge server for executing application-specific processing at the boundary between sensor networks and the Internet. This enables sensor based application to scale as the application server is not required to treat every message emitted from the sensors. The edge server uses messaging for both the propagation of sensor data and the dissemination of control information allowing the addition and removal of edge servers dynamically without manual configuration. We have identified the need for multiple messaging system due to the rather different nature of communication with sensors and application servers. We have described our implementation of a lightweight messaging system and given figures for its performance.

References [1] Bosch. Control area network specification, version 2.0. Technical report, Robet Bosch GmbH, D-70422 Stuttgart, 1991.

[2] G. Brose. Jacorb: Implementation and design of a java orb. In Procs. of DAIS’97, IFIP WG 6.1 International Working Conference on Distributed Aplications and Interoperable Systems, Cottbus, Germany, Sept. 1997. [3] S. Graham, P. Niblett, D. Chappell, A. Lewis, N. Magaratnam, J. Parikh, S. Patil, S. Samdarshi, S. Tuecke, W. Vambenepe, and B. Weihl. Web services notification (ws-base notification) version 1.0, Mar. 2004. www-106.ibm.com/developerworks/ library/specification/ws-notification/. [4] IBM. Telemetry integration. www-306.ibm.com/ software/integration/mqfamily/integrator/ telemetry/. [5] IBM. Websphere mq family. http://www-3.ibm. com/software/integration/mqfamily/. [6] IBM. Tspaces programmer’s guide. 2002. http:// www.almaden.ibm.com/cs/TSpaces. [7] IEEE. IEEE Standard for a Smart Transducer Interface for Sensors and Actuators Network Capable Application Processor (NCAP) Information Model, June 1999. IEEE Std 1451.1-1999. [8] OMG. CORBA Notification Service Specification. OMG, August 2002. [9] OMG. Smart Transducers Interface Specification. OMG, January 2003. [10] D. Schmidt, M. Stal, H. Rohnert, and F. Buschmann. Pattern-Oriented Software Architecture, volume 2, chapter 3, pages 215–260. Wiley, 2001. ISBN 0-47160695-2. [11] Sun Microsystems. Rio architecture overview, v1.0, 2000. [12] The OSGI Alliance. Osgi service platform, release 3, 2003. http://www.osgi.org.