ControlHost Distributed Data Handling Package Abstract ... - Nikhef

ControlHost Distributed Data Handling Package Abstract In an arbitrary complex multihost/multiprocess/multiarchitecture system, the transparent and universal means for the organisation of data traffic and message passing are provided. This software runs on the wide variety of platforms, its only requisites are the C(C++)-compilers and TCP/IP support. The set of interface routines allows users to create the distributed applications with a minimal effort; the platform-independent programming scheme ensures the portability of applications across the variety of operating systems. The routines are callable from programs written in C, C++.

Copyright Notice ControlHost Copyright 1993-1995 CASPUR Consortium All rights reserved. Permission is hereby granted, without written agreement and without license or royalty fees, to use, copy, modify, and distribute this software and its documentation for any purpose, provided that the above copyright notice and the following two paragraphs and the team reference appear in all copies of this software. In no event shall the CASPUR Consortium be liable to any party for direct, indirect, special, incidental, or consequential damages arising out of the use of this software and its documentation, even if the CASPUR Consortium has been advised of the possibility of such damage. The CASPUR Consortium specifically disclaims any warranties, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. The software provided here under is on "as is" basis, and the CASPUR Consortium has no obligation to provide maintenance, support, updates, enhancements, or modifications. CASPUR Inter-University Computing Consortium c/o Universita’ "La Sapienza" 5 Piazzale Aldo Moro 00185 Rome, Italy The ControlHost Team: Ruten Gurin, Andrei Maslennikov.

Contents Terminology and principles More detail Reference Section The C interface The C++ interface Implementation notes The Java interface The Dispatcher The test suite

Terminology and principles The Space of Processes The fundamental term in this project is the Process; we define the Process as a Running Task, which runs on one of the computers, with any type of operating system. Processes make the Space of Processes. This Space may be divided in the two Wholes: The Whole of Data Generators, and The Whole of Data Receivers. The two Wholes usually intersect, i.e. some of the Processes function simultaneously as Data Generators and Data Receivers.

The Tagged Data The Data is any amount of bytes which one of Generators produced somewhere in the Space of Processes, and is going to ship it into the network (or within the same machine), to share with other Processes. Sending the data, each Generator provides a small string of bytes (1-8). This string is called "Tag", and the data then is called the Tagged Data.

The Dispatcher This is a key stand-alone process in the ControlHost scheme. This process (and in principle you can have many of them on many machines which constitute your system, but not more than one per machine), is handling the fluxes of Tagged Data within the Space of Processes. The Generators are always sending the Tagged Data to one of the Dispatchers in the system. The role of the Dispatcher then is to redistribute the Tagged Data between the Receivers. This is done in several different ways, and Dispatcher knows at any given moment to whom it must send the given Tagged Data, thanks to the Subscription Mechanism (see below). The Dispatcher is a data-driven process, and it consumes no system resources when idle. By all means it may be treated just as an extension for Operating System; it is usually started at the boot time and is always alive.

The Subscription Mechanism Each Receiver has to inform the Dispatcher that it is willing to receive the certain subset of the Tagged Data, upon it is available in the system from one or several Generators. This is done via the Subscription Mechanism: the Receiver which wills to get the certain Tagged Data communicates to the Dispatcher the list of Data Tags it would like to receive.

More Detail How the Dispatcher redistributes the Tagged Data The Dispatcher may ship the data to Receivers in several different ways, we list them below: via the shared memory (only for the Processes which run on the same host with the Dispatcher). via the plain TCP socket (to the whole Space of Processes);

via the multiple sockets

The Synchronous Subscription It is often important to make sure that some of the processes within the Space of Processes work with the same portion of the Tagged Data. This may be achieved subscribing Synchronously for some of the Tags. The Dispatcher then delivers the next portion of the data to all synchronous subscribers only upon they all finish with the previous portion. This mechanism allows the creation of distributed commands etc. The C++ interface doesn’t support this kind of subscription yet!

The Dispatcher as the Process Watchdog The special routine allows the Process to communicate a short string of data, its nickname, to the Dispatcher. Upon receiving the process nickname, the Dispatcher redistributes it with the tag "Born" to whoever is subscribed to this tag in the system. Here Dispatcher itself acts as a Generator. Next, when the process with a given nickname closes/loses the link to Dispatcher, the Dispatcher again redistributes the nickname of this process, this time with a tag "Died". This feature of Dispatcher is quite important, as it may be used for the process monitoring.

Structuring the data traffic The Dispatcher serves as a data multiplexer/multiplier - one data portion may be sent to any number of processes. It provides also a data buffer in memory - keeps many data portions in its buffer space. The typical communication scenario is as follows: Connect to the Dispatcher communicating the list of data tags that you are willing to receive in this process loop Check whether some data arrived (or wait for their arrival) Process the data (in case something arrived) Send some data endloop

Subscription The Tags The Tag is a character string composed of 1-8 printing characters. The "space" character is not allowed. The Subscription List The Subscription List is a character string of the following form:

stype TAG stype TAG ....

In this notation, stype stands for the Subscription Type. Listed below are the valid values for the Subscription Types: a - subscription for ALL the data with the given tag. When used, this type of subscription prevents the Dispatcher from dropping the data from its memory until it is delivered to the calling process. This type of subscription is extremely important but also ports an immanent danger (your application MUST be fast enough to process all the incoming data, otherwise the Dispatcher will stop accepting the new data, having its buffer full). s - SYNCHRONOUS subscription. If there are several processes which use this type of subscription for the same data tag, the next data portion will be delivered to them only when all of them are ready to accept it. Do not use send.me.always routine (see below) when coding such a process, otherwise the Dispatcher will treat the process as "always ready", and it might happen that next data portion will be shipped to other synchronous partner processes while the current process has yet not finished with it. Use send_me_next routine instead. The C++ interface does not support this subscription mode (yet). m - subscription via the shared MEMORY. Dispatcher will try to send to process only the pointer into shared memory instead of the data buffer, as a rule. In certain cases (e.g. when data buffer is very small, or the process is connected to the Dispatcher remotely, thus not having an access to the Dispatcher shared memory), the Dispatcher will still send the buffer rather than a pointer. Any combination of a,m,s may be used, e.g.: "am EVENT". If you don’t need any of this options, you must use w for the Subscription Type; w is just a place holder, for the empty Subscription Type is not allowed. The whole Subscription List may also be empty. This means that the process will only send some data to the Dispatcher, without receiving anything from it (acting as a pure Generator).

The ControlHost Library The ControlHost library provides the user with a set of routines allowing him to implement the various communication schemes. The routines may be used from C, C++. The C/C++ library is currently implemented for: Linux, Solaris and VxWorks operating systems. The Dispatcher program is available for Linux/Solaris systems only.

Reference Section The C interface (obsolete) . The user C-code must contain the line: #include "dispatch.h"

and linked with:

libch.a On some old systems it happens to be: libconthost.a

Most of the routines return an integer result - return code. Negative value of result means an error (wrong parameters, broken connection, invalid calling sequence etc).

The C++ interface The user C++ code must contain the line: #include "choo.h"

and linked with: libchoo.a

The CHOO library offers multiple connections, optional exceptions and some additional swapping mechanism. In general the interface covers the original ControlHost library. The new and the old library can be used together in the same program if you like. Almost all routines return an integer result. Negative return code means an error (wrong parameters, broken connection, invalid calling sequence etc). From those routines (like: GetDataAddr) that don’t return an integer value the return diagnostics are noted in the ’Action’ part.

An overview class ControlHost { ControlHost ControlHost ControlHost ~ControlHost int Connected int Subscribe int SendMeNext int SendMeAlways int MyId int WaitHead int CheckHead

(string host, int mode, string id, string tag = ""); (string host = ""); (string host, int port); (void); (void); (const string subscr); (void); (void); (const string nickname); (string &tag, int &nbytes); (string &tag, int &nbytes);

int int

GetFullData (void *data, int GetFullString (string &str);

int int

PutFullData (const string tag, const void *data, int nbytes); PutFullString (const string tag, const string str);

int int

PutPartData (const string tag, const void *data, int nbytes); PutPartString (const string tag, const string str);

int int int

AddSwapInfo PutSwapData GetSwapData

void *GetDataAddr int UnlockData static static static static

nbytes);

(const string tag, const string format); (const string tag, const void *data, int nbytes); (const string tag, void *data, int nbytes); (void); (void *ptr);

bool SwapSyntax int Swap ControlHost *SelectRead ControlHost *SelectWrite

(const string (const string (int timeout, (int timeout,

fmt); fmt, void *dst, const void *src, int nbytes); ...); ...);

static static static static

ControlHost *SelectRead ControlHost *SelectWrite void Throw int WhereIs

static int static string

(int timeout, ControlHost *selch []); (int timeout, ControlHost *selch []); (bool except = true); (const string disphost, const string nickname, string &replybuf); IpNumber (const string name = ""); IpDecimalDot (int ip);

class TagStream { TagStream (const string tag) } } class ControlServ { ControlServ (int port); ~ControlServ (void); ControlHost *AcceptClient (void); } class ControlShare { static void Throw (bool except = true); ControlShare (string host, string myid = ""); ~ControlShare (); int int int

Connected (); PutFullData (const string tag, const void *buf, int nbytes); PutFullString (const string tag, const string str);

};

ControlHost (string host, int mode, string id, string tag = "") Action The constructor makes the connection to the dispatcher, while the destructor will drop the connection implicitly. This contructor does do an implicit Connected () and will default be in de SendMeAlways mode. When the mode argument is ORed with CHOO_NEXT the connection will be in SendMeNext mode. In general this default constructor is recommanded. The other constructor calls should only be used in very special circumstances. Input parameters host String with the Internet address of the host where the dispatcher runs, symbolic like: "antares.nikhef.nl", or decimal dot notation like: "128.141.202.74" When the program and the dispatcher run on the same host, "local" can be used. mode Announce the client as:

Generator Receiver Both

CHOO_WRITE CHOO_READ CHOO_READ|CHOO_WRITE

SendMeNext Mode CHOO_READ|CHOO_NEXT

id A string to identify the client. tag Only when a Receiver (CHOO_READ) this is the Subscription List, otherwise it’s omitted. Example ch1 = new ControlHost ("auge", CHOO_WRITE, "Conn 1"); ch2 = new ControlHost ("auge", CHOO_WRITE|CHOO_READ, "Conn 2", "a LOG w CHK"); ...... ...... delete ch1; delete ch2;

ControlHost (string host = "") Action This constructor is obsolete now please use the default. The constructor makes the connection to the dispatcher, while the destructor will drop the connection implicitly. Input parameters host String with the Internet address (symbolic or decimal dot) of the host where the dispatcher runs. When no host argument is given only an object without any connection is created. Example ControlHost *ch = new ControlHost ("tricot.nikhef.nl"); ...... ...... delete ch;

ControlHost (string host, int port) Action This constructor makes the connection to an user constructed server on an arbitrary port. This call is equivalent with the previous call when port equals DISPATCH_PORT. This call should only be used in very special circumstances. Input parameters

host String with the Internet address of the host where the server runs. When the program and the server run on the same host, "local" can be used. port The (not default) user defined port number.

int Connected (void) Action The Connected call can be used to check the connection in fact the socket descriptor or on failure -1 is returned. On success a message is send to the dispatcher, there the version of the libchoo software is checked against the version of the dispatcher. An eventually mismatch is logged by the dispatcher. When this happens, please re-compile the dispatcher or the client software after a ’CVS’ update.

void ControlHost::Throw (bool except = true) Action Throw will decide on throwing exceptions. This is a static member function. The default mode is no exceptions. Because some network error’s raise the SIGPIPE signal the application can be killed before any exception handling takes place. To avoid this behaviour one can ignore the SIGPIPE by: #include { struct sigaction

sa;

sa.sa_handler = SIG_IGN; sa.sa_flags = 0; sigemptyset (&sa.sa_mask); sigaction (SIGPIPE, &sa, NULL); }

The default error handling (printout) can be to an ostream or string like: catch (ControlHost::Exception error) { cout WhereIs ("local", "SLOW CONTROL", reply); if (rc > 0) ...

int WaitHead (string &tag, int &nbytes) Action Waits for incoming data header and returns the data tag and size. After a successful WaitHead call one should read the corresponding data. Output parameters tag Reference to tag string. nbytes Reference to an integer to keep the size of the data portion. Example char histoBuf [BUFSIZE]; string tag; int nbytes; ch -> WaitHead (tag, nbytes); if (tag == "HISTO") ch -> GetFullData (histoBuf, nbytes); ...

int CheckHead (string &tag, int &nbytes) Action Same as the WaitHead, but does not wait if there is no data. Returns 0 in this case, and a positive value when the data has been arrived. After a successful CheckHead call one should read the corresponding data. Output parameters tag Reference to tag string.

nbytes Reference to an integer to keep the size of the data portion.

int GetFullData (void *data, int nbytes) Action Gets the data. May be called after successful call to WaitHead or CheckHead. Input parameters nbytes Specifies the upper bound on the number of bytes to fill in data. Output parameters data Pointer to a buffer to keep the data. The stored data will be truncated in accordance with the nbytes parameter.

int GetFullString (string &str) Action Gets a string. May be called after successful call to WaitHead or CheckHead. Input parameters nbytes Specifies the upper bound on the number of bytes to fill in string. Output parameters str Reference to string.

int PutFullData (const string tag, const void *data, int nbytes) Action Sends data to the Dispatcher. Input parameters tag The tag string. data Pointer to data buffer to be sent. nbytes Size of data buffer to be sent.

int PutFullString (const string tag, const string str) Action Sends a string to the Dispatcher. Input parameters tag The tag string. str The string to be sent.

int PutPartData (const string tag, const void *data, int nbytes) Action The non-blocking variant of PutFullData. It may happen that the dispatcher refuses to accept data (e.g. has its buffer full), or the TCP/IP socket accepted only a fraction of data While the PutFullData routine returns only after the whole buffer was sent, the PutPartData routine returns immediately after the first try. When the routine did not complete we had to repeat this call until it’s ready. Possible results are: less

then zero zero greater then zero

error data transfer not complete data transfer complete

Input parameters tag The tag string. data Pointer to Data buffer to be sent. nbytes Size of data buffer to be sent. Example fill the buf int nbytes = ...; for(;;) { rc = ch->PutPartData ("HISTO", buf, nbytes); if (rc != 0) break; // finished or error do something // transfer not complete } checkReturnCode (rc)

int PutPartString (const string tag, const string str) Action The non-blocking variant of PutFullString (see the comment for PutPartData) Returns as a result: less

then zero zero greater then zero

error string transfer not complete string transfer complete

Input parameters tag The tag string. str String to be sent. Example string str = "test string"; for (;;) { rc = PutPartString ("HIST1", str); if (rc != 0) break; // finished or error do something // transfer not complete } checkReturnCode (rc)

void *GetDataAddr (void) Action Returns a pointer to the data buffer in the Dispatcher shared memory area. May be called after successful call to WaitHead or CheckHead (which give you the tag and size) only in case the data portion was shipped via the shared memory. NULL return value means an error. Originally GetDataAddr and UnlockData had to be used in pairs, but these primitives can be used more randomly. Only the number of occupied shared memory buffers (difference between GetDataAddr and UnlockData calls) is limited per client. These two calls are portable, but there will be a performance penalty when running on not local (in respect to the dispatcher) machines. An ’data address’ returned by GetDataAddr had to be used as parameter of the UnlockData call.

int UnlockData (void *ptr) Action Can be used after calls to GetDataAddr notifies the dispatcher that it can reuse the buffer space in its shared memory, previously locked by the GetDataAddr call.

Example for (x=0 .. n) dataPtr [x] = ch->GetDataAddr(); ... process the data pointed to by dataPtr [0 .. n] ... then release them in any order ... for (x=n .. 0) ch->UnlockData (dataPtr [x]);

static ControlHost *SelectRead (int timeout, ...) static ControlHost *SelectWrite (int timeout, ...) static ControlHost *SelectRead (int timeout, ControlHost *selch []) static ControlHost *SelectWrite (int timeout, ControlHost *selch []) Action The SelectRead/Write call will select a ’ready’ connection or a timeout. A returned NIL pointer will indicate the timeout, this is not considered as an error. The priority is conform the order (left to right OR low- to high-index) of the object parameters. The user should built some ’round robin’ like mechanism on successive calls. (To be changed) The practical use of these select primitives is limited because a selected connection is able of handling data but it is not clear how much. Input parameters timeout The timeout in micro-seconds. A timeout of zero will return immediately while a timeout of minus one will be equivalent to infinite timeout. varargs ... An arbitrary number of pointers to ControlHost objects ended by NULL. selch The address of an array of pointers to ControlHost objects ended by NULL. Example string tag; int nby; ControlHost *ch1 = new ControlHost ("dcs.nikhef.nl", CHOO_READ, "1", "a CAL w DET"); ControlHost *ch2 = new ControlHost ("daq.nikhef.nl", CHOO_READ, "2", "a DATA w SPY"); ControlHost *chsel = ControlHost::SelectRead (0, ch1, ch2, NULL); if (chsel) { chsel->WaitHead (tag, nby); ... } ...

int AddSwapInfo (const string tag, const string format) Action The AddSwapInfo call will connect swap format information to a certain tag. This swap info is given in the syn meta language . syn syn syn syn syn syn syn

swapinfo swappair swaptype comptype simptype lowertype uppertype

= = = = = = =

; [swaptype + "number"]; (simptype/comptype); [’(’+ swappair + ’)’]; (lowertype/uppertype); (’l’/’s’/’c’/’d’/’f’/’x’); (’L’/’S’/’C’/’D’/’F’/’X’);

Input parameters tag The tag string. format The format string. Example When transporting a message structured like: struct { struct { long exa_l [3]; short exa_s [2]; } exa_m [5]; char exa_c [20]; }

The corresponding swap format string looks like: ControlHost *ch = new ControlHost ("dcs.nikhef.nl"); ch -> AddSwapInfo ("EXAMPLE", "(l3s2)5c");

This swap format string is interpreted as: the combination of 3 longs and 2 shorts will be repeated 5 times and the remaining message will be chars. When the swap info ends on a type (without a number) the rest of the message will be considered that type. Depending on the machine architectures involved there should be some padding of data structures. Longs should start on a modulo sizeof (long) byte address boundary and shorts should start at a modulo sizeof (short) boundary. Only the PutSwapData and GetSwapData routines will act conform this swap format information when applied on a Big Endian machine. In other words, at the moment, unlike the Ethernet standard, Little Endian is the default.

int PutSwapData (const string tag, const void *data, int nbytes) Action Sends data to the dispatcher and eventually swap the message conform the format. Input parameters tag The tag string. data Pointer to data buffer to be sent. nbytes Size of data buffer to be sent.

int GetSwapData (const string tag, void *data, int nbytes) Action Gets and eventually swaps the data. May be called after successful call to WaitHead or CheckHead. The swap will be performed when there has been an previous call to AddSwapInfo connecting this tag to a swap format and the ’Endian’ of this machine is such that swapping is needed. Input parameters tag The tag string. (from a previous WaitHead/CheckHead) nbytes Specifies the upper bound on the number of bytes to fill in data. Output parameters data Pointer to a buffer to keep the data. The stored data will be truncated in accordance with the nbytes parameter and swapped conform the to the tag connected format. Example #include "controlhost/choo.h" ControlHost::Throw (true); try { ControlHost *ch1 = new ControlHost ("dcs.nikhef.nl"); ch1 -> AddSwapInfo ("CAL", "((L4C8)16L4)3S2C"); ch1 -> AddSwapInfo ("DET", "L1(L1C256L512)2C256L1"); ch1 -> Subscribe ("a CAL a DET"); ch1 -> SendMeAlways (); ControlHost *ch2 = new ControlHost ("daq.nikhef.nl", CHOO_READ, "", "a DATA w SPY"); while (true) { string tag;

int int

buf [ControlHost::MAX_DATA]; nby;

ControlHost *chsel = ControlHost::SelectRead (0, ch1, ch2, NULL); if (chsel) { chsel->WaitHead (tag, nby); chsel->GetSwapData (tag, buf, nby); if (tag == "CAL") ... } ... } } catch (ControlHost::Exception error) { cout