Control and monitoring of data acquisition and trigger system (TRIDAQ ...

Control and monitoring of data acquisition and trigger system (TRIDAQ) for backing calorimeter (BAC) of ZEUS experiment Tomasz Jeżyński, Zbigniew Łuszczak, Krzysztof T. Poźniak, Ryszard S. Romaniuk, Institute of Electronic Systems WUT, Nowowiejska 15/19, 00-665 Warsaw, Poland Michał Pietrusiński Institute of Experimental Physics WU, Hoża 69, 00-681 Warsaw, Poland ABSTRACT The paper presents idea and realization of hardware (photonics and electronics) and software diagnostic layers of TRIDAQ for BAC calorimeter of ZEUS experiment at HERA accelerator. The aim of diagnostic system (DS) is to provide high reliability of the BAC. The diagnostic system localizes, in the fastest way, place, time and cause of any breakdown inside the BAC. Functionalities of diagnostic system were debated. Construction of distributed, diagnostic hardware layer was presented. Object oriented virtual classes were defined. These classes control electronics and photonics devices and sub-systems. Application examples were presented. The implementation results of diagnostic system for TRIDAQ were quoted and discussed. Keywords: High energy physics experiments, hardware data bases, object programming, FPGA, photonics for HEP

1. INTRODUCTION A ring accelerator, called HERA from Hadron Electron Ring Accelerator, with counter-propagating beams of electrons (or positrons) and protons was made operable in DESY/Hamburg in 1990 [5]. HERA researches the structure of proton. Very rare interactions are searched. The counter-propagating particle bunches are crashed against each other every 96ns (very frequently) to obtain sufficient bunch crossing and interaction statistics. There are four international experiments on the circumference of HERA. One of these is ZEUS [1]. ZEUS consists of many detectors enabling measurements of energies and registration of particle paths. The experiment goes practically all the time. Every several hours the accelerator is filled with new particle bunches. The filling procedure lasts for 30min. All the electronic and photonic systems, serving the experiment, have to be monitored and supervised during their nominal work conditions. All technical breaks, lasting several hours, are used to do more versatile diagnostic tests and for fast exchange of not efficiently working hardware modules. This work concerns the TRIDAQ system. It is a data acquisition and trigger for backing calorimeter (BAC) of ZEUS experiment. TRIDAQ is an example of a very large, distributed, measurement system using state of the art electronics and photonics technologies. TRIDAQ reads data from 2200 analog channels and 40000 digital channels. Data reading clock is the same as the experiment clock and equals to 96ns. TRIDAQ system delivers data for Global Triggering System of ZEUS experiment. Efficient and intelligent diagnostic system (DS) is a kind of supplement to TRIDAQ. DS increases considerably TRIDAQ reliability to provide its constant availability. DS should detect all uncertainties in the processes of triggering and data acquisition. DS fulfills constant and precise diagnostics of particular blocks and the whole measurement channels. DS provides calibration of data processing and trigger paths. Diagnostic tests data are gathered. This enables tracing of detector state. The time coincident detector state is used while analyzing registered physical data off line. General functional diagram of TRIDAQ with DS for BAC detector are presented in figure 1. The increase in TRIDAQ reliability is done through: 1.

Constant uninterrupted diagnostics of TRIDAQ, during its nominal work conditions, through monitoring and physical and test data analysis. Data irregularities and calibration mismatches are looked for and registered.

1

2.

Dedicated tests and simulations, during shut downs of ZEUS experiment, through evaluation of TRIDAQ hardware status. The registered data irregularities are combined with place of their generation and the hardware is checked and replaced, in case of need.

This work is devoted to present solutions used to realize system tests and simulations during ZEUS shut down – off-line diagnostics. The problems of on-line diagnostics are associated with the structure and functionality of DAQ software and will be published in other paper Front-end electronics

Pulser Pulser Power Power

Scanner

GFLTBI

Master

Trigger BAC

Pulser Pulser Power

Distributor

Master

Positional readout

Slow control (low & high)

Main VME crate

1 from 6 VME crates

High voltage

Power supply

Scanner

1 from 13 VME crates

Energy readout

BAC

Master

Cooling

Temperature To other VME crates

OS 9 Database

To other components

PC VAX (VMS)

Diagnostic software

Access to VME crates

TCP/IP

Data aquisition

Fig. 1. Functional structure of TRIDAQ and DS for BAC detector.

2. FUNCTIONAL STRUCTURE OF DIAGNOSTIC SYSTEM The functional structure of DS was tailored to the nominal work needs of TRIDAQ. This was presented in figure 2. The figure presents what kind of diagnostics can be done for which part of TRIDAQ. The DS was built much later than TRIDAQ. It was necessary to fit the DS to existing apparatus, computer and software infrastructure used in ZEUS experiment. Thus, a design of an open, distributed and multithreaded SD was proposed. Its architecture may be considered within the frames of three combined functional layers: • Hardware, embraces diagnostic modules placed in particular electronic and photonic blocks of TRIDAQ. Taking into account the method of analysic, one can distinguish:

2

• •

Continuous diagnostics, analyzing the state of monitored functional module for each consecutive bunch crossing [4]. Continuous monitoring is done for data signals (frequency measurements, histograming, etc.) and steering signals (synchronization, formatting, etc.); Selective diagnostics, done on the basis of the first trigger level of the experiment. Data registered by TRIDAQ are destined for further computer analysis and archiving.

Rys. 2. DS functional structure (white boxes) and its connections with TRIDAQ of BAC detector and global trigger system of ZEUS experiment (grey boxes) • Software, embraces three basic parts, connected by standard Ethernet network: •

Performing, including transputer applications. These programs control electronic and photonic apparatus via VME interfaces [6] and do data acquisition. The on-line experiment DAQ functions are done by DAQ channel. The off line, test functions are done by specific software described below. A TCP/IP server is used to communicate with all remaining parts of the DS residing on a VAX computer. This computer is a part of transputer network. • Monitoring, basing on a group of application servers and object oriented software. Monitoring layers may work simultaneously and nondependently on various operational platforms. Their task is to give a relevant expert the opportunity of convenient insight and command over the whole DS. Other task is to do current data processing in order to assess on-line status of TRIDAQ; • Client, resting on standard WWW browsers and Java applets, PHP, etc. Such solution gives free access for numerable users to information from arbitrary place, depending on the level of authorization; • Database, which major role is fast data gathering and providing access to data stored in complex connected structures. These structures are pictures of all objects inside the BAC. Database also insures archive and data processing safety. Full description of the database layer is to be found elsewhere [7].

3

3. HARDWARE LAYER OF DIAGNOSTIC SYSTEM

WTT STT HITS GFLTBI SCANNER DISTRIBUTOR LT ADDER RACE BITS EMBAC RMBAC BMBAC PULSER

Energy readout through analog response processing Energy readout, Signal source for trigger Energy readout, Signal source for trigger Position readout, Signal source for trigger Readout control for First Level Trigger of the experiment Energy readout control for First Level Trigger Position readout control for First Level Trigger Energy readout correction in correlation with position readout Calculates sum of energy and transverse energy from a single region of detector Searches two towers from the region, where the biggest energies were detected consecutively Prepares gathered information about the number and types of muons from a region Calculates total energy and transverse energy registered by detector Indicates two towers from detector, which registered consecutively the largest energies Prepares information about number and types of muons registered by detector; Generates local trigger signal for testing purposes Programmable generator of test analog pulses

Simulator

D

X

T&D

X

T&D

X

T&D

X

X

X

X

C

X

X

X

X

C C

X X

X X

X X

X X

Control

DAQ

16FADC

Board function in TRIDAQ

Function

Name of the board

Monitoring

TRIDAQ is very diversified according to realized processes by particular hardware modules and PCBs. TRIDAQ has been built during at least a decade. Its functional parts differ considerably in fabrication technology and performance. This causes the DS to be also diversified to serve optimally the needs of particular TRIDAQ blocks. The dedicated DS subsystems have to check continuously particular TRIDAQ blocks during their normal operation conditions. The early blocks did not have any diagnostic functionality at all. The diagnostic components are used as external ones n such a case. The latest PCBs basing on FPGA Altera chips, and used for BAC detector, contain a lot of inbuilt diagnostics on board. In particular such hardware functionalities were included as: hardware histograming circuits, event frequency meters, diagnostic readout, etc. The new functional boards have ability to send test information during experiment data gathering. Thus, they enable full real time testing on-line, during experiment activity. The functionalities and diagnostic duties of particular boards in the TRIDAQ are presented in table 1.

X

T

X

T

X

X

X

X

T

X

X

X

X

T

X

X

X

X

T

X

X

X

X

T

X

X

X

X

T

X

X

X

X

P

X

Table 1. Electronic and photonic boards of TRIDAQ for BAC detector with their functionalities and diagnostic properties The particular fields are: • Function realized in the system: • • •

T – participation in trigger channel, D – realization of data gathering process, synchronous with first level trigger S – Service control of first level trigger

4

•

P – Board used for generation of test pulses

• DAQ – board realizes synchronous data acquisition synchronous with first level trigger, • Monitoring – board gives on-line data about process during the experiment, • Testing – board allows to do configuration tests, • Simulator – board is equipped with simulator block. Testing process for TRIDAQ was realized in agreement to diagram presented in fig. 3. A programmable pulse generator is a source of testing signals (PULSER). Introduction of a test pulse to the input of front-end electronics provides testing of all the analog and digital channel. The channel starts with amplifier circuits, position readout comparators, ADC converters for energy readout and digital circuits of data acquisition.

Fig. 3. Block diagram of data acquisition channel for functional tests. Discovering signal discrepancies form given patterns classifies tested channel for more precise static tests. These tests include the following procedures: • Checking possible failures in hardware control sub-systems like: checking access, via VME, to single registers and memory areas, etc., • Verifying settings and work status of control block; • Performing simulation processes; excitation of inputs with logical state generator in monitored circuit, using one of the previous blocks to check the following one under investigation.

4. SOFTWARE LAYER OF DIAGNOSTIC SYSTEM The dedicated software, which realizes tests and simulations during ZEUS experiments shutdowns, consists of the following components: • Software residing on transputers; enabling basic VME operations, special VME command sequences responsible for functions of tested block; •

TCP/IP server on VMS/PC; the task is to gain access to operations offered by transputers through a local Ethernet;

5

•

Client software on PC; VME and communications libraries, BAC access library, control application;

The client software was done in C++, in environment Borland C++ Builder.

Transputer Transputer VME Server

Transputer

Process VME Server Process Transputer VME Server Process VME Server Process

Transputer

Tranputer Link Transputer

Transputer VME Server

VME Server Process

Process VME Server Process

DB Server

VAX

BAC DB

TCP/IP Server

TCP/IP

TCP/IP

Client PC Hardware System

VME Interface BAC Test Application

Rys. 4. Deployment diagram of the BAC test system. VME library was designed in such a way, that moving the system between different test environments is done in a transparent way. All calls are done through the collection of a few abstract C++ classes (interfaces). Each VME crate has corresponding object in the system, which realizes the interface (VMEInterface). The work with such interface goes as if the operations are done locally, instead of calling TCP. Exchange of this calling to TCP packet, receiving the packet by TCP/IP server, sending message to transputer network, performing VME operation, returning the result is done „behind the scenes”. Group VME operations were introduced to increase system efficiency. It is possible to register observers of particular crate (Observer GoF pattern) in the case of working with VME interrupts. The observers are notified about each interrupt and receive data relevant to this event. The BAC access library consists of the following, co-working components: 1.

Class groups mirroring BAC system in hierarchical way. Class object THardwareSystem (singleton) includes information about all VME crates (TCrate) in the system. Each of TCrate objects includes objects mirroring devices present in particular crate. These objects, in turn, have relevant access and monitoring methods for particular devices. These objects fulfill the roles of containers and enable system control as a whole. They enable setup of the

6

whole detector or chosen types of devices like event building, run control, etc. Object hierarchy is constructed dynamically basing on the information included in database. 2.

Class groups implementing specialized tests (THardwareTest). These classes use objects described above. They perform tests, simulations and system calibration. Test results may be automatically written to database. Test factory (THardwareTestFactory) has methods to build, accessible for particular device or sub-system, objects THardwareTest.

3.

Classes realizing graphical user’s interface.

The libraries described above are kernel of application serving for tests, simulations and calibration of BAC detector. This kernel may be used to work with chosen devices in nondependent testing stand.

5. DIAGNOSTIC SYSTEM – EXAMPLES OF APPLICATIONS TRIDAQ breakdowns should be discovered and repaired as fast as possible, because of short technical breaks. Depending on the break length in physical data acquisition, the tests may be done in the following variants: 1.

Test is done for the whole data acquisition channel,

2.

Particular components of TRIDAQ are tested, so as to test the whole detector during 24H,

3.

Detector calibration is done one time per 24H,

Now working version of the diagnostic system enables: •

Doing testes for the whole detector or its particular blocks,

•

Investigation of continuity in analog channel,

•

Searching for channels not fulfilling technical specifications,

•

Investigation of linearity in analog channel,

•

Reporting of discovered breakdowns,

•

Generation of configuration file for data acquisition including detector state,

•

Calibration of amplifiers in energy readout,

•

Setting comparators thresholds in position readout,

•

Saving test data for off-line analysis,

•

Information about localization of the blocks qualified by the testing software as broken (or acting improperly),

•

Trigger calibration.

Below there is description of DS usage in real work conditions within ZEUS experiment. The most important test types were debated on chosen examples.

5.1. OPERATOR’S SERVICE DS enables static access to all registers of TRIDAQ devices. It is possible to monitor continuously particular work parameters of all modules and electronic boards. Operator’s service is the lowest control level of apparatus. It enables direct access to registers and I/O memory areas. Primarily, experts use this kind of diagnostics after discovering breakdowns of chosen module or during repair work. Access to the resources of particular board is done always through precisely designed operator’s panel. Maintenance personnel have the ability of full control over configuration and work parameters of chosen electronic/photonic module. Introduced changes may be saved as configuration data for chosen board.

7

DS gives access to full hardware configuration of TRIDAQ. Operator’s panel presents all data of the board during the on-line operation. To facilitate some parameters reading, they are presented in graphical form like: status register set of flags. Exemplary panel of GFLTBI board, realized in programmable FPGA ACEX K50 Altera chips, is presented in fig.5. Complex functional structure of the board was realized in a form of folders. Folder choice makes respective board resources available, together with control tools. Left side of the panel contains tree of accessible TRIDAQ modules and enables convenient user’s navigation.

Fig. 5. Steering panel for GFLTBI board

5.2. APPARATUS TESTS Apparatus tests monitor particular modules of TRIDAQ. Tests are done automatically on the basis of preset parameters. Module resources are tested first. Module functions are tested next. There are used inbuilt stimulators for functional tests of the boards. Figure 6 presents result of apparatus test of a big block of position readout [3]. This block consists of 15 HITBOX modules placed on ZEUS detector. The modules are steered by CONTROLLER board placed in VME crate. The example embraces block tests of the base address 0x100000 placed in SH1 crate. The resources are investigated of particular parts of monitored apparatus. The results are imaged in the panel. The information on the actual configuration of monitored block is stored in database [7]. Database gathers a large number of configurations of particular blocks. The results of apparatus tests are transferred to database. It allows creation of the history and the map of current breakdowns

8

in the BAC detector. This information is taken into account in the TRIDAQ work during the experiment as well as during the analysis of physical data.

Fig. 6. Results from controller tests

5.2. FUNCTIONAL TESTS The last level of TRIDAQ investigation are functional tests. The aim is to evaluate the activities of particular electronic/photonic blocks. They are done during realization of nominal experiment functions of the system. The quality of signal processing, data acquisition and trigger generation is investigated. Synchronization of particular channels and time correlation of data are monitored.

Fig. 7. Time dependences registered by DS

9

Figure 7 presents test readout example for two trigger channels of BAC detector. Upper part of figure is BAC decision. The BMBAC board takes the decision on the basis of muon position readout. Lower part of figure is passing muon energy readout registered by EMBAC board. Time correlation is visible between registered energy and trigger decision. This diagram enables synchronization and amplification monitoring in particular channels. Full functional testing of TRIDAQ is done through automatic investigations of energy pulses maximum and position detection. Results of tests and blocks configurations are stored in database [7], similarly to apparatus tests. These data are taken into account during TRIDAQ work in the experiment and during physical data analysis. A number of newer TRIDAQ electronic/photonic boards may do diagnostic activities during physical data acquisition (see table 1). The implemented functions, mainly in programmable FPGA Altera chips, enable hardware histograming, frequency measurements of chosen signals, monitoring of realized processes, monitoring data reading for data acquisition channel.

Fig. 8. Time functions registered by the DS Figure 8 presents histograming example. Time of data acquisition is histogramed on the basis of information provided by by GFLTBI board. The readout is done synchronously with first level trigger of the ZEUS experiment [2]. The readout concerns, among others: ACCEPT an BUSY signals duration and all registered mistakes in this information. Continuous monitoring of time duration of ACCEPT and BUSY signals enables estimation of average time of event processing by TRIDAQ. Thus, introduced latencies are assessed. Data acquisition process monitoring enables to register breakdowns. The aim is to localize the very place of error generation [3,4]. This, in turn, serves to exclude from the physical data analysis these data, which were registered improperly or erroneously.

6. CONCLUSIONS Suggested idea and assumed solution of the diagnostic system (DS) in a form of a collection of object oriented software modules enabled to build reliable and functionally expandable practical system for BAC. DS enables making precise tests of BAC detector. The test results are saved in database. These results are used off-line for analysis and comparisons. The results give the history record of BAC status. Data are used for seeking correlations between all hardware breakdowns. DS consists of hardware layer (electronic/photonic boards, photonic transmission) and dedicated software.

10

7. REFERENCES 1. 2. 3. 4. 5. 6. 7.

ZEUS Collaboration, The ZEUS Detector Status Report, 1993 K.Poźniak, I Kudla, R. Kupczak, GFLT signal testing and distribution within the BAC system, ZEUS-Note 96-116 K. Poźniak, „Software control and testing of the HIT-READOUT for the BAC detector”, ZEUS-Note 96-117, 1996 M.Adamus, K.Poźniak: “Proposition of Veto Wall Electronics System development based on the model of Elementary Functional Unit”, ZEUS-Note 98-071 http:\\www.desy.de W. Petersen “The VMEbus Handbook” Z.Łuszczak et.al. “Functional structure of database system for DQM in BAC detector, ZEUS experiment, HERA accelerator, published in this Volume

11