Applications of the scalable coherent interface to data acquisition at LHC

EUROPEAN ORGANIZATION FOR NUCLEAR RESEARCH

CERN DRDC/91-45

. ’ R QE N

E 6

CERN LIBRARIES, GENEVA

Il l l l l l Il l l l lil l l lil l l l l l l l

S

Pow?

SCOOOOOIZZ

~5-' 3 c. P “'

"

Applications of the Scalable Coherent Interface to Data Acqu1s1t1on at LHC Q; ug

A. Bogaertsl, J. Buytaertz, R. Divia, H. Miillerl, C. Paxkman, P. Ponting CERN, Geneva, Switzerland

B. Skaali, G. Midttun, D. Wormald, J. Wikne University of Oslo, Physics Department, Norway

S. Falciano, F. Cesaroni

INFN Sezione di Roma and University of Rome, La Sapienza, Italy

V.I. Vinogradov Institute of Nuclear Research, Academy of Sciences, Moscow, USSR

E.H. Kristiansen, B. Solbergz Dolphin Server Technology A.S., Oslo, Norway A. Guglielmi

Digital Equipment Corporation (DEC}, Joint Project at CERN

F-H. Worm, J. Bovier

Creative Electronic Systems (CES), Geneva, Switzerland

C. Davis

Radstone Technology plc, Towcester, UK joint spokesmen Fellow at CERN

Scientific associate at CERN, funded by Norwegian Research Council for Science and Humanities OCR Output

ZKLUIU

We propose to use the Scalable Coherent Interface (SCI) as a very high speed interconnect between LHC detector data buffers and farms of com

mercial trigger processors. Both the global 2"‘f and 3"d level trigger can be based on SCI as a reconfigurable and scalable system. SCI is a proposed IEEE standard which uses fast point-to-point links to provide computer bus-like services. It can connect a maximum of 65536 nodes (memories or processors), providing data transfer rates of up to 1 Gbyte / s. Scalable data acquisition systems can be built using either simple SCI rings or complex switches. The interconnections may be flat cables, coaxial cables, or optical fibers. SCI protocols have been entirely implemented in VLSI, resulting in a significant simplification of data acquisition software. Novel SCI features allow efficient implementation of both data and processor driven readout

architectures. In particular, a very efficient implementation of the 3"d level trigger can be achieved by combining SCI’s shared and distributed mem ory with the virtual memory of modern RISC processors. This approach avoids complex event building hardware and software. Collaboration with computer and VLSI manufacturers is foreseen to assure the production of maintainable components. The proposed studies on SCI hardware and soft ware will be made in collaboration with other LHC R&D projects to provide a basis for future, standard SCI-based data acquisition systems. OCR Output

Contents LHC Detector Readout 1.1 Introduction

..........................

1.2 LHC Detect ors

........................

1.3 Size of the SCI Readout System ...............

1.4 SCI Readout Node Implementation ............

1.5 Data Streams after 2”“ Level Trigger ............ .nerace o everer 16 Itft2”‘ Ll Tigg ................

Application of SCI to Global'“* 2and"' 3Level Triggers 2.1 Status of the SCI standard ..........

2.2 Impact of SCI on Data Acquisition Systems

2.3 Coherent Caching ...........

10

2.4 Use of SCI for the 2'“‘ Level 'Irigger

11

2.5 Use of SCI for the 3"d Level 'Trigger

.11

2.6 Demonstration Systems .......

12

Research and Development Program

14

3.1 General Purpose SCI Interface . . .

14

3.2 SCI Ringlet Test System .......

14

3.3 Direct SCI-Computer Interface . . .

15

3.4 SCI Memory .............

16

3.5 SCI Bridges and Interfaces ......

16

3.6 SCI/ VME Single Board Computer .

18

3.7 Intelligent Data Controller ......

18

3.8 Diagnostics, using a Protocol Tracer

18

39 Sf .otware ................

19

3.10 Modelling and Simulation ......

20

Collaboration with industry

22

Budgets

23

Responsibilities

25

Timescales, Milestones

26 OCR Output

1 LHC Detector Readout 1.1 Introduction

According to ECFA studies [1], the event rate for a general purpose LHC detector, operated

at a luminosity of 2 >•· 103‘cm2s‘1 will not exceed 105 Hz after the 1" level trigger. The data volume generated by such a detector is estimated as:

Inner Tracking: 1 Mbyte per 15 ns bunch, 20 million channels Calorimeter: 200 kbyte per 15 ns bunch, 200 000 channels

Muon Tracking: negligible amount from up to 106 sparsely filled channels

A possible readout scheme for such a detector, according to the current understanding [1] [2], is illustrated in fig. 1. A first stage of data concentration after the 1** level trigger decision is implemented by electronics located close to the detector. Next, data is carried off the detector by point-to-point links, further concentrated in bus units and stored in data buffers. The data volume of each segment is sufficiently small that these buffers can be implemented using conventional backplane bus systems

Event data is further filtered by a 2"" level trigger which is implemented in two stages. The first stage consists of local trigger processors which have access to data of one segment of a detector only. These produce reformatted, reduced events complemented by trigger data

which are stored in output buffers. The overall trigger decision is taken by global 2"d level trigger processors which can correlate the pre·processed event and trigger data of the first

stage. The event rate at the input stages of the local and global 2"‘ level trigger is estimated at z 100 kHz; the output rate after the global 2**** level trigger at z 1 kHz. With an average of up to 1 ms for the global 2'"' level decision, such a reduction could be obtained by a farm of 100 processors.

Final event rejection is accomplished by a 3"" level trigger, based on complete event data, further reducing the rate to z 100 Hz. Both the 3'“ level trigger and data logger can be implemented by a processor farm.

There are several reasons not to use buses for the readout system after the local 2"“ level trigger [3]: the expected data rates exceed the capacity of existing buses, the required con nectivity over large distances is very problematic and event building methods based on buses

[4] cannot be scaled to the size of an LHC system.

We propose to implement the global 2'“’ level trigger, the 3"' level trigger and the data logger using a uniform SCI network: copying of data is avoided and events

are stored only in the (local) 2"‘ level output buffers from where data can be accessed by the trigger processors. These and the data logger are all implemented as farms of commercially available computers. A significant simplification of both hardware and software can be ob tained by using the processor’s virtual memory hardware for implicit event building, whilst caches avoid repetitive data access. OCR Output

iii$:`—·$