XDAQ Software Framework for Distributed Trigger and ... - INFN-LNL

XDAQ Software Framework for Distributed Trigger and Data Acquisition Systems in Physics Experiments Andrea Petrucci – CERN (PH/CMD)

Legnaro National Laboratories - Istituto Nazionale di Fisica Nucleare (LNL-INFN) - Seminar 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro (PD) - ITALY

Outline ¤  CMS Data Acquisition System ¤  Design Parameters and Requirements ¤  Architecture ¤  Upgrade for LHC Run 2 ¤  Front-End Redout Link ¤  Data Concentrator 10/40 GE ¤  Infiniband Core Event Builder ¤  File-based Filter Farm ¤  Run Control & Monitoring System (RCMS)

¤  CMS Online Software Framework (XDAQ) ¤  Motivation and Requirements ¤  Architecture ¤  Software Product Line ¤  Outlook ¤  Memory Management ¤  Multithreading ¤  Data Transmission ¤  Integrated Web Technologies ¤  XDAQ Monitoring & Alarm System (XMAS)

¤  Conclusion XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

2

CMS DAQ Experiment

CMS DAQ Requirements for LHC Run 2

Parameters

Data Sources (FEDs) Trigger levels First Level rate

~ 620 2 100 kHz

Event size Readout Throughput High Level Trigger Storage Bandwidth

1 to 2 MB 200 GB/s 1 kHz 2 GB/s

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

4

CMS DAQ Upgrade Motivations ¤ Aging equipment: ¤  Run1 DAQ uses some technologies which are disappearing ¤  PCI-X cards, Myrinet ¤  Almost all equipment reached the end of the 5 year lifecycle

¤ CMS detector upgrades ¤  Some subsystems move to new front-end drivers ¤  Some subsystems will add more channels

¤ LHC performance ¤  Expect higher instantaneous luminosity after LS1 → higher number of interactions per bunch crossing (‘pileup’) → larger event size, higher data rate XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

5

CMS  DAQ  architecture  for  LHC  run  1  

Custom  

DAQ Event builder changes

Front-end Readout Links

Super-fragment builder (Myrinet)  

100 kHz

Replace  with     Readout Builders (Gigabit Ethernet) 8x 12.5 kHz Infiniband     Remove  8  slices  EB  

DAQ Slice 1

DAQ Slice 2

Commercial  hardware  

Replace  with  10/40  GE  

DAQ Slice 8

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

6

CMS DAQ 2

Overview of DAQ2 ¤  Detector front-end readout (custom) ¤  ~700 front-end drivers (FEDs) with ~2kB/fragment at 100 kHz ¤  Data transmission to surface (~200m) ¤  TCP/IP over 576 optical 10 GbE links ¤  Data-concentrator network (2.9 Tbs) ¤  Route & aggregate 10 GbE links into 40 GbE links ¤  Deliver to 84 readout-unit PCs (RUs) ¤  Synchronize and combine data into super-fragments ¤  Buffer super-fragments for O(seconds) ¤  Event-builder network (2.7 Tbs) ¤  Infiniband FDR network to ~64 Builder Unit PCs (BUs) ¤  BUs build complete events (~1 MB event size) ¤  Write events to temporary storage O(minute) ¤  High-level trigger farm (~13000 cores in ~800 boxes) ¤  Select ~1% of the events for permanent storage XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

8

New or upgraded detectors in CMS ¤  Several detectors / online-systems being upgraded to cope with higher luminosity ¤  Increase of event size ¤  Readout electronics of upgraded systems based to µTCA

n  n  n  n  n 

2014: New Trigger Control and Distribution System 2014: Stage-1 calorimeter trigger upgrade 2014/15: new HCAL readout electronics 2016: Full trigger upgrade 2017: New pixel detector and readout electronics

SLINK sender mezzanine plugged onto VME electronics

SLINK express sender = IP-core µTCA electronics “AMC-13” card used by many subsystems

Fragment size 1..4 kB SLINK-64 copper cable 400 MB/s Myrinet NIC

Fragment size 2..8 kB Optical SLINK-express 6 Gb/s (soon 10 Gb/s) - retransmit

FrontendReadout Link

640 Legacy Links: SLINK-64 (600 after pixel upgrade)

FrontendReadout Optical Link

+ 50 new Links: SLINK-express (170 after pixel upgrade)

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

9

FrontEnd Readout Link (FEROL)

FrontEnd Readout Link (FEROL)

11

¡  Replace Myrinet card 10 GBit/s Ethernet (upper half) by a new Slink Express custom card from µTCA ¡  PCI-X interface to legacy FEDs slink receiver card (lower half) Slink64 ¡  10 GBit/s Ethernet output to from FEDs central event builder ¡  Simplified TCP/IP protocol engine inside the FPGA

¡  Additional optical links (inputs) for future μTCA based Front End Drivers (6-10 GBit/s; custom, simple point to point protocol) ¡  Allows to use industry standard 10 GBit/s transceivers, cables and switches/routers ¡  Only commercially available hardware further downstream

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

11

Simplified TCP/IP protocol for FPGA DAQ hardware for readout

TCP State Diagram

PCI-x

48 x 10 Gb/s

6 x 40 Gb/s

10 Gb/s simplified TCP/IP from an FPGA Data concentration: 10/40 Gb/s Ethernet switch PC running standard TCP/IP Linux stack

Simplified unidirectional TCP/IP only needs 3 states XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

12

10 Gb/s TCP/IP link from FPGA to PC

Performs at 9.7 Gb/s for fragments > 1 kB (receiving PC with 10 Gb/s NIC, performance tuned) XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

13

FrontEnd Readout Link Rack

FRL/Myrinet

FRL/FEROL10 Gb/s Ethernet

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

14

Data concentrator (10/40 GE)

FEROL Aggregation into One RU ¤  Aggregation n-to-1, for example

12/16 Streams

¤  12 Streams from 12 FEROLs each sending fragment size from 2 to 4 kB ¤  16 Streams from 8 FEROLs each sending fragment size from 1 to 2 kB

¤  Concentrated in one 40 GbE NIC in RU PC ¤  Mellanox SX 1024 with 48 ports at 10 GbE and 12 ports at 40 GbE ¤  Reliability and congestion handled by TCP/IP ¤  Post frames are enabled in the 10/40 GbE switch

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

16

Performance Factors I/O Interrupt affinity

CPU affinity

Memory affinity

TCP Custom Kernel Settings

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

17

Link saturation with affinity

P2P Example: Out of the Box vs Affinity

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

18

DAQ Test Bed Readout Unit 56 Gb/s Infiniband FDR

I/O

CPU

CPU

I/O

40 Gb/s Ethernet

¤  Final performance indicator of the RU is based upon the performance achieved while executing simultaneous input and output ¤  Test bed up to 47 FEROLs (10 GbE) in input

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

19

Readout Unit Machine Affinities Readout Unit CPU Socket 0 NUMA Node 0 (16 GB)

I/O

56 Gb/s Infiniband

CPU Socket 1

0

2

4

6

8

10

12

NUMA Node 1 (16 GB) 14

1

3

5

7

9

11

13

I/O 15

40 Gb/s Ethernet

Affinities 40 GbE - I/O Interrupts 40 GbE - Socket reading Infiniband - I/O Interrupts DAQ threads DAQ Memory allocation Cores available to operating system XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

20

Av. Throughput per RU (MB/s)

Test with 1Data Source from FEROL (I) 40 GE

5000

4000

3000

2000

1000 12 FEROLs x 1 RU x 4 BUs 100 kHz (12 streams)

0

300 400

1000

2000

3000

10000 Fragment Size (bytes)

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

21

Av. Throughput per RU (MB/s)

Test with 1Data Source from FEROL (II) 40 GE

5000

4000

90% efficiency of 40 GE at 4.5 GB/s

3000

working range

2000 12 FEROLs x 1 RU x 4 BUs 24 FEROLs x 2 RUs x 4 BUs

1000

47 FEROLs x 4 RUs x 8 BUs 100 kHz (12 streams)

0

300 400

1000

2000

3000

10000 Fragment Size (bytes)

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

22

Av. Throughput per RU (MB/s)

Test with 2 Data Sources from FEROL 40 GE

5000

4000

90% efficiency of 40 GE at 4.5 GB/s

3000

working range

2000

8 FEROLs x 1 RU x 4 BUs 16 FEROLs x 2 RUs x 4 BUs

1000

32 FEROLs x 4 RUs x 8 BUs 100 kHz (16 streams)

0

300 400

1000

2000

3000

10000 Fragment Size (bytes)

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

23

Data concentrator patch panels and switches

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

24

Infiniband Core Event Builder

Infiniband Core Event Builder 4 Tb/s in 84x64 Event Builder – 56 Gb/s FDR Infiniband Clos network

3.5 Tb/s out Infiniband •  reliable in hardware at link level (no heavy software stack needed) •  supports credit-based flow control •  switches do not need to buffer •  can construct large network from smaller switches •  cost effective

6 spine switches

6 Tb/s per direction 12 leaf switches

Inputs and outputs mixed on leafs to better utilize leaf-to-spine connections XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

26

Av. Throughput per RU (MB/s)

First Measurement of Core EVB Network 6000

EvB InputEmulator/IBV 5000

4000

3000

CMS Preliminary

Shuffled send/recv, split config, daq2 4x4 8x8 16x16 32x32 54x54 72x54 100 kHz

2000

1000

0 1000 15 Nov 2014

2000

3000

10000

20000 Super-Fragment Size (bytes)

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

27

Infiniband Clos Network

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

28

File-based Filter Farm

File-based Filter Farm ¤  Filter Farm runs CMSSW for event reconstruction / filtering ¤  CMSSW input/output is file based ¤  Write the raw data to RAM disk on BU (256 GB/BU) ¤  Run standard CMSSW jobs (with cmsRun) with output to disk ¤  8-16 FUs mount RAM disk via NFS4 and run up to 2 processes per core reading the files ¤  FU processes merge their outputs into a single file per FU and then write it back to a disk on the BU

BU-FU Appliance: 1 BU + 8..16 FUs XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

30

Measurement of EVB Test Setup 4 RUs

1 BU to/from RAM disk

3.5 GB/s x 64 BUs: ~ 220 GB/s 8 FUs BU’s RAM disk mounted with NFS-4 32 CMSSW processes per FU XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

31

Merging and Storage ¤  File-Based Filer Farm produces output files ¤  After merging on FU: 800 files x 10 streams scattered over 64 BUs every lumi section (23s) ¤  To be merged to 1 file per stream and lumi section in a central place

¤  Merging is done by a Global File System (Lustre) ¤  Merger process on BU reads data from all FUs in appliance ¤  Data are written directly from the BUs to a single output file in the global file system ¤  Merged data is then transferred to Tier 0 or to consumers (e.g. DQM/Event Display) at pt.5 XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

32

Merging and storage BU1 merger

One file per event filter PC on hard disk of Builder Unit

2 TB hard disk

BU2

…

BU64

need: ~ 2 GB/s write ~ 1 GB/s read

Single output file in the cluster file system Recent measurement on a test setup (at NetApp) 4 Object storage servers (final system size may be larger) 14 clients Lustre 2.4 4.1 GB/s write speed achieved XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

33

Legacy readout link SLINK-64

Optical readout link SLINK express

Frontend Readout Optical Links

10 Gb/s TCP/IP links from FPGA Data Concentrator: Individual 10/40 Gb/s Ethernet switches

Core Event Builder: 56 Gb/s FDR Infiniband Event Filter attached by 1/10/40 Gb/s Ethernet

Clos network

Storage: Cluster file system

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

34

Run Control & Monitoring System (RCMS)

RCMS Control Structure for Run 2

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

36

RCMS Links and References ¤ RCMS home page: ¤  http://cmsdoc.cern.ch/cms/TRIDAS/RCMS/

¤ RCMS Wiki: ¤  https://twiki.cern.ch/twiki/bin/view/CMS/RunControl (especially User Area – Documentation)

¤ RCMS Hypernews (announcements): ¤  hn-cms-run-control [email protected]

¤ SVN Repository: ¤  svn.cern.ch/reps/rcms

¤ Trac: ¤  https://svnweb.cern.ch/trac/rcms XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

37

CMS Online Software Framework (XDAQ)

XDAQ Framework ¤  The XDAQ is software platform created specifically for the development of distributed data acquisition systems ¤  Implemented in C++, developed by the CMS DAQ group ¤  Provides platform independent services, tools for inter-process communication, configuration and control ¤  Builds upon industrial standards, open protocols and libraries, and is designed according to the object-oriented model

For further information about XDAQ see: J. Gutleber, S. Murray and L. Orsini, Towards a homogeneous architecture for high-energy physics data acquisition systems published in Computer Physics Communications, vol. 153, issue 2, pp. 155-163, 2003 http://www.sciencedirect.com/science/article/pii/S0010465503001619 XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

39

Context Diagram

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

40

Motivation ¤  CMS consists of a set of sub-projects ¤  Similar to a coordinated set of small experiments ¤  Many scenarios: central DAQ, subdetector DAQ, testbeams, etc.

¤  Geographically dispersed participants ¤  Autonomous developments ¤  High personnel turnover ¤  High performance requirements ¤  Long lifetime and need to survive technology generations ¤  Similar tasks to be performed in each sub-detector

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

41

Functional Requirements (TDR) ¤  Communication and Interoperability ¤  Transparent use of communication protocols ¤  Possibility to add new protocols ¤  Concurrent use of multiple protocols ¤  Device Access ¤  Access to custom devices ¤  Hardware abstraction layer ¤  Configuration, Control and Monitoring of Applications ¤  Inspect and modify simple/complex parameters ¤  Allow coordination of application components ¤  Record structured information ¤  Uniform logging, error reporting, monitoring ¤  Interface to persistent data stores

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

42

Non-Functional Requirements (TDR) ¤  Maintainability and Portability ¤  Portability across operating system and hardware platforms ¤  Add new electronics without functional changes in user software ¤  Application code shall be invariant with respect to the physical location and the network ¤  Encourage working with re-usable building blocks

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

43

Scalability ¤  Scalability ¤  Operate within requirements if size or volumes change ¤  Take advantage of additional resource availability ¤  Overhead introduced by the software environment must be constant for each transmission operation and small with respect to the underlying communication hardware in order not to introduce unpredictable behavior

Performance

linear

good within requirements

bad System size XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

44

Architecture

XDAQ Architecture Foundation

Plugin Interface

Core Executive

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

46

XDAQ Architecture Foundation

XML Configuration

Plugin Interface

Application Plugin

Core Executive

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

47

XDAQ Architecture Foundation

XML Configuration

Plugin Interface

Application Plugin

Core Executive

RESTful

HTTP

SOAP Control and Interface XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

48

XDAQ Architecture Foundation

XML Configuration

Plugin Interface

Application Plugin

Core Executive

RESTful

HTTP

Hardware Access SOAP

Control and Interface

PCI

VME

Memory

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

49

XDAQ Architecture Foundation ¤  Uniform building blocks - One or more executives per computer contain application and service components Protocols and Formats XML Configuration

Plugin Interface

Application Plugin

Core Executive

RESTful

HTTP

Hardware Access SOAP

Control and Interface

PCI

VME

Memory

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

50

Software Product Line

Software Distribution Core Tools

Core framework

Power Pack

Reusable applications

Work Suite

CMS specific applications

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

52

Layered View

Work Suite

Event builders

Front-end controllers

External system interfaces

Detector specific applications

Power Pack

Data monitoring

Error and alarming

Job Control

User interfaces

Core Tools

OS abstraction

Executive framework

Hardware access

Communication subsystems

Platforms

Operating systems

Networking infrastructures

Hardware device interfaces

Configuration management support

Online Software

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

53

Timeline Well consolidated after fourteen years of development and use 2014

Upgrade of CMS developing phase: support for NUMA architcture, IB and 40 GE network technologies

2012

Successfully used in the first three years of LHC

2008

Commissioning and first beam event successfully achieved XMAS – monitoring, orthogonal to applications

2006 2004

XDAQ 3 – Experiment wide adoption

2002

XDAQ 2 – Web enters DAQ (SOAP)

2000

First version of XDAQ - I2O communication kernel

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

54

Outlook ¤  Memory Management ¤  Multithreading ¤  Data Transmission ¤  Integrated Web Technologies ¤  XDAQ Monitoring & Alarm System (XMAS) ¤  Monitoring System ¤  Discovery Service (heartbeat) ¤  Tracer Service (timeline) ¤  Errors and Alarms System

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

55

Memory – Memory Pools

CPU

All memory allocation can bound to specific NUMA node

¤  Memory pools allocate and cache memory

Memory Pool

¤  log2 best-fit allocation

¤  No fragmentation of memory over long runs ¤  Buffers are recycled during runs – constant time for retrieval

Cached buffers

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

56

Memory – Buffer Loaning ¤  Buffer loaning allows zero-copy of data between software layers and processes Step 1

Step 2

Allocates reference

Task A

Step 3

Releases reference

Loans reference

Task B

Task A

Task B

Task A

Task B

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

57

Multithreading I/O Controller

Workloop Thread

CPU

Assignment of work

I/O Controller

Application Thread

¤  Workloops can be bound to run on specific CPU cores by configuration ¤  Work assigned by application ¤  Workloops provide easy use of threads XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

58

Data Transmission – Peer Transports Logical communication

User Application

User Application

XDAQ

XDAQ

Application uses network ProtocolXDAQ transparently through level framework

Peer Transport

Peer Transport

Peer Transport

NIC NIC

¤  User application is network and protocol independent ¤  Routing defined by XDAQ configuration ¤  Connections setup through Peer to Peer model XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

59

HyperDAQ •  Access through embedded HTTP •  Navigate and inspect the whole cluster

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

60

HyperDAQ Framework

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

61

Navigation Menu – Control Panel

XDAQ, LNL-INFN Seminar, 27 November 2014, Ceolin meeting room, Viale dell'Università, 2 - 35020 Legnaro, Italy – Andrea Petrucci CERN (PH/CMD)

62

XDAQ 12 - HyperDAQ Framework ¤  Web technology has evolved ¤  HTML5 and CSS3 provide much more natively ¤  jQuery – used by 60-80% of top 100,000 websites

¤  Browser support has improved ¤  Better acceptance of standards ¤  Better rendering engines

¤  XDAQ