CMS Note

Internal Note 2004/000

Available on CMS information server

The Compact Muon Solenoid Experiment

CMS Note

Mailing address: CMS CERN, CH-1211 GENEVA 23, Switzerland

6 January 2004

The CERN CMS Tracker Control System F. Drouhin




Universit´e de Haute-Alsace, Mulhouse, France

L. Gross, D. Vintache Institut de Recherches Subatomiques de Strasbourg, France

A. Marchioro, C. Paillard, C. Ljuslin, P. Siegrist, A. Tsirou European Organization for Nuclear Research, Geneva, Switzerland

L. Mirabito Institut de Physique Nucl´eaire, Lyon, France

P.G. Verdini Istituto Nazionale di Fisica Nucleare, Pisa, Italy

Abstract Due to the high integration level of the experiments planned for the Large Hadron Collider (LHC) of the European Organization for Nuclear Research (CERN), the data acquisition and the control systems need complex developments both in hardware and software. The purpose of this paper is to describe the control system of a sub-detector of one of the CERN experiments, the Tracker of the Compact Muon Solenoid (CMS) The CMS Tracker control system is based on dedicated hardware and software. The hardware is based on the Front End Controller (FEC), an interface board that hosts token rings for the communication with the Control and Communication Unit (CCU) modules. These in turn contain dedicated I/O channels for the front end readout and control chips. The software is built in layers: one device driver, a C++ dedicated application program interface (API) plus a database for the storage of all the information needed for the front end electronics. This system will also be adopted in some other CMS sub-detectors. Submitted to IEEE Transaction for Nuclear Science Preliminary version 

Corresponding author: [email protected], Institut de Recherches Subatomiques de Strasbourg, Batiment 21, 23 rue du Loess, BP28, F67037 Strasbourg Cedex

1 Introduction The CMS Tracker [1] will operate in a hostile environment, where the temperature will be around -20 ˚ C and the integrated particle fluence will reach     1 MeV neutron equivalent per cm  . The CMS Tracker control system must handle the temperature, humidity and voltage on the detector and at the same time ensure the download of correct parameters to the front end electronic chips. A dedicated control system both in hardware and software has been developed to face this challenge.

Figure 1: A Tracker module with two silicon wafers and the electronic hybrid that hosts the front end electronic chips The Single Event Upset (SEU) phenomenon can change register values (bit changes) due to the radiations. With increasing radiation exposure, the response of the silicon sensors and of the readout electronics is bound to change. Therefore, for the CMS Tracker special electronics has been developed whose mode of operation can be modified by programming specific registers. There are five kinds of front end chips shown in figure 1: - The APV [2]: front end chip for the multiplexed readout of 128 strips; - The APV Multiplexer (APV Mux): chip that multiplexes the outputs of two APV on a single optical link; - The PLL: programmable delay and clock rephaser; - The Laserdriver: used for the transfer of data between the front end electronics and the Front End Driver (analogue to digital converter with memory) over an optical fiber; - The DCU (Detector Control Unit): this chip is very important in the CMS Tracker control system because it gives condition parameters for voltage, current and temperature. It contains an eight channel analog to digital converter, two constant current sources and a temperature sensor. It will read two set of thermistors, on the detector and on the hybrid, its internal temperature, the silicon detector leakage current and the two (1.25 V and 2.5 V) low voltages. So there are three temperatures, 2 voltages and one current measurements for each hybrid. Moreover, each DCU has a unique hardware identification number that can be read by software. This identification number will be used in order to identify in the final Tracker construction the position of each module and to act as a link between the construction database that stores the module information (pedestal, calibration, ...), the condition database and the online parameter database. The APV chips (4 or 6 per module), APV Mux, PLL and DCU (in the following text referenced to generically as “devices”) each have several registers to configure. All registers are accessed over an Inter Integrated Circuit (I  C) bus with the standard modes described in the specification [3], normal I  C addressing mode shown on table 1, extended addressing mode shown on table 2, and a specific addressing mode for the APV with a non standard extension of the I  C extended mode, the RAL mode shown on table 3. Before starting data taking through the data acquisition system [4], the front end electronics must be configured properly and each device register must be set through the I  C bus. For the Tracker, about 16 000 modules will be produced and installed. Thus the potential problems of the system (both in hardware and software) come from its complexity and from its dimensions. In fact, the parameter set and 2

therefore the number of registers to access is about 1 680 000. Most of the device registers must be tuned in order to find the best conditions for taking physics data. In parallel to this electronic design, a control system was developed to set and retreive all the device parameters. The next chapters describe the hardware and software architecture. Note that this hardware and software control architecture is also used in other parts of the CMS experiment with the same FEC/CCU rings but different front end electronic chips. This use is discussed in the conclusions (chapter 8). Table 1: Normal I  C addressing Start

Slave address [6:0]

R/W [0]

Ack. [0]

Data [7:0]

Ack. [0]

Stop

Table 2: Extended I  C addressing Start

Slave address [6:0] 1 1 1 1 0 X X

R/W [0]

A1

Slave address 2 [7:0]

A2 [0]

Data [7:0]

Ack. [0]

Stop

Table 3: RAL I  C addressing Start

Slave address [6:0]

R/W [0]

Ack. [0]

Sub Address [7:0]

3

Ack. [0]

Data [7:0]

Ack. [0]

Stop

2 Hardware architecture 2.1 Hardware description CCU: Control and Communication Unit FEC: Front End Controller

CCU

CCU

FEC

CCU CCU

CCU Token Ring CCU

CCU i2c channels

16 − i2c channels Hybrids / Devices

1 − Memory channel 1 − JTAG channel 4 − PIO channel

1 − Trigger channel

Figure 2: Hardware architecture. Note than one FEC can manage up to 8 token rings. Figure 2 shows the hardware architecture of the control system. This architecture is based on a specific board, the Front End Controller (FEC) [5], shown in figure 3, which hosts a token ring with modules, communication and control units (CCU) [6], shown in figure 4. Each CCU manages several kinds of channels: - Sixteen I  C channels for the access to the front end chips. Each channel can be configured for different speeds (100, 200, 400 KHz and 1 MHz). - Four Input/Ouput-like parallel bus controllers (Parallel input output (PIO)) such as the ones used in the Motorola PIA, etc. These channels are used in the Tracker in order to reset the front end readout electronics; - One memory channel, memory-like bus controller to access devices such as static memories, analog digital converters, etc.


- One JTAG  master controller ; - One trigger distribution controller ; - One node controller (the CCU control itself is seen as a special channel capable for instance to report the status of the other CCU channels). The memory, JTAG and the trigger channels are not used in the Tracker control system, but some other parts of the CMS experiment use the memory channel and check the possibility to use the trigger channel. The CCU is built to be radiation resistant but the SEU can also involve bit changes, so a cyclic redundancy check (CRC) and an auto-correction algorithm are implemented in case of errors. For SEU and noise reasons, the ring is optical with laserdrivers. In the Tracker, the CCU is completed with a DCU for the voltage and temperature measurements which must also be read. In order to prevent any problem due to the loss of a CCU that would break the ring and consequently lead to lose all the CCU on the ring, a second ring is implemented. This redundancy ring makes it possible to bypass one CCU by configuring the input and output of each CCU and FEC. The two rings are presented in figure 5. Once the ring is built, for each access, whatever the channel is, a frame is built with a FEC number, a CCU address, a channel, a transaction and a command number. The channel number specifies one of the CCU channels. The transaction number allows to recover the action which initiates the frame, for example, a read command. The  

IEEE Standard 1149.1 (JTAG), IEEE Standard Test Access Port and Boundary-Scan Architecture

4

Figure 3: Current prototype of the PMC Front End Controller Board for PCI support

Figure 4: A token ring with Control and Command Unit module with 6 I  C channels

5

command number gives the command to be executed (read, write, read modify write). For the I  C accesses, the frame must also contain an address and possibly some data for write or read-modify-write operations. Note that the ring number is not given, it is configured via a specific register into the FEC and CCU. Token Ring A: Default ring (without error)

IN A

OUT A

IN A

CCU 1 B

OUT A

CCU 2

B

B

B

A OUT B A IN B

Token Ring B: Redundancy ring

FEC A

CCU 3

IN B

FEC: Front End Controller CCU: Control and Communication Unit

OUT A

IN A

OUT A

IN A

CCU 4

CCU 5 B

A OUT B

B

B

B

Figure 5: CCU token ring with the redundancy channel (A/B) For research and development reasons, two versions of the FEC have been developed, one with an electrical ring and one with an optical ring. Both FEC are developed under the Peripheral Component Interconnect (PCI) bus but
the final version of the FEC will be on the VME  bus with eight optical rings. An intermediate version will be available in mid-2004. For the same reasons two versions of CCU exist (CCU6 and CCU25).

2.2 Mode of operation The normal mode of operation for the FEC is as follows, where all the FEC control and status registers described are shown in figure 6. When an action is to be performed on a unit connected to the ring one first makes sure that the link is initialized and that no pending operation is active. A data frame is then prepared and written word by word into the transmit FIFO (figure 6). When the frame has been written the SEND bit in the control register 0 (figure 6, CR0) is toggled. The FEC then waits for a token to replace it by the data frame present in the transmit FIFO. Once the token is grabbed the status register 0 (figure 6, SR0) indicates TRANSMIT RUNNING. Depending on the ring delay the transmitted data will then start returning to the FEC at which point the status register 0 indicates RECEIVE RUNNING. Data will then be assembled and written to the FIFO Receive. Normal ring delay is around (8  25  number of CCU) ns so the delays are normally rather short, i.e. of the order of microseconds. Once the whole message (direct acknowledge) is returned the last word is written together with the status of the frame, i.e. Error, Address Seen and Data Copied (figure 6, FIFO Receive). If errors have been detected in the FEC the status is signaled as CRC Error, Illegal sequence or Illegal Data depending on the case. In case packets return with the Data Copied bit set to 0, that indicates that the addressed channel was busy and the packet should be retransmitted. The retransmission is not done automatically in the hardware but should be taken care of in the software, in this case the FIFO return is used. When the last word has been written into the receive FIFO the FEC sends an interrupt request (IRQ) to the PCI which can be fed through the host computer if this function is enabled in the FEC PCI-to-local-bus bridge. Once the IRQ has been sent the CLEAR IRQ has to be sent to control register 1 (figure 6, CR1) to indicate that the IRQ has been taken care of. Together with the IRQ there is an indication in the status register telling if the frame is a returned frame or a frame destined for the FEC. In case of a write operation, the CCU I  C channels can be configured in order to send an additional frame, called  

Versa Module Eurocard

6

force acknowledge frame, with a status. This status ensures that the frame has arrived or not to at a device. The direct acknowledge ensures only that the data frame arrives at the correct CCU and is sent over the correct CCU channel. Note that the FIFO Receive can receive frames with warning information from devices (like the front end electronics) or from external alarms through the CCU or the PIO channels. All the FEC, CCU and channels have status and command registers to report errors or to take configuration actions: one of the CCU registers is used to enable or disable channels in order to reduce the power consumption; an I  C channel register also exists to configure the speed of the bus; a control register in the FEC and CCU is used to configure the input/output for the redundancy ring. The principle looks like the TCP/IPX protocol and the C++ API looks like the layers of the Open System Inter
connection (OSI  ) model defined in the networking community. The next sections describe the complete software structure developed on the hardware architecture in order to meet all the requirements for the Tracker control system and for other parts of the CMS experiment.

 

ISO 7498

7

End Of Message

Reset OUT

CRC Error

Link Initialized Tra. Empty Tra. Full

9

Tra. Half F.

8 7

Ret. Full Reset TTCRx

Ret. Empty

Ret. Empty

SEL IRQ Mode

Address Seen

Rec Full

SEL SER IN

4

Data Copied

Rec Half F.

SEL SER OUT

3

CRC Error

Rec. Empty

FEC Release

Resend

2

Illegal Sequence

Rec. Running

Clear Error Indicators

Send

Illegal Data

Tra. Running

Clear IRQ

Enable FEC

1

Ret Half F.

5

DATA

0

LAST DATA

DATA

LAST DATA

Error

6

FIFO Transmit (Tra.)

Destination Source Length Channel Transaction Command Data ...

Frame

Destination Source Length Channel Transaction Command Data ...

FIFO Return (Ret.)

Low

FIFO Receive (Rec.)

Ill Sequence

10

Illegal Data

11

Pending IRQ

12

Data Copied

13

Data to FEC

14

Address Seen

Used in case of automatic retransmission

TTCRx

Direct Acknowledge Force Acknowledge or Answer

8

CR0

CR1

SR0

SR0

Bits

Figure 6: FEC status and control registers with the three FIFOs.

Error

15

End Of Message

3 Software architecture Device Driver

PCI bus

FEC

i2c access

Hybird / devices

FEC Supervisor OCCI

Oracle

C++ API Device Driver

FEC

Hybird / devices

Device Driver

FEC

Hybird / devices

Device Driver

FEC

Hybird / devices

Device Driver

FEC

Hybird / devices

Device Driver

FEC

Hybird / devices

SOAP (Finite State Machine)

Database FEC Supervisor C++ API

CMS General Run Control IMS

FEC Supervisor C++ API

subscribe

DCU Analyser

Database access through Oracle Call C++ Interface (OCCI) Finite state machine: SOAP message

Diagnosis system

Hardware access Device driver access (software: open, read, write, ioctl, close)

Figure 7: Software architecture The software architecture presented in figure 7 is built upon the hardware architecture presented in the previous section. The software is split up into several parts: - Run Control [7]: controls the data acquisition and control system with the help of the Detector Control System. - Information Message service (IMS) integrated in the run control: dedicated task that receives all the messages (information, error, ...) belonging to the run control. - FEC device driver: a Linux device driver for the FEC. - FEC supervisor (C++ API, the FecSoftware API): central part of the control, it manages several FECs in order to read and write values into the front end electronics. - Database: Oracle database used to store all the values needed for the hardware. It manages all the parameters to set the hardware with the help of a versioning system. - DCU analyzer: dedicated process to analyze all the values provided by the DCU. - diagnostic system: system that diagnoses the source of errors and possibly takes a decision and signals it to the run control system. The software control is built around the FEC device driver, database, FEC C++ API (FecSoftware API), DCU analyzer and the diagnostic system. The control software issue is divided in two actions: - Download and upload parameters in the front end electronics: these two operations require that all FEC / CCU / channels (PIO and I  C channels) needed for the access, are enabled and configured. Once the control hardware is configured, the software sends the value needed to each register of the front end electronics (download). At the end of the download operation, the parameters set are read back from each device and compared to the values downloaded. If a difference appears the faulty device register is stored in the database and an error is reported to the run control through the IMS. A similar operation can be done directly between the event builder and the FEC control processes (FecSupervisor) to make calibration or a delay curve. During this kind of operation, the event builder asks the control system to set PLL or APV values. The download/upload system must be done in a relatively short time (about 2 minutes for the current measurements, see figures 10, 15, 16) in order to be able to acquire data as quickly as possible. 9

- Control the environmental parameters: this operation consists in reading all DCU on the hybrid and CCU and to send all this information to the IMS. By a mechanism of web subscription, the information is sent to the DCU analyzer to check the correctness of the parameters (voltages and temperatures). These readings are performed every thirty seconds but this rate can be changed dynamically depending on the needs of the system. This system must detect a failure in any system in terms of temperatures and power. A second system exists but it is controlled through hardware locks to switch off all the Tracker to prevent, burning for example. Note that the DCU system is important to avoid the switching off of the complete system. If a problem occurs in temperature or humidity, the DCU analyzer must detect the divergence of the system and must send a message or take the decision to switch off the system correctly one part at a time. In the next sections, each part of the software control system will be described with the requirements of each part, the mode of operation and performances: FEC device driver, Database, C++ FecSoftware API, DCU analyzer and finally the diagnostic system.

10

4 FEC Device Driver 4.1 Overview of the device driver The device driver developed in the framework of the control/command system of the CMS Tracker is a low-level program, aimed to give users and programmers a basic but easy and complete access to the hardware components used for the configuration and the control of the Tracker. Therefore, this device driver (called further on the FEC driver) can be seen as a glue layer that sticks together the hardware side and the software side of the control/command system of the CMS Tracker. The functionalities offered by the FEC driver can be accessed through functions defined in a user interface, a library which provides all the necessary entry points for accessing the FEC board, the CCU and all the devices attached to the CCU. This library will be called, further on, the glue layer.

4.2 Designing the driver One of the main characteristics of the FEC driver is inherent in the constraints given by the framework of the control/command team: this driver had to be designed to run under Linux OS. Therefore, it has been developed as a software module, i.e. an independent part of code that can be added or removed on request from the OS kernel, at run time. The other characteristics of this driver have been defined during its design stage, following two main guidelines: fully exploit the hardware potential and keep the source code as modular and adaptable as possible. 4.2.1 Hardware requirements Seen from the hardware side, this device driver has to comply with different requirements in order to be able to use all the functionalities of the FEC board efficiently. Management of asynchronous events coming from the hardware, like trapping and processing incoming data frames on the token ring, implies the use of an interruptdriven event manager. Furthermore, thanks to the buffering capabilities of the FEC via its FIFO, sending a frame of the kind ”data requested” on the ring is an action totally dissociated from receiving the answer to this frame. We can then have one user that accesses the driver in order to write a frame on the ring and, at the same time, several users that access the driver in order to check if the data they requested are present. This implies that the FEC device driver must support multi-user access by blocking write mode and unblocking read mode. Paragraph 4.3 details the reasons why a blocking write mode has been preferred to a non-blocking write mode. 4.2.2 Software requirements Seen from the software side, keeping the source code as modular and adaptable as possible has been achieved by two means. First of all, the driver’s internal functions have been categorized into well-separated layers, as independent as possible. Each layer is composed of a group of not too complex functions, dedicated to one specific level of hardware. The FEC driver is actually composed of a PCI layer (configuration of the PCI bus of the computer), a PCI-to-local-bus bridge (PLX PCI 9080 Master chip) layer, a FEC layer (read/write accesses to the FEC registers), and a driver administration layer (code sanity checking, data consistency checking, error management and emergency commands). This logical architecture of the driver gives a good modularity, useful for debugging, for maintenance purposes and for quick adaptation in case of hardware evolution. The adaptability of the code was the second important point to take into account. It seemed that code adaptation would be needed not only to follow the hardware evolution, but also to follow the user’s requests. In order to be able to face these constraints without consuming a large amount of development time at each change, all the constant values and all the physical characteristics of the hardware components (registers size, bits positions, bits meaning, etc.) have been defined externally in a header file. Complex or inter-correlated values have been defined via macros in this header file. For the same reasons, all the data types used in the driver source code are home made data types, defined in a second header file. That way, it is possible to change totally the physical properties of the hardware attached to the FEC driver without changing one single line of code. Furthermore, a back compatibility had to exist between the different released versions of the driver. With the use of the glue layer (see paragraph 4.1), it is possible to change totally the internals of the driver while keeping the same user interface with the same data types, the same returned values and the same sent parameters. This allows to acheive a back compatibility level close to 100%.

11

4.3 Device driver read/write internals Basically, writing or reading data from the devices attached to the CCU token ring is performed as described below in 4.3.1 and 4.3.2. All the synchronization and multi-users protections are performed via semaphore. The multi
user operating mode for the write method is managed by a mutex  , and the multi-users operating mode for the read method is managed by a 127-users-max semaphore. This limit fits the highest usable transaction number, i.e. the highest number of transactions that can be active simultaneously. A mutex has been chosen for the write method, in order to ensure that no incomplete or badly formed frame could be released on the ring due to race conditions. The time needed to fill or read data from the FEC FIFO is far higher (nearly two order of magnitude) than the software pre-processing time. Furthermore accesses to the FEC FIFO are, by hardware conception, single-user accesses. So, there are nearly no differences between locking the whole write method or locking only the critical sections of this same method. The simplest way often being most powerful, the choice has been made to lock the whole method with a single mutex. All the hardware-related operations described in points 4.3.1 and 4.3.2 are protected by watchdogs. If, after a given delay, an operation remains answerless, all the resources used for achieving this operation are freed and a timeout status is sent back to the user-side calling process (if it exists). 4.3.1 Writing a frame A primary frame is built in the user space (opposed to kernel space), and sent to a write function of the glue layer. The driver checks that no other write operation is currently running and that the system state is nominal, assigns a transaction number to this frame and stores this frame into the FEC FIFO Transmit. The FEC FIFO Transmit content is then sent on the ring and the write method of the driver is put on sleep. When the data frame has passed all along the ring and has been received in the FEC FIFO Receive, an interrupt is raised. This returned frame will be called, further on, the ”direct acknowledge” frame. The interrupt manager reads this returned frame from the FEC FIFO Receive, checks its sanity, and wakes up the waiting write method. The frame is then copied to the calling process (user space) with a status report about the good or bad achievement of the operation and with the transaction number assigned to this frame. If, after a while, no frame is returned, a timeout error is returned to the user. If the written frame does not lead to the incoming of a second one in the FEC FIFO Receive (mainly the case of the CCU configuration frames), the assigned transaction number is tagged free and can be used for a further write request. If the written frame leads to the incoming of a second one in the FEC FIFO Receive (in case of data requested or forced acknowledge mode), the tag ”busy” is kept on the transaction number assigned to the direct acknowledge frame. When this second frame arrives and falls into FEC FIFO Receive, an interrupt is raised and the interrupt manager stores this frame in memory. If, for any reason the awaited second frame does not arrive in the receive FIFO, a timeout is issued and the transaction is tagged “free”. 4.3.2 Reading a frame If a user has sent a write command and knows that following this write command a data frame must be issued from the hardware, he has to call a read function of the glue layer with the help of the transaction number issued from the write command in order to catch this data frame. The read method checks if a data frame with the appropriate transaction number has been received by the driver, and if yes returns it to the user process. If such a data frame does not exist, the read method is put on sleep for few milliseconds and the existence of the frame is checked again. If the requested data frame is still not present, the method returns a timeout error; else, the data frame is returned to the user.

4.4 Error management and integrated securities The driver code embeds low-level error counters and a health monitoring system. System health is to be understood from a software point of view. Users can know, via the glue layer, how many errors have occurred on the hardware and what kind of errors they were. It is possible to know if some warning frames have been emitted by a device, and to read these frames. One can check if the device driver state is still nominal, if a reset (hardware or software) has been triggered recently, and what the current interrupt manager state is.  

semaphore with a user counter limited to 1 as defined in Portable Operating System Interface (POSIX)

12

This last information is useful in a particular case. If too many interrupts are generated too quickly by the PLX 9080 because of a noisy ring (hardware and ground problem) or broken ring, these interrupts will flood the ring and consume the whole computing time i.e., the computer will look ”frozen”. If such a situation occurs for more than one second, the interrupt line of the PLX 9080 is disabled in order that the operator can take the hand again on the system. In that case, all the next read/write requests addressed to the driver will issue a timeout error. If such a behavior is detected, the first thing to do will be to check the interrupt line status and to turn it back on if it has been auto-disabled.

4.5 FEC Timings 4.5.1 Delays measurement conditions The FEC timings can be subdivided in different parts, detailed further on. The timings exposed below start from an initial time defined in user-space, when a client program sends a data frame to the glue layer in order to write it on the token ring. The timing measurements end when the direct acknowledge frame has been fully sent back to the client program. The operation that consists in calling a kernel side function from the user side space, or opposite, is defined further on as a context switch action. All the measurements performed in these timings and performances analysis have been made on an Intel Pentium III 800MHz, equipped with 256 MBytes of RAM, and running a Linux RedHat 7.3 operating system (distribution and kernel supported by CERN). The measurement values given for frames of length higher than 200 elements are extrapolated values, as the actual physical frame length limitation is set to 200 elements. This is due to Linux kernel internal limitations regarding the device driver ioctl functions used for performing context switches between kernel and user space. All measurements have been done with the following status: one element of the frame is 16 bits long. 4.5.2 Measured delays 35

context switch time in microseconds

30

25

20

15

10

5 0

500

1000

1500

2000

2500

3000

3500

4000

frame length (number of 16 bits elements)

Figure 8: Context switch delay versus frame size The first measurement in figure 8 represents the time needed to perform a context switch between user side and kernel side, as a function of the size of the frame to transfer from the user side toward the kernel side. A linear fit yields   "!$#&%')(* + ',  .-0/214365278- . The second measurement in figure 9 represents the time needed to perform a context switch between user side and kernel side plus the time needed to pre-process this frame (i.e. prepare the frame for writing into the FEC transmit FIFO). A linear fit yields 9:0; )#?;" @ + , A B-/21"36578- . The third measurement in figure 10 represents the total of the time needed to perform a context switch between user side and kernel side, the time needed to pre-process the frame, the time needed to send this frame on the ring and to read it back plus the time needed to perform a context switch between kernel side and user side. 13

180

context switch + preprocessing time in microseconds

160

140

120

100

80

60

40

20

0 0

500

1000

1500

2000

2500

3000

3500

4000

frame length (number of 16 bits elements)

Figure 9: Context switch plus pre-processing delay versus frame size This time can be considered as the overall time needed to successfully issue a command on the CCU ring. A linear fit yields C  ;4 >#?"C @ DE , A .-0/214365278- .

total delay in microseconds for full frame processing

3500

3000

2500

2000

1500

1000

500

0 0

500

1000

1500

2000

2500

3000

3500

4000

frame length (number of 16 elements)

Figure 10: Complete write frame cycle delay versus frame size One should notice here that the overall delay is significantly higher than the context switch or pre-processing delays. Except for frames of very small size, most of the time is consumed during FIFO writing/reading - each read or write action takes 0.25 microseconds per frame element. Instead of rough measurements, the most interesting criterion to study is certainly the bit rate at which the data can be written on the ring. This information is given in figure 11 for write operations performed on the ring. It is evident from the last curve that the available data rate is very quickly limited, and that working with very long frames does not improve this rate very much. Frames of 200 elements allow to work at over 1Mbyte/s, which is enough for our needs.

4.6 Conclusions In conclusion, we can say that the device driver developed for driving the FEC board and the token ring attached to this board fulfills our needs. Even better, the original specifications (exposed in paragraph 4.2) have been covered, both in hardware and in software. We have demonstrated high-performance, modular low level software, that can 14

1.2 1.1 1

data rate in Mbytes/s

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

500

1000

1500

2000

2500

3000

3500

4000

frame length (number of 16 elements)

Figure 11: Bit rate for write operations versus frame size easily follow the user requirements and future hardware evolution, and is capable of fully exploiting the hardware for which it has been developed.

15

5 Database Due to the large number of parameters (about 1 680 000 values) to set in the hardware, a database management system (DBMS) is mandatory. It is used in order to store all the information needed and for a possible modification of one register on one device. The device parameters can change depending on the data to be retreived and on the timing. In order to store also the conditions for a data capture run, a versioning system is added. This section is about the database solution used to store all these parameters with a relational model, the way to retrieve information from the database and the way to store the information into a database. Another problem we have to face, is the performance of the database. We do not have a specified target performance to reach, but we assume that the final user of this software/database should not wait too long in order to download all the parameters in the Tracker devices. So a part of this section describes the solution chosen in terms of performance.

5.1 Relational Model The model of the database, described in figure 12, is a representation of the software and hardware chain and other concepts like version and partition managements. Module ModuleId ModuleName I2CChannel PartitionId CcuId

CCU

Ring RingId RingSlot FecId

CcuId CcuAddress RingId

FEC FecId FecSlot XdaqInstanceId

XdaqInstance XdaqInstanceId ClassName SupervisorId

Supervisor SupervisorId SupervisorIp

Device DeviceId I2CAddress ModuleId

ApvMux DeviceId VersionMajorId VersionMinorId ValueType ApvMuxParameters

Apv25 DeviceId VersionMajorId VersionMinorId ValueType Apv25Parameters

Laserdriver DeviceId VersionMajorId VersionMinorId ValueType LaserdriverParameters

Pll DeviceId VersionMajorId VersionMinorId ValueType PllParameters

DCU DeviceId TimeStamp ValueType DCUParameters

Philips DeviceId VersionMajorId VersionMinorId ValueType PhilipsParameters

Version VersionMajorId VersionMinorId VersionName VersionDate Table Name Partition PartitionId PartitionName

State VersionMajorId VersionMinorId PartitionId

Primary Key Column Name Reference + Primary Key Reference

Figure 12: relational database model

5.1.1 Software and Hardware chain The tables Supervisor, XdaqInstance (specific process identification), FEC, Ring, CCU, Module represent the software and hardware chain. A Device is a generic electronic device: an Apv25, an ApvMux, a Pll, a Laserdriver,
a DCU or a Philips F chip. 5.1.2 Partition, version and state management Partition A partition is a set of modules. A module belongs to one and only one partition; it can be moved from a partition to another. A supervisor can manage one or several partitions but a partition is managed by only one G 

Philips Semiconductor chip used for research and development

16

supervisor. A partition is tagged by: - partitionId: identifier of the partition (primary key) - partitionName: name of the partition We assume that all modules, belonging to a partition with partitionId greater than one, are in an active state and partition one is used for unused modules (disabled). Version The versioning is the way to keep a history of all parameter value changes. Suppose we use just one version number for the front end parameters. If between two runs just one parameter value is changed then a new complete version must be created: all parameters, even those which have not been changed, are copied. This mechanism implies a waste of disk space. To reduce the disk space usage, we have implemented an incremental mechanism with two numbers which characterize the version: - VersionMajorId: major version number - VersionMinorId: minor version number - VersionName: name of the version - VersionDate: creation date When a version N.n is created with a new minor number (n) but same major number (N), only the values different from version N.(n-1) are stored in the database. To retrieve the values of version N.n from the database: - we take the values from version number N.n if these values are present in the database. - else we take the values from version number N.(n-1). If these values are not present in the database, we take the value from version number N.(n-2), etc. For example a major number 2.0 references a full set of all parameters values for all devices and a minor number 2.2 references a possibly incomplete set of parameters for a subset of devices. We will use the major number 0 to define a version corresponding to a read from hardware values. We just keep few “hardware” versions. State A state is used in order to correlate a version with a partition. So several partitions can download different version parameters. The CMS Tracker has different parts that can need different configurations depending on the hardware, so a version for each partition can be used. A state is tagged by: - stateId: identifier of the state (primary key) - stateName: name of the state - versionMajorId: major version number - versionMinorId: minor version number - partitionId: identifier of the partition The XdaqInstance downloads the parameter values for the devices they control. Looking at the State table, it can know for each partitionId the major version number N and the minor version number n. For each module, it will then take the parameter values corresponding to the major number N and the highest existing minor version number less than or equal to n.

5.2 Database Management System Oracle was chosen as a database management system for two main reasons. On one hand, Oracle provides useful features that we will discover in the further subsections and on the other hand Oracle is a well known software company, in the research and industry world. Moreover, the CERN Oracle division provides a support service that is very efficient and will be used to build the Oracle database cluster for LHC experiments. We use the Oracle9i Enterprise Edition (see reference [8]). The next paragraphs discuss the solution and tools used to save time in terms of performance. We use several tools; some of them only exist with Oracle DBMS: 17

- View: A view is a logical representation of one or more tables or views in Structured Query Language (SQL); - Oracle Programming Language over Structured Query language (PL/SQL): specific programming language embedded in Oracle; - Oracle Package: entity that groups logically related PL/SQL types, items and subprograms; - Trigger: Triggers are procedures written in PL/SQL, Java, or C that run (fire) implicitly whenever a table or view is modified or when some user actions or database system actions occur; - Extensible Markup Language (XML) technology: PL/SQL packages for XML processing. This document format is used commonly in software technologies and the CMS data acquisiton and control system has chosen XML as the format to be used.

5.3 View Views can be considered as virtual tables, like windows through which you see data stored in your real tables. Simplified data access, data independence, data privacy are some benefits of using views. We create a view for each type of device; the next example is the definition of the view for the PLL device type. CREATE OR REPLACE VIEW ViewPll AS SELECT DISTINCT supervisorIp, className, Fec.fecId, fecSlot, Ring.ringId, ringSlot, Ccu.ccuId, ccuAddress, Module.moduleId, Module.partitionId, i2cChannel, Device.deviceId, i2cAddress, Pll1.valueType, Pll1.versionMajorId, Pll1.versionMinorId delayCoarse, delayFine FROM Supervisor, XdaqInstance, Fec, Ring, Ccu, Module, Device, State, Pll Pll1 WHERE Supervisor.supervisorId = XdaqInstance.supervisorId AND XdaqInstance.xdaqInstanceId = Fec.xdaqInstanceId AND Fec.fecId = Ring.fecId AND Ccu.ringId = Ring.ringId AND Module.ccuId = Ccu.ccuId AND Device.moduleId = Module.moduleId AND Pll1.deviceId = Device.deviceId AND Pll1.valueType = ’Set’ AND Pll1.versionMajorId = State.versionMajorId AND State.partitionId = Module.partitionId AND Module.partitionId > 1 AND Pll1.versionMinorId = ( SELECT MAX(Pll2.versionMinorId) FROM Pll Pll2 WHERE Pll2.deviceId = MonPll1.deviceId AND Pll2.versionMajorId = State.versionMajorId AND Pll2.versionMinorId