Fermilab
FERMILAB-Conf-01/292-E November 2001 1
A Two-Level Fanout System for the CDF Silicon Vertex Tracker A. Bardi, M .Bari, S. Belforte, A. Cerri, M. Dell'Orso, S. Donati, S. Galeotti, P. Giannetti, C. Magazzu, F. Morsani, G. Punzi, L. Ristori, F. Spinella, A. M. Zanetti
Abstract | The Fanout system is part of the Silicon Vertex Tracker, a new trigger processor designed to reconstruct charged particle trajectories at Level 2 of the CDF trigger, with a latency of 10 s and an event rate up to 100 kHz. The core of SVT is organized as 12 identical slices, which process in parallel the data from the 12 independent azimuthal wedges of the Silicon Vertex Detector (SVXII). Each SVT slice links the digitized pulse heights found within one SVXII wedge to the tracks reconstructed by the Level 1 fast track nder (XFT) in the corresponding 30 angular region of the Central Outer Tracker. Since the XFT tracks are transmitted to SVT as a single data stream, their distribution to the proper SVT slices requires dedicated fanout logic. The Fanout system has been implemented as a multiboard project running on a common 20 MHz clock. Track fanout is performed in two steps by one \Fanout A" and two \Fanout B" boards. The architecture, design, and implementation of this system are described.
I. Introduction
The main functional blocks of each Silicon Vertex Tracker slice are the Hit Finders, the Associative Memory system, the Hit Bu er and the Track Fitter [1] [2]. Every time an event is accepted by the Level 1 trigger, the digitized pulse heights in the Silicon Vertex Detector [3] are sent to the Hit Finders which calculate hit positions. The hits found by the Hit Finders and the raw tracks found in the Central Outer tracker by the Level 1 Fast Track nder [4] are then fed to both the Associative Memory system and to the Hit Bu er [5]. The Associative Memory system performs pattern recognition by selecting for further processing the combinations of XFT tracks and SVXII hits that represent good track candidates. This is done by comparing the input data with a stored set of patterns in a completely parallel way, using a dedicated custom VLSI chip (AM chip [6]). The AM system outputs a list of Roads. Each Road is de ned as a combination of SuperStrips on ve di erent detector layers that can be traversed by a single track. The SuperStrips used to de ne the Roads correspond to the hit positions on four silicon layers, while the fth SuperStrip is a function of the track parameters reconstructed by the XFT (track curvature and azimuthal Manuscript received on November 4, 2000. Corresponding author S. Donati is with INFN-Pisa, Via Livornese 1291, 56010 San Piero a Grado, Pisa, Italy (phone +39 050 880239, email
[email protected]). M. Dell'Orso, Dipartimento di Fisica, Universita di Pisa, Piazza Torricelli 2, 56100 Pisa, Italy A. Bardi, S. Belforte, S. Galeotti, P. Giannetti, C. Magazzu, F. Morsani, L. Ristori, F. Spinella, INFN Pisa, Via Livornese 1291, 56010 S.Piero A Grado (PI), Italy A. Cerri, G. Punzi, Scuola Normale Superiore, 56100 Pisa, Italy M. Bari, A. M. Zanetti, INFN Trieste, Area di Ricerca - Padriciano 99, 34012 Trieste, Italy
angle measured at the sixth superlayer of the COT at 106 cm from the beamline). To reduce the amount of required memory this pattern recognition process is carried out at a coarser resolution than the full available resolution. The Roads found by the AM system are sent to the Hit Bu er, that retrieves the original full-resolution silicon hit coordinates and the XFT track associated to each Road and delivers them to the Track Fitter system for full-precision computation of track parameters. The Fanout system has two main functions: performing the distribution of the XFT tracks to the proper SVT slices and mapping the two XFT track parameters to a single coordinate which is binned in SuperStrips and is used by the AM system. Simulation studies have shown that the most ecient use of the Associative Memory is obtained by using as SuperStrip the azimuthal angle of the XFT track extrapolated to a distance of about 10 cm from the beamline and a SuperStrip size of 25 mrad. For each XFT track received in input, the Fanout outputs a two word packet: the rst word contains the SuperStrip number used by the AM system, while the second word is a copy of the input word and is used by the Track Fitter for the fullprecision reconstruction of the track. For each event the Fanout receives also from the Global Level 1 trigger the information of which among the 64 CDF Level 1 triggers were red. This information is summarized and output from the Fanout as two bits in the special word used within the SVT system to mark the end of each event and to record diagnostic information (End Event word). II. Architecture and data format
The Fanout system is organized as a set of three (one of type \A" and two of type \B") 9U400 VME boards. Communication between the Fanout A and the two Fanout B boards takes place through a customized P3 backplane, with data owing from A to B. All the input streams to the Fanout system are received by the A board through three connectors placed on its front panel. One stream carries the XFT tracks and two carry the L1 trigger decisions. Data to the rest of the SVT are output only from the Fanout B boards. Each Fanout B has 6 output connectors on the front panel and feeds 6 SVT slices. Both the XFT data and the data output from the B boards conform to the standard SVT data format while L1 trigger data have a di erent format. SVT and XFT data occupy 25 bit words. In each word there are 21 data bits (the 21 least signi cant bits), an End Packet (EP), an End Event (EE), a Data Strobe (DS ) and a Hold (HD) signal
2
(Table I). The EP bit speci es if the same data packet occupies only one or more words. Its value is set to 1 for the XFT tracks since each packet (track) occupies one single word. The EE bit marks the end of data corresponding to one event. The Hold and Data Strobe are used for the communication protocol among boards. The DS signal is driven by the source board and indicates when the data become valid and can be latched by the destination board. The Hold signal is driven by the destination board. When the destination board does not keep up with the incoming data rate, the Hold signal is set active and the source responds by suspending the data ow, which is resumed as soon as the Hold goes out. This protocol is uniform among all the SVT boards. The word with the EE bit set to 1 is called End Event word (EE word), its data eld contains no information related to tracks but it is used to keep track of possible errors occurred during data transmission or processing and to record global information on the event. TABLE I
SVT data word format
24 23 22 21 20...0 HD DS EE EP Data eld The data eld of the XFT word contains 12 bits specifying the azimuthal angle, 6 the curvature and 1 respectively the charge, the isolation and the pseudorapidity of the track. The data eld in the End Event word contains the Event Tag (8 bits), the Parity (PA) of the event (1 bit), the Error Flags (8 bits), the L1 Trigger information (L1T) generated by the Fanout system (2 bits) and the Level 2 Bu er number (L2B) assigned to the event by the Trigger Supervisor (2 bits) [8]. The format of the End Event word data eld is reported in Table II, where the additional EP, EE (both set to 1), DS and HD signals are omitted. The meaning of the information recorded in the End Event word relevant to the Fanout system will be discussed in Sec. II-A.1. TABLE II
End Event data field format
20...19 18...17 16...9 8 7...0 L2B L1T Error Flags PA Event Tag Two streams from the Global Level 1 trigger carry the 64 Level 1 trigger decisions to the Fanout A board. This information is transmitted using one word per stream per event. The format of data is the same in both streams and uses a 33 bit data eld with a DS signal to indicate when data become valid. No End Packet, End Event and Hold signal are present. The data eld contains 32 trigger decisions (the 32 least signi cant bits) and 1 bit of the Level 2 Bu er number (Table 3). The combination of the two L2B bits from the two streams must match the L2B number recorded in the EE word of the XFT data.
TABLE III
Global Level 1 trigger word format
33 32 31...0 DS L2B L1 Trigger decisions For each XFT track received in input, the Fanout system outputs a two word packet through the B boards to the proper SVT slices, depending on the azimuthal angle of the track. The rst word of the packet is used by the AM system while the second one by the Track Fitter. The format of the two output words is the standard SVT data format. In the word used by the AM system only 15 out of the available 21 bits of the data eld are actually used: 12 bits contain the SuperStrip number, 3 bits the Layer number (set to 5) associated to the XFT tracks. The remaining 6 bits of the data eld are spare. The word used by the Track Fitter is a copy of the XFT input word. Since in this case one data packet is logically made of two words, the EP bit is set to 0 in the rst word and to 1 in the second one. The end of the output event is marked by the EE word, which is sent simultaneously to all the SVT slices. All the 12 End Event words are identical except for the Parity bit which can change from stream to stream because di erent streams transmit di erent data sets (Sec. II-B).
A. Fanout A board
The Fanout A board performs the entire data handling within the Fanout system. It receives and processes all the input streams and constructs the output data packets as well as the output End Event word (Sec. II-A.1). The B boards perform simply the fanout function. XFT tracks and L1 trigger bits are received asynchronously and are stored on three sets (1 for the XFT tracks and 2 for the L1 Trigger bits) of two 4K18 bit FiFos each. The set of FiFos receiving XFT tracks provides an Almost Full signal that is sent back to the source meaning Hold. If the Fanout A does not keep up with the incoming data rate, the FiFos become Almost Full and the Hold signal is asserted. The signal is active while the amount of unread data in the FiFos remains above the Almost Full threshold. Although this communication protocol is uniform among all the SVT modules, no handshake is implemented with the Global Level 1 trigger. This is allowed by the fact that the Level 1 trigger information is received at a much lower rate (one word per stream per event) than the XFT tracks. The Fanout A board transmits data to the B board through a customized P3 backplane. The data stream contains one standard SVT format word (21 bit data eld plus EP, EE, DS and HD bits), 12 \enable" bits, used by the B boards to select which SVT slices (one or more) must receive the word, and 2 clock lines routed on the backplane to have each B board receiving one line (Sec. IV). When the Data Strobe bit is active, the B boards receive one word for each clock cycle, validated at the positive edge. The Hold line is driven and is common to both the B boards. It is implemented as a wired-or chain. When the Hold line
3
* C207
R47
R45
R185
R184
R192
R183
R193
C97
C236 C237 R206
R205
R204
P42
P88
P94
C265
C229
IC81
C61
C62
R244 P97
C59
R207
Vbias Fuse: 1A R208 R31
P41 R209
VCC
R32
C64
P98
P91
D1
C260
C60
P92
BYTEBLASTER CONNECTORS P223 R78
C53 C57
RS22
C254
R8
C22
IC39
C267
IC19
R176
R210
R213
C297
C43
C77
C86
C78 C79
C88 C87
C80
C41
C218 C184
R82 C219 C185
C175
C174
R10
P147
R140
C31
C82
C83
C81
C89
P61
P143 P144 R87
P120
C110
C144
R84
C143
C145 P154
P452
IC20 R85
C187
C147
C138
C137
C136
C135
C188 P125
R13
R97
C26
IC44 IC45 IC46
P443
R16
P477
R15
P124
IC93
R14
P183
R96
C28
R141 C149 C148
P480 R134
C111
P481
C11
R44
P479 P485 P182
R95
IC43
C15
IC40 C13
C100
P186 C150
C23
P126
P487
P478
R46
C16 C27 C25
C116
R93
C139
C24
R92
C112
C30
P187 C113
P196
R137
P179 C10
P439
RD
C109
C114
P195
C140
IC72
C164
C12
R86 C115
P15
C141
P194
C163
C151 R136
P193
C162
C152
P462
P438
P192 C142
C153
P466
P155
P191 P28
C161
C154
P437
P475
C160
P476
C146
C155
C268
RS45
IC75
C159
IC47
P416
P503 P505 P417
IC92
P29
P408 P415
R30
IC41 C17
IC42
R139 R133
C158
P489
C36
C29
IC94
R26
C32
C38
R3
C37
R4
P442
IC52
P407
C220
R9 C157
C156
P493
C35 P436
P5
P173
P504 P468
R138
P172
C84
IC83
C176
IC73
J8
P491
RS43
R128
R18
C173
P114 C177
P488
P409
C
P136 R215
P142 R242
IC51
R88
IC76
D
P133
P434
C14
C350 J3
R81 R217
IC82
U10
RS44
P3
R17
R216
C172
C39
R89
IC200
C242 R68
R165
R241
P414
C244
P37
IC84
C171
C178
R240
C34
C90
C85
C247
P132
R161
C1
R164 R163 P134
C170
C179
P512
RS41
P131
C169
C180
R237
U4
R63
C76
R162
C168
C181
C215
R235
R239
P435
C9
IC90
C182
R130
R231 R233
R238
P410
RS42
C8
C183
C18
R232 R234 R236 P411
IC50
C246
P54 C7
IC74
S2 R27
P413
R28
IC85 C2
R175
R174
C298
C48
C42
R172 R173
R171
R157
R169 R170
R166
C166
R194
C19
R168 R167
P540 C165
P552
C33 P412
C217
P541
P538
P553
P554
R129
R156
P544
IC89
C243 R29
C45
IC77
P533 P539
P122
IC35 R155
IC2
R58
P4
IC91
R159
R158
P537
IC37
R142
C40
P483 P492
RD
RD
P1
P10 P8
TEST CLOCK R160
C238
C44
IC36
P128 P129
C20 P293
GND
C245
C221
C222
C223
C224
P495 C186
P542
R55
U1
ST10
P145
P181
P180 P534
C249
P556
P294
IC3
RESET
*
R196
P148
C216
IC99
R197
R195
IC88
IC86
C46
IC22
RS2
P106
R198
R200
C248
C21
C47 R109
P292
IC4
RS3
P149
P130
P146
P543
IC38
TEST
P14 P151
R7 P536
IC107
R201
C269
P535
P105
P301
IC5
R66
ST2
C251 P461
C50
P57
ST1
20 MHz
C258
R21
C55
C51
R6
C49
P300 P299
ST4 ST3
R25
P555 P250
P296 P295
ST6 ST5
FS
IC78
P247
P244 P342
IC6
RS4
IC106
ST8
F02A F13A
J5
P341 P9
P297 RS6
ST7
F01A F12A
F03A
OSCILLATOR
C58
C259
C54
R99 C52
IC78
P509
C108
C257
R103 IC79
RS21
C250
EPC2 EEPROM P207
P245
RS5
F10A F00A F11A
R199
P211 P175
P246
R5 P248 P117
P152
C107
P209
C126
P240
P249
IC7
C106
RS20
C253
ST9
P212
C125
C127
P116
R212
R102
6
IC79
C124
J10
C255
C128
P338 P118
IC100
0
P213 C129
P334 P237
P298
P104
P103
J6
C130
IC23
IC8
J2
IC98
P55
1
P208
RS8
RS7
R211
RS27 5
P243 P340
C56
P273
IC1
EPC2 EEPROM
C123
P7
R108 P272
P256
P242
C131
C256
RS1
C132
P339
P96 R214
4
R22
P43
S3
C252
IC9
P352
C122
RS9 U13
C133
RD
C121
P221 P353
GROUND
all switches to 0
GEOADD GEOADD GEOADD GEOADD GEOADD
J4
C120
P241 P322
R56 U6
K
P89
VIPA crates:set
S1
direct
C66 C134
IC11
P90
3
C213R106R107C214
2
FLEX10K P257 P258
R67
ON
C117
IC29 P336
P93
P113
1
C118 C119
P115
P219 C65
J0
C262 C261
R243
P38
P6
R76
P365
P323
fuse!
F1 P35
P326
IC12
P95
VCC Fuse: 10A !mount only one
P20
C296
P348
IC13
Z
P102
P445
P266
R77
IC27
RS12
B A
C104
C96 C94
R36
C92
R202 R203
C74
C71
C95
C73
C70
C234
C98
R38
R49
R53 R64
R23 C72
C75
C99
P393 R127 C198
F4
R65
C228
R220 R219 R218 C197
R1
P331
RS11
Z
P60 IC95
P157
C203
C196
C199
R2 C63
F3
C206
R54
R222 R221
C69
P161
P33 P39
P455
P460
P32
IC103
P22 P101 P86
F2
R62
R191
R182 IC31
IC58
C263
P40
C91
IC201
C204 C266
R34
C93
P459
P31 P356
P23
P83
P99 P100 P156
IC80
P85 P456
C195
C202 C200
P123
RS13
B A
C231
P84
P30
C6
IC24
C201 P119
P349P71
IC14
C
C233
P171
P19 P330 P328
R111 P325 P58
D
P21
P81
P448
R37
C194
C4
C68
P48
P50
P158
C193
C3
P357
C67
P80
C105
C264
P329
IC104
C235
IC68
R61
IC60 IC54
C5
C192
R39
P404
IC102
P109 P79 P24
P355
P327
IC15
P110
P108
P25 P27
P82 P508 P354
IC28
IC16
P18
P26 P507
P383
R41
R51 R60
IC69
R12 C191
C103
C190
IC30
P351
P324
R19
C232
C189
R223 R110
RS14
Z
P46
R33
P34
GND
S/N
P392
R125
RS10
RS15
B A
R20
P67 R35
august 2000
C230
C226
C225
C227 R112
P402
IC17
P47
P78
R226 R225 R224
RS16
P45
C102
P69
A.BARDI
P394
R229 R228 R227
P347
RS17
C
C101
R181
R189
R180
R188
P112
design:S.DONATI
P121
R230
R24
IC10
D IC101
IC105
INFN PISA
P406
P384 P386
IC18
R179
P2
R187
IC34
P385
R48
C205
RS18
R178
P388
CDF-SVT
P16
U12
J11
There are two identical Fanout B boards whose identity is set by an onboard switch. Their function is to receive the data from the Fanout A board through the P3 backplane and to distribute the 23 data bits to the proper SVT wedges according to the list of enable bits. They have also the
P401
IC33
P391
RS46
B. Fanout B board
IC32
P387
R186
IC53
C212
P53
C208
IC56
P390
R11
U5
P107
XTF-A V1.0 P389
P403
R50
P111 P52
IC59
IC67
R52
C210
R42
R59
C209
C211
R190
RD
U2
R57
A.1 Error handling and data ow monitoring There is a number of error conditions that can be detected by SVT while processing data, for example something can go wrong in the data transfer and the input FiFo on one board can become full, or some of the data received in input by one board are outside the valid range. The system can both set error ags in the VME register of the single board and propagate error ags in the data stream setting appropriate error bits on the End Event word [7]. The Fanout A can detect 4 out of the 8 possible SVT error conditions: the Parity Error (if the parity of the input XFT data does not match the Parity bit of the XFT End Event word), the Lost Synch (if the L2B of the XFT End Event word does not match the L2B of the L1 Trigger words), the FiFo Over ow (if one input FiFo gets full) and the Invalid Data (if the parameters of one XFT track are outside the valid address range of the Look Up Table). When the A board detects one of these conditions, it sets the corresponding error ag in the output End Event word. If one error bit is already asserted in the input EE word, it is propagated in the output EE word. The SVT boards have also a system for data ow monitoring. The input and the output data streams from the boards are continuously copied to circular memories called Spy Bu ers. In the Fanout A board these memories are 128K static RAMs for the XFT data and 64K both for the L1 Trigger data and for the output Spy Bu er, which spies data owing to the P3 connector. These bu ers act as built-in logic state analizers and help system monitoring and diagnostics. As a consequence of error conditions these bu ers can be frozen and read from VME with no interruption of the normal data ow.
RD J1
RD
IC57
is active, the Fanout A stops reading new data from the FiFos. The SuperStrip number (12 bits) inserted in the word used by the AM system and the list of 12 enable bits associated to each XFT track are determined using a Look Up Table. The Look Up Table is implemented as a set of 6 (512K8 bit) static RAMs where 20 bits summarizing the XFT track parameters are used as an address and the SuperStrip number and the list of enables are stored in each location. To obtain a 1M24 bit Look Up table, the 6 memories are organized as two blocks of 3 units, with bit 20 of the XFT parameters actually used as a chip select to enable either the rst or the second block of RAMs. The 2 L1T bits are generated by the A board as a programmable combinatorial function of the 64 Level 1 Trigger decisions. The logic of the board is implemented on seven 240 pin FLEX10K20 FPGAs from ALTERA. The rmware is fully recon gurable on board.
SILKSCREEN TOP LAYER
Fig. 1. PCB drawing of the Fanout A board. Left side is front panel side, right side is VME connectors side.
function to set the Parity bit in the End Event words sent to the 12 outputs. The Parity of each stream is computed by the A board and is transmitted to the B boards: when the EE word is sent from A to B, the 12 enable bits have a special meaning, each of them is the parity bit of the corresponding output stream. The B boards insert each PA bit in the EE word sent to the corresponding output stream. In addition the B boards control the Hold and Data Strobe bits of each output stream. Data ow out of each Fanout B board through the 6 connectors on the front panel. If the Hold signal is received from any output, the line on the P3 backplane meaning Hold is set active and the Fanout A stops reading new data. Each Fanout B receives the clock from the Fanout A board through the P3 backplane and its logical functions are implemented on one 240 pin FLEX10K20 FPGA from Altera. For simplicity this board has no Spy Bu ers and no VME interface. III. Operation modes
The Fanout system has two di erent operation modes: running mode and VME mode. In running mode it processes the data received through the input connectors and outputs the proper data to each SVT wedge. The VME mode is used to write the content of the Look Up Table used to map the XFT tracks into SuperStrips and for testing purposes. In particular in VME mode the content of any Spy Bu er can be written from the VME interface and the board can be forced to process these data as if they were the actual input streams. In this operating mode data contained in the input FiFos are disregarded. This can be done for both the input streams simultaneously as well as for only one. This allows an almost complete test
4
have been optimized for the maximum symmetry among similar functional blocks. References [1] S. Belforte, M. Dell'Orso, S. Donati, G. Gagliardi, S. Galeotti, P. Giannetti, et al. \The CDF Trigger SVT", IEEE Trans. Nucl. Sci., vol. 42, pp. 860 - 864, 1995. [2] W. Ashmanskas, A. Bardi, M. Bari, S. Belforte, J. Berryhill, M. Bogdan, et al. \The CDF Silicon Vertex Tracker: online precision tracking of the CDF Silicon Vertex Detector", Il Nuovo Cimento, vol. 112A, no. 11, pp. 1239 - 1243, 1999. [3] J. A. Valls, \The SVX II Silicon Vertex Detector at CDF", Nucl. Phys. B, vol. 78, pp. 311 - 314, 1999. [4] C. Ciobanu, J. Gerstenslager, J. Hoftiezer, R. Hughes, M. Johnson, P. Koehn, et al. \Online Track Processor for the CDF Upgrade", IEEE Trans. Nucl. Sci., vol. 46, pp. 933 - 937, 1999. [5] S. Belforte, M. Dell'Orso, S. Donati, G. Gagliardi, S. Galeotti, P. Giannetti, et al. \The SVT Hit Bu er", IEEE Trans. Nucl. Sci., vol. 43, pp. 1810 - 1813, 1996. [6] S.R. Amendolia, S. Galeotti, F. Morsani, D. Passuello, L. Ristori, N. Turini, \The AMchip: a Full-custom CMOS VLSI Associative Memory for Pattern Recognition", IEEE Trans. Nucl. Sci., vol. 39, pp. 795 - 797, 1992. [7] M. Bari, S. Belforte, A. Cerri, M. Dell'Orso, S. Donati, S. Galeotti, et al., \Error Handling for the CDF Silicon Vertex Tracker", these Proceedings. [8] CDF II Collaboration, \Technical Design Report", FERMILABPUB-96/390-E (1996). Fig. 2. PCB drawing of the Fanout B board. The orientation of the picture is as in Fig. 1.
of the system in absence of one or both the real input data sources. IV. Clock distribution
Clock signals are generated on the Fanout A board and are distributed to the two Fanout B boards through two dedicated lines of the P3 backplane. Each Fanout B board receives one clock line. Clock generation on the Fanout A is accomplished by using one quartz oscillator connected to one PLL based clock bu er from Quicklogic (\Roboclock"). The Roboclock produces eight outputs with 4 adjustable phase shifts with respect to the input clock, also allowing zero or negative delays. The timing of each phase is adjusted by controlling the individual programmable delays of the Roboclock, in steps of about 1 ns. Each of the three Spy Bu ers uses one Roboclock output to generate the write enable signals to the RAMs. Two Roboclock outputs are sent to the Fanout B boards and one is sent to a set of two PLL logic fanout chips which provide clock signals to the input FiFos and to the Altera FPGAs on the Fanout A. Each Fanout B board is equipped with a system of two Roboclocks which receive in parallel the clock line from the Fanout A board. One Roboclock output is used by the Altera programmable chip. Each of the six output channels from the board uses one Roboblock output to generate the Data Strobe signal on the cable. This allows adjusting the Data Strobe timing with respect to the data bits. V. PCB design
The Fanout A and B boards have been implemented as a six layers Eurocard 9U PCB. Critical lines have controlled impedance. The placement of components and routing
