Performance and System Flexibility of the CDF Hardware Event Builder

Fermi National Accelerator

Laboratory

FERMILAB-conf-9l/314

Performance and System Flexibility CDF Hardware Event Builder

of the

T. Shaw and K. Schurecht Fermi National Accelerator Laboratory P.O. Box 500, Batavia, Illinois 60510

P. Sinervo University of Toronto, Dept. of Physics 60 St. George, Toronto, ON, Canada M5S IA7

November 1991 * Presented at the IEEE Nuclear Science Symposium, Santa Fe, New Mexico, November

e

Operated

by Universities

Research

Asaoclatlon

Inc. under Contract

No. DE-ACW-76CHO3000

2-9, 1991.

with the United States Department

of Energy

Disclaimer

This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof, nor any of their employees, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefullness of any information, apparatus, product, or pmcessdisclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Gouernment or any agency thereof. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof

PERFORMANCE

AND SYSTEM FLEXIBILITY HARDWARE

OF THE CDF

EVENT BUILDER

T. M. Shaw and K. Schurecht Fermi National AcceleratorLaboratory’ P. 0. Box 500, Batavia, Il. 60510 P. K. Sinervo University of Toronto Dept. of Physics, 60 St. George, Toronto, ON, CanadaMSS lA7 fiBSTRACT The CDF Hardware Event Builder [l] is a flexible systemwhich is built from a combination of three different 68020-basedsingle width Fastbw modules. The systemmay containasfew asthreeboardsor asmany BSfifieen, dependiug on the specificapplication. Functionally,the boardsreceivea commandto read out the raw eventdata from a set of Fastbus baseddatabuffers(“scanners”),reformatthe dataandthenwrite the datato a Level 3 txigger/processingfarm which will decide to throw the event away or to write it to tape. The data acquisitionsystemat CDF will utilize two nine board systems which will allow an eventrate of up to 35 Hz into the Level 3 trigger. This paperwill presentdetailedperformancefactors, systemand individual board amhitechre, and possiblesystem oonfiglNations.

Tom the four cable segmentsis performed simultaneously. When completed, the Event Builder informs the Buffer Manager. The Buffer Managerthenwaits for the EventBuilder to inform it that it has reformatted the event into a YBOS [4,5] structure which will be recognized by Level 3. The Buffer Managerthen tells the Event Builder to push the event to a given addressin Level 3. The protocol for B single event concludeswhen the EventBuilder informsthe Buffer Manager that it has completedthe push.

7, I

,

t

INTRODUCTION The data acquisition system at Fermilab’s Collider DetectorFacility (CDF) consistsof a large F&bus network. Figure 1 gives an abbreviatedoverview of the network. The hardwareEventBuilder resideson this nehvorkand its job is to readout the front end scanners,reformatthe raw dataandship the gathered and reformatted event to a Level 3 trigger/processor farmwhich will thendecidewhetherthe event will be written to tape or thrown away. Approximately 85 front end scanners,each with four internal buffers, must be read out by the Event Builder. Thesescannersare distributed amongfour Fastbuscable segmentsthrough which the Event Builderwill readthemout. The Event Builder is directed in its actions by the Buffer Manager [2], a sofhvareprocessrunning on a VAX workstation which gains access to the Fastbus network througha Q-busProcessorInterface(QPI) module. The Buffer Fig. 1 Overviewof CDF Data Acquisition Network Mangerprovidescentralizedcontrol and communicationwith the trigger system, Level 3 and the Event Builder. Event readout [3] is initiated when the Buffer Managerinforms the maximum event processing rate of the Event Event Builder, through a Fastbusinterrupt message,that an Builder The is defmed as the rate at which the overall data event is present in one of the fiat end buffers. The Event acquisition system contributes no more than 5% overall Builderwill theninitiate a readoutofthe scanners.Tlx readout deadtime(deadtimeis accruedwhen all soarmabufferscontain event data waiting to be read out.) The expectedeventsizefor * Operatedby UniversityResearchAssociation,Inc., the CDF detector is 180 kilobytes. Simulation results have undercontractwith the U.S. Departmentof Energy predictedthat a maximumeventrate of 35 Hz will be achieved

by installing two Event Builders in the CDF data acquisition valuablebandwidth on F&bus. The bus physically consists network. The Buffer Managerwould alternateeventreadout of two 50 conductorcableswhich attachto the front panelsof the Event Builder boards. Electrically, it uses RS-485 betweenthe two EventBuilders. transceiversandsupportsup to 15 systemboards. A minimal Event Builder systemwould consist of threeboards- one Crate Controller [6], one Cable Controller, and one Reformatter. A maximum of fifteen boards can be THECRATECONTROLLER installed in a single system. The Crate Controller communicateswith the Buffer Managerand controlsthe push, The Crate Controller acts as the “traffio cop” for the or write, to Level 3. The Cable Controllers control the pull, EventBuilder System. Only one Crate Controller is required or reading, of raw data from the scanners. The Reformatters in the EventBuilder system. AU communicationbetweenthe are responsiblefor reformatting the raw event data. Boards Event Builder and the Buffer Manager is done through the communicateamongthemselvesthrough a specially designed Crate Controller. The Crate Controller is also the internal t?ontpanel bus. The proposed9-boardEventBuilder systems bookkeeperfor the Event Builder. It keepstrack of which of which areplannedfor installationat CDF would consistof one its internalbuffersor “engines”are free,reformatting,or ready Crate Controller, four Cable Controllers, and four to be written out to Level 3. The Crate Controller also Reformat&s, seefigure 2. ContIolsthe writing of datato Level 3. The Crate Controller has both F&bus master and slave capabilities on the Fastbuscrate segment. Its Fastbus interfaceis implementedas a coprocessor[7] to the MC68020. A microsequenceroperatingwith an 88 bit field controls the actual gating of the Fastbussignals. A block diagram of the CrateControlleris found in Figure 3. --

Fig. 2 ProposedCDF EventBuilder Confwation This paper will discussthe board and system level architecture,possiblesystemconfigurationsand performance of the Event Builder, and outline how we chosethe system configuration best suited for CDF’s application. The CDF terms of Buffer Manager and Level 3 wiU be used to help describethe Event Builder system, but, these terms really imply the needfor somecentraltaskmanager(BufferManager) and destinationbuffer (Level 3) for use with a generic data acquisitionnetwork.

I

a=II, ~,“‘, ,!, !II !II I I Ivq I .Icg+Y(-‘x1 ~___ __

EVENT BUILDER MODULES ‘Ilx EventBuilder consistsof three different typesof modules: a Crate Controller, a Cable Controller and a Reformatter. The Controller boardswere basedon the Aleph EventBuilder, but their designwas adaptedto CDF’s specific needs.The Reformat& boardwasessentiallya new design. All boards are basedon Motorola’s 68020 processor andrun a versionof Motorola’smonitor sol&we. Eachof the EventBuilder boardshas 512 kilobytes of staticRAM and512 kilobytes of PROM. The modules also featuretwo RS-232 ports through which downloading and communication are possible. A Front Panel bus was designedso that the boards could communicate among themselves without tying up

Fig. 3 CrateController Block Diagram

THECABLECONTROLLER The Cable Controller has the primary responsibility of controllingthe readoutof the front-endscanners.The Cable Controller has the same Fastbus functionality as the Crate Controller, exceptthat its electrioal co~eotion to Fastbusis to the Fastbuscablesegment.

THEREFORMTTER The Event Builder performsa DMA operationon the cable segmentfrom each scannerto the Reformatter event buffer. The CableController performsbus arbitrationand the addrm cycles to selecta given scannerand then togglesDS (data strobe) transitions on the cable segmentto have the scannerplace the data on the bus. The Reformatterspieson the bus and latchesthe data placed on the bus each time the scannertogglesDK (dataacknowledge.) The Reformatter itself can perform no Fastbus transactions,but it doeshaveconneotionsto the dataand parity lines on both the Fastbus crate and cable segments. This allows it to spy on the segmentsand either puU data in or push it out on commandfrom one of the Controller boards. The Reformatterreadsdata in t?om a Fastbuscable segment under commandof the Cable Controller and writes data out undercommandof the CrateController onto the Fastbuscrate segment.

Internally, the Reform&x has two “engines” or blocks of memory for receiving events;thus, it is capableof buffering two events at s time. After the data has been receivedby the Reformatter,the processorsoausthrough the data and builds a DMA table that lists the order in which individual blocks of data are to be written out. This table includesthe “header”blocks that identify the type of data sod the length of the blocks. Theseheaderblocks are constructed by the processoras it scansthe data. The arrangementof the data and headerblocks in the DMA list ensuresthat the event informationhasbeenproperly rearrrmgedto group datafrom a oommondetector subsystemtogether, even though this data mayhavebeenreadout from severalscanners. The design of the Reformatter allows it to simultaneouslyperform two of its three iitrwtions at any one time, i.e. reading from the front-end scanners,formaning or writing data to Level 3. Figure 4 shows a block diagram of the Reformat& board POSSIBLE SYSTEM CONFIGURATIONS A maximumof tifieeu boards may be presentin an Event Builder system. One Crate Controller is required and from one to four Cable Controllers are to be included Each Cable Controller requires at least one Reformat%, giving it two eventbuffers. Additional Refonuatterscould be addedto work with a given Cable Controller if additional eventbuffers arerequiredat the EventBuilder level. A minimum Event Builder systemwould consist of three boards: a Crate Controller. a Cable Controller, and a Refommtter. This would meantl& the &+s data aoquisition systemwould be set up such that all front-end scannersare addressableandcau be read out through a singleFastbuscable segmenton which the CableControllerandReformator would reside. Sometask manager(in the caseof CDF, this would be the Buffer Manager)would then communicatewith the Crate Controller over the Fastbus crate segment and direct its actions. In addition,somehigher levelbuffering device(Level 3 in the CDF system)would have to be addressableby the Crate Contiollor throughthe Fastbuscrate segmentso that the reformatteddatacouldbe dumpedto it A larger exampleof an Event Builder systemwould includethe following boards: one CrateController,four Cable Controllersand eight Reformatters. Thus, front-end scanners could be distributed along four separate Fastbus cable segments,each of which would be connectedto one of the CableConWollers.Also two Reformatterswould be connected to eachFastbuscablesegmentto work with the residentCable c0nti011er.

1

Figure 2 illustrates the Fastbus crate and cable segmentconnectionsof a possibletie board configuration.

Fig. 4 Reformat& Block Diagram

THE CDF EVENT BUILDER CONFIGURATION

THEORIGINALCDFCONFIGURATION The original CDF configuration was a five board systemwhich had one CrateController,two CableControllers andtwo Reformat&s. Its throughputduring the last CDF run was 4-7 Hz into Level 3, as documentedin a previouspaper [S]. Many of the possible board level, system level and sofiware improvementswhich were describedin that paper such as doubling the Front Panel Bus bandwidth, adding a secondDMA table to the Reformatter,addingpipelinetransfer capability, and improving the DMA mechanism were implementedin the new generationof Event Builder Boards that we arc using now. However, it was anticipatedthat the original configurationwould still be limited by an Event rate of 15-18 Hz. Since we wanted to squeezethe maximum performanceout of the system,we beganto investigateother possiblesystemconfigurations.

In addition, five to ten milliseconds arc spent waiting for variousmessagesfor the buffer manager. Oneof the first decisionsto be madewas to distribute the front-end scauuerson four cable segmentsinstead of the original two. Factorsthat influencedthis distributionwere the fact that certain data banks could not be split between cable segmentsand that the large overhead for reading out each scannerhad to be factoredin with the amountof dataexpected In the initial system,the suumorswere distributedbasedon the amount of data alone, which led to very unbalancedreadout timeson the two cablese~neuts. A fairly balancedsot of pull timeswasarrivedat by t&ii thesefactorsinto aocamt.

From that point, we decidedto focuson four possible systemsand use Verilog results as well as system tests to determinethe best solution. Systemsunder study included a nine board system,shown in figure 2. The tie board system wasnow our minimum choicesincetherewere now four cable segmentsto be read out. Also studied was a thhteen board systemwhich would includeau additiomdReformatteron each Fastbuscablesegment.The t-malquestionwhich we askedthe CDFSY~~EMSTUD~~ANDPROPOSED simulatorto resolvewas whether or not it made senseto run CONFIGURATION two Event Builder systems with the Buffer Manager distributing events between them. The results of the In order to determine the best system for CDF’s simulationsarc reportedbelow. apphcationwe begandoing extensivetiming measurements on au existing five board system. We also developeda Vorilog 1 EventBuilder 9 Board System 21 Hz behavioral model of the CDF data acquisition system. 1 EventBuilder 13 BoardSystem 27 Hz Detailed information about the form and results of this 2 EventBuilder 9 Board Systems 35 Hz simulationis discussedin anotherpaperbeing presentedat this 2 EventBuilder 13 BoardSystems 35 Hz conference[9]. The simulationwork was aidedby the factthat we were ableto measurethe behaviorof the componentsof the It was discoveredthat at 35 Hz, the Buffer Managerbeginsto system.We foundthat the simulationsaccuratelypredictedthe be the bottleneck of the system. Therefore, a decision was behavior of the five board system,and subsequentlya nine madethat the best solution for CDF was to run two 9 board board system. We therefore felt confident to let it analyze Event Builder systems. Figure 5 illustrates how the other configurationsand beganto be able to seethe resultsof connectionsto the scannersand Level 3 look to the Event certaintradeoffs. For example,did it makesouseto add more Builder. Reformattersto the cable segmentsor were the additional bufferingthey providedusedso infiequeutly that the overhead CONCLUSIONS in the CrateController for keepingtrack ofthom did not make it warranted? ‘Ibe EventBuilder is a configurablesystemcapableof reading data J?omfront-end scannerswhich are distributed Someof the timing parametersthat form the input to amongone to four Fastbuscable segmentsand which writes the sim”lation are: formatted data out onto the Fastbus crate segment. A minimum system would include three boards, one Crate ReadingScanners- Pull Time Controller, one Cable Controller and one Reformatter. Up to -2 ms in preparingfor the read fifieen boardsmay be presentin a system,which would have -200 us setupper scauuerto be read one Crate Controller, from one to four Cable Controllersand -350 11sper word read at leastoneReformat&xper CableController. Reform&t& of data Systemstudies for the currently implementeddata -5 msoverhead acquisition system at CDF have indicated that a 35 Hz -500 us per YBOS Bank built throughputrate shouldbe possibleby running two tie board -100 us per DMA pointer (1 pointerper component EventBuilder systems.Thesesystemshavebeeninstalledand block in scanners) will be usedin the upcomingCDF datarun in 1992. Writing to Level 3 - PushTime -4 mspreparingpush -2 ms overheadduringpush -270 ns per word written

[9] K. Schurecht, et al, “A Verilog Simulation of the CDF DAQ System”, paper submiti to this symposium.

Fig. 5

Two Parallel Nine Board Event Builder Systems REFERENCES

Ul A. W. Booth, M. Bowden and H. Gonzalez, “Specification for a Hardware Event Builder”, CDF Note 452, F&lab.

PI C. Day, “Buffer Manager Software Design”, CDF Note 326, Fermilab. [31 P. K. Sinervo, et al, “Fast Data Acquisition with the CDF Event Builder”, IEEE Transactions on Nuclear Science, Vol 36, No. 1, Febmary 1989. L41 J. T. Carroll, “YBOS Scanner Bank Format”, CDF Note 264, Fermilab. bl

D. Quark, et all, “CDF Event Structure”, CDF Note 152, Familab.

PI

r. M. Shaw, “Hardware Description of the Controller Board”, Event Builder Technical Note EVBMIl, Fennilab.

[71 T. M. Shaw, “The Coprocessor Interface in the CDF Event

Builder Technical Note EVBIHI4, Fermilab.

PI

f. M. Shaw, et al, “Architecture and Development of the CDF Hardware Event Builder”, IEEE Transactions on Nuclear Science, Vol. 36, No. 1, February 1989.