IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 49, NO. 2, APRIL 2002
501
A Remote Control System for FPGA-Embedded Modules in Radiation Environments K. Hasuko, C. Fukunaga, R. Ichimiya, M. Ikeno, Y. Ishida, H. Kano, H. Kurashige, K. Mizouchi, Y. Nakamura, H. Sakamoto, O. Sasaki, and K. Tanaka
Abstract—A remote control system has been developed for versa module Eurocard (VME) modules located in a radiation environment. Two new VME modules—the remote controller (RC) and local interface modules—are introduced to mediate between the local host and remote slave modules. These two modules are connected with optical links and the local host can master the remote VME bus to access the slave modules through these intermediate modules. This control system can perform watchdog for field programmable gate array (FPGA)-embedded modules whose configuration data are susceptible to single event upsets (SEUs). The architectural study and first prototyping of this system are discussed. Index Terms—ATLAS, control, field programmable gate array (FPGA), large hadron collider (LHC), radiation, versa module Eurocard (VME).
I. INTRODUCTION
T
HE ATLAS experiment [1] at the large hadron collider (LHC), scheduled to start in 2006, is presently under construction at the European Center for Nuclear Research (CERN). The ATLAS is a general-purpose detector designed to study proton–proton collisions. One of the signatures used to select important candidate events is high transverse momentum muon. Such events can be taken only by a multilevel trigger system well optimized for identification of muons with wide momentum range. Because of the high bunch crossing frequency of 40 MHz and the large number of detector channels, the ATLAS trigger system consists of three levels of online event selection. In the first-level trigger electronics for the endcap muon (T1ME) system, versa module Eurocard (VME) modules are used and located in the detector [2]. Field programmable gate arrays (FPGAs) are actively used in these modules to install complex circuits in limited space as well as flexibility of development and cost performance [3]. However, since FPGAs are susceptible to radiation-induced upset [single event upset Manuscript received June 7, 2001; revised December 8, 2001. This work was supported by the Japanese ATLAS project. K. Hasuko, H. Kano, Y. Nakamura, and H. Sakamoto are with the International Center for Elementary Particle Physics (ICEPP), University of Tokyo, Tokyo 113-0033, Japan (e-mail:
[email protected]). C. Fukunaga, Y. Ishida, and K. Tanaka are with the Department of Physics, Tokyo Metropolitan University, Hachioji 192-0397, Japan. R. Ichimiya and H. Kurashige are with the Department of Physics, Kobe University, Kobe 657-8501, Japan. M. Ikeno and O. Sasaki are with the Institute of Nuclear and Particle Physics, High Energy Accelerator Research Organization (KEK), Tsukuba 305-0801, Japan. K. Mizouchi is with Department of Physics, Kyoto University, Kyoto 6068502, Japan. Publisher Item Identifier S 0018-9499(02)03928-X.
(SEU)], the firmware configurations will be corrupted and the functionalities will be crushed because of SEUs. Since the radiation level where T1ME system is placed is not expected to be extremely harsh [4] and cost performance should be considered, the system is built using only standard electronics components [commercial off the shelf (COTS)]. SEU rates in COTS are expected to be low in T1ME system, but they are not yet negligible. Hence, an on-detector remote control system is necessary to detect SEUs and restore the lost functionalities of FPGAs. In this paper, the architectural idea and design of the remote control system is mainly introduced in Section II. Prototype implementation of the system is described in Section III and then the functionality is summarized in Section IV. Performance tests and results are discussed in Section V. Finally, the conclusion is given in Section VI. II. SYSTEM ARCHITECTURE AND DESIGN The idea of SEU watchdog is accomplished using VME functionality and Xilinx Virtex series FPGAs supporting byte-based configuration scheme [5]. The remote control system initiates VME functionality as a master. FPGAs on VME modules are always watched via VME bus by periodical readback procedure. However, there is risk of losing VME functionality itself if VME controllers are implemented using COTS devices. This remote control system should implement the strategy for restoration of VME functionality from SEUs. An overall architecture of the remote control system is shown in Fig. 1. The goals of the system are as follows: 1) to control and configure the remote slave modules in a radiation area from a local host via VME bus and 2) to restore VME functionality if it failed. Two additional VME modules—remote controller (RC) and local interface (LI) modules—are introduced between the local host and slave modules as shown in the figure. The RC module is in a VME crate inside a radiation area (called remote crate). As a master module, it controls all the slave modules in the remote crate. The LI module is a slave module in another VME crate in a control room (called local crate). It is managed by a local host via a VME master module. The RC and LI modules are connected with dedicated links for a full-time controlling. Optical links based on agilent transmitters and receivers (G-LINK) [7] are chosen for the system. The host can control the slave modules with an instruction set described later. Details of the two new modules are described in Sections II-A and B.
0018-9499/02$17.00 © 2002 IEEE
502
Fig. 1.
IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 49, NO. 2, APRIL 2002
Overall architecture of the remote control system.
the backplane of the remote crate. Dedicated instructions are prepared to control the JTAG bus. The RC module has two blocks to encode/decode instructions to/from the LI module. One is called primary instruction converter (PIC) and works for JTAG control; the other called secondary instruction converter (SIC) for VME control. At least, the PIC and JTAG controller should be robust to SEUs and other radiation effects. Functionalities of the SIC and VME controller are implemented in CPLDs, which can be restored with JTAG via the PIC and JTAG controller. Instructions to reset functional blocks are interpreted at PIC. B. LI Module
Fig. 2. Block diagram of the RC and LI modules.
A. RC Module Fig. 2 shows a block diagram of the RC and LI modules. When the instructions on VME control are received, the RC module can master the VME bus to access slave modules in the remote crate. All the control and configuration to the slave modules are performed on the basis of the VME accesses. FPGA-embedded modules are also configurable; configuration bit streams can be read and written via the VME bus. Both the RC and slave modules have implemented VME protocol converters in complex programmable logic devices (CPLDs). These CPLDs have serial ports to be configured and IEEE 1149.1 [Joint Test Action Group (JTAG)] is usually used as the serial protocol. The RC module provides JTAG signals for the CPLDs. The JTAG signals are put in a bus structure on
A block diagram of the LI module is also shown in Fig. 2. Instructions from the host are at first stored into instruction registers on the module and then transmitted to the RC modules. Responses from the RC module are stored into response registers and kept for the local host. There are some other registers for the control/status and special VME functionality such as interrupts. All the registers can be accessed via VME bus. III. PROTOTYPE IMPLEMENTATION Prototypes of the RC module, backplane of the remote crate and LI module have been implemented. A. RC Module and Remote Crate Backplane The RC module has been built as special 9U VME module, whose size is 367.7 mm height 160 mm depth. This size is required by the area limitation. For optical links, Infineon EO/OE converter [6], G-LINK transmitter and receiver are used to support 16-bit duplex transmission. A 40-MHz oscillator provides the synchronization clock to operate at 640-Mbits/s serial data rate. These components consists of G-LINK unit. The main functionality has been implemented using two different programmable logic devices manufactured by Altera
HASUKO et al.: REMOTE CONTROL SYSTEM FOR FPGA-EMBEDDED MODULES
503
(a) Fig. 3.
(b)
Photograph of the (a) RC and (b) LI modules.
Corp [8]. Both are MAX7000AE devices; one is a QFP144 for the PIC and the other a QFP208 for both the SIC and VME controller. In the final version of the RC module, the device for the PIC will be replaced by an ASIC, which is expected to be much more tolerant to the radiation effects, especially SEUs. In the PIC, JTAG signals together with an internal address for the destination are packed in an eight-bit bus data and stored for the multidrop JTAG bus system with embedded JTAG controller [embedded test bus controller (eTBC)] [9]. These JTAG signals are distributed to an appropriate device specified by the address on the bus. The bus is built into the VME P3 backplane. TCK is provided by dividing the 40-MHz oscillator clock. The CPLD-based VME controller gives access to resources in the slave modules using VME A32/A24 D32/D16 data transfer. No block transfer access is supported. The controller supports VME interrupts (see below). It can provide the VME bus system clock and the bus-error (BERR) global timeout (100 s). Neither the bus arbiter nor the power monitor are configured in this controller. A customized backplane of the remote crate has been built and mounted in the position of the J3 backplane in a 9U VME crate. This backplane also provides for 3.3 V required by the devices on the RC module. B. LI Module The LI module has been built in 6U VME64 extension module required from the specification of the local crate. The same G-LINK unit is mounted to communicate with the RC module. The main functionality has been built using a programmable logic device, ACEX1K, manufactured by Altera Corp. The CPLD-based VME controller supports A32/A24/A16 D32/D16 data transfer. Fig. 3 shows a photograph of the RC and LI modules with optical fibers.
IV. FUNCTIONALITY A. Control Sequence Control using the RC and LI modules is based on an instruction set summarized in Table I. It is defined with a 14-bit control word of G-LINK. Each instruction is executed from the local host with a control word. It includes some parameters such as bit data for eTBC, etc., to reAM code for VME access duce the system overhead. When reading data, writing data, or handling interrupts, two additional 16-bit data words are also used. After execution of the instruction, the RC module returns a response to the LI module and the sequence is completed. The instruction and its response are stored in dedicated registers of the LI module (Fig. 2). The response register has an update bit to indicate whether the register has been renewed with the response corresponding to the last instruction or not. The local host can check the result and status of the corresponding instruction through the LI module. B. Application Procedure A main flow in the ATLAS T1ME system is shown in Fig. 4. After the initialization by a reset instruction, the remote control system is ready to access the slave modules in the remote crate. The FPGAs in the slave modules are configured via VME write accesses. A write access is accomplished by a series of instructions, setVMEA, set VMED and configVME [Fig. 4(a)]. During experimental runs, the local host executes a polling program to read back periodically the FPGAs’ configuration data via VME read accesses. A read access is accomplished by setVMEA and configVME; the target data are sent to the LI module through the G-LINK; the response control word from the RC module is answered back [Fig. 4(b)]. If an incorrect bit is detected (probably due to SEU), the target FPGA is instantly reconfigured with
504
IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 49, NO. 2, APRIL 2002
TABLE I SUMMARY OF THE INSTRUCTION SET
Fig. 5. Flow of interrupt handling. (a) Interrupt enable. (b) Interrupt disable in the LI module. (c) Interrupt disable in both the LI and RC modules.
from the RC module. A flow chart of interrupt handling is shown in Fig. 5(a). The functionality to handle interrupt requests can be disabled in both the RC and LI modules independently via instructions [Fig. 5(b)–(c)]. If the RC module receives an IRQ when interrupt handling is disabled, it sends a special instruction noting that the IRQ is provisionally inhibited [Fig. 5(c)]. V. PERFORMANCE TESTS AND RESULTS A. Performance Tests Fig. 4. Application flow in the endcap muon trigger system. (a) Configuration of slave modules. (b) Readback of configured data. (c) Reconfiguration of VME controllers. (d) Reconfiguration of the SIC in the RC module.
VME write accesses. The polling frequency can be optimized by considering the SEU rate of the whole endcap muon trigger system. If a VME access is failed and a time-out response is reported from the RC module, the VME controller on the target slave module has probably failed because of SEU. The corresponding CPLD is reconfigured using JTAG in the backplane of the remote crate with configeTBC instruction [Fig. 4(c)]. If there is no response to VME instructions in the RC module, the SIC of the RC module is assumed to have failed and it is reconfigured in the same manner as shown in Fig. 4(d). C. VME Interrupt Handling The remote control system supports VME interrupts from any of the slave modules in the remote crate. The RC module receives an interrupt request (IRQ) and status ID from an interrupter and then sends the interruption information to the LI module using a control word and two data words for the IRQ and status ID, respectively. Upon receipt of the interrupt information, the LI module activates the interrupt request to the local host with its independent IRQ level and the status ID derived
Four prototype sets of RC and LI modules have been constructed. Performance tests have been done using the following setup: The VME bus of the local crate is connected via the PCI bus to an IBM PC/AT compatible PC via an SBS PCI-VME adapter [10]. The host PC runs Linux OS with vmehb [11] as a device driver for the PCI-VME adopter. The test software has been written in C++ and uses a VME library, vmelib [12]. RC and LI modules are connected with a duplex optical fiber. B. Results All the implemented instructions have been issued from the ) has been meaPC. The response time of the RC module ( is defined as the time between sured with 1-ns accuracy. the output of the G-LINK Rx (instruction control word) and the input of the G-LINK Tx (response) in the RC module. The meaare summarized in Table II. of the SIC sured values of is longer than that of the PIC because the PIC has an arbiter and the cost for the instructions is about four clocks. Additional 11 of clocks of configVME are due to the VME read cycle. configeTBC includes the response of the eTBC. Fig. 6 shows a waveform of configVME instruction and its response to “access VME” and “read a 32-bit data.” The performance of the remote control system in the endcap muon trigger has been estimated from the test results shown above. The RC and LI modules are connected via 90-m optical
HASUKO et al.: REMOTE CONTROL SYSTEM FOR FPGA-EMBEDDED MODULES
505
Fig. 6. A waveform of configVME instruction and its response. TABLE II RESPONSE TIME OF THE RC MODULE
TABLE III EXPECTED RADIATION LEVELS IN THE ATLAS ENDCAP MUON SYSTEM
Working Group has determined the simulated radiation levels for the whole experimental operation of the LHC with some safety factors. The expected radiation levels in the endcap muon system are summarized in Table III [4]. The unit for total neutron fluence is the number of 1-MeV equivalent neutrons/cm ; the unit for hadron fluence refers to hadrons with energy of 20 MeV or more, which are the primary source of SEUs. All the COTS components on the RC module have an unknown tolerance to the radiation level shown above. The irradiation tests for the components are planned to be performed. VI. CONCLUSION
links to the trigger system. Considering the propagation, the bandwidth can be estimated to be 1 MB/s. In the current design of the readout module, the total capacity of FPGA configuration is 18 MB/crate; the total time of the FPGA configuration/readback is estimated to be 18 s. This estimation indicates that the remote control system can perform a watchdog for the FPGAs with sufficient polling frequency and can provide quick reconfigurations for damaged FPGAs. In the endcap muon trigger, the system-wide SEU rate is roughly estimated to be a couple of upsets every hour, but this rate is acceptable for the remote control system. C. Radiation Tolerance The remaining issues on stability of the remote control system are radiation tolerance of the COTS components on the RC modules. The ATLAS Radiation Hardness Assurance
We have developed a remote control system for VME modules located in a radiation environment. The architectural study has been discussed in this paper. The primary purpose of our development is the first-level trigger for the endcap muon system of ATLAS. The main components of the system are the RC and LI modules. These modules are connected with optical links based on G-LINK, Agilent transmitters and receivers, and the local host can communicate with the slave modules in the radiation area through these new modules. The control system can perform watchdog for FPGA-embedded modules whose configuration data are susceptible to SEUs. FPGAs on the slave modules are configured, read back and reconfigured, if necessary, via the VME bus using byte-based configuration scheme. In case VME controllers using CPLDs on the RC and slave modules loose their functionalities, these CPLDs are configurable with JTAG provided on the backplane of the remote crate. These JTAG signals also can be managed by the control system. Prototypes of the RC module, backplane of the remote crate and LI module have been implemented. These modules have been built using CPLDs and performance tests have been done. They work according to the instructions submitted from the host PC and the corresponding responses are correctly reported. The bandwidth of the control system is estimated to be 1 MB/s and it allows the remote control system to act as a watchdog for the FPGAs.
506
IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 49, NO. 2, APRIL 2002
The system is built using only COTS components. Although the tolerance of the COTS components to the radiation level in our application is unknown, the architectural idea of the COTSbased VME control system has the great advantage of reducing the development cost and is applicable to other VME systems.
ACKNOWLEDGMENT The authors would like to thank Prof. T. Kondo, High Energy Research Organization (KEK), Japan, and Prof. T. Kobayashi, ICEPP, University of Tokyo, for their support and encouragement. The authors would like to thank the staff of Mitsui Zosen Systems Research Inc. (MSR) for their help and advice. The RC and LI modules were implemented by MSR.
REFERENCES [1] W. W. Armstrong et al.. (1994) ATLAS: Technical proposal for a general-purpose PP experiment at the large hadron collider at CERN. ATLAS Collaboration. [Online]. Available: http://atlasinfo.cern.ch/ATLAS/TP/tp.html [2] K. Hasuko et al., “First-level endcap muon trigger system for ATLAS,” in Proc. 6th Workshop Electronics for LHC Experiments, Cracow, Poland, Sept. 2000, CERN/LHCC/2000-041, pp. 328–331. [3] H. Sakamoto et al., “Readout system for the ATLAS end cap muon trigger chamber,” Nucl. Instrum. Meth., vol. A453, pp. 430–432, Oct. 2000. [4] M. Dentan. (2000) ATLAS policy on radiation tolerant electronics. ATLAS Project Document. [Online]. Available: http://atlas.web.cern.ch/Atlas/GROUPS/FRONTEND/radhard.html [5] [Online]. Available: http://www.xilinx.com/ [6] [Online]. Available: http://www.infineon.com/ [7] [Online]. Available: http://www.agilent.com/ [8] [Online]. Available: http://www.altera.com/ [9] [Online]. Available: http://www.ti.com/ [10] [Online]. Available: http://www.sbs.com/ [11] [Online]. Available: http://www.nikhef.nl/~natalia/projects/vmehb.html [12] [Online]. Available: http://onlax2.kek.jp/~nakayosi/BIT3/index_e.html
