As fault tolerant monitoring systems in Double Chooz, Gaibu. (external in .... Each event goes through a higher level of monitoring stage, where not only viewing ...
Online monitoring system for Double Chooz experiment T. Konno, A. Cabrera, M. Ishitsuka, M. Kuze, Y. Nagasaka, Member, IEEE, Y. Sakamoto, Member, IEEE, and A. Shigemori
Abstract—The Double Chooz is a reactor-neutrino experiment at Chooz nuclear power plant in France to measure the unknown neutrino mixing angle θ13 with a better sensitivity than the current best limit. A remote-monitoring framework has been developed to check status of DAQ systems, which enables collaborators off-site to access online monitoring data though the internet. As for these graphical user interfaces (GUI) are developed with platform independent technologies to be available through the internet. As fault tolerant monitoring systems in Double Chooz, Gaibu (external in Japanese) system and Log-Message system have been developed. Gaibu system works as an alarm message provider working outside DAQs and Log-Message system is a log file manager Developments of these systems are almost finished and we are preparing software commissioning at the far detector site now.
Thus, neutrino oscillation experiments become one of major topics to explore the new physics model. This document shows a brief outline of a reactor neutrino experiment Double Chooz, and then shows an online monitoring system with the experiment. II. T HE D OUBLE C HOOZ
EXPERIMENT
I. I NTRODUCTION
T
HE Standard Model in particle physics has been established along with improvements of detector technologies and increased accelerator energy. On the other hand, breakthrough to the new physics beyond the Standard Model was given by neutrino experiment by the discovery of neutrino oscillations. Observed data taken with various neutrino sources strongly support the existence of neutrino oscillations. The neutrino oscillations are explained as a consequence of leptonic mixing expressed by MNS matrix eq. (1) and non-zero neutrino mass as:
(1) sij = sin θij , cij = cos θij Among three neutrino mixing angles, θ12 and θ23 are measured up to 5% percesion by neutrino oscillation experiments using solar, reactor and atmospheric neutrinos as well as artificial neutrino beam from accelerator, while only the upper limit is given to sin2 2θ13 < 0.15 [1]. The CP violating phase δCP is totally unknown. In order to measure the value of δCP via neutrino oscillation in future experiments, θ13 should be measured not to be zero or too small. T.Konno, M.Ishitsuka, and M.Kuze are with the Department of Physics, Tokyo Institute of Technology, Tokyo, Japan. A.Cabrera is with CNRS/IN2P3-APC Laboratory (Paris), Paris, France Y.Nagasaka is with Hiroshima Institute of Technology, Hiroshima, Japan, CO40238718 Y.Sakamoto is with Tohoku Gakuin University, Sendai, Japan, CO85037950 A.Shigemori is with Niigata University, Niigata, Japan
Fig. 1.
Neutrino oscillation probability for reactor neutrino (E = 4MeV)
The survival provability of anti-electron neutrino from a reactor as a function of the distance from the neutrino source is shown as Fig 1 assuming sin2 2θ13 = 0.1. If the distance from the neutrino sources (reactor cores) is less than 10km, θ13 is the dominant parameter and described as in eq. (2). P (¯ νe → ν¯e ) = 1 − sin2 2θ13 sin2
„
∆m213 L 4E
«
+ O(10−3 )
(2)
The Double Chooz experiment is a new reactor-neutrino experiment. A reactor experiment is thought as one of the most effective approaches to measure neutrino mixing parameters. The Double Chooz experiment, which aims to measure θ13 with a better sensitivity than the current best limit is currently (as of November 2009) constructing the detector at Chooz nuclear power plant in France (Fig. 2). The Double Chooz experiment will construct two identical detectors with different baselines from the reactor cores. Systematic uncertainties associated with the neutrino flux and detection efficiency are largely canceled by comparing the measurements from two detectors. The neutrino targets in the both detectors which are gadolinium loaded liquid scintillator filled in acrylic vessels. They have identical shape to reduce systematic errors and are placed in a different distance from reactor cores. The distance
of the near one is about 400 m from the reactor core, where effect of neutrino oscillation is less effective, and 1.05km for the far detector.
Fig. 2.
inner regions by the steel tank, and worked as cosmic ray veto counter as well as shield to environmental gammas and fast neutrons from the rock surrounding the detector. Outside the inner veto, a 1.5 m thick low activity steel shielding protects the detector from the natural radioactivity of the rock. In addition, there is also an active tracking outer-veto to identify cosmic ray muons using several layers of plastic-scintillator strips on the top of the main detector.
The experiment site (Chooz reactor)
A. Detection technique The detection of anti neutrinos is via inverse-beta decay reaction, which gives a powerful identification method known as, ’delayed coincidence’. In this approach (Fig.3, a neutrino is detected as two correlated signals; the prompt signal is the positron followed by the delayed signal from neutron captured on gadolinium (or hydrogen) nucleon. This delayed coincidence method significantly reduces accidental backgrounds mainly from natural radioactivity. Double Chooz employs gadolinium loaded liquid scintillator as the neutrino target since the thermal neutron capture crosssection of gadolinium is very high.
Fig. 3.
Fig. 4.
The neutrino detector for Double Chooz experiment.
Neutron-Gd capture can only happen in the target, while the gamma catcher regions is used to detect total 8 MeV gamma rays due to neutron capture on Gd. Long-term stability of liquid scintillator is necessary to enable the experiment to run for several years without degradation of light yield. C. Readout system The DC detector is divided into three, optically independent, sub-detectors: Inner-Detector (ID; consists of target, gammacatcher, and buffer volumes), the inner-veto (IV) and the outerveto (OV).
Neutrino detection (delayed coincidence)
B. Detector structure
Fig. 5.
The data flow of DC readout systems
The Double Chooz detector consists of four concentric cylindrical tanks. Fig.4 shows the four volumes of the detector: target (Gadolinium doped liquid scintillator filled in acrylic tank), gamma-catcher (liquid scintillator inside acrylics), buffer (non-scintillating oil) and inner-veto (liquid scintillator). The inner veto is optically separated from the
The readout of the ID and IV is based on large PMTs and waveform-FADC based electronics. ID and IV PMTs are the Hamamatsu [2] R7081MODASSY (10-inch, 390 per detector, Fig.6, left) and R1408 (8-inch, 78 per detector), respectively. The ID and IV PMTs are run at a 107 gain delivering single
photo-electron pulses of about 10 mV. The DC custom made ”Front-End” electronics (FEE) has been developed to fit the dynamic range between output of PMTs and input of FADC, performing pre-amplification (single-PE about 50 mV), large signal clipping, baseline stabilization and noise reduction. The FEE delivers signals to two FADC electronics systems. One of the FADC system, named ’Nu-DAQ’ since it is optimized for the neutrino interaction dynamic range: single photo electron to 15MeV. The FADC cards are CAEN-V1721 [3] (Fig.6, right), sampling at 500 MHz and holding a buffer of up to 4µs per trigger per channel, co-developed by CAEN and the APC laboratory (Paris). The systems are controlled by Emerson [4] VMEbus CPU board, MVME 3100. Fig. 7.
Overview of DC data monitoring
III. O NLINE MONITORING
Fig. 6.
Hardware devices for the DC main DAQ
Left : Low-BG 10inch PMT R7081MOD-ASSY (Hamamatsu) Right : Flash ADC V1721 (CAEN)
Another FADC system (sampling at 125 MHz, also DC custom, covered 1/3 of ID PMTs) is named ’Mu-DAQ’ because its purpose is to read out large energy depositions in the detector (higher than 40 MeV), tagging high energy muons. In addition, The OV readout system (’OV-DAQ’) uses the Hamamatsu M64 multi-pixel-PMTs and measures the charges and timing from the nearby passing muons not hitting directly the detector, which is effiently monitored by the IV PMTs). D. Online systems All online systems for the Double Chooz experiment are controlled remotely because control room for shifters are placed away from the detector site. Nu-DAQ, Mu-DAQ and OV-DAQ, working independently, are synchronized and controlled by Run-Control systems. Each DAQ has an independent system of online-monitoring to allow shifters to assess whether the DAQs are running correctly. Subsequently, the Event Builder is responsible for merging all data streams into one event (using clock information mainly) and converting the data format for offline analysis. Each event goes through a higher level of monitoring stage, where not only viewing the data acquisition process is running properly, but a higher degree of diagnosis and reconstruction can be performed using the offline developments. This stage of powerful monitoring is called ’Pseudo-Online Monitor’. In addition, the Slow Control system controls systematic effects that could impact the experiment, to allow automated scans of parameters such as tempereture and high voltages, and to provide alarms to shifters.
SYSTEM
A general software framework for monitoring all DAQ systems of the Double Chooz experiment (Fig. 7) is developed in order to send various types of information (see Table I) from all monitors to shifters viewers in the same way. There are three types of actors; online monitors, Monitor Server, and viewer and they are connected by TCP sockets. This framework allows online monitor developers transportation and visualization of data. The data sources think about only what data to be sent and how often the data to be updated. On the other hand, online monitor users, usually shift persons, can easily see plots of the data through the internet. TABLE I C ATEGORIES OF DARA SOURCES FOR MONITORING Source types Three DAQs Event Builder Slow Control
Monitoring items Status of DAQs High level data quality checks using offline analysis codes Stabilities of high voltage supply or temperatures etc
Refresh rate 1 sec / refresh 10 sec / refresh 2 mins / refresh
An online monitor transfers a collection of histograms and graphs, named ”histogram package”. All information, even configuration of GUIs, is contained in the histogram packages and no other information sources, such as no configuration files, are needed for viewers. In addition, the histogram package also has its name and an update serial number, useful for identification and checking data updates. As an environment of the system development, a C++ library, handling histograms and inter process communication, has been developed. No ROOT [5] dependence was envisaged because Monitor Server will communicate with some processes based on other programming languages (Monitor Viewer is based on Java). Currently, the library can handle 1-D, 2-D histograms and 1-D graphs and all online monitors of Double Chooz have adapted to use this library. A. Monitor Server Monitor Server (the core of the frame work) is a system consisting of two processes connecting with each other by a shared memory, named ’Monitor Skeleton’ and ’Monitor
with other monitoring processes. Commands in the protocol are listed below: • Configure the package • Update the package • Reset the package • Close the connection C. Performance of data transferring Fig. 8.
The detail structure of Monitor Server
Server’ (Fig.8). The shared memory works as a buffer to absorb differences of performance between DAQ machines and shifters computers. And it also helps to avoid that DAQ processes stop due to troubles from shifters viewers even if the small Monitor Server goes down. Both processes have a TCP socket and create one thread every time they accept a connection. When Monitor Skeleton receives a new package, it allocates a region in the shared memory including a mutex and a condition variable for exclusion control to store the package A thread in the small ’Monitor Server’ checks update serial numbers of all packages in the shared memory, and it sends the updated packages to viewers. All updated packages are sent through one connection and viewers identify what package is coming by checking the package names. B. Data transition and Communication protocol Data passing of the online monitoring is in semi-’data push’ style. It means that online monitors have triggers to start updating data. The online framework does not support real data-push-style since systems are connected by TCP and it is difficult for senders to notify receivers of updating data by such as interrupts. In order to make data passing system similar to data-push-style, the online system translates data from online monitors to viewers in the following order (Fig.9) : 1) A thread in Monitor Skeleton always waits for data coming from Skeletons socket. 2) The mutex forbids reader threads to access the shared memory until the writer thread finishes writing and releases the lock. 3) A thread in Monitor Server sends only updated packages 4) Monitor viewer has a thread always waiting for data coming.
Fig. 9.
The Double Chooz online monitoring framework has been tested for the performance of data transferring (see Table II, summary of results). This test has three steps; writing seed in the shared memory, that in socket communication, and total performance from online monitors to monitor viewer. During the test, viewer is single and read all data from online monitors. The data size for the test is about 4MB per online monitor. From the result of the test, we have confirmed the system has enough performance for the experiment. TABLE II P RELIMINARY RESULTS OF THE PERFORMANCE TEST Speed [MB/sec/package] in shared memory in socket Total
1 monitor 2667 8.70 8.58
10 monitors 225 3.54 2.59
IV. D EVELOPMENT OF GUI S The monitor viewers for Double Chooz experiment are aimed to be able to check data qualities from wherever a computer and internet connections available. To realize this concept, platform independent technologies with some functions to get data from Monitor Sever through the network and to make plots of histograms and graphs, is necessary. Two types of technologies for graphical user interfaces (GUIs) are suitable; Java and web browsers. They are very powerful because both technologies have functions to communicate with Monitor Server through the internet and to draw graphics by themselves on viewer windows with general computer environments.
Data passing flow in DC monitoring system Fig. 10.
A communication protocol is also defined for the monitoring framework. This protocol is designed to be independent of programming languages so all systems can easily communicate
Data stream for monitor viewers
The data stream of the viewer side is summarized Fig.10. Java based viewers can handle TCP binary stream so that they
get monitoring data from Monitor Server directly. On the other hand, web browser based viewers cannot communicate directly since JavaScript is usually difficult to deal with binary data. Therefore, a process on the web server machine reads binary data instead of web browsers and converts it to text files. The web viewers read the text files and make plots with JavaScript technologies. For the developments, source codes can be shared between both types because of ’Google web toolkits (GWT)’, a new Java based development environments to compile Java codes into web pages with JavaScript [6]. Then core classes such as histograms, abstract classes for I/O and graphics, are common (see Fig.11). Only concrete I/O and graphic functions are needed to be implemented for each system. Sharing the common core classes helps to reduce the efforts not only to make similar classes but also to maintain the system in future. It also makes the layout of viewers same between both types.
Fig. 11.
Class relations for viewer GUIs
A. Java based viewer
Fig. 12.
Screen shots for Java based viewer
to draw vectored pictures on web pages without any picture files [10]. GWT enables to develop web applications by preparing only Java codes. Developers do not need to write JavaScript but the application works as web pages with JavaScript after compiling. It means that the same codes of Java version can be reused for the web browser version. GWT also prepares graphical widgets on the web pages, such as panels, buttons, and tabs. Concrete classes for I/O and graphics have been developed, based on Ajax and Canvas for web browser version of viewers. The performance of web browser version is not as fast as the Java based version but this type is much easier to see histograms since it is really not necessary to use any applications except web browsers. Many browsers, Internet explorer, Firefox, Safari Opera and Google Chrome have been tested. Screenshots of web browser based viewer are shows in Fig. 13.
The Java version of viewers is based on usual Java environments, supported by many operating systems (Screenshots shows in Fig.12). This type of viewer can get binary data by using TCP socket and make visible plots of histograms and graphs on the windows by AWT (Abstrace Window Toolkits) package [7], which supports two-dimension graphics on GUI components. So, concrete classes for Java version are implemented with socket binary stream and AWT APIs. The Java based viewer is not only more powerful than the web browser version and can update at higher refresh rates than 1Hz but also easy to access data because the viewer applications are available from web by ’Java web start’ framework [8]. It has tested to work on Windows, Mac OS, and Linux. B. browser based viewer There are several technologies to realize desktop-like applications on web browsers without any additional software. Ajax (Asynchronous JavaScript + XML) and Canvas are typical examples for these technologies based on JavaScript. Ajax supports non-refresh and background communications with web servers [9]. Canvas is a new HTML standard (HTML5)
Fig. 13.
Screen shots for web browser based viewer
V. FAULT MONITORING
SYSTEM
The fault monitoring system in Double Chooz experiment, which has very important roles in remote process controlling, consists of two schemes. One is a ’Gaibu system’ named
after ’external system’ in Japanese, and the other is ’Log Message system’. The role of these systems is accumulating all information corresponding to statuses of online processes via networks. The accumulated information is passed to each shifter’s GUI based on Java. It enables to watch the system status easily from outside the experimental site.
Thus, Gaibu system enables even non-expert shifters to find a reason and a recovering way of a fault as sooner as they can. B. Log Message system
A. Gaibu system The online systems in Double Chooz are designed as network distributed systems. Usually, a scheme of fault detection is the most dedicate part in such a distributed system. Gaibu system is the main fault monitoring system of Double Chooz and has a role to deliver notifications from each online system to the shifters. Gaibu system is working independently of all other online systems in Double Chooz and it looks as if Gaibu system is standing outside of these systems. Therefore, the name of ’Gaibu’, a Japanese word of ’external’, is given. When a fault happens in a certain online system during the data taking, usually the system itself knows a reason of the fault and how to solve the problem in the most cases. For examples, breaking of a network connection, occurring a timeout in communication, no high voltage power supply for a trigger system and so on. Gaibu system provides a common functionality telling the fault information to shifters for the all online systems in Double Chooz.
Fig. 14.
Fig. 15.
Message flow of Log message system
Log Message System bears a responsibility to manage all log messages generated in online processes, an important role in the fault monitoring mechanism. The overview of the system is shown in Fig.16. Log Message System uses syslogd, which is a well-known daemon process on a various UNIX clone OS by its high reliability and stability , plays a role of message accumulator. A syslogd receives local messages from online processes via a system call, and then passes the messages to a remote syslogd as a destination. Because the syslogd can also pass these messages to another process via a FIFO (a named pipe), Log Message Server, which is the main part of the system, can receive all message in real time. A data processing in LMS server is very similar to Gaibu server, and messages passed by a syslogd is sent to all shifters’ GUIs by using TCP sockets.
Structure of Gaibu system
The overview of Gaibu server, a main process in Gaibu system, is shown in Fig.14. The server contains two kinds of threads for an online processes and shifters viewer, and these threads are designed as C++ classes and contains TCP sockets to deal with messages. When a certain online process tries to connects, a receiver thread is yielded in the server by a main thread. The receiver thread receives messages passed by the online process and put it into a queue in the deliverer thread, which is single thread. Thus, all messages generated in Double Chooz online systems are handled with keeping time order sequence. On the other hand, the deliverer thread gets the message from the queue and tosses to all viewers sequentially. Then, on the viewer side, a new popup window is appeared on a shifters’ computer. The viewers GUI can change its behaviors depending on the message priority.
Fig. 16.
Screen shots of Log message viewer GUI
The viewer of Log Message System stores and shows messages sequentially on its window. However, a large number of messages shown in the window are too hard for shifters to understand what is going on remote online systems. Therefore, the viewer has a message filter to reduce an amount of messages based on a message priority and it enables for shifter to get only important messages.
VI. C ONCLUSION Neutrino oscillation is one of the potential paths to open new physics beyond the Standard Model, Measurement of θ13 is essential to search for CP violation in neutrino sector in future experiments. The Double Chooz experiment is a new neutrino reactor experiment to achieve precise measurement of θ13 using two identical detectors with different base lengths. The far detector of the Double Chooz experiment is currently under construction at Chooz reactor site. DAQ systems are controlled apart from the experimental sites so that monitoring systems for status and faults of online systems are critically important for stable data taking. A new monitoring framework has been developed to allow monitoring data through the internet, handling various types of information from all DAQ systems in the same way. Monitor viewers GUI are based on both Java and web browser to be available far from the experimental site. The monitoring framework can integrate efforts for transportation and visualization of data. For fault monitoring, ’Gaibu’ system and Log message system handle messages about status of online processes as a whole. GUIs for fault monitoring are also based on Java and available from wherever the internet connections are prepared. Currently, the developments of the monitoring systems are almost finished and Double Chooz online developers are preparing for DAQs and online software commissioning, starting at the beginning of 2010. R EFERENCES [1] [2] [3] [4] [5] [6] [7] [8]
M. Apollonio et al. hep-ex/0301017 Hamamatsu Photonics Corporation: http://www.hamamatsu.com/ CAEN Corporation : http://www.caen.it Emerson Corporation : http://www.emerson.com/ ROOT : http://root.cern.ch/ Google web toolkit : http://code.google.com/intl/en/webtoolkit/ Java AWT package: http://java.sun.com/products/jdk/awt/ Sun Java web start : http://java.sun.com/javase/technologies/desktop/javawebstart/ [9] OpenAjax : http://www.openajax.org/ [10] HTML5 : http://www.w3.org/TR/html5/
