REDACLE: A flexible database for traceability and ...

ARTICLE IN PRESS JID:COMPHY

AID:3092 /FLA

COMPHY:3092

[m5+; v 1.59; Prn:3/04/2006; 15:42] P.1 (1-7)

Computer Physics Communications ••• (••••) •••–••• www.elsevier.com/locate/cpc

REDACLE: A flexible database for traceability and workflow management for detector construction Luciano M. Barone a , Simone Cau c , Francesca Cavallari b , Silvia Costantini a , Ioan Dafinei b , Marcella Diemoz b , Remo Moro c , Giovanni Organtini a,∗ , Riccardo Paramatti a , Fabio Pellegrino b a Dipartimento di Fisica Università di Roma “La Sapienza” & INFN Sezione di Roma, Roma, Italy b INFN Sezione di Roma, Roma, Italy c Dipartimento di Scienze dell’Informazione Università di Roma “La Sapienza”, Roma, Italy

Received 21 October 2005; accepted 25 February 2006

Abstract The REDACLE Project was initiated as a simple, flexible and fast workflow management system to assist the construction of a high energy physics detector: CMS. Its database structure has been designed to be flexible enough to manage the construction of virtually any kind of product. In fact it can be used for a very different business process. One of the key element of the project was the complete decoupling between the database structure and the workflow process software as well as the separation between objects and their description.  2006 Elsevier B.V. All rights reserved. PACS: 89.80; 07.05; 29.40 Keywords: Workflow Management; Traceability; Database

1. Introduction CMS (Compact Muon Solenoid) is a high energy physics experiment [1] currently being assembled for the operations of LHC [2] (Large Hadron Collider) at CERN (Geneva, Switzerland). LHC experiments are huge: CMS (Fig. 1) has approximately the shape of a cylinder with a 15 m base diameter and 21.5 m length, for a total weight of about 12 500 t. It is composed of thousands of parts assembled together and the signals provided by them are read through millions of electronic channels. One of its subdetectors, the electromagnetic calorimeter [3] (ECAL), devoted to the measurement of electrons and photons, is partly being assembled in Rome. The construction of the detector, in fact, is a distributed process worldwide for which a high degree of coordination is required between construction sites (Regional Centres). Moreover, the data collected on parts during quality control measurements, regularly per* Corresponding author. Tel.: +39 06 49914329.

E-mail address: [email protected] (G. Organtini). 0010-4655/$ – see front matter  2006 Elsevier B.V. All rights reserved. doi:10.1016/j.cpc.2006.02.003

formed at each stage, have to be stored permanently to be used in the future for the detector calibration and analysis. Workflow Management Systems are widely used in the fields of business and manufacturing, and is becoming customary to adopt them for the management of the construction of particle physics detectors, the reason being the complexity of the task resembling very much an industrial production. 2. Detector description and assembly The electromagnetic calorimeter is composed of 75 848 lead tungstate (PbWO4 ) crystals (PWO): a scintillating medium with very short radiation length (X0 = 0.89 cm), providing compactness and very good resistance to the high radiation LHC environment. CMS crystals are 23 cm long and have a slightly tapered shape. Their cross section is approximately 2 × 2 cm2 . They will be assembled (Fig. 2) in such a way they will form a 6 m long cylindrical structure (called the barrel) closed at the two ends by circular shaped detectors (endcaps) whose radius is 1.75 m. Each crystal is equipped with a photodetector (capsule) glued on one of its small faces to measure the emitted

ARTICLE IN PRESS JID:COMPHY 2

AID:3092 /FLA

COMPHY:3092

[m5+; v 1.59; Prn:3/04/2006; 15:42] P.2 (1-7) L.M. Barone et al. / Computer Physics Communications ••• (••••) •••–•••

Fig. 1. An isometric exploded view of the CMS experiment.

Fig. 2. An isometric view of part of the CMS electromagnetic calorimeter.

light and the crystal–capsule pair is called a subunit. In the barrel region subunits are grouped ten by ten in glass fiber alveolar structures called submodules (Fig. 3). Groups of submodules are then assembled over an aluminum structure (called the grid) to form modules (Fig. 4) of 40 or 50 submodules, depending on their position within the detector. Four modules (1700 crystals) are mounted together to form a supermodule, 36 of which, mounted side by side, will form the entire calorimeter barrel. In the end-caps crystals are assembled into glass fiber structures called supercrystals that, in turns, are mounted over four D-shaped supporting structure, shown in Fig. 5. The construction process consists in receiving parts (crystals, capsules, alveola, etc.) from providers, verifying their qual-

ity according to CMS specifications, and assembling them in super-parts. Each activity must be done by operators in the right order and most of them are mandatory. It is required to keep track of each activity made on parts. Quality control is required to ensure the detector quality. To this aim, a set of measurements is done on every crystal by an automatic machine [4] (ACCOR) to precisely determine its size, its transmission properties and its light yield. Subunits are subject to the measurement of some properties of the photo-detectors too, in order to early detect defects in the glue. In order to assist this process (both for human operators and automatic instruments) we designed and realized the REDACLE system (Relational Ecal Database for Construction LEvel). In fact REDACLE was

ARTICLE IN PRESS JID:COMPHY

AID:3092 /FLA

COMPHY:3092

[m5+; v 1.59; Prn:3/04/2006; 15:42] P.3 (1-7) L.M. Barone et al. / Computer Physics Communications ••• (••••) •••–••• 3

Fig. 5. The two D-shaped structures forming one end-cap.

Fig. 3. Barrel crystals are mounted into a glass fiber alveolar structure to form submodules.

Fig. 4. Electromagnetic calorimeter modules assembly scheme.

meant as a replacement for the previously used product, called CRISTAL [5], to improve for speed of operation, flexibility and scalability. REDACLE is composed of both the database used to store and retrieve the data collected during the assembly, and the sequence of activities performed on parts as well as by several software components allowing the realization of the workflow management. 3. The REDACLE design strategy The REDACLE project started in late January 2003. From the experience gained after about 1 year in the construction process, the basic requirements for REDACLE were established to be the following: 1. its development should be very fast (no more than 3 months);

2. its implementation must include Open Source software only (the project is currently fully supported on Linux. All the software components, however, exist for Windows too, but the installation scripts and few maintenance tools); 3. it has to be as simple as possible, keeping the flexibility required to introduce new products and new activities during the construction process; 4. it must be fast for the construction operations as well as for queries; 5. it only stores data (the outcomes of the activities) and metadata (the description of both activities and products) leaving the control of the composition and of the workflow to the interface software. Here we want to stress that this decision was taken both to keep simple the design of the database and to apply the functional decomposition methodology [6] that keeps the software as modular as possible; 6. it has to be open to a large variety of access methods and languages; 7. it must be able to handle existing data. By design we do not support user authentication and GUI tools for data analysis. However, thanks to its open nature and to the high degree of decomposition, virtually any tool can be developed by users, if needed, using their preferred language. In our case the REDACLE database is accessed through the following methods: 1. web interfaces with PHP scripts for the construction process; 2. a C++ program for the interaction with instruments; 3. Java applications for computing activities; 4. Perl scripts for queries and information import/export. The ability to interact with the database using many technologies makes easier the deployment of the product to different Regional Centres, profiting of the local know-how.

ARTICLE IN PRESS JID:COMPHY 4

COMPHY:3092

AID:3092 /FLA

[m5+; v 1.59; Prn:3/04/2006; 15:42] P.4 (1-7) L.M. Barone et al. / Computer Physics Communications ••• (••••) •••–•••

4. The REDACLE tables

on other platforms, since all the employed technologies exist for them. As can be seen in Fig. 6, all tables have an identifier field. Identifiers are used to manage relationships, so they are defined as auto-incrementing integer numbers, with the exception of the part identifier, which is a unique string that is assigned from outside to any single part involved in the construction (in our case a 14 digit barcode attached to each part). A part is any single sub-product that will be used during the realization of the final product, which, in turn, is itself a part. A part may be composed of other parts and must be uniquely

To model the database structure we used the UML language [7], since we recognized it as a very powerful, expressive and flexible tool. In this model tables are represented as classes. Each field in a table is represented as an attribute of the class and relationships as arrows, whose name is the name of the field representing the relationship. The chosen DBMS was MySQL® [8], an Open-Source database freely available for many different platforms, easy to manage, fast and robust. Even if all our development was made on Linux, the product can easily be used

Fig. 6. The UML diagram showing the structure of the REDACLE database tables. In this diagram one-to-one relationships are represented as arrows, while many-to-one relationships are represented as lines starting with a diamond. Fields in tables are represented as attributes.

ARTICLE IN PRESS JID:COMPHY

AID:3092 /FLA

COMPHY:3092

[m5+; v 1.59; Prn:3/04/2006; 15:42] P.5 (1-7) L.M. Barone et al. / Computer Physics Communications ••• (••••) •••–••• 5

identified. As an example, for the ECAL construction, crystals, photo-detectors, subunits, sub-modules, modules, grids, etc. are parts. Some of them (e.g., sub-modules) are composed. Parts belong to a Regional Centre and have a status (a NULL status is the default status and means that the part is in the construction process). Many parts may share the same Part Definition: the description of a part contains its name, sub-name and type. The use of this triplet makes it possible to define generic classes of parts with different degrees of detail: for example, we define crystals as parts whose name is “crystal”, that may have two different sub-names (“barrel” and “end-cap” according to the sub-detector to which they will belong) and different types according to their shape (e.g., “6L”, “8R”, etc.). Each part is also related to many activities. Activities are defined as any atomic operation made on a part whose execution must be stored into the database. For each activity we store the date and the time of the execution and the status. The definition of each activity is kept into a separate table as for part definition. Activities may generate outcomes, called characteristics. A characteristic is just a placeholder for the real data and its definition is kept into a separate table as well. Data generated by an activity and stored as a characteristic have, at the moment, three different kinds of outcome: value, multivalue and charValue. The first is a single numeric field, a multivalue is a collection of three numeric values and charValue is a string field. Ntuples (i.e. collection of values) can be obtained associating more than one outcome to a single characteristic. We do not support other outcome types, that however, can be freely added at any time, if necessary, without perturbing the design of the database. New data types, in fact, result in the addition of a new table in the database, and no changes are required to existing tables. Table 1 shows an example of a single field (the Light Yield of a crystal as measured at 8X0 from its front), a collection of multivalues (the Transversal Transmission of a crystal as measured at 2.5 cm from the front, showing the Transmission—in %— versus the wavelength—in nm) and of a charValue field (the comment of an operator after the visual inspection of a crystal). The workflow is stored into the REDACLE database in a table as a list of all the activity definitions corresponding to the operations that have to be done to complete the construction. The right sequence in which the activities have to be done is computed sorting them according to their label. The database itself, however, does not impose a given sequence (to cope with the decomposition statement): only the management software can do it. It has to be noted too that in this preliminary version the workflow can be just a sequence of actions, some of which can be performed in parallel, but no AND- or OR-splits are supported.1 With time, the workflow definition may change. In this case a new workflow definition can be defined and associated to parts. If the change in the workflow concerns the sequence of the ac1 More complex workflows will be supported by the next versions of the project, for which we are currently designing a new set of tables and software that will allow the description of more complex workflows.

Table 1 Example of entries in REDACLE tables Value id

X

char_id

34

8.21

57

multiValue id

X

Y

z

char_id

45 46 47 48

25 25 25 25

420 440 460 480

20 29 45 58

58 58 58 58

id

Value

char_id

87

Few bubbles

59

charValue

Characteristics id

part_id

act_id

charDef_id

57 58 59

33101000001234 33101000001234 33101000001234

243 364 125

8 9 10

CharDefinition id

Name

Unit

Description

8 9 10

LY TTO VIS INSP

pe/MeV mm#nm#%

Light yield Transv. transmission Operator comment

In this table we show an example of few entries in the REDACLE database tables with some values filled in to make it clear how the data is organized. For each database table we show the name in bold; the names of the fields are given in italic. All the tables have a numeric field called id acting as a primary key for the table. Foreign keys have a name ending with the string “id”. The Value, multiValue and charValue tables refer to the Characteristics table via the char_id field. The characteristics definition of the latter can be obtained from the charDef_id field. In this example the LY of crystal 33101000001234 has the value of 8.21 pe/MeV, its TTO is measured at point 25 mm at 4 different wavelength (420, 440, 460 and 480 nm) and its value varies from 20 to 58%, depending on the wavelength. The Visual Inspection activity lead to the generation of the characteristic whose value is stored into the charValue table as the string “few bubbles”.

tivities, formally new activities must be defined with different labels, in the current version. 5. Use of REDACLE This section is devoted to the illustration of the steps needed to work with REDACLE. Let us define Ak as an activity whose definition Dk has a label such that, in the workflow definition, it represents the kth activity to be performed in the sequence. Any agent (either a human operator or a software interface) wishing to perform some activity whose description is Dk on a given part, looks for the workflow definition associated to it and retrieves the list L = {Di } of the definitions of the activities to be done. Then it retrieves the list L0 = {Aj } of activities already done on the part and gets their definitions L = {Dj }. If Dk ∈ L and Dk ∈ / L and the last activity in L0 , AN , is such that k = N + 1 and has the status “FINISHED/Ok”, then the activity described by Dk can be performed. At the start of the activity a new record is inserted in the activity table giving the start time, the part identifier and

ARTICLE IN PRESS JID:COMPHY 6

AID:3092 /FLA

COMPHY:3092

[m5+; v 1.59; Prn:3/04/2006; 15:42] P.6 (1-7) L.M. Barone et al. / Computer Physics Communications ••• (••••) •••–•••

the activity definition associated to it. A NULL end time defines a pending activity. Once the activity is finished the latter record is updated: the end time field is filled with the proper timestamp and the status is changed into “FINISHED/Ok” or “FINISHED/Bad”, according to the result of the operation. If characteristics are collected during the activity, a new record is inserted into the characteristic table containing the part identifier and the characteristic description. Let I being the identifier of that record. Then, appropriate records are inserted into the suitable table, according to the data type; each record has a field containing the corresponding characteristic identifier I . If new parts are built during the activity (for example, by assembling two existing parts), a new record is inserted into the part table and the component parts are updated in such a way that their superPart_id field contains the identifier of the composite. The REDACLE database can then be used by physicists to check the status of the construction looking in the activity table, or to analyze characteristics. To search for given characteristics, users determine their definition identifier IdDef from the charDefinition table (see Table 1), then selects characteristics whose charDefinition_id is equal to IdDef and part_id is equal to the part identifier. Corresponding data are stored in different tables (called Value, multiValue and charValue), according to the data type, whose char_id equal to the characteristics id. Of course, the queries described above can be inserted into scripts to simplify operator-database interaction. As a matter of fact, in our case, most of the physics analysis is performed via web interfaces for which we have realized PHP scripts that do the queries on the REDACLE database, so that no direct access to the it is needed and no SQL commands has to be composed by users. 6. Interface examples In our applications there are three interface categories: human interfaces, instruments interfaces and computing applications interfaces. For each category we chose the best connection method to the database, demonstrating how different access methods can be used with REDACLE. Human interfaces consist of PHP scripts. Operators locate web pages with any Internet browser and perform activities filling HTML forms with data. Data provided by the operators always contain the identifier(s) of the part(s) involved in the activity. The scripts verify that activities are allowed at the time of execution, by checking the status of the previous activities as given by the workflow stored in the database. During construction, super-parts are composed according to a set of rules depending on the type of parts involved. For example, for the construction of a sub-unit, crystals of a given type can only be coupled with capsules of the same type. The interface performs this kind of test getting the part type from the database and does not allow the activity to continue if their type mismatch. Quality control measurements are taken by different instruments, each one having its own software. Some of them are controlled by C++ programs, others by LabView VIs [9]. Both

can easily connect with REDACLE using a TCP socket and sending and receiving character strings over the network. The dialog between instruments and REDACLE is made possible by a dedicated software agent listening on a given Internet port, providing a direct connection to REDACLE, thanks to the Mysql++ [10] library. Data collected by instruments are sent to that software agent as XML-formatted strings, who interpret it and generates characteristics filling the REDACLE tables. After the measurements, we compute new characteristics from the data and we apply a quality control on them. This step is made with a Java application that, in turn, retrieves data measured by instruments from the database, fits them with predefined functions and computes new characteristics. Then, it checks that computed values are within the allowed range defined by the ECAL-CMS Collaboration. The dialog with REDACLE is made, in this case, through a direct connection via the Java JDBC [11] module. 7. CRISTAL data import/export Data produced in Regional Centres have to be exchanged between them as soon as parts are shipped from one Centre to another. The REDACLE software then, should be able to import data produced by other Regional Centres and export data for them. Moreover, data produced in the past must be included into the REDACLE database for consistency. It was agreed by the Regional Centres representatives that data have to be exported from one Centre to another as XML formatted files. An import/export data interface has been envisaged to that purpose: this interface is available to the Centre Supervisor as a set of scripts that allows data to be extracted from the REDACLE database and formatted as XML files following the CRISTAL specifications and/or import data using the same format. A prototype of it was developed to insert in the REDACLE database the data collected for crystals during the construction of the first calorimeter module built in Rome. The data amounts to about 18 MB for more than 1500 parts and superparts, 7743 activities, 61 323 characteristics, 481 338 multiValues, 26 377 values and 813 charValues. The total elapsed time to populate the database was about 11 minutes, including all the overhead coming from the negotiation of the connection over the network. The CPU time was estimated to be about 25% of the elapsed one. We also measured the performance of the system during queries. To extract data for the Transversal Transmission for one single crystal it took 0.64 seconds elapsed time. The operation involved queries on the charDefinition, characteristic and multiValue tables. Again this time includes all overheads. The total size of the database for the complete detector is expected to be of about 5–6 GB. Since data is distributed over several files the file size limit will never be reached: however, in order to make the administration simpler, we envisaged the possibility to archive data for each module in different physical MySQL databases.

ARTICLE IN PRESS JID:COMPHY

AID:3092 /FLA

COMPHY:3092

[m5+; v 1.59; Prn:3/04/2006; 15:42] P.7 (1-7) L.M. Barone et al. / Computer Physics Communications ••• (••••) •••–••• 7

8. Generalization to other activities Keeping the separation between products, activities and their description, the REDACLE system is able to drive the realization of virtually any kind of product (from food to vehicles, etc.). The only requirement being that each part belonging to the construction process, whose characteristics have to be stored, must be uniquely identified. The identifier itself, being a generic string, can be composed in a number of different ways. In fact, ignoring the construction related fields in the table (such as the superpart identifier, the relative location, etc.), the system is much more generic and can be used for completely different business processes (from mail delivery management to classroom activities management). 9. Future and prospects The current status of the REDACLE database is not yet fully compatible with the widely accepted definition of a workflow management system, given by the Workflow Management Coalition [12]. In particular it does not support the concepts of users and roles. Moreover the activity definitions table is conceived in such a way that workflows can be defined as a sorted list of activities and no complex schemas with AND- or ORsplits and joins can currently be introduced. Our team is currently involved in the definition of a set of tables that will allow REDACLE to be fully compatible with the requirements of a workflow management system. Thanks to the high level of separation of concepts in REDACLE, the modifications will be limited to few tables so that the new version will be backward-compatible with the current one. 10. Conclusion In this paper we described the REDACLE system: a relational database to assist the construction of products. It has been developed as a backup solution to the CRISTAL system for the construction of the CMS e.m. calorimeter, but it turned out to be much more generic and usable to drive the realization of any kind of product.

The REDACLE system is in production in the Rome ECAL Regional Centre and is freely available to anyone willing to use it (both for scientific and commercial purposes), via the popular SourceForge [13] portal or by directly contacting the authors. It performed very well both in terms of performance and resource needs. Its simple and flexible structure allowed us to define new interfaces in few hours as soon as they are needed, so that no direct access to the database is made by operators. The time needed for the development of its first production version was less than 4.2 men months, mostly thanks to the availability of many open source products. References [1] The Compact Muon Solenoid, Technical Proposal, CERN/LHCC 94-38, 1994. [2] LHC Study Group, The Large Hadron Collider: Conceptual Design, CERN/AC/95-05 (LHC), 1995. [3] The Electromagnetic Calorimeter Project—Technical Design Report, CERN/LHCC 97-33, 1997. [4] S. Baccaro, et al., An automatic device for quality control of large-scale crystal’s production, NIM A 459 (2001) 278–284. [5] J.M. Le Goff, et al., CRISTAL: Concurrent Repository & Information System for Tracking Assembly and production Lifecycles—A data capture and production management tool for assembly and construction of the CMS ECAL detector, CERN CMS Note 96/003, 1996. [6] G. Organtini, Software Tools at the Rome-ECAL Regional Centre, in: Proceedings of the CHEP2000 Conference. [7] G. Booch, I. Jacobson, J. Rumbaugh, The Unified Modeling Language User Guide, Addison-Wesley, 1999. [8] MySQL: MySQL is a trademark of MySQL AB in the United States and other countries. Documentation can be obtained on http://www.mysql. com. [9] LabView is a trademark of “National instruments”: see http://www.ni.com. [10] mysql++: a C++ library for connections to the MySQL database server. The library and its documentation can be downloaded from http:// www.mysql.com. [11] JDBC: an API for database connections from java applications. Library and documentation can be downloaded from http://java.sun.com. [12] D. Hollingsworth, The Workflow Reference Model, WfMC Document no. TC00-1003, 1995. Available on http://www.wfmc.org. [13] SourceForge is a popular portal for distributing free software: http:// sourceforge.net/projects/redacle.