Implementation of an operating High Energy Physics Data grid

Implementation of an operating High Energy Physics Data grid B. Abbott,a P. Baringer,b T. Bolton,c Z. Greenwood,d E. Gregores,e H. Kim,f C. Leangsuksan,d D. Meyer,f N. Mondal,g S. Novaes,e B. Quinn,h H. Severini,a P. Skubic,a,* J. Snow,i M. Sosebee,f J. Yuf a

University of Oklahoma, Norman, OK 73019, USA b

c

University of Kansas, Lawrence, KS 66045 , USA

Kansas State University, Manhattan, KS 66506 , USA d

Louisiana Tech University, Ruston, LA 71272, USA

e

Universidade Estadual Paulista, Sao Paulo, Brazil

f

University of Texas at Arlington, Arlington, TX 76019, USA g h

Tata Institute of Fundamental Research, Bombay, India

The University of Mississippi, University, MS 38677, USA i

Langston University, Langston, OK 73050, USA

Abstract Hadron Collider experiments in progress at Fermilab’s Tevatron and under construction at the Large Hadron Collider at CERN will record many petabytes of data in pursuing the goals of understanding nature and searching for the origin of mass. Computing resources required to analyze these data far exceed the capabilities of any one institution. Moreover, the widely scattered geographical distribution of collaborators poses further serious difficulties for optimal use of human and computing resources. The computing grid has long been recognized as a solution to these problems. The success of the grid solution will crucially depend on having high-speed internet connections from the central laboratory to the remote institutions, the ability to use general-purpose computer facilities, and the existence of robust software tools. A consortium of eleven universities in the Southern US, Brazil, Mexico and India are developing a fully realized grid that will test this technology. These institutions are all members of the DØ experiment at the Tevatron and form the DØ Southern Analysis Region, the centerpiece of which is a data and resource hub, a Regional Analysis Center (RAC). Each member institution constructs an Institutional Analysis Center (IAC), which acts as a gateway to the grid for users within that institution. These IACs combine dedicated rack-mounted servers and personal desktop computers into a local physics analysis cluster. We have been working on establishing an operational regional grid, DØSARGrid, using all available resources within it and a home-grown local task manager, McFarm. In this paper, we will describe the architecture of the DØSAR-Grid implementation, the use and functionality of the grid, and the plans for operating the grid for simulation, reprocessing and analysis of data from a currently running HEP experiment.

1.

Introduction

Particle physicists employ high energy particle accelerators and complex detectors to probe ever smaller distance scales in an attempt to understand the nature and origin of the universe. The Tevatron proton-anti proton collider located in the Department of Energy’s Fermi National Accelerator Laboratory [1], Batavia, Illinois, currently operates at the “energy frontier” and has an opportunity of providing new discoveries through the DØ [2] and CDF [3] high energy physics (HEP) experiments. Future experiments, such as *

Corresponding author. Tel.: +1-405-325-3961; fax: +1-405-325-7557; e-mail: [email protected].

1

ATLAS [4] at the Large Hadron Collider (LHC) [5] in the European Organization for Nuclear Research (CERN) [6] and a proposed electron-positron linear collider [7] have the prospects of either building on discoveries made at Fermilab, or making the discoveries themselves if nature has placed these new processes beyond the energy reach of the Tevatron. The TeV energies accessed by these experiments coincide with the predicted scale of electroweak symmetry breaking (EWSB) [8], lending weight to the belief that new breakthroughs are just around the corner. This EWSB process, although yet to be fully understood, is needed to explain the observation that weak vector bosons are very heavy while photons are massless. Elucidating the mechanism for EWSB stands as one of the top priorities in elementary particle physics. Current theoretical models of EWSB, including the Higgs mechanism, various super-symmetry schemes, and the presence of extra spatial dimensions share the common feature of predicting the existence of massive new particles [8]. These objects typically decay into complex finals states and are very rarely produced. Selecting these few complicated interesting events from the enormous volume of data requires a detailed understanding of the detector and physics that can only be quantified by the development of sophisticated analysis algorithms and generation of large sets of detailed Monte Carlo (MC) simulations. Even more exciting would be unexpected discoveries, perhaps connected to problems in cosmology such as dark matter and dark energy [9]. Given the small production probabilities of these interesting events, expediting searches for such processes and other critical analyses will require fast, efficient, and transparent delivery of large data sets, together with an efficient resource management and software distribution system. In addition to problems associated with the magnitude of the data sets, the situation is further complicated by the distribution throughout the world of the collaborating institutions in these experiments. To set the scale, the DØ experiment has 650 collaborators from 78 institutions in 18 countries. The broad geographical distribution of the collaborators, combined with several petabytes of data, poses serious difficulties in data sharing and optimal use of human and computing resources. These problems not only affect high energy physics experiments, but are also shared by other scientific disciplines that generate large volume data sets, such as biology (genomics), meteorology (forecasting), and medical research (3Dimaging). These challenges have been documented by the NSF sponsored GriPhyN initiative [10] and its follow-up iVDGL [11] project, and the DOE sponsored PPDG project [12]. The DOE and NSF efforts have combined to study the computing needs of U.S. physics experiments through the Trillium Collaboration [13]. Ideas and software from these efforts are being incorporated into experiments, such as DØ and ATLAS, to manage and provide access to their petabytes of data. As part of the DØ experiment’s effort to utilize grid technology for the expeditious analysis of data, nine universities have formed a regional virtual organization (VO), the DØ Southern Analysis Region (DØSAR) [14]. The centerpiece of DØSAR is a data and resource hub called a Regional Analysis Center (RAC) [15], constructed at the University of Texas at Arlington with the support of NSF MRI funds. Each DØSAR member institution constructs an Institutional Analysis Center (IAC), which acts as a gateway to other RACs and to the grid for the users within that institution. These IACs combine dedicated rack-mounted servers and personal desktop computers into a local physics analysis cluster. In order to pursue full utilization of grid concepts, we are establishing an operational regional grid called DØSAR-Grid using all available resources, including personal desktop computers and large dedicated computer clusters, in six DØSAR universities. We will initially construct and operate DØSAR-Grid utilizing a framework called SAM-Grid [16] being developed at Fermilab. DØSAR-Grid will subsequently be made interoperable under other grid frameworks such as LCG [17], Teragrid [18], Grid3 [19] and Open Science Grid [20]. Wherever possible, we will exploit existing software and technological advances made by these other grid projects. We plan to develop and implement tools to support easy and efficient user access to the grid and to ensure its robustness. Tools to transfer binary executables and libraries efficiently across the grid for environment-independent execution will be developed. DØSAR will implement the Grid for critical physics data analyses, while at the same time subjecting grid computing concepts to a stress test to its true conceptual level, down to personal desktop computers. Many of the proponents of this project are members of the LHC ATLAS [4] experiment and a future experiment under study by the American Linear Collider Physics Group [21], and will apply the experience gained from DØSAR to these projects.

2

2.

Current Environment

We outline here the current environment that this project will be building upon. We discuss various DØ software packages, the DØ Remote Analysis Model, the virtual organization, and current simulation efforts. SAM. The data access system for DØ offline analyses is managed by a database and cataloging system called Sequential Data Access via Metadata, or SAM [22]. SAM is used in conjunction with the combined software and hardware systems, known as Enstore [23, 24], that provide actual access to the data stored in the tape robot. It is designed to accommodate a distributed computing environment, and the combination of Enstore and SAM is well suited to the grid. SAM provides high-level collective services for reliable data storage and replication through efficient cataloging, recording, and retrieval of data. In service for the past four years, SAM’s strengths and weaknesses have been thoroughly assessed; opportunities to improve upon its features and stability through the actual use of the system in data analyses have been identified. The system also allows for offsite remote access to the data, a critical functionality for a successful computing grid. Localized offsite data access features also have been in use through the past three years, primarily in MC production. SAM-Grid Framework. The SAM-Grid team was formed at Fermilab in 2001 to incorporate existing grid tools like Globus [25] into the SAM system [26,27]. SAM-Grid provides the Tevatron grid framework utilizing the global Job and Information Management system (JIM) [28] supported by the Particle Physics Data Grid (PPDG) project [12] at Fermilab. JIM is designed to provide job submission and management tools and information services in a grid environment utilizing the SAM data management system. Fundamental features of JIM include: 1) remote job submission through Condor-G [29], 2) interfaces built upon Globus, 3) grid job brokering, and 4) a rudimentary web-based monitoring system [30]. The functionality of JIM was demonstrated at both the Super Computing 2002 and 2003 conferences [31]. JIM is currently in its trial implementation stage for practical use in the experiment.

Figure 1 A schematic diagram of the DØ Remote Analysis Model (DØRAM). Central Analysis Center (CAC) in this diagram is located at Fermilab.

DØRAM. In order to prepare the experiment for the transition to grid computing, a DØ Remote Analysis Model (DØRAM) [32] was proposed by the PI of this proposal at about the time of the formation of the SAM-Grid team. Figure 1 shows a schematic diagram of the DØRAM, which employs an architecture that is similar to the tiered scheme of the LHC experiments’ grid computing model for data and resource distribution [17]. Regional Analysis Centers (RACs) in the DØRAM are equivalent to Tier-I sites for LHC experiments. Given the expected network traffic generated by sharing petabytes of data and the limited network resources, it seemed prudent to select a limited number of collaborating institutions to act as RACs in order to minimize network load. RACs then act as data storage and CPU resource hubs for the users within their regions. Each RAC stores a few data sets with different levels of detail (and size) for

3

physics analyses and serves several institutional centers (IACs) within the region. The IACs act as gateways to desktop users at a given institution. This aggressive approach to establish a remote analysis system for DØ was commended at the Fermilab Director’s review by the Bird Committee in June 2002 [33]. Since that review, a proposal [15] to establish RACs for DØ was submitted to the collaboration, whereupon it received a strong endorsement from the DØ leadership. Following these two endorsements, DØ management formed the DØ Off-site Analysis Task Force to define and establish regional analysis centers. European RACs have subsequently been formed at Karlsruhe (GridKa), Germany; Manchester University, United Kingdom; NIKHEF, the Netherlands; and IN2P3, France. The University of Texas-Arlington (UTA) has constructed the first and only DØ RAC in the United States. DØSAR. As the first step of DØRAM implementation in the U.S. beyond a RAC, the DØ Southern Analysis Region was established in April 2003 to pool resources and expertise for the analysis of real data and production of simulated data for the DØ experiment, and to promote remote computing contributions to the experiment. Initially consisting of UTA, Oklahoma University, Langston University, and Louisiana Tech. University, DØSAR has grown to encompass nine universities from six states. It holds semi-annual workshops that provide the opportunity to explore technical issues in depth and to plan for the future direction of the virtual organization. Bi-weekly web-based videoconferences among member institutions help in coordinating activities, exchanging information, and reporting the status of each site. An issue tracker [34] facilitates inter-meeting communication and serves as an archive. DØSAR has been very effective in rapidly mobilizing significant additional computing resources into the production of MC events for DØ, helping to meet a critical need of the experiment. The four original DØSAR institutions have been operating MC production facilities (“farms”), and the remainder will have operational farms by spring 2004. These farms are localized computing clusters that share a common storage and user space with external Internet connections. Some of the farms are comprised of dedicated rack systems while others use linked personal desktop computers [35]. These farms operate as independent entities that communicate to the central site (Fermilab) via SAM to transfer MC requests and output data. DØ Software. The DØ software infrastructure [36] is installed and maintained regularly on each farm using the Fermilab UPS/UPD package management and retrieval system [37]. Each institution has a SAM station [22] for data handling between the institution and the central data repository at Fermilab. The SAM system allows the farms to process official DØ MC requests [38] from the physics analysis groups, with produced files and metadata transferred back to Fermilab for storage. For MC production, the DØ software releases contain four programs that are run serially to produce the simulated data and its reduction for analysis, as well as low-level MC management software that controls the chain of programs and necessary data files. To set the scale, a single event typically takes about 3 minutes to generate on a 2.2 GHz Pentium 4 Xeon CPU. The total number of MC events for DØ analysis through the end of this decade is expected to be over 10 billion events. MC Production and Management. The MC_Runjob package [39] provides a low-level MC manager that coordinates data files used by and produced with the executables of the MC chain, via scripts produced for the job. This package is combined with MC task management software called McFarm [40], developed at UTA, for automated, highly efficient MC production. McFarm provides high-level services such as bookkeeping, job parallelization, data retrieval and storage at the Fermilab central facility, interfacing to the specific cluster batch system, and farm performance and request status monitoring information. The McFarm system is capable of obtaining an official request from the central site, breaking the request into small jobs to better utilize computing resources, communicating with SAM to acquire any necessary input files from Fermilab, submitting the jobs to the local batch system, collecting and storing the output data and metadata at Fermilab via SAM, and monitoring job progress and farm status. The high degree of automation allows efficient management and oversight of an operating farm. MC Production Monitoring. DØSAR utilizes three web accessible applications to monitor the farm clusters in a graphical manner. The Ganglia Cluster Toolkit [41] is used to provide resource monitoring of all the DØSAR clusters. McFarmGraph [42] and McPerM [43], developed at UTA, interface to McFarm to provide job status information and the overall farm performance metrics. Both applications use features in Globus to transfer information produced by McFarm at each site to the monitoring server at UTA. Since its creation, DØSAR MC farms have produced and stored several million events, a significant portion of the total DØ MC sample.

4

3.

The Relationship of DØSAR to Other Ongoing Grid Activities

Processing large data sets collected by the DØ experiment and future HEP experiments will require enormous computing resources that only a grid is capable of providing. DØSAR is by no means the first project to employ grid techniques for scientific applications, nor even for applications in high energy physics. Indeed, ATLAS, for example, already has an operational prototype grid [44]. The strengths of this project are its promise to deliver practical grid experience with an operating experiment, and an ability to demonstrate that grid tools tested on DØ have wider applicability by applying them to ATLAS and LC software. More specifically, DØSAR can test grid integration concepts with DØ, an operating high energy physics experiment which has: (1) a large and growing data set that must be analyzed expeditiously; (2) complex simulation tools under continuous development; and (3) real-world data reconstruction issues that involve multiple evolving versions of software packages and challenging database query and retrieval problems. By comparison, the ATLAS grid can now be used only for relatively immature simulation tools, and is not testable on real data because the LHC is still under construction. Significant progress towards a grid-like system for the remote processing of data has recently been reported by the DØ experiment in the Fermilab periodical FermiNews [45]. Several non-U.S. institutions led by the Fermilab team participated in this effort of re-processing over 100 million events, transported through SAM. This achievement by Fermilab and a consortium of European universities was very human resource intensive. It underscores the complexities of implementing a grid solution and the tremendous challenge of expanding the scope to include data analysis. It also establishes a clear direction for future work. DØSAR plans to address the next steps to establish a fully grid-enabled virtual organization operating in a production mode for a running experiment. These steps include establishing a capability for simulation, data reconstruction, and analysis job submissions without specifying the execution site, enhanced monitoring, and greatly improved reliability. It is important to note that the institutions involved in the successful DØ remote reprocessing project were large RACs with computing power in the range of 102-103 CPUs. This is a rather atypical capability for a university; and the true grid cannot be limited to such big facilities. DØSAR has a significant investment in the resources of much smaller institutions. These resources, a mixture of personal desktop workstations and dedicated computer farms, have proven to be valuable to the experiment. The full resource utilization that can be achieved through grid computing is of great importance to the overall computing capacity of the community and, most importantly, in the realization of the complete grid computing concept. Engaging the resources of smaller facilities will allow the further decentralization of global distributed computing tasks, enabling research in institutions that would otherwise be unable to contribute materially, and bringing this research to scientists and students who might otherwise be disenfranchised. DØSAR has advanced to the stage where further resource efficiencies will be gained by bringing the individual, independent farms into one regional grid, the DØSAR Grid, which is based on user desktop computers and other larger computing resources. 4.

Future Plans

We intend to integrate the resources in DØSAR institutions into an operating grid to be utilized in critical data analyses at the DØ experiment, taking grid computing one step further to the realization of the conceptual grid. We will implement a system that aims at robustness and portability. We will develop and implement monitoring capabilities and diagnostics for this system. We will develop smart binary transport tools to dramatically increase the efficiency of the grid and interfaces to simplify use of the grid. These activities will also provide system services that complement and improve existing functionality. In order to complete the implementation of DØSAR Grid, we have submitted a proposal to the ITR program of the National Science Foundation. This project, called PHANTOM (A PHysics Analysis Network at the Tevatron Operation Mode), proposes to establish an operational regional grid among a consortium of six southern U.S. universities. As discussed below, this grid will use all locally available computing resources, including personal desktop computers and large dedicated computer clusters. It will leverage software advancement by other grid projects, and be applied to cutting-edge high energy physics research.

5

4.1

DØSAR and SAM-Grid

SAM serves as the tool by which DØ collaborators access both real and simulated data. A current limitation of SAM is that this tool is not transparent; users must interact with a local SAM station, a software component that must be set up at each DØ analysis site. Local DØ computing is then tied to this particular station. Remote computing is possible, but it involves connecting to a specific off-site SAM station. The SAM-Grid project seeks to remove this need for an explicit user-SAM connection so that a DØ user anywhere can execute software requiring data access via SAM without being tied to a particular local computing system. Since SAM-Grid must service the worldwide DØ community, its development is an extremely complex undertaking. PHANTOM proposes to enable more rapid testing of the proposed features of SAM-Grid within the more manageable environment of DØSAR. In utilizing the SAM-Grid framework, PHANTOM will, as a by-product, help validate tools of use to the larger SAM community, but it aims to do much more. Indeed, to be successful, PHANTOM must expand on its first goal of establishing DØSAR-Grid. It is essential that PHANTOM’s tools have wider applicability and that their implementation be tested. These requirements will be met by adapting PHANTOM tools to simulation tasks for the LHC ATLAS experiment and the Linear Collider whose software is largely based at the Stanford Linear Accelerator Center [46]. These projects, both of which involve several collaborators from PHANTOM, differ substantially from DØ and from one another. 4.2

Robustness, Stability and Monitoring

A major initiative in this project is to study and implement ways to provide greater stability and robustness in operations using the grid. System failures are unavoidable in the information technology field, and their elimination has been the subject of much work. If not resolved in a timely fashion, failures can often result in unavailability of service, thereby significantly reducing productivity. We propose to address such unavailability issues while maintaining a high-performance computing environment. The ultimate goal to strive for is zero downtime. A common weakness in many cluster designs is the existence of sources of single-point failures. The idea is illustrated in Fig. 2 from Ref. [47]. In the panel on the right, a single-point failure can occur as a result of the remote site gatekeeper malfunctioning. If this unit is lost, the site is no longer available for job distribution. The chance of single-point failures can be greatly reduced by including multiple-head nodes in each site. For PHANTOM, this entails augmenting SAM stations with a second node at each site that will transparently assume station functions in case of the loss of the primary node. Similarly the present McFarm design will be extended to include a back-up head node that will automatically assume control in case of loss of the primary head node. In both examples described above, reliability is improved without changing or interfering with grid resource management. Performance can be sustained in the entire grid by providing each local site with transparently self-healing and redundancy mechanisms, e.g., the implementation of multiple head gatekeepers and other important services. Much research in this area that has been performed on non-grid systems [48] can be applied in a similar fashion to PHANTOM. A particularly attractive idea is that of Leangsuksun and his collaborators on HA-OSCAR (High Availability-Open Source Cluster Application Resources) [49]. A dual-head architecture has been created that includes reliable communications with a heartbeat mechanism to maintain cluster health. Periodic transmission of heartbeat messages traverse across a dedicated serial link, as well as an IP channel, and monitor the health of the working server. When a failure of the primary server occurs, the system can automatically transfer the control of services to the failover server, allowing services to remain available with minimal interruption.

6

Figure 2 A schematic diagram of grid communication between various software components. A single point failure in the Globus gatekeeper, shown in the diagram on the right, can disable the entire remote site.

Figure 3 A schematic diagram of HA-OSCAR architecture that utilizes automatic fail over head-nodes with self-healing daemons [48].

7

Figure 3 illustrates the HA-OSCAR self-healing architecture. Each head node runs monitoring, selfhealing, and System-Imager daemons. These daemons work together forming adaptive self-healing mechanisms that detect undesirable events and attempt to shield and recover potential site outages, especially at the head nodes. There are three types of daemons monitoring system health, services, and resources, respectively. The health daemon detects hardware, OS and network outages, and performs recovery with a failover server. Several critical services such as grid gatekeeper, GRIS services, local schedulers such as PBS and MAUI, as well as NFS and DHCP, are monitored by the HA-OSCAR service monitoring daemon. Once a service failure occurs, a corresponding service recovery procedure is triggered, smoothly taking over the operations. PHANTOM will apply the high-availability computing techniques that have been successively implemented in the OSCAR program and elsewhere to DØSAR. This work will involve the creation of monitoring and diagnostic tools featuring daemons which function in a similar manner to those described above. We envision that these features will initially be implemented at the local IAC level and then later applied to groups of IACs within DØSAR. 4.3

Data Management and Proxy Utilities

The DØ experiment uses very large programs for data processing and analyses; for example, the experiment’s software libraries and other executables occupy more than seven gigabytes [50] of storage space. As understanding of the detector is improved and more sophisticated data analysis algorithms are developed, the size of executables will only increase. Large program size requires significant time to transfer updates over networks to remote analysis sites, which must be done frequently in an evolving experiment. This utilizes large amounts of network bandwidth and time. Network bandwidths at many remote analysis centers improve at a much slower pace than the overall network, and also typically suffer from restrictive network policies. These issues pose difficulties in keeping the data and libraries up-to-date at remote sites. Problems will be exacerbated in the ultimate implementation of grid since computing tasks must be performed in a self-contained and self-sufficient manner in foreign environments that do not have prior knowledge of particular software executables. This necessitates the need to reduce and manage the transfer of large repetitive libraries. One of the key requirements in resolving these issues is efficient transport of software and other binary data. We thus propose to build an effective data management system, which transports only the minimal required software needed to perform a given computing task. We will investigate various existing compression [51] and string-matching techniques [52] to identify the most optimal techniques. We intend to analyze these existing software components and, if necessary, develop new ones to complement them in order to handle the transport of software executables in the DØSAR Grid environment. We will exploit the RAC at UTA and the DØRAM architecture as a binary management hierarchy where software libraries may be temporarily or semi-permanently cached, and where decisions on what version to transfer to which remote sites within the region will be optimized. We will then develop methods and algorithms for efficient transport of software executables. The software components employed in these methods include binary difference discovery and management, purging and caching, packaging and deployment, file transportation, authentication, validity checks, and network monitoring and utilization. Discovery mechanisms identify available software libraries and executables on the execution site and, subsequently, dispatch the collected information to the code repository, where the appropriate libraries and executables are prepared for deployment. Purging and caching of releases requires the creation of heuristics for efficiency both on the central code server and the remote machines, where computational and network resources are utilized. Old releases are purged, and the most recent and most used libraries are cached. Packaging creates a “patch” of the differences between two releases and applies that patch on the remote execution site to create the software version required for the given task. Algorithms for creating the required packages and reassembling them on the client sites will be developed. Metadata and profile services that handle message communication will be designed and implemented. Actual transportation, including stop-resume partial transfers, of patches in multiple streams completes the process.

8

4.4

Utilities for Ease of User Communication

Distributed and heterogeneous computing on a global scale is a complex task. The exercise of grid technologies to date has been the province of experts. This presents an imposing barrier for the use of these technologies to the application level user who, for example, is interested in the physics discoveries lying within the data, not the mechanisms of data transport, processing power, scheduling, and authorization necessary to accomplish the examination of the data. Moreover, success in using grid computing on a large scale will require more user-friendly interfaces. To accomplish this goal and to overcome the barrier of “experts-only” grid environments for the user community, we propose to develop user-level utilities to enhance and aid the users’ interaction with the grid. We will develop these tools in a modular and universal manner so that the underlying grid layer, in SAM-Grid or any other grid framework, is interchangeable to accommodate the evolving environment. The application user would benefit from an intelligent agent (IA) to aid in decision-making regarding job preparation and submission to maximize efficiency in human and grid resources. The IA would consist of distributed monitoring processes at grid sites that provide real-time information on the status of the systems. This information will be interpreted and presented to the user for perusal and as sets of possible actions. The IA would provide a self-aware virtual organization (VO) level system status in easily understood terms with points of failure, congestion, incompatibilities, and dependencies. As discussed in Section 2, DØSAR already has developed expertise in the application of grid monitoring software. We plan to exploit this experience in the PHANTOM monitoring and communication interface development. We envision the creation of a graphical user interface to serve as a grid portal (GP). The GP will consist of these components: job submission and status, certificate and key management, log file access, electronic notebook, user configurable database, error reporting, extensive online help, and VO “yellow and white pages.” The GP will be portable across operating systems to provide a standard grid user tool environment. The volume and complexity of information characterizing a computing task running on the grid can be large; therefore, analyzing this information to ascertain the cause of failure can be difficult. In the case of SAM-Grid, information concerning a job is returned to the user via a web interface [30], with a compressed version of archive files of standard output and standard error for each sub-task and some additional files. We will develop log analysis engines to facilitate user access and understanding of job logs, reporting and diagnosing of problems, and promulgating suggested solutions to the user. These utilities will be deployed initially in the DØSAR Grid, but they will be designed and implemented with general use in mind. These products should lower the entry threshold to productive use of distributed and heterogeneous computing.

5.

Conclusions

Historically, experimental particle physics has pushed information technology in directions that have had profound broader impact. The most spectacular example is CERN’s development of the World Wide Web, originally intended to facilitate communication among collaborating physicists spread around the globe. DØSAR’s goal of efficiently using computing resources scattered across the south-central United States for tasks such as complex data reduction and simulation will produce grid computing solutions with broad applicability. This project will also likely affect how we disseminate and analyze data from future experiments at facilities such as the LHC, and perhaps fundamentally enhance the collaborative nature of high energy physics and other large international scientific endeavors. DØSAR will help improve the cyber-infrastructure within the region; five participating institutions are located in states that are traditionally under-funded for R&D activities. In addition to technology development, there will also be significant development of human resources in the region. Students at the undergraduate, graduate, and postdoctoral levels will receive interdisciplinary training in physics and computer science. Faculty involved with this proposal will be able to pass on the knowledge gained to a wide spectrum of students in their classrooms. The broader scientific and technical

9

community will be reached by student theses, refereed journal articles, and conference talks resulting from this work. Information for the broader public will be disseminated on the web. Finally, DØSAR will realize the conceptual grid to the level of personal desktop computers, demonstrating the performance of a grid at its fullest level. The DØSAR Grid will be exploited in critical physics analyses of real and simulated data from the DØ experiment, and later from LHC and other future experiments. These data will serve to advance our understanding of nature at the smallest distance scales.

References Cited Fermilab National Accelerator Laboratory, http://www.fnal.gov/. The DØ Experiment, http://www-d0.fnal.gov/. The CDF Experiment, http://www-cdf.fnal.gov/. The ATLAS (A Toroidal LHC ApparatuS) Experiment, http://atlasinfo.cern.ch/Atlas/. The Large Hadron Collider (LHC) project, http://lhc-new-homepage.web.cern.ch/lhc-newhomepage/. 6. European Organization for Nuclear Research (CERN), http://www.cern.ch/. 7. The Linear Collider Research and Development Working Group, http://www.hep.uiuc.edu/LCRD/. 8. For reviews, see The Review of Particle Physics, K. Hagiwara et al., Phys. Rev. D66, 010001 (2002); E.D. Commins and P.H. Buchksbaum, Weak Interactions of Leptons and Quarks, (Cambridge Univ. Press, Cambridge, 1983); J.M. Conrad, M.H. Shaevitz, and T. Bolton, Rev. Mod. Phys 70, 1341 (1998).

1. 2. 3. 4. 5.

9. V. Trimble, Ann. Rev. Astron. Astrophys. 25, 425 (1987); B. Sadoulet, Rev. Mod. Phys. 71, S197 (1999); M.S. Turner and J.A. Tyson, Rev. Mod. Phys. 71, S197 (1999); N.A. Bahcall et al., Science 284, 1481 (1999). 10. The Grid Physics Network (GriPhyN) project, http://www.griphyn.org/ 11. The International Virtual Data Grid Laboratory (iVDGL) project, http://www.ivdgl.org/ 12. The Particle Physics Data Grid (PPDG) project, http://www.ppdg.net/ 13. The Trillium Collaboration, http://www.hicb.org/TrilliumNewsletter/ 14. The DØ Southern Analysis Region (DØSAR), http://www-hep.uta.edu/d0-sar/d0-sar.html 15. I. Bertram, R. Brock, F. Filthaut, L. Lueking, P. Mattig, M. Narain, P. Lebrun, B. Thooris, J. Yu, C. Zeitnjitz, “Proposal for DØ Regional Analysis Centers,” DØ Note #3984, unpublished (2002),

http://www-d0.fnal.gov/computing/D0Race/d0rac-wg/d0rac-final.pdf

16. The DØ Grid Group, http://www-d0.fnal.gov/computing/grid/ 17. The Computing Grid Project, http://lcg.web.cern.ch/LCG/ 18. Wired News, http://www.wired.com/news/technology/0,1282,45977,00.html 19. The Grid 3 project, http://www.ivdgl.org/grid2003/ 20. Open Science Grid, http://www.opensciencegrid.org/ 21. American Linear Collider Group (ALCPG), http://blueox.uoregon.edu/~lc/alcpg/ 22. The Sequential Data Access via Metadata (SAM) project, http://d0db.fnal.gov/sam/ 23. J. Baken, E. Berman, C.H. Huang, A. Moibenko, D. Petravick, R. Rochenmacher, K. Ruthmansdorfer, “Enstore Technical Design Document,” Fermilab-JP0026 (1999). 24. The Enstore project, http://www-isd.fnal.gov/enstore/ 25. The Globus project, http://www.globus.org/; I.T. Foster, and C. Kesselman, “The Globus Toolkit,” pp. 256 – 278 in “The Grid: Blueprint for a Future Computing Infrastructure,” Morgan Kaufmann Publishers, ISBN 1-55860-475-8 (1999). 26. A. Baranovski, G. Garzoglio, L. Lueking, D. Skow, I. Terekhov, and R. Walker, “SAM-GRID: A System Utilizing Grid Middleware and SAM to Enable Full Function Grid Computing,” Beauty 2002, Santiago de Compostela, Spain, June (2002), http://d0db.fnal.gov/sam_talks/talks/20020618beauty/Beauty.ps. 27. The SAM-Grid project, http://www-d0.fnal.gov/computing/grid/; G. Garzoglio, “The SAM-Grid Project,” talk presented at the VIII International Workshop on Advanced Computing and Analysis

10

Techniques in Physics Research (ACAT02), Moscow, Russia, June (2002),

http://acat02.sinp.msu.ru/presentations/garzoglio/ACAT02.ppt

28. The DØ Job and Information Management (JIM), http://samadams.fnal.gov:8080/ 29. The Condor-G project, http://www.cs.wisc.edu/condor/condorg/; J. Frey, T. Tannenbaum, I.T.Foster, M. Livny, and S. Tuecke, “Condor-G: A Computational Management Agent for Multiinsitutional Grids,” Proceedings of the Tenth IEEE Symposium on High Performance Distributed Computing (HPDC), pp. 7-9, San Francisco, CA, August (2001). 30. SamGrid Monitoring Page, http://samgrid.fnal.gov:8080/ 31. Super Computing 2002 conference, Baltimore, Maryland, Nov. (2002),http://www.sc-

conference.org/sc2002

32. J. Yu, “Proposal for A DØ Remote Analysis Model (DØRAM),” http://wwwhep.uta.edu/~d0race/meetings/jan-17-2002/jae-d0ram-011702.pdf, unpublished (2002). 33. I. Bird, E. Elizabeth Buckley-Geer, J. Hobbs, R. Mount, P.

Sinervo, “Run II Computing Review,” (2002), http://www.fnal.gov/cd/reviews/runii2002/ 34. DØSAR Issue Tracker, http://physics.lunet.edu/cgi-bin/roundup.cgi/d0sar/

35. H. Severini and J. Snow, “Preparation of a Desktop Linux Cluster for DØ Monte Carlo Production”, DØ Note #4208, unpublished (2003), http://www-

hep.nhn.ou.edu/d0/grid/docs/DesktopMcFarmHowto.pdf The DØ Code Management System, http://www-d0.fnal.gov/software/cmgt/cmgt.html The UPS/UPD project, http://www.fnal.gov/docs/products/ups/ DØ Run2 Monte Carlo Production, http://www-d0.fnal.gov/computing/mcprod/mcc.html MC_Runjob Home Page, http://www-clued0.fnal.gov/runjob/current/ McFarm Home Page, http://hepfm000.uta.edu/documentation.html The Ganglia project, http://ganglia.sourceforge.net/ A. Nyshanda, S. Jain, et al., “McFarmGraph, a web based monitoring tool for McFarm jobs,” UTA-HEP/Computing-0026, unpublished (2003); McFarm Monitoring Page, http://hepfm000.uta.edu/monitor.html

36. 37. 38. 39. 40. 41. 42.

43. P. Bhamidipati, “McPerM, MC Farm Performance Monitoring System,” UTA-HEP/Computing-0028, unpublished (2004); http://hepfm000.uta.edu/new_m.htm 44. The ATLAS Testbed project, http://atlasgrid.bu.edu/atlasgrid/atlas/testbed/ 45. FermiNews, http://www.fnal.gov/pub/ferminews/ferminews04-02-01/index.html 46. Stanford Linear Accelerator Laboratory, http://www.slac.stanford.edu/ 47. The Condor Project, http://www.cs.wisc.edu/condor 48. C. Leangsuksun, et al, “High-Availability and Performance Clusters: Staging strategy, selfhealing, experiment and dependability analysis”, submitted to The 10th International Conference on Parallel and Distributed Systems July 7-9, 2004, Newport Beach, CA 49. C. Leangsuksun, et al, “Availability Prediction and Modeling of High Availability OSCAR

Cluster”, IEEE International Conference on Cluster Computing, December 1-4 2003, Hong Kong. 50. The DØ Offsite Reprocessing project, http://www-d0.fnal.gov/computing/reprocessing/ 51. Adaptive Huffman Encoding, http://www.nist.gov/dads/HTML/adaptiveHuffman.html; 52. An extremely fast Ziv-Lempel data compression algorithm, Williams, R.N., http://ieeexplore.ieee.org/xpl/abs_free.jsp?arNumber=213344.

11