OSG–doc–965 June 24, 2010 www.opensciencegrid.org
Report to the US National Science Foundation June 2010
Miron Livny Ruth Pordes Kent Blackburn Paul Avery
University of Wisconsin Fermilab Caltech University of Florida
1
PI, Technical Director Co-PI, Executive Director Co-PI, Council co-Chair Co-PI, Council co-Chair
Table of Contents 1. Executive Summary............................................................................................................... 3 1.1 What is Open Science Grid? .......................................................................................................... 3 1.2 Usage of Open Science Grid .......................................................................................................... 4 1.3 Science enabled by Open Science Grid.......................................................................................... 5 1.4 OSG cyberinfrastructure research .................................................................................................. 6 1.5 Technical achievements in 2009-2010 ........................................................................................... 7 1.6 Challenges facing OSG .................................................................................................................. 9 1.7 Preparing for the Future ............................................................................................................... 10 2. Contributions to Science ..................................................................................................... 12 2.1 ATLAS ......................................................................................................................................... 12 2.2 CMS ............................................................................................................................................. 16 2.3 LIGO ............................................................................................................................................ 19 2.4 ALICE .......................................................................................................................................... 22 2.5 D0 at Tevatron.............................................................................................................................. 22 2.6 CDF at Tevatron........................................................................................................................... 26 2.7 Nuclear physics ............................................................................................................................ 33 2.8 MINOS ......................................................................................................................................... 37 2.9 Astrophysics ................................................................................................................................. 38 2.10 Structural Biology ........................................................................................................................ 38 2.11 Multi-Disciplinary Sciences ......................................................................................................... 42 2.12 Computer Science Research......................................................................................................... 42 3. Development of the OSG Distributed Infrastructure....................................................... 44 3.1 Usage of the OSG Facility ........................................................................................................... 44 3.2 Middleware/Software ................................................................................................................... 46 3.3 Operations .................................................................................................................................... 48 3.4 Integration and Site Coordination ................................................................................................ 49 3.5 Virtual Organizations Group ........................................................................................................ 50 3.6 Engagement and Campus Grids ................................................................................................... 52 3.7 Security......................................................................................................................................... 54 3.8 Metrics and Measurements........................................................................................................... 56 3.9 Extending Science Applications................................................................................................... 57 3.10 Scalability, Reliability, and Usability .......................................................................................... 58 3.11 Workload Management System ................................................................................................... 59 3.12 Condor Collaboration ................................................................................................................... 60 3.13 High Throughput Parallel Computing.......................................................................................... 64 3.14 Internet2 Joint Activities .............................................................................................................. 64 3.15 ESNET Joint Activities ................................................................................................................ 65 4. Training, Outreach and Dissemination ............................................................................. 68 4.1 Training and Content Management.............................................................................................. 68 4.2 Outreach Activities....................................................................................................................... 69 4.3 Internet dissemination .................................................................................................................. 70 5. Participants .......................................................................................................................... 71 5.1 Organizations ............................................................................................................................... 71 5.2 Partnerships and Collaborations................................................................................................... 71 6. Cooperative Agreement Performance ............................................................................... 74 Sections of this report were provided by: the scientific members of the OSG Council, OSG PI-s and Co-PIs, and OSG staff and partners. Paul Avery and Chander Sehgal acted as the editors.
2
1. 1.1
Executive Summary What is Open Science Grid?
Open Science Grid (OSG) aims to transform processing and data intensive science by operating and evolving a cross-domain, self-managed, nationally distributed cyber-infrastructure (Figure 1). OSG’s distributed facility, composed of laboratory, campus and community resources, is designed to meet the current and future needs of scientific Virtual Organizations (VOs) at all scales. It provides a broad range of common services and support, a software platform, and a set of operational principles that organizes and supports users and resources in Virtual Organizations. OSG is jointly funded, until 2011, by the Department of Energy and the National Science Foundation.
Figure 1: Sites in the OSG Facility
OSG does not own any computing or storage resources. Rather, these are contributed by the members of the OSG Consortium and used both by the owning VO and other VOs. OSG resources are summarized in Table 1. Table 1: OSG computing resources Number of Grid interfaced processing resources on the production infrastructure Number of Grid interfaced data storage resources on the production infrastructure Number of Campus Infrastructures interfaced to the OSG Number of National Grids interoperating with the
3
113 65 9 (GridUNESP, Clemson, FermiGrid, Purdue, Wisconsin, Buffalo, Nebraska, Oklahoma, SBGrid) 3 (EGEE, NGDF, TeraGrid)
OSG Number of processing resources on the Integration infrastructure Number of Grid interfaced data storage resources on the integration infrastructure Number of Cores accessible to the OSG infrastructure Size of Disk storage accessible to the OSG infrastructure CPU Wall Clock usage of the OSG infrastructure
1.2
21 9 ~54,000 ~24 Petabytes Average of 37,800 CPU days/ day during May 2010
Usage of Open Science Grid
The overall usage of OSG continues to grow (Figure 2) though utilization by each stakeholder varies depending on its needs during any particular interval. Overall use of the facility for the 12 month period ending June 1, 2010 was 272M hours, compared to 182M hours for the previous 12 months, a 50% increase. (Detailed usage plots can be found in the attached document on Production on Open Science Grid.) During stable normal operations, OSG provides approximately 900,000 CPU wall clock hours a day (~37,500 cpu days per day) with peaks occasionally exceeding 1M hours a day; approximately 250K – 300K opportunistic hours (~30%) are available on a daily basis for resource sharing. Based on our transfer accounting (which is in its early days and ongoing in depth validation), we measure approximately 400 TB of data movement (both intra- and inter-site) on a daily basis with peaks of 1000 TB/day. Of this, we estimate 25% is GridFTP transfers between sites and the rest is via LAN protocols.
Figure 2: OSG Usage (hours/month) from June 2007 to June 2010
Non-HEP usage has increased substantially over the past year (Figure 3), primarily due to the increased ability of LIGO to submit Einstein@Home jobs supporting the pulsar analysis. From June 2009 to June 2010, the fraction of non-HEP usage increased from 4.5% to 20%, more than a 4-fold increase in the fractional use. LIGO accounts for approximately 95K hours/day (11% of the total), and non-physics use now averages 80K hours/day (9% of the total), reflecting efforts
4
over the past year to support SBGRID and incorporate new resources such as Nebraska’s Holland Computing Center.
Figure 3: OSG non-HEP weekly usage from June 2007 to June 2010, showing more than a quadruple fractional increase. LIGO (shown in red) is the largest non-HEP contributor.
1.3
Science enabled by Open Science Grid
OSG’s infrastructure supports a broad scope of scientific research activities, including the major physics collaborations, nanoscience, biological sciences, applied mathematics, engineering, computer science, and, through the Engagement program, other non-physics research disciplines. The distributed facility is heavily used, as described below and in the attachment “Production on Open Science Grid” showing usage charts. A strong OSG focus in the last year has been supporting the ATLAS and CMS collaborations preparations for LHC data taking that re-started in March 2010. Each experiment ran significant preparatory workload tests (including STEP09), data distribution and analysis challenges while maintaining significant ongoing simulation processing. As a result, the OSG infrastructure has performed well during current data taking. At the same time, OSG partnered with ATLAS and CMS to develop and deploy mechanisms that has enabled productive use of over 40 U.S. Tier-3 sites that were added over the past year. OSG made significant accomplishments in the past year supporting the science of the Consortium members and stakeholders (Table 2). Considering first the large experiments, in late 2009 LIGO significantly ramped up Einstein@Home production on OSG to search for gravitational radiation from spinning neutron star pulsars, publishing 28 papers on this and other analyses. The D0 and CDF experiments used the OSG facility for a large fraction of their simulation and analysis processing in publishing 28 and 61 papers, respectively, over the past 12 months. The LHC experiments ATLAS and CMS also had a productive year. CMS submitted for publication 23 physics papers based on cosmic ray analyses as well as a charged particle measurement from the December 2009 first collision dataset. ATLAS submitted a total of 113 papers. The STAR experiment had 29 publications during this time.
5
Smaller research activities also made considerable science contributions with OSG support. Besides the physics communities the structural biology group at Harvard Medical School, groups using the Holland Computing Center and NYSGrid, mathematics research at the University of Colorado, protein structure modeling and prediction applications have sustained (though cyclic) use of the production infrastructure. The Harvard paper was published in Science. As Table 2 shows, approximately 367 papers were published over the past 12 months (listed in the attachment Publications Enabled by Open Science Grid). Non-HEP activities accounted for 92 (25%), a large increase from the previous 12 month period. These publications depended not only on OSG “cycles” but OSG-provided software, monitoring and testing infrastructure, security and other services. Table 2: Science Publications Resulting from OSG Usage
VO Accelerator Physics ATLAS CDF CDMS CIGI CMS D0 Engage GLOW HCC LIGO Mini-Boone MINOS nanoHUB NYSGRID SBGRID STAR Total 1.4
# pubs 2 113 61 4 1 27 28 18 35 1 28 6 7 3 3 1 29 367
Comments
+ 3 accepted
+Ph.D thesis 92 non-HEP
OSG cyberinfrastructure research
As a comprehensive collaboratory OSG continues to provide a laboratory for research activities to deploy and extend advanced distributed computing technologies in the following areas: •
Research on the operation of a large scalable heterogeneous cyber-infrastructure in order to improve its effectiveness and throughput. As part of this research we have developed a comprehensive set of “availability” probes and reporting infrastructure to allow site and grid administrators to quantitatively measure and assess the robustness and availability of the resources and services.
•
Deployment and scaling in the production use of “pilot-job” or “resource overlay” workload management system – ATLAS PanDA and CMS glideinWMS. These developments were crucial to the experiments meeting their analysis job throughput targets.
6
•
Scalability and robustness enhancements to Condor technologies. For example, extensions to Condor to support Pilot job submissions were developed, significantly increasing the job throughput possible on each Grid site.
•
Scalability and robustness testing of enhancements to Globus grid technologies. For example, testing of the alpha and beta releases of the Globus GRAM5 package provided feedback to Globus ahead of the official release, in order to improve the quality of the released software.
•
Scalability and robustness testing of storage technologies - BeStMan, XrootD, dCache, Lustre and HDFS at-scale to determine their capabilities and provide feedback to the development team to help meet the needs of the OSG stakeholders.
•
Operational experiences with a widely distributed security infrastructure that assesses usability and availability together with response, vulnerability and risk.
•
Support of inter-grid gateways that support transfer of data and cross- execution of jobs, including transportation of information, accounting, service availability information between OSG and European Grids supporting the LHC Experiments (EGEE/WLCG). Usage of the Wisconsin GLOW campus grid “grid-router” to move data and jobs transparently from the local infrastructure to the national OSG resources. Prototype testing of the OSG FermiGridto-TeraGrid gateway to enable greater integration and thus enable easier access to appropriate resources for the science communities.
•
Integration and scaling enhancement of BOINC-based applications (LIGO’s Einstein@home) submitted through grid interfaces.
•
Further development of effective resource selection services - a hierarchy of matchmaking services (OSG MM) and Resource Selection Services (ReSS) that collect information from most OSG sites and provide community based matchmaking services that are further tailored to particular application needs.
•
Investigations and testing of policy and scheduling algorithms to support “opportunistic” use and backfill of resources that are not otherwise being used by their owners, using information services such as GLUE, matchmaking and workflow engines including Pegasus and Swift.
•
Investigations to integrate of Shiboleth as an end-point Identity Management system, and unified client tools to handle identity tokens across web and grid clients.
•
Comprehensive job and initial data accounting across most OSG sites with published summaries for each VO and Site. This work also supports a per-job information finding utility for security forensic investigations.
1.5
Technical achievements in 2009-2010
More than three quarters of the OSG staff directly support (and leverage at least an equivalent number of contributor efforts) the operation and software for the ongoing stakeholder productions and applications (the remaining quarter mainly engages new customers and extends and proves software and capabilities; and also provides management and communications etc.). In 2009, some specific technical activities that directly support science include:
7
•
OSG released a stable, production-capable OSG 1.2 software package on a schedule that enabled the experiments to deploy and test the cyberinfrastructure before LHC data taking. This release also allowed the Tier-1 sites to transition to be totally OSG supported, eliminating the need for separate integration of EGEE gLite components and simplifying software layering for applications that use both EGEE and OSG.
•
OSG carried out “prove-in” of reliable critical services (e.g. BDII) for LHC and operation of services at levels that meet or exceed the needs of the experiments. This effort included robustness tests of the production infrastructure against failures and outages and validation of information by the OSG as well as the WLCG.
•
Collaboration with STAR continued toward deploying the STAR software environment as virtual machine images on grid and cloud resources. We have successfully tested publish/subscribe mechanisms for VM instantiation (Clemson) as well as VM managed by batch systems (Wisconsin).
•
Extensions work on LIGO resulted in adapting the Einstein@Home for Condor-G submission, enabling a greater than 5x increase in the use of OSG by Einstein@Home.
•
Collaborative support of ALICE, Geant4, NanoHub, and SBGrid has increased their productive access to and use of OSG, as well as initial support for IceCube and GlueX.
•
Engagement efforts and outreach to science communities this year have led to work and collaboration with more than 10 additional research teams.
•
Security program activities that continue to improve our defenses and capabilities towards incident detection and response via review of our procedures by peer grids and adoption of new tools and procedures.
•
Metrics and measurements effort has continued to evolve and provides a key set of functions in enabling the US-LHC experiments to understand their performance against plans; and assess the overall performance and production across the infrastructure for all communities. In addition, the metrics function handles the reporting of many key data elements to the LHC on behalf of US-ATLAS and US-CMS.
•
Work in testing at-scale of various software elements and feedback to the development teams to help achieve the needed performance goals. In addition, this effort tested new candidate technologies (e.g. CREAM and ARC) in the OSG environment and provided feedback to WLCG.
•
Contributions to the PanDA and GlideinWMS workload management software that have helped improve capability and supported broader adoption of these within the experiments.
•
Collaboration with ESnet and Internet2 has provided a distribution, deployment, and training framework for new network diagnostic tools such as perfSONAR and extensions in the partnerships for Identity Management.
•
Training, Education, and Outreach activities have reached out to numerous professionals and students who may benefits from leverage of OSG and the national CI. The International Science Grid This Week electronic newsletter (www.isgtw.org) continues to experience significant subscription growth.
8
In summary, OSG continues to demonstrate that national cyberinfrastructure based on federation of distributed resources can effectively meet the needs of researchers and scientists. 1.6
Challenges facing OSG
We continue to work and make progress on the challenges faced to meet our longer-term goals: •
Dynamic sharing of tens of locally owned, used and managed compute and storage resources, with minimal additional human effort (to use and administer, and limited negative impacts on the communities owning them.
•
Utilizing shared storage with other than the owner group - not only more difficult than (the quantized) CPU cycle sharing, but also less well supported by the available middleware.
•
Federation of the local and community identity/authorization attributes within the OSG authorization infrastructure.
•
The effort and testing required for inter-grid bridges involves significant costs, both in the initial stages and in continuous testing and upgrading. Ensuring correct, robust end-to-end reporting of information across such bridges remains fragile and human effort intensive.
•
Validation and analysis of availability and reliability testing, accounting and monitoring information. Validation of the information is incomplete, needs continuous attention, and can be human effort intensive.
•
The scalability and robustness of the infrastructure has not yet reached the scales needed by the LHC for analysis operations in the out years. The US LHC software and computing leaders have indicated that OSG needs to provide ~x2 in interface performance over the next year or two, and the robustness to upgrades and configuration changes throughout the infrastructure needs to be improved.
•
Full usage of available resources. A job “pull” architecture (e.g., the Pilot mechanism) provides higher throughput and better management than one based on static job queues, but now we need to move to the next level of effective usage.
•
Automated site selection capabilities are inadequately deployed and are also embryonic in the capabilities needed – especially when faced with the plethora of errors and faults that are naturally a result of a heterogeneous mix of resources and applications with greatly varying I/O, CPU and data provision and requirements.
•
User and customer frameworks are important for engaging non-Physics communities in active use of grid computing technologies; for example, the structural biology community has ramped up use of OSG enabled via portals and community outreach and support.
•
A common operations infrastructure across heterogeneous communities can be brittle. Efforts to improve the early detection of faults and problems before they impact the users help everyone.
•
Analysis and assessment of and recommendations as a result of usage, performance, accounting and monitoring information are key needs which require dedicated and experienced effort.
9
•
Transitioning students from the classroom to be users is possible but continues as a challenge, partially limited by the effort OSG can dedicate to this activity.
•
Use of new virtualization, multi-core and job parallelism techniques, scientific and commercial cloud computing. We have two new satellite projects funded in these areas: the first on High Throughput Parallel Computing (HTPC) on OSG resources for an emerging class of applications where large ensembles (hundreds to thousands) of modestly parallel (4to ~64- way) jobs; the second a research project to do application testing over the ESnet 100Gigabit network prototype, using the storage and compute end-points supplied by the Magellan cloud computing at ANL and NERSC.
•
Collaboration and partnership with TeraGrid, under the umbrella of a Joint Statement of Agreed Upon Principles signed in August 2009. New activities have been undertaken, including representation at one another’s management meetings, tests on submitting jobs to one another’s infrastructure and exploration of how to accommodate the different resource access mechanisms of the two organizations. An NSF-funded joint OSG – TeraGrid effort, called ExTENCI, is expected to commence in July or August 2010.
These challenges are not unique to OSG. Other communities are facing similar challenges in educating new entrants to advance their science through large-scale distributed computing resources. 1.7
Preparing for the Future
At the request of our stakeholders to continue to sustain their dependence on the OSG services, we started a planning exercise within the OSG Council. We have started initial discussions of transition planning to ensure sustainability of the organization, its contributors and staff. Council sub-committees are charged to revisit the existing Consortium mission statement and the organizational (management) plan. The following draft documents are have had an initial review but are not in final form: •
National CI and the Campuses (OSG-939)
•
Requirements and Principles for the Future of OSG (OSG-938)
•
OSG Interface to Satellite Proposals/Projects (OSG-913)
•
OSG Architecture (OSG-966)
OSG contributed to a paper written by US ATLAS and US CMS “Assessment of Core Services provide to U.S. ATLAS and U.S. CMS by OSG”. We have started preliminary thinking and discussions in the following areas of research and development identified as needed to extend and improve the OSG services and capabilities for our science stakeholders: •
Configuration Management across services on different hosts – subject of a Blueprint discussion in May 2010. This is planned as future work by the software teams.
•
Integration of Commercial and Scientific Clouds – Initial tests with the Magellan scientific Clouds at ANL and NERSC are underway and we have started a series of follow up technical meetings with the service groups every six weeks; explorations with EC2 are ongoing. We are using this work and the High Throughput Parallel Computing (HTPC) satellite project to
10
understand how new resources and compute capabilities and capacities can be best integrated smoothly into the OSG infrastructure. •
Usability for collaborative analysis – evaluations of extended data management, opportunistic storage management, the IRODS data management technologies are underway. Requirements for “Dynamic VOs/Workgroups” are being gathered from several stakeholders.
•
Active management of shared capacity, utilization planning and change – an active analysis of available cycles is underway; we have started discussions of resource reservation and application of more dynamic “OSG usage priorities”.
•
End-to-End Data Management challenges in light of advanced networks – we continue to work with I2 and ESNET research arms to look for opportunities for collaboration.
We are starting to look forward to the next generation challenges for the field, in preparation for the increases in LHC luminosity and upgrades, Advanced LIGO, and new communities such as Glue-X. For example, those challenges from the HEP Scientific Grand Challenges Report (Dec 2008): •
Extreme wide area networking
•
Data management at the Exabyte scale
•
Management of large-scale global collaboratories
•
Planning for long-term data stewardship
•
Change management for muli-decade long projects.
•
Enabling advantages from new technologies: multicore, GPU etc
•
Exploitation of multi-level storage hierarchies.
11
2. 2.1
Contributions to Science ATLAS
Following the startup of LHC collider operations at CERN in November 2009, the ATLAS collaboration has taken several hundred terabytes of collision data with their detector. This data was, after prompt reconstruction at the Tier-0 center, distributed to regional Tier-1 centers for archival storage, analysis and further distribution to the regional Tier-2 centers. Re-reconstruction of the data taken during the short 2009 run was conducted at the Tier-1 centers during the Christmas holidays while users started to analyze the data using resources at the Tier-1 site, all Tier-2 centers and their own institutional computing facilities. As the amount of initial data taking was small we observed a spike in resource usage at the higher levels of the facility with users running data reduction steps followed by transfers of the derived, condensed data products to compute servers they use for interactive analysis, resulting in a reduced utilization of grid resources for a few months until LHC operations resumed in March 2010. As machine luminosity ramped up rather quickly much more data was taken in the March to May timeframe, which was re-reconstructed at the Tier-1 centers in April and May. Figure 4 shows the charged particle multiplicity published by ATLAS in the first data taken at 900 GeV.
Figure 4: ATLAS charged particle measurement of 900 GeV data
According to the data distribution policy as it was defined for the US region Event Summary Data (ESD) and Analysis Object Data (AOD) along with their derived versions were replicated in multiple copies to the Tier-2 centers in the U.S. Given that the replication process of several hundred terabytes of data from the Tier-1 center to the Tier-2s needed to be completed within the shortest possible period of time, the data rates the network and the storage systems had to sustain 12
rose to an aggregate rate of 2 gigabytes per second. User analysis on the data started instantly with the arrival of the datasets at the sites. With more data becoming available the level of activity in the analysis queues at the Tier-1 and the Tier-2 centers was almost constant with a significant backlog of jobs waiting in the queues at all times. The workload management system based on PanDA distributed the load evenly across all sites that were prepared to run analysis on the required datasets. On average, the U.S. ATLAS facility contributes 30% of worldwide analysisrelated data access. The number of user jobs submitted by the worldwide ATLAS community and brokered by PanDA to U.S. sites has reached an average number of 600,000 per month peaking occasionally at more than 1 million submitted jobs per month. Monte Carlo production is ongoing with some 50,000 concurrent jobs worldwide, and about 10,000 jobs running on resources provided by the distributed U. S. ATLAS computing facility comprising the Tier-1 center at BNL and 5 Tier-2 centers located at 9 different institutions spread across the U.S.
ATLAS
Figure 5: OSG CPU hours (71M total) used by ATLAS over 12 months, color coded by facility.
The experience gained during the computing challenges and at the start of ATLAS data taking gives us confidence that the tiered, grid-based, computing model has sufficient flexibility to process, reprocess, distill, disseminate, and analyze ATLAS data. We have found, however, that the Tier-2 centers may not be sufficient to reliably serve as the primary analysis engine for more than 400 U.S. physicists. As a consequence a third tier with computing and storage resources located geographically close to the researchers was defined as part of the analysis chain as an important component to buffer the U.S. ATLAS analysis system from unforeseen, future problems. Further, the enhancement of U.S. ATLAS institutions’ Tier-3 capabilities is essential and is planned to be built around the short and long-term analysis strategies of each U.S. group. An essential component of this strategy is the creation of a centralized support structure to handle the increased number of campus-based computing clusters. A small group within U.S. ATLAS spent considerable effort over Summer 2009 developing a low maintenance Tier-3 computing implementation. OSG joined this effort soon after and helped in two key areas: packaging of batch processing (Condor) and storage management components (xrootd), both of which are eas-
13
ily installable and maintainable by physicists. Because this U.S. initiative (driven by Rik Yoshida from Argonne National Laboratory and Doug Benjamin from Duke University in collaboration with OSG) made rapid progress in just a few months, ATLAS Distributed Computing Management invited the initiative leaders to develop a technical and maintainable solution for the Tier-3 community. A very successful CERN workshop addressing Tier-3 issues was organized in January 2010, with good representation from around the globe. Major areas of work and interest were identified during the meeting and short lived working groups were formed to address issues associated with in software installation, data and storage management, data replication and data access. Most of these working groups are close to delivering their reports. Open Science Grid has organized regular Tier-3 Liaison meetings between several members of the OSG facilities, U.S. Atlas and U.S. CMS. During these meetings, topics discussed include cluster management, site configuration, site security, storage technology, site design, and experiment-specific Tier-3 requirements. Based on information exchanged at these meetings several aspects of the U.S. Atlas Tier-3 design were refined. Both the OSG and U.S. Atlas Tier-3 documentation was improved and enhanced. Following several workshops conducted in the U.S., Yoshida and Benjamin installed an entire Tier-3 cluster using virtual machines on a single multi core desktop machine. This virtual cluster is used for documentation development and U.S. Atlas Tier-3 administrator training. Marco Mambelli of OSG has provided much assistance in the configuration and installation of the software for the virtual cluster. OSG also supports a crucial user software package (wlcg-client with support for xrootd added by OSG) used by all U.S. Atlas Tier-3 users, a package that enhances and simplifies the user environment at the Tier-3 sites. Today U.S. ATLAS (contributing to ATLAS as a whole) relies extensively on services and software provided by OSG, as well as on processes and support systems that have been produced or evolved by OSG. This dependence originates partly from the fact that U.S. ATLAS fully committed to relying on OSG several years ago. Over the past 3 years U.S. ATLAS invested heavily in OSG in many aspects – human and computing resources, operational coherence and more. In addition, and essential to the operation of the worldwide distributed ATLAS computing facility, the OSG efforts have aided the integration with WLCG partners in Europe and Asia. The derived components and procedures have become the basis for support and operation covering the interoperation between OSG, EGEE, and other grid sites relevant to ATLAS data analysis. OSG provides software components that allow interoperability with European ATLAS sites, including selected components from the gLite middleware stack such as LCG client utilities (for file movement, supporting space tokens as required by ATLAS), and file catalogs (server and client). It is vital to U.S. ATLAS that the present level of service continues uninterrupted for the foreseeable future, and that all of the services and support structures upon which U.S. ATLAS relies today have a clear transition or continuation strategy. Based on its observations U.S. ATLAS made a suggestion for OSG to develop a coherent middleware architecture rather than continue providing a distribution as a heterogeneous software system consisting of components contributed by a wide range of projects. Difficulties we encountered included inter-component functional dependencies that require communication and coordination between component development teams. A technology working group, chaired by a member of the U.S. ATLAS facilities group (John Hover, BNL) has been asked to help the OSG Technical Director by investigating, researching, and clarifying design issues, resolving ques-
14
tions directly, and summarizing technical design trade-offs such that the component project teams can make informed decisions. In order to achieve the resulting goals, OSG needs an explicit, documented system design, or architecture, so that component developers can make compatible design decisions and virtual organizations (VO) such as U.S. ATLAS can develop their own application based on the OSG middleware stack as a platform. A design roadmap is now under development. The OSG Grid Operations Center (GOC) infrastructure at Indiana University is at the heart of the operations and user support procedures. GOC services are integrated with the GGUS infrastructure in Europe making the GOC a globally connected system for worldwide ATLAS computing operation. Middleware deployment support provides an essential and complex function for U.S. ATLAS facilities. For example, support for testing, certifying and building a foundational middleware for our production and distributed analysis activities is a continuing requirement, as is the need for coordination of the roll out, deployment, debugging and support for the middleware services. In addition, some level of preproduction deployment testing has been shown to be indispensable. This testing is currently supported through the OSG Integration Test Bed (ITB) providing the underlying grid infrastructure at several sites along with a dedicated test instance of PanDA, the ATLAS Production and Distributed Analysis system. These elements implement the essential function of validation processes that accompany incorporation of new and new versions of grid middleware services into the VDT, which provides a coherent OSG software component repository. U.S. ATLAS relies on the VDT and OSG packaging, installation, and configuration processes to provide a well-documented and easily deployable OSG software stack. U.S. ATLAS greatly benefits from OSG’s Gratia accounting services, as well as the information services and probes that provide statistical data about facility resource usage and site information passed to the application layer and to WLCG for review of compliance with MOU agreements. An essential component of grid operations is operational security coordination. The coordinator provided by OSG has good contacts with security representatives at the U.S. ATLAS Tier-1 center and Tier-2 sites. Thanks to activities initiated and coordinated by OSG a strong operational security community has grown up in the U.S. in the past few years, ensuring that security problems are well coordinated across the distributed infrastructure. No significant problems with the OSG provided infrastructure have been encountered since the start of LHC data taking. However, there is an area of concern that may impact the facilities’ performance in the future. As the number of job slots at sites continues to increase the performance of pilot submission through CondorG and the underlying Globus Toolkit 2 (GT2) based gatekeeper must keep up without slowing down job throughput, particularly when running short jobs. When addressing this point with the OSG facilities team we found that they were open to evaluating and incorporating recently developed components such as the CREAM Computing Element (CE) provided by EGEE developers in Italy. Intensive tests were conducted by the Condor team in Madison and integration issues were discussed in December 2009 and again in May 2010 between the PanDA team, the Computing Facilities and Condor developers. In the area of middleware extensions, US ATLAS continued to benefit from the OSG’s support for and involvement in the U.S. ATLAS-developed distributed processing and analysis system (PanDA) layered over the OSG’s job management, storage management, security and information system middleware and services. PanDA provides a uniform interface and utilization 15
model for the experiment's exploitation of the grid, extending across OSG, EGEE and Nordugrid. It is the basis for distributed analysis and production ATLAS-wide, and is also used by OSG as a WMS available to OSG VOs, as well as a PanDA based service for OSG Integrated Testbed (ITB) test job submission, monitoring and automation. This year the OSG’s WMS extensions program continued to provide the effort and expertise on PanDA security that has been essential to establish and maintain PanDA’s validation as a secure system deployable in production on the grids. In particular PanDA’s glexec-based pilot security system, implemented last year, was deployed and tested this year as glexec-enabled sites came available. After several further iterations of improvements and fixes on both the glexec and PanDA pilot fronts, by the end of 2009 the glexec functionality was supported in ATLAS’s production pilot version and ready for production validation. Another important extension activity during the past year was in WMS monitoring software and information systems. ATLAS and U.S. ATLAS are in the process of merging what were distinct monitoring efforts, a PanDA/US effort and a WLCG/CERN effort, together with the new ATLAS Grid Information System (AGIS) that integrates ATLAS-specific information with the grid information systems. A technical approach for this integration was developed and a first framework put in place, based on Django, the python based web application framework, together with json and jQuery. Coherency of this effort with OSG systems is vital and during the year ATLAS and US ATLAS managers visited OSG principals to discuss the monitoring work and the integration of AGIS with OSG information systems and monitoring. The picture gained there of how AGIS can best interface to OSG systems will guide future work, particularly once a new US-based developer is added in the coming year (at UT Arlington). 2.2
CMS
US-CMS relies on Open Science Grid for critical computing infrastructure, operations, and security services. These contributions have allowed US-CMS to focus experiment resources on being prepared for analysis and data processing, by saving effort in areas provided by OSG. OSG provides a common set of computing infrastructure on top of which CMS, with development effort from the US, has been able to build a reliable processing and analysis framework that runs on the Tier-1 facility at Fermilab, the project supported Tier-2 university computing centers, and opportunistic Tier-3 centers at universities. There are currently 24 Tier-3 centers registered with the CMS computing grid in the US which provide additional simulation and analysis resources to the US community. Since data taking commenced on March 30, 2010 there has been a tremendous push to produce physics results at 7 TeV center of mass energy using worldwide CMS computing resources. Figure 6 shows the charged multiplicity vs pseudorapidity η, taken from the first CMS physics paper using 7 TeV data. A number of other papers are under preparation.
16
Figure 6: Plot of charged multiplicity vs η in the first paper produced by CMS for 7 TeV running.
In addition to common interfaces, OSG provides the packaging, configuration, and support of the storage services. Since the beginning of OSG the operations of storage at the Tier-2 centers have improved steadily in reliability and performance. OSG is playing a crucial role here for CMS in that it operates a clearinghouse and point of contact between the sites that deploy and operate this technology and the developers. In addition, OSG fills in gaps left open by the developers in areas of integration, testing, and tools to ease operations. The stability of the computing infrastructure has not only benefitted CMS. CMS’ use of resources (see Figure 7 and Figure 8) has been very much cyclical so far, thus allowing for significant use of the resources by other scientific communities. OSG is an important partner in Education and Outreach, and in maximizing the impact of the investment in computing resources for CMS and other scientific communities. OSG also plays an important role in US-CMS operations and security. OSG has been crucial to ensure US interests are addressed in the WLCG. The US is a large fraction of the collaboration both in terms of participants and capacity, but a small fraction of the sites that make-up WLCG. OSG is able to provide a common infrastructure for operations including support tickets, accounting, availability monitoring, interoperability and documentation. Now that CMS is taking data, the need for sustainable security models and regular accounting of available and used resources is crucial. The common accounting and security infrastructure and the personnel provided by OSG represent significant benefits to the experiment, with the teams at Fermilab and the University of Nebraska providing the development and operations support, including the reporting and validation of the accounting information between the OSG and WLCG
17
CMS
Figure 7: OSG CPU hours (62M total) used by CMS over 12 months, color-coded by facility.
Figure 8: Number of unique CMS users of the OSG computing facility.
In addition to these general roles OSG plays for CMS, there were the following concrete contributions OSG has made to major milestones in CMS during the last year: •
Accounting and Availability reporting to the agencies as part of the monthly WLCG reporting has become routine. In addition, the daily availability reporting is the first email many of the site admins and managers read in the morning to verify that all is well. The transition to the new WLCG accounting metric went very smoothly.
•
Three of the CMS Tier-2 sites successfully transitioned to the hadoop file system as underpinning for their storage software, and BeStMan SRM and Globus gridftp as grid interfaces. 18
This transition went very well, and has proven to reduce the cost of ownership of storage at those sites. The reduced human effort allowed us to add effort to other areas, especially in Analysis Operations (Performance Metrics), and glideinWMS operations. •
CMS is benefiting from the joint gfactory operations of OSG and CMS at UCSD. We now have two active VOs other than CMS at a scale of 400,000 hours per week. This has forced us to think through the operations model more clearly, thus clarifying the responsibilities and operational procedures in ways that CMS benefits from as well. In addition, the new customers are pushing more aggressively for improved maintenance and operations tools that also CMS benefits from.
•
Work has continued in the area of scalability and reliability, especially with regards to Condor as needed for glideinWMS operations, and BeSTMan SRM deployment and configuration with an eye towards performance tuning, and IO performance in general.
Finally, let us comment on the US Tier-2 infrastructure as compared with the rest of the world. All 7 US Tier-2s are in the top 10-15 of the 50 Tier-2s globally as measured by successfully executed data analysis, data transfer volume ingested, and data transfer volume exported. US Tier-2s provide the best performing site in all three categories, and typically provide two of the best three sites worldwide, and three or four of the top five. In addition, more than 50% of the entire successful CMS MC production last year was done on OSG. The OSG infrastructure continues to be the most reliable region for CMS worldwide. 2.3
LIGO
LIGO continues to leverage the Open Science Grid for opportunistic computing cycles associated with its Einstein@Home application, known as Einstein@OSG for its customized grid based job submission and monitoring tools, which are a superset of the original code base. This application is one of several in use for an “all-sky” search for gravitational waves of a periodic nature attributed to elliptically deformed pulsars. Such a search requires enormous computational resources to fully exploit the science content available within LIGO’s data during the analysis. As a result, volunteer and opportunistic computing based on the BOINC (Berkeley Open Infrastructure for Network Computing) has been leveraged to utilize as many computing resource worldwide as possible. Since porting the grid based Einstein@OSG code onto the Open Science Grid more than a year ago, steady advances in both the code performance, reliability and overall deployment onto the Open Science Grid have been demonstrated (Figure 9). For a period of time early in 2010, Einstein@OSG was the number one computational application running on the Open Science Grid. In terms of the scientific contributions to the overall worldwide Einstein@Home analysis, the Open Science Grid is routinely the world leader for weekly scientific credits. Since beginning to run on the OSG, close to 400 million E@H credits, units of scientific computation as defined by the Einstein@Home team (approximately 1 TeraFlops-second per credit) have been attributed to the OSG. Figure 10 shows an Einstein@Home search for pulsar gravitational candidates.
19
Figure 9: OSG Usage by LIGO's Einstein@Home application for the past 12 months. Increases in opportunistic usage have come primarily from being able to reach a growing numbers of OSG sites. Greater competition for opportunistic cycles beginning in the early spring of 2010 has resulted in a steady decline in average throughput per interval of time in the past few months.
Figure 10: Search for LIGO pulsar sources. Each angular “cell” is analyzed using Einstein@Home, with the results color coded by coincidences in frequency and frequency changes.
20
One of the most promising sources of gravitational waves for LIGO is from the inspiral of a system of compact black holes and/or neutron stars as the system emits gravitational radiation leading to the ultimate coalescence of the binary pair. The binary inspiral data analyses typically involve working with tens of terabytes of data in a single workflow. Collaborating with the Pegasus Workflow Planner developers at USC-ISI, LIGO continues to identify changes to both Pegasus and to the binary inspiral workflow codes to more efficiently utilize the OSG and its emerging storage technology, where data must be moved from LIGO archives to storage resources near the worker nodes on OSG sites. One area of particular focus has been on the understanding and integration of Storage Resource Management (SRM) technologies used in OSG Storage Element (SE) sites to house the vast amounts of data used by the binary inspiral workflows so that worker nodes running the binary inspiral codes can effectively access the LIGO data. The SRM based Storage Element established on the LIGO Caltech OSG integration testbed site is being used as a development and test platform to get this effort underway without impacting OSG production facilities. This site has 120 CPU cores with approximately 30 terabytes of storage currently configured under SRM. The SE is based on BeStMan and Hadoop for the distributed file system shared across the worker nodes. Using Pegasus for the workflow planning, workflows for the binary inspiral data analysis application using close to ten terabytes of LIGO data have successfully run on this site. After effectively demonstrating the ability to run the binary inspiral workflows on the Caltech integration testbed, additional globus based services, in particular a replication locator service (RLS) and an OSG match-making service (OSG-MM) were set up, with support from USC-ISI and RENCI to allow workflows to be generated the Pegasus Planner for running binary inspiral analysis on OSG production sites. A careful analysis of the capabilities and policies of OSG sites identified one site as the key to making this effort successful. LIGO data based on the S6 (current science) run was cataloged and transferred to the SE at this production site. Up-to-date versions of the science code were deployed and have successfully run on the OSG production site with one minor step failing to port at this time. Code development is underway to resolve this issue. Another area of investigation is the overall performance of an Open Science Grid site relative to a LIGO Data Grid site for which the code was originally developed to run. A three hundred percent increase in run-time has been seen on OSG production sites relative to run-times on the LIGO Data Grid. However, this same discrepancy is not seen when running on the Caltech OSG Integration testbed. More work is needed to carry this effort to the level of full production. A very recent development is the porting of the pulsar powerflux application onto the Open Science Grid. This application use custom data sets currently available at the LIGO Data Grid sites in Europe. These data sets have now been cataloged and transferred onto the Caltech integration testbed and one of the OSG production site’s storage element (SE). The application has successfully been demonstrated to run on small parameter ranges on both the integration testbed and the production site. Improvements on performance have been identified as an area for development, but the technology looks to be well matched at this time. LIGO continues working closely with the OSG Security team, DOE Grids, and ESnet to evaluate the implications of its requirements on authentication and authorization within its own LIGO Data Grid user community and how these requirements map onto the security model of the OSG.
21
2.4
ALICE
The ALICE experiment at the LHC relies on a mature grid framework, AliEn, to provide computing resources in a production environment for the simulation, reconstruction and analysis of physics data. Developed by the ALICE Collaboration, the framework has been fully operational for several years, deployed at ALICE and WLCG Grid resources worldwide. ALICE US collaboration is currently in the process of deploying significant compute and storage resources in the US, anchored by tier centers at LBNL/NERSC and LLNL. That effort makes use of work carried out in 2009 by the ALICE VO and OSG for integration of OSG resources into the ALICE Grid. In early 2009, an ALICE-OSG Joint task force was formed to support the inclusion of ALICE Grid activities in OSG. The task force developed a series of goals leading to a common understanding of AliEn and OSG architectures. The OSG Security team reviewed and approved a proxy renewal procedure common to ALICE Grid deployments for use on OSG sites. A jobsubmission mechanism was implemented whereby an ALICE VO-box service deployed on the NERSC-PDSF OSG site, submitted jobs to the PDSF cluster through the OSG interface. The submission mechanism was activated for ALICE production tasks and operated for several months. The task force validated ALICE OSG usage through normal reporting means and verified that site operations were sufficiently stable for ALICE production tasks at low job rates and with minimal data requirements. ALICE is in the process of re-doing these validation tasks as larger local resources are being deployed at LBNL and LLNL. The ALICE VO is currently a registered VO with OSG, supports a representative in the OSG VO forum and an Agent to the OSG-RA for issuing DOE Grid Certificates to ALICE collaborators. ALICE use of OSG will grow as ALICE resources are deployed in the US. These resources will provide the data storage facilities needed to expand ALICE use of OSG and add compute capacity on which the AliEn-OSG interface can be utilized at full ALICE production rates. 2.5
D0 at Tevatron
The D0 experiment continues to rely heavily on OSG infrastructure and resources in order to achieve the computing demands of the experiment. The D0 experiment has successfully used OSG resources for many years and plans on continuing with this very successful relationship into the foreseeable future. This usage has resulted in a tremendous science publication record, including the intriguing CP violation measurement shown in Figure 11.
22
Figure 11: Plot showing new D0 measurement of CP violation, which gives evidence for significant matter-antimatter asymmetry beyond what is expected in the Standard Model.
All D0 Monte Carlo simulation is generated at remote sites, with OSG continuing to be a major contributor. During the past year, OSG sites simulated 400 million events for D0, approximately 1/3 of all production. The rate of production has leveled off over the past year as almost all major sources of inefficiency have been resolved and D0 continues to use OSG resources very efficiently. The changes in policy at numerous sites for job preemption, the continued use of automated job submissions, and the use of resource selection has allowed D0 to opportunistically use OSG resources to efficiently produce large samples of Monte Carlo events. D0 continues to use approximately 20 OSG sites regularly in its Monte Carlo production), The total number of D0 OSG MC events produced over the past several years is nearing 1 billion events (Figure 12). Over the past year, the average number of Monte Carlo events produced per week by OSG continues to remain approximately constant. Since we use the computing resources opportunistically, it is interesting to find that, on average, we can maintain an approximately constant rate of MC events (Figure 13). The dip in OSG production in December and January was due to D0 switching to a new software release which temporarily reduced our job submission rate to OSG. Over the past year D0 has been able to obtain the necessary opportunistic resources to meet our Monte Carlo needs even though the LHC has turned on. As the luminosity of the LHC increases and the computing demands, increases, it will be crucial to have very efficient computing Therefore D0 will continue to work with OSG and Fermilab computing to continue to improve the efficiency in any way possible of Monte Carlo production on OSG sites. Over the past year D0 has been able to use LCG resources to produce Monte Carlo events. The primary reason that this is possible, is over the past year LCG began to use some of the infrastructure developed by OSG. Because LCG was able to easily adopt some of the OSG infrastructure, D0 is now able to produce approximately 5 million events/week on LCG.
23
The primary processing of D0 data continues to be run using OSG infrastructure. One of the very important goals of the experiment is to have the primary processing of data keep up with the rate of data collection. It is critical that the processing of data keep up in order for the experiment to quickly find any problems in the data and to keep the experiment from having a backlog of data. Typically D0 is able to keep up with the primary processing of data by reconstructing nearly 6 million events/day (Figure 14). However, when the accelerator collides at very high luminosities, it is difficult to keep up with the data using our standard resources. However, since the computing farm and the analysis farm have the same infrastructure, D0 is able to move analysis computing nodes to primary processing to improve its daily processing of data, as it has done on more than one occasion. This flexibility is a tremendous asset and allows D0 to efficiently use its computing resources. Over the past year D0 has reconstructed nearly 2 billion events on OSG facilities. In order to achieve such a high throughput, much work has been done to improve the efficiency of primary processing. In almost all cases, only 1 job submission is needed to complete a job, even though the jobs can take several days to finish, see Figure 15. OSG resources continue to allow D0 to meet is computing requirements in both Monte Carlo production and in data processing. This has directly contributed to D0 publishing 29 papers (11 additional papers have been submitted/accepted) from July 2009-to June 2010, see http://wwwd0.fnal.gov/d0_publications/.
Figure 12: Cumulative number of D0 MC events generated by OSG during the past year.
24
Figure 13: Number of D0 MC events generated per week by OSG during the past year. The dip in production in December and January was due to D0 switching to a new software release which temporarily reduced our job submission rate to OSG
Figure 14: Daily production of D0 data events processed by OSG infrastructure. The dip in September corresponds to the time when the accelerator was down for maintenance so no events needed to be processed.
25
Figure 15: Submission statistics for D0 primary processing. In almost all cases, only 1 job submission is required to complete the job even though jobs can run for several days.
2.6
CDF at Tevatron
In 2009-2010, the CDF experiment produced 42 new results for summer 2009 and 48 for winter 2010 using OSG infrastructure and resources, including the most recent upper limit on searches for the Standard Model Higgs (Figure 16).
26
Figure 16: Upper limit plot of recent CDF search for the Standard Model Higgs
The OSG resources support the work of graduate students, who are producing one thesis per week, and the collaboration as a whole, which is submitting a publication of new physics results every ten days. About 50 publications have been submitted in this period. A total of 900 million Monte Carlo events were produced by CDF in the last year. Most of this processing took place on OSG resources. CDF also used OSG infrastructure and resources to support the processing of 1.9 billion raw data events that were streamed to 2.5 billion reconstructed events, which were then processed into 4.8 billion ntuple events; an additional 1.9 billion ntuple events were created from Monte Carlo. Detailed numbers of events and volume of data are given in Table 3 (total data since 2000) and Table 4 (data taken from June 2009 to June 2010). Table 3: CDF data collection since 2000
Data Type Raw Data Production MC Stripped-Prd Stripped-MC MC Ntuple Total
Volume (TB) 1673 2011 880 89 0 371 5024
# Events (M) 11397 14519 6069 786 3 5722 37496
27
# Files 1922838 1927936 1016638 85609 533 334458 5288012
Table 4: CDF data collection from June 2009- June 2010
Data Type Raw Data Production MC Stripped-Prd Stripped-MC Ntuple MC Ntuple Total
Data Volume (TB) 306.2 404.0 181.3 14.140 0 149.5 116.8 1172.0
# Events (M) 1892.2 2516.3 893.9 80.2 0 4810.9 1905.8 12099.3
# Files 340487 331081 224156 11360 0 120416 100308 1127808
The OSG provides the collaboration computing resources through two portals. The first, the North American Grid portal (NamGrid), covers the functionality of MC generation in an environment which requires the full software to be ported to the site and only Kerberos or grid authenticated access to remote storage for output. The second portal, CDFGrid, provides an environment that allows full access to all CDF software libraries and methods for data handling. CDF, in collaboration with OSG, aims to improve the infrastructural tools in the next years to increase the usage of Grid resources, particularly in the area of distributed data handling. Furthermore the portal distinction will be eliminated in favor of qualifiers that are translated to Class Ads specifying the DH requirements, opportunistic usage or not, and CDF software requirements. CDF operates the pilot-based Workload Management System (glideinWMS) as the submission method to remote OSG sites. Figure 17 shows the number of running jobs on NAmGrid and demonstrates that there has been steady usage of the facilities, while Figure 18, a plot of the queued requests, shows that there is large demand. The highest priority in the last year has been to validate sites for reliable usage of Monte Carlo generation and to develop metrics to demonstrate smooth operations. Many sites in OSG remain unusable to CDF because of preemption policies and changes in policy without notification. Any site that becomes unstable is put into a test instance of the portal and removed if the problem is shown to be due to a preemption that prevents jobs from completing. New sites are tested and certified in an integration instance of the NAmGrid portal using Monte Carlo jobs that have previously been run in production. A large resource provided by Korea at KISTI is in operation and provides a large Monte Carlo production resource with high-speed connection to Fermilab for storage of the output. It will also provide a cache that will allow the data handling functionality to be exploited. The system is being commissioned just now. We are also adding more monitoring to the CDF middleware to allow faster identification of problem sites or individual worker nodes. Issues of data transfer and the applicability of opportunistic storage is being studied as part of the effort to understand issues affecting reliability. Significant progress has been made by simply adding retries with a backoff in time assuming that failures occur most often at the far end.
28
Figure 17: Running CDF jobs on NAmGrid
Figure 18: Waiting CDF jobs on NAmGrid, showing large demand, especially in preparation for the 42 results sent to Lepton-Photon in August 2009 and the rise in demand for the winter 2010 conferences.
A legacy glide-in infrastructure developed by the experiment was running through December 8, 2009 on the portal to on-site OSG resources (CDFGrid). This system was replaced by the same glideWMS infrastructure used in NAmGrid. Plots of the running jobs and queued requests are shown in Figure 19 and Figure 20. The very high demand for the CDFGrid resources observed during the summer conference season (leading to an additional 42 new results) and again during the winter conference season (leading to 48 new results), is noteworthy. Queues exceeding 30,000 jobs can be seen.
29
Figure 19: Running CDF jobs on CDFGrid
Figure 20: Waiting CDF jobs on CDFGrid
A clear pattern of CDF computing has emerged. There is high demand for Monte Carlo production in the months after the conference season, and for both Monte Carlo and data starting about two months before the major conferences. Since the implementation of opportunistic computing on CDFGrid in August, the NAmGrid portal has been able to take advantage of the computing resources on FermiGrid that were formerly only available through the CDFGrid portal. This has led to very rapid production of Monte Carlo in the period of time between conferences when the generation of Monte Carlo datasets are the main computing demand. In May 2009, CDF conducted a review of the CDF middleware and usage of Condor and OSG. While there were no major issues identified, a number of cleanup projects were suggested. These have all been implemented and will add to the long-term stability and maintainability of the software.
30
A number of issues affecting operational stability and operational efficiency have arisen in the last year. These issues and examples and solutions or requests for further OSG development are cited here. •
Architecture of an OSG site: Among the major issues we encountered in achieving smooth and efficient operations was a serious unscheduled downtime lasting several days in April. Subsequent analysis found the direct cause to be incorrect parameters set on disk systems that were simultaneously serving the OSG gatekeeper software stack and end-user data output areas. No OSG software was implicated in the root cause analysis; however, the choice of an architecture was a contributing cause. This is a lesson worth consideration by OSG, that a best practices recommendation coming from a review of the implementation of computing resources across OSG could be a worthwhile investment.
•
Service level and Security: Since April, 2009 Fermilab has had a new protocol for upgrading Linux kernels with security updates. An investigation of the security kernel releases from the beginning of 2009 showed that for both SLF4 and SLF5 the time between releases was smaller than the maximum time allowed by Fermilab for a kernel to be updated. Since this requires a reboot of all services, this has forced the NAmGrid and CDFGrid to be down for three days for long (72 hour) job queues every 2 months. A rolling reboot scheme has been developed and is deployed, but careful sequencing of critical servers is still being developed.
•
Opportunistic Computing/Efficient resource usage: The issue of preemption has been important to CDF this year. CDF has the role of both providing and exploiting opportunistic computing. During the April downtime already mentioned above, the preemption policy as provider caused operational difficulties characterized vaguely as “unexpected behavior”. The definition of expected behavior was worked out and preemption was enabled after the August 2009 conferences ended. From the point of view of exploiting opportunistic resources, the management of preemption policies at sites has a dramatic effect on the ability of CDF to utilize those sites opportunistically. Some sites, for instance, modify the preemption policy from time to time. Tracking these changes requires careful communication with site managers to ensure that the job duration options visible to CDF users are consistent with the preemption policies. CDF has added queue lengths in an attempt to provide this match. The conventional way in which CDF Monte Carlo producers compute their production strategy, however, requires queue lengths that exceed the typical preemption time by a considerable margin. To address this problem, the current submission strategies are being re-examined. A second common impediment to opportunistic usage is the exercise of a preemption policy that kill jobs immediately when a higher priority user submits a job, rather than a more graceful policy that allows completion of the lower priority job within some time frame. CDF has removed all such sites from NAmGrid because the effective job failure rate is too high for users most users to tolerate. This step has significantly reduced the OSG resources available to CDF, which now essentially consists of those sites at which CDF has paid for computing. Clear policies and guidelines on opportunistic usage, publication of the policies in human and computer readable form are needed so that the most efficient use of computing resources may be achieved.
31
•
Infrastructure Reliability and Fault tolerance: Restarts of jobs are not yet completely eliminated. The main cause seems to be individual worker nodes that are faulty and reboot from time to time. While it is possible to blacklist nodes at the portal level, better sensing of these faults and removal at the OSG infrastructural level would be more desirable. We continue to emphasize the importance of stable running and minimization of infrastructure failures so that users can reliably assume that failures are the result of errors in their own processing code, thereby avoiding the need to continually question the infrastructure.
•
Job and Data handling interaction: Job restarts also cause a loss of synchronization between the job handling and data handling. A separate effort is under way to improve the recovery tools within the data handling infrastructure. Tools and design are needed to allow for the Job and data handling to be integrated and to allow fault tolerance for both systems to remain synchronized.
•
Management of input Data resources: The access to data both from databases and data files has caused service problems in the last year. The implementation of opportunistic running on CDFGrid from NAmGrid, coupled with decreased demand on CDFGrid for file access in the post-conference period and a significant demand for new Monte Carlo datasets to be used in the 2010 Winter conference season led to huge demands on the CDF Database infrastructure and grid job failures due to overloading of the databases. This has been traced to complex queries whose computations should be done locally rather than on the server. Modifications in the simulation code were implemented in January 2010. Up to the modification of the code throttling of the Monte Carlo production was required. During the conference crunch in July 2009 and again in March 2010 there was huge demand on the data-handling infrastructure and the 350TB disk cache was being turned over every two weeks. Effectively the files were being moved from tape to disk, being used by jobs and deleted. This in turn led to many FermiGrid worker nodes sitting idle waiting for data. A program to understand the causes of idle computing nodes from this and other sources has been initiated and CDF users are asked to more accurately describe what work they are doing when they submit jobs by filling in qualifiers in the submit command. Pre-staging of files was implemented but further use of file management using SAM is being made default for the ntuple analysis framework. There is a general resource management problem pointed to by this and the database overload issue. Resource requirements of jobs running on OSG should be examined in a more considered way and would benefit from more thought by the community at large.
The usage of OSG for CDF has been fruitful and the ability to add large new resources such as KISTI as well as more moderate resources within a single job submission framework has been extremely useful for CDF. The collaboration has produced significant new results in the last year with the processing of huge data volumes. Significant consolidation of the tools has occurred. In the next year, the collaboration looks forward to a bold computing effort in the push to see evidence for the Higgs boson, a task that will require further innovation in data handling and significant computing resources in order to reprocess the large quantities of Monte Carlo and data needed to achieve the desired improvements in tagging efficiencies. We look forward to another year with high publication rates and interesting discoveries.
32
2.7
Nuclear physics
The STAR experiment has continued the use of data movement capabilities between its established Tier-1 and Tier-2 centers and between BNL and LBNL (Tier-1), Wayne State University and NPI/ASCR in Prague (two fully functional Tier-2 centers). A new center, the Korea Institute of Science and Technology Information (KISTI) has joined the STAR collaboration as a full partnering facility and a resource provider in 2008. Activities surrounding the exploitation of this new potential have taken a large part of STAR’s activity in the 2008/2009 period. The RHIC run 2009 was projected to bring to STAR a fully integrated new data acquisition system with data throughput capabilities going from 100 MB/sec reached in 2004 to1000 MB/sec. This is the second time in the experiment’s lifetime STAR computing has to cope with an order of magnitude growth in data rates. Hence, a threshold in STAR’s Physics program was reached where leveraging all resources across all available sites has become essential to success. Since the resources at KISTI have the potential to absorb up to 20% of the needed cycles for one pass data production in early 2009, efforts were focused on bringing the average data transfer throughput from BNL to KISTI to 1 Gb/sec. It was projected (Section 3.2 of the STAR computing resource planning, “The STAR Computing resource plan”, STAR Notes CSN0474, http://drupal.star.bnl.gov/STAR/starnotes/public/csn0474) that such a rate would sustain the need up to 2010 after which a maximum of 1.5 Gb/sec would cover the currently projected Physics program up to 2015. Thanks to the help from ESNet, Kreonet and collaborators at both end institutions this performance was reached (see http://www.bnl.gov/rhic/news/011309/story2.asp, “From BNL to KISTI: Establishing High Performance Data Transfer From the US to Asia” and 1 http://www.lbl.gov/cs/Archive/news042409c.html, “ESnet Connects STAR to Asian Collaborators”). At this time baseline Grid tools are used and the OSG software stack has not yet been deployed. STAR plans to include a fully automated job processing capability and return of data results using BeStMan/SRM (Berkeley’s implementation of SRM server). Encouraged by the progress on the network tuning for the BNL/KISTI path and driven by the expected data flood from Run-9, the computing team re-addressed all of its network data transfer capabilities, especially between BNL and NERSC and between BNL and MIT. MIT has been a silent Tier-2, a site providing resources for local scientist’s research and R&D work but has not been providing resources to the collaboration as a whole. MIT has been active since the work made on Mac/X-Grid reported in 2006, a well-spent effort which has evolved in leveraging additional standard Linux-based resources. Data samples are routinely transferred between BNL and MIT. The BNL/STAR gatekeepers have all been upgraded and all data transfer services are being re-tuned based on the new topology. Initially planned for the end of 2008, the strengthening of the transfers to/from well-established sites was a delayed milestone (6 months) to the benefit of the BNL/KISTI data transfer. A research activity involving STAR and the computer science department at Prague has been initiated to improve the data management program and network tuning. We are studying and testing a multi-site data transfer paradigm, coordinating movement of datasets to and from multiple locations (sources) in an optimal manner, using a planner taking into account the performance of the network and site. This project relies on the knowledge of file locations at each site and a known network data transfer speed as initial parameters (as data is moved, speed can be reassessed so the system is a self-learning component). The project has already shown impressive gains over a standard peer-to-peer approach for data transfer. Although this activity has so far impacted OSG in a minimal way, we will use the OSG infrastructure to test our implementation 33
and prototyping. To this end, we paid close attention to protocols and concepts used in Caltech’s Fast Data Transfer (FDT) tool as its streaming approach has non-trivial consequence and impact on TCP protocol shortcomings. Design considerations and initial results were presented at the Grid2009 conference and published in the proceedings as “Efficient Multi-Site Data Movement in distributed Environment”. The implementation is not fully ready however and we expect further development in 2010. Our Prague site also presented their previous work on setting up a fully functional Tier-2 site at the CHEP 2009 conference as well as summarized our work on the Scalla/Xrootd and HPSS interaction and how to achieve efficient retrieval of data from mass storage using advanced request queuing techniques based on file location on tape but respecting faire-shareness. The respective asbtract “Setting up Tier-2 site at Golias/ Prague farm” and “Fair-share scheduling algorithm for a tertiary storage system” are available as and http://indico.cern.ch/abstractDisplay.py?abstractId=432&confId=35523 http://indico.cern.ch/abstractDisplay.py?abstractId=431&confId=35523. STAR has continued to use and consolidate the BeStMan/SRM implementation and has continued active discussions, steering and integration of the messaging format from the Center for Enabling Distributed Petascale Science’s (CEDPS) Troubleshooting team, in particular targeting use of BeStMan client/server troubleshooting for faster error and performance anomaly detection and recovery. BeStMan and syslog-ng deployments at NERSC provide early testing of new features in a production environment, especially for logging and recursive directory tree file transfers. At the time of this report, an implementation is available whereas BeStMan based messages are passed to a collector using syslog-ng. Several problems have already been found, leading to strengthening of the product. We hoped to have a case study within months but we are at this time missing a data-mining tool able to correlate (hence detect) complex problems and automatically send alarms. The collected logs have been useful however to, at a single source of information, find and identify problems. STAR has also finished developing its own job tracking and accounting system, a simple approach based on adding tags at each stage of the workflow and collecting the information via recorded database entries and log parsing. The work was presented at the CHEP 2009 conference (“Workflow generator and tracking at the rescue of distributed processing. Automating the handling of STAR's Grid production”, Contribution ID 475, CHEP 2009, http://indico.cern.ch/ contributionDisplay.py?contribId=475&confId= 35523). STAR has also continued an effort to collect information at application level, build and learn from in-house user-centric and workflow-centric monitoring packages. The STAR SBIR TechX/UCM project, aimed to provide a fully integrated User Centric Monitoring (UCM) toolkit, has reached its end-of-funding cycle. The project is being absorbed by STAR personnel aiming to deliver a workable monitoring scheme at application level. The reshaped UCM library has been used in nightly and regression testing to help further development (mainly scalability, security and integration into Grid context). Several components needed reshape as the initial design approach, too complex, slowed down maintenance and upgrade. To this extent, a new SWIG (“Simplified Wrapper and Interface Generator”) based approach was used and reduced the overall size of the interface package by more than an order of magnitude. The knowledge and a working infrastructure based on syslog-ng may very well provide a simple mechanism for merging UCM with CEDPS vision. Furthermore, STAR has developed a workflow analyzer for experimental data production (simulation mainly) and presented the work at the CHEP 2009 conference as “Automation and Quality Assurance of the Production Cycle” (http://indico.cern.ch/abstractDisplay.py?abstractId=475&confId=35523) now accepted for publication. The toolkit developed in this activity allows extracting independent accounting and sta34
tistical information such as task efficiency, percentage success allowing keeping a good record of production made on Grid based operation. Additionally, a job feeder was developed allowing automatic throttling of job submission across multiple site, keeping all site at maximal occupancy but also detecting problems (gatekeeper downtimes and other issues). The feeder has the ability to automatically re-submit failed jobs for at least N times, bringing the overall job success efficiency for only one re-submission to 97% success. This tool was used in the Amazon EC2 exercise (see later section). STAR grid data processing and job handling operations have continued their progression toward a full Grid-based operation relying on the OSG software stack and the OSG Operation Center issue tracker. The STAR operation support team has been efficiently addressing issues and stability. Overall the grid infrastructure stability seems to have increased. To date, STAR has however mainly achieved simulated data production on Grid resources. Since reaching a milestone in 2007, it has become routine to utilize non-STAR dedicated resources from the OSG for the Monte-Carlo event generation pass and to run the full response simulator chain (requiring the whole STAR framework installed) on STAR’s dedicated resources. On the other hand, the relative proportion of processing contributions using non-STAR dedicated resources has been marginal (and mainly on the FermiGrid resources in 2007). This disparity is explained by the fact that the complete STAR software stack and environment, which is difficult to impossible to recreate on arbitrary grid resources, is necessary for full event reconstruction processing and hence, access to generic and opportunistic resources are simply impractical and not matching the realities and needs of running experiments in Physics production mode. In addition, STAR’s science simply cannot suffer the risk of heterogeneous or non-reproducible results due to subtle library or operating system dependencies and the overall workforce involved to ensure seamless results on all platforms exceeds our operational funding profile. Hence, STAR has been a strong advocate for moving toward a model relying on the use of Virtual Machine (see contribution at the OSG booth @ CHEP 2007) and have since closely work, to the extent possible, with the CEDPS Virtualization activity, seeking the benefits of truly opportunistic use of resources by creating a complete pre-packaged environment (with a validated software stack) in which jobs will run. Such approach would allow STAR to run any one of its job workflow (event generation, simulated data reconstruction, embedding, real event reconstruction and even user analysis) while respecting STAR’s policies of reproducibility implemented as complete software stack validation. The technology has huge potential in allowing (beyond a means of reaching non-dedicated sites) software provisioning of Tier-2 centers with minimal workforce to maintain the software stack hence, maximizing the return to investment of Grid technologies. The multitude of combinations and the fast dynamic of changes (OS upgrade and patches) make the reach of the diverse resources available on the OSG, workforce constraining and economically un-viable.
35
Figure 21: Corrected STAR recoil jet distribution vs pT
This activity reached a world-premiere milestone when STAR made used of the Amazon/EC2 resources, using Nimbus Workspace service to carry part of its simulation production and handle a late request. These activities were written up in iSGTW (“Clouds make way for STAR to shine”, http://www.isgtw.org/?pid=1001735, Newsweek (“Number Crunching Made Easy Cloud computing is making high-end computing readily available to researchers in rich and poor nations alike“ http://www.newsweek.com/id/195734), SearchCloudComputing (“Nimbus cloud project saves brainiacs' bacon“ http://searchcloudcomputing.techtarget.com/news/article/ 0,289142,sid201_gci1357548,00.html) and HPCWire (“Nimbus and Cloud Computing Meet STAR Production Demands“ http://www.hpcwire.com/offthewire/Nimbus-and-CloudComputing-Meet-STAR-Production-Demands-42354742.html?page=1). This was the very first time cloud computing had been used in the HENP field for scientific production work with full confidence in the results. The results were presented during a plenary talk at CHEP 2009 conference (Figure 21), and represent a breakthrough in production use of clouds. We are working with the OSG management for the inclusion of this technology into OSG’s program of work. Continuing on this activity in the second half of 2009, STAR has undertaken testing of various models of cloud computing on OSG since the EC2 production run. The model used on EC2 was to deploy a full OSG-like compute element with gatekeeper and worker nodes. Several groups within the OSG offered to assist and implement diverse approaches. The second model, deployed
36
at Clemson University, uses a persistent gatekeeper and having worker nodes launched using a STAR specific VM image. Within the image, a Condor client then register to an external Condor master hence making the whole batch system is completely transparent to the end-user (the instantiated VM appear as just like other nodes, the STAR jobs slide in the VM instances where it finds a fully supported STAR environment and software package). The result is similar to configuring a special batch queue meeting the application requirements contained in the VM image and then many batch jobs can be run in that queue. This model is being used at a few sites in Europe as described at a recent HEPiX meeting, http://indico.cern.ch/conferenceTimeTable.py ?confId=61917. A third model has been preliminarily tested at Wisconsin where the VM image itself acts as the payload of the batch job and is launched for each job. This is similar to the condor glide-in approach and also the pilot-job method where the useful application work is performed after the glide-in or pilot job starts. This particular model is not well matched to the present STAR SUMS workflow as jobs would need to be pulled in the VM instance rather than integrated as a submission via a standard Gatekeeper (the GK interaction only starts instances). However, our MIT team will pursue testing at Wisconsin and attempt to a demonstrator simulation run and measure efficiency and evaluate practicality within this approach. One goal of the testing at Clemson and Wisconsin is to eventually reach a level where scalable performance can be compared with running on traditional clusters. The effort in that direction is helping to identify various technical issues, including configuration of the VM image to match the local batch system (contextualization), considerations for how a particular VM image is selected for a job, policy and security issues concerning the content of the VM image, and how the different models fit different workflow management scenarios. Our experience and results in this cloud/grid integration domain are very encouraging regarding the potential usability and benefits of being able to deploy application specific virtual machines, and also indicate that a concerted effort is necessary in order to address the numerous issues exposed and reach an optimal deployment model. All STAR physics publications acknowledge the resources provided by the OSG. 2.8
MINOS
Over the last three years, computing for MINOS data analysis has greatly expanded to use more of the OSG resources available at Fermilab. The scale of computing has increased from about 50 traditional batch slots to typical user jobs running on over 2,000 cores, with an expectation to expand to about 5,000 cores (over the past 12 months we have used 3.1M hours on OSG from 1.16M submitted jobs). This computing resource, combined with 120 TB of dedicated BlueArc (NFS mounted) file storage, has allowed MINOS to move ahead with traditional and advanced analysis techniques, such as Neural Network, Nearest Neighbor, and Event Library methods. These computing resources are critical as the experiment has moved beyond the early, somewhat simpler Charged Current physics, to more challenging Neutral Current, ν+e, anti-neutrino and other analyses which push the limits of the detector. We use a few hundred cores of offsite computing at collaborating universities for occasional Monte Carlo generation. MINOS was also successful at using TeraGrid resources at TACC in Fall 2009 for a complete pass over our data. MINOS recently made a disappearance νμ measurement (shown at Neutrino 2010) comparing the energy spectra of a neutrino interactions in a near and far target that fits well to a mass difference model.
37
Figure 22: Recent MINOS measurement comparing neutrino rates in a near and far detector.
2.9
Astrophysics
The Dark Energy Survey (DES) used approximately 80,000 hours of OSG resources during the period July 2009 – June 2010 to generate simulated images of galaxies and stars on the sky as would be observed by the survey. The bulk of the simulation activity took place during two production runs, which generated a total of over 7 Terabytes of simulated imaging data for use in testing the DES data management data processing pipelines as part of DES Data Challenge 5 (DC5). The DC5 simulations consist of over 5,000 mock science images, covering some 300 square degrees of the sky, along with nearly another 2000 calibration images needed for data processing. Each 1-GB-sized DES image is produced by a single job on OSG and simulates the 300,000 galaxies and stars on the sky covered in a single 3-square-degree pointing of the DES camera. The processed simulated data are also being actively used by the DES science working groups for development and testing of their science analysis codes. In addition to the main DC5 simulations, we also used OSG resources to produce 1 TB of simulated images for the DES supernova science working group, as well as to produce a number of smaller simulation data sets generated to enable quick turnaround and debugging of the DES data processing pipelines. 2.10
Structural Biology
Biomedical computing Activities supported under the OSG NSF award are complementary to our independent NSF award to support the Research Coordination Network (RCN) for structural biology. While the work carried under the RCN award allowed us to establish the computational workflow, and a grid submission portal, the OSG support allowed us to scale computations to OSG and support an outreach effort to the biomedical community.
38
The SBGrid VO was able to achieve a peak weekly usage of 246,225 hours during the last week of February. During the first four months of 2010 we averaged over 50,000 hours per week. Specifically, the global siMR searches would commonly execute 3000- 4000 concurrent processes at 20+ computing centers, allowing a single global search to complete in less than 24 hours. The key software components for OSG are provided by VDT, Condor, and Globus and provide the basic services, security infrastructure, data, and job management tools necessary to create, submit, and manage computations on OSG. To balance computation time with grid infrastructure overhead it was necessary to set time limits on individual molecular replacement instances (typically 30 minutes), and also to group instances into sub-sets to have grid jobs that required 0.5-12 hours to complete. Scheduling of jobs to sites was managed through a combination of Condor DAGMan and the OSG Match Maker. To reduce network traffic at the job source, the necessary applications and common data (e.g. SCOPCLEAN corpus) were pre-staged to each computing center. Maintenance systems ensure these stay up to date. Individual job execution was handled by a wrapper that configures the system environment appropriately and retrieves any job-specific files, such as the reflection data or pre-placed structures (for second and subsequent round searches on the same structure). Although both Condor and DAGMan provide mechanisms for error recovery it was still typically the case that 1-5% of results would not be returned from a particular search, due to various forms of failure. Even these failure rates were only achieved after initial experience of >50% job failure rate, and the consequent introduction of system tuning and fault tolerance mechanisms. A semi-automated mechanism was developed to retry any missing results until >99.8% of results were available. All results were then aggregated, filtered, and sorted, then augmented with results from other searches (such as TMAlign comparison, Reforigin placement, or Molrep), and with “static” data related to each individual SCOP domain (such as the SCOP class, the domain size, or the domain description). This process resulted in large tabular data sets that could be processed into reports or analyzed with the assistance of visualization software. In accordance with the most recent OSG recommendation the SBGrid VO is transitioning job submission system to a pilot mechanism. We have interlinked our job submission setup with the OSG GlideinWMS factory in San Diego, and reconfigured our DAGMAN workflows. We are currently finetuning job submission rates, and will try to replicate and surpass peak utilization of 4000 concurrent CPUs that was previously achieved with OSGMM system (current peak utilization with GlideinWMS is 1000 CPUs). Our early experience with GlideinWMS is very positive, and in comparison to OSGMM we find the system easier to configure and manage. We have also engaged in activities of the newly established Biomed HPC Collaborative. The initiative aims to coordinate efforts of High Performance Biomedical Computing groups from Boston area (participants include Beth Israel Deaconess Medical Center, Boston University, Brown University, Dana Farber Cancer Institute, Harvard and several affiliated schools, Northeastern University, Partners Healthcare, The Broad Institute, Tufts University, University of Massachusetts, University of Connecticut Health Center and Wyss Institute for Biologically Inspired Engineering). SBGrid RCN has been providing guidance on Open Science Grid integration, and in collaboration with the OSG we have seeded a supporting initiative to interlink existing biomedical resources in the Boston area. Biomedical computing examples
39
SBGrid VO has deployed a range of applications onto over a dozen OSG sites. The primary computing technique for structure determination which was developed and implemented by our group is called sequence independent molecular replacement (siMR). The technique will allow structural biologists to determine the 3-D structure of proteins by comparing imaging data from the unknown structure to that of known protein fragments. Typically a data set for an unknown structure is compared to a single set of protein coordinates. SBGrid has developed a technique to do this analysis with 100,000 fragments, requiring between 2000 and 15,000 hours for a single structure study, depending on the exact application and configuration parameters. Our early analysis with Molrep indicated that signals produced by models with very weak sequence identity are, in many cases, too weak to produce meaningful ranking. The study was repeated utilizing Phaser - a maximum likelihood application that requires significant computing resources (searches with individual coordinates take between 2-10 minutes for crystals with a single molecule in an asymmetric unit, and longer for molecules with many copies of the same molecule). With Phaser a significant improvement in sensitivity of global molecular replacement was achieved and we have recently identified several very encouraging cases. A Phaser analysis example is shown in Figure 23.
Figure 23: Global Molecular replacement. A - after searching with 100,000 SCOP domains four models form a distinct cluster (highlighted). B - one of the SCOP domain in the cluster (teal) superimposes well with 2VZF coordinates (grey), although sequence identity between two structures is minimal. C - SBGrid molecular replacement portal deploys computations to OSG resources. Typical runtime for an individual search with a single SCOP domain is 10 minutes.
In order to validate the siMR approach we have develop a set of utilities which can verify correct placement of models, while correcting for symmetry and origin-shift deviations. We find that while Phaser LLG and TFZ scores combine to provide good discrimination of clusters, other measures such as R-factor improvement or contrast as provided by Molrep are not suitable for a robust cross-model comparison. We can further augment the sensitivity of the Phaser scoring function by incorporating additional dimensions, such as rotation function Z-score (RFZ).
40
The sequence independent molecular replacement approach has been validated on several cases, and a publication describing our method has been now submitted to the journal PNAS. Several members of our community tested the siMR method: • • • • •
Prof. Karin Reinisch -Yale University Prof. Ben Spiller * - Vanderbilt University Prof. Amir Khan * - Trinity College Dublin Jawdat Al-Bassan * - Harvard Medical School Uhn-Soo Cho * - Harvard Medical School
Cases with a star denote examples where grid computing provided strong results which immediately impacted research in user's laboratory. Typically, users run siMR through the Open Science Grid portal (average case: 12,000 CPU hours, 90,000 individual jobs). Further dissemination of siMR is pending awaiting publication of our method, although we expect that several collaborating laboratories will be testing our portal in the near future. Outreach EMBO Practical Course: in collaboration with a structural biology scientist (Daniel Panne) from the European Molecular Biology Organization (EMBO) Piotr Sliz (PI) organized a 2010 EMBO Practical Course in Heildelberg, Germany. The Practical Course maintained the format of the SBGrid Computing School: three nanocourses in Python, Molecular Visualization and OSX Programming were offered. A special lecture on grid computing in HPC was delivered by a member of OSG Consortium, John McGee. Presentations by members of SBGrid VO: •
July 28, 2009 - Web portal interfaces to HPC and collaborative e-science environments (Ian Stokes-Rees)
•
October 6, 2009 - Harvard Computer Society: "The web, the grid, and the cloud: intersecting technologies for computationally intensive science"
•
October 13 2009 - Open Grid Forum 27: "e-Infrastructure Interoperability: A perspective from structural biology"
•
December 7 2009 - Center for Research on Computation and Society, School of Engineering and Applied Sciences: "Security related challenges for collaborative e-Science and Federated Cyberinfrastructure".
•
February 16 2009 - Scientific Software Development Workshop (invited paper and speaker) "Development, deployment, and operation of a life sciences computational grid environment".
•
March 10 2009 - Open Science Grid All Hands Meeting 2010: "Global molecular replacement for protein structure determination".
•
May 3rd, 2009 – Leibniz-Institut fur Molekulare Pharmakologie, Berlin, Germany. Structural Biology on the Grid: Sequence Independent Molecular Replacement.
•
May 7th, 2010 – EMBO Course on Scientific Programming and Data Visualization. Coordinated computing in structural biology. 41
•
May 11th, 2010 – Rutherford Appleton Laboratory, Oxfordshire, UK. Structural Biology on the Grid: Sequence Independent Molecular Replacement.
•
May 12th, 2010 – Structural Genomics Consortium, University of Oxford, UK. X-ray structure determination by global sequence-independent molecular replacement.
2.11
Multi-Disciplinary Sciences
The Engagement team has worked directly with researchers in the areas of: biochemistry (Xu), molecular replacement (PRAGMA), molecular simulation (Schultz), genetics (Wilhelmsen), information retrieval (Blake), economics, mathematical finance (Buttimer), computer science (Feng), industrial engineering (Kurz), and weather modeling (Etherton). The computational biology team led by Jinbo Xu of the Toyota Technological Institute at Chicago uses the OSG for production simulations on an ongoing basis. Their protein prediction software, RAPTOR, is likely to be one of the top three such programs worldwide. A chemist from the NYSGrid VO using several thousand CPU hours a day sustained as part of the modeling of virial coefficients of water. During the past six months a collaborative task force between the Structural Biology Grid (computation group at Harvard) and OSG has resulted in porting of their applications to run across multiple sites on the OSG. They are planning to publish science based on production runs over the past few months. 2.12
Computer Science Research
OSG contributes to the field of Computer Science via research in job management systems and security frameworks for a grid-style cyber-infrastructure. We expect this work to have near-term impact to OSG but also be extensible to other distributed computing models. A collaboration between the Condor project, US ATLAS, and US CMS is using the OSG to test new workload and job management scenarios which provide “just-in-time” scheduling across the OSG sites; this uses “glide-in” methods to schedule a pilot job locally at a site which then requests user jobs for execution as and when resources are available. This approach has many advantages over the traditional, "push" model, including better resource utilization, reduced error rates and better user prioritization. However, glideins introduce new challenges, like two-tiered matching, two-tiered authorization model, network connectivity, and scalability. The two-tiered matching is being addressed within the glideinWMS project sponsored by US CMS. The twotiered authorization is addressed by the gLExec component developed in Europe by NIKHEF, and partially supported by Fermilab for OSG. The network and scalability issues are being addressed by Condor. Cybersecurity is a growing concern, especially in computing grids, where attack propagation is possible because of prevalent collaborations among thousands of users and hundreds of institutions. The collaboration rules that typically govern large science experiments as well as social networks of scientists span across the institutional security boundaries. A common concern is that the increased openness may allow malicious attackers to spread more readily around the grid. Mine Altunay of OSG Security team collaborated with Sven Leyffer and Zhen Xie of Argonne National Laboratory and Jeffrey Linderoth of University of Wisconsin-Madison to study this problem by combining techniques from computer security and optimization areas. The team framed their research question as how to optimally respond to attacks in open grid environments. To understand how attacks spread, they used OSG infrastructure as a testbed. They developed a
42
novel collaboration model observed in the grid and a threat model that is built upon the collaboration model. This work is novel in that the threat model takes social collaborations into account while calculating the risk associated with a participant during the lifetime of the collaboration. The researchers again used OSG testbed for developing optimal response models (e.g. shutting down a site vs. blocking some users preemptively) for simulated attacks. The results of this work has been presented at SIAM Annual Conference 2010 at Denver, Colorado and also submitted to the Journal of Computer Networks.
43
3.
Development of the OSG Distributed Infrastructure
3.1 Usage of the OSG Facility The OSG facility provides the platform that enables production by the science stakeholders; this includes operational capabilities, security, software, integration, testing, packaging and documentation as well as engagement capabilities and support. We are continuing our focus on providing stable and reliable production level capabilities that the OSG science stakeholders can depend on for their computing work and get timely support when needed. The stakeholders continue to increase their use of OSG. The two largest experiments, ATLAS and CMS, after performing a series of data processing exercises last year that thoroughly vetted the end-to-end architecture, were ready to meet the challenge of data taking that began in February 2010. The OSG infrastructure has demonstrated that is up to the challenge and continues to meet the needs of the stakeholders. Currently over 1 Petabyte of data is transferred nearly every day and more than 4 million jobs complete each week.
Figure 24: OSG facility usage vs. time broken down by VO
During the last year, the usage of OSG resources by VOs increased from about 4.5M hours per week to about 6M hours per week; additional detail is provided in the attachment entitled “Production on the OSG.” OSG provides an infrastructure that supports a broad scope of scientific research activities, including the major physics collaborations, nanoscience, biological sciences, applied mathematics, engineering, and computer science. Most of the current usage continues to be in the area of physics but non-physics use of OSG is a growth area with current usage exceeding 200K hours per week (averaged over the last year) spread over 17 VOs.
44
Figure 25: OSG facility usage vs. time broken down by Site. (Other represents the summation of all other “smaller” sites)
With about 80 sites, the production provided on OSG resources continues to grow; the usage varies depending on the needs of the stakeholders. During stable normal operations, OSG provides approximately 850K CPU wall clock hours a day with peaks occasionally exceeding 1 M CPU wall clock hours a day; approximately 200K opportunistic wall clock hours are available on a daily basis for resource sharing. In addition, OSG continues to provide significant effort and technical planning devoted to enabling the large influx of CMS (~20 new) and Atlas (~20 new) Tier-3 sites that have been funded and will be coming online in the second half of 2010. These ~40 Tier-3 sites are notable since many of their administrators are not expected to have formal computer science training and thus special frameworks are needed to provide effective and productive environments. To support these sites (in collaboration with Atlas and CMS), OSG has been focused on creating both documentation as well as a support structure suitable for these sites. To date the effort has addressed: •
Onsite help and hands-on assistance to the ATLAS and CMS Tier-3 coordinators in setting up their Tier-3 test sites including several multi-day meetings to bring together the OSG experts needed to answer and document specific issues relevant to the Tier-3s. OSG hosts regular meetings with these coordinators as well to discuss issues and plan steps forward.
•
OSG packaging and support for Tier-3 components such as Xrootd that are projected to be installed at over half of the Tier-3 sites (primarily ATLAS sites). This includes testing and working closely with the Xrootd development team via bi-weekly meetings.
•
OSG support for the Canadian and WLCG clients that have been selected as the mechanism for deploying ATLAS software at T3 sites. This involves adding features to the VDT to meet
45
the ATLAS requirement of strict versioning, as well as features to the WLCG Client tool to support specific directory and log file changes to support ATLAS. •
Many OSG workshops have been updated to draw in the smaller sites by incorporating tutorials and detailed instruction. A site admins workshop is currently planned for August 2010. One new feature we will be adding is a tutorial on the Hadoop file system.
•
OSG documentation for Tier-3s has been extended to support T3s beginning with installation on the bare hardware. Sections for site planning, file system setup, basic networking instructions, and cluster setup and configuration are being updated and maintained together with more detailed explanations of each step (https://twiki.grid.iu.edu/bin/view/Tier3/WebHome). This documentation is used directly by CMS, and serves as the reference documentation that was used by ATLAS to develop more specific documentation for their T3s.
•
OSG is working directly with new CMS and ATLAS site administrators as they start to deploy their sites, in particular security. We have made arrangements to work directly with local site administrators to work through security issues and barriers that many T3 sites are beginning to encounter as they attempt to setup T3 sites for the first time.
•
The OSG Security team is in the process of setting up a PAKITI server that will centrally monitor and enable all the CMS sites to find security loopholes.
•
Regular site meetings geared toward Tier-3s in conjunction with the ongoing site coordination effort including office hours held three times every week to discuss issues that arise involving all aspects of the sites.
In summary OSG has demonstrated that it is meeting the needs of US CMS and US ATLAS stakeholders at all Tier-1’s, Tier-2’s, and Tier-3’s, and is successfully managing the uptick in job submissions and data movement now that LHC data taking has resumed in 2010. 3.2
Middleware/Software
To enable a stable and reliable production platform, the middleware/software effort has increased focus on support and capabilities that improve administration, upgrades, and support. Between July 2009 and June 2010, OSG’s software efforts focused on developing, releasing, and supporting OSG 1.2, a new focus on native packaging, and supporting the upcoming LHC Tier-3 sites. As in all major software distributions, significant effort must be given to ongoing support. In early 2009, we developed OSG 1.2 with a focus on improving our ability to ship small, incremental updates to the software stack. Our goal was to release OSG 1.2 before the restart of the LHC so that sites could install the new version. We had a pre-release in June 2009, and it was formally released in July 2009, which gave sites sufficient time to upgrade if they chose to do so, and roughly nearly all of the sites in OSG have done so; since the initial pre-release in June we have released 17 software updates to the software stack. Most of the software updates to the OSG software stack were “standard” updates spanning general bug fixes, security fixes, and occasional minor feature upgrades. This general maintenance consumes roughly 50% of the effort of the OSG software effort. There have been several software updates and events in the last year that are worthy of deeper discussion. As background, the OSG software stack is based on the VDT grid software distribu-
46
tion. The VDT is grid-agnostic and used by several grid projects including OSG, TeraGrid, and WLCG. The OSG software stack is the VDT with the addition of OSG-specific configuration. •
OSG 1.2 was released in July 2009. Not only did it significantly improve our ability to provide updates to users, but it also added support for a new operating system (Debian 5), which is required by LIGO.
•
Since summer 2009 we have been focusing on the needs of the upcoming ATLAS and CMS Tier-3 sites. In particular, we have focused on Tier-3 support, particularly with respect to new storage solutions. We have improved our packaging, testing, and releasing of BeStMan, Xrootd, and Hadoop, which are a large part of our set of storage solutions. We have released several iterations of these, and are now finalizing our support for ATLAS Tier-3 sites.
•
We have emphasized improving our storage solutions in OSG. This is partly for the Tier-3 effort mentioned in the previous item, but is also for broader use in OSG. For example, we have created new testbeds for Xrootd and Hadoop and expanded our test suite to ensure that the storage software we support and release are well tested and understood internally. We have started regular meetings with the Xrootd developers and ATLAS to make sure that we understand how development is proceeding and what changes are needed. We have also provided new tools to help users query our information system for discovering information about deployed storage systems, which has traditionally been hard in OSG. We expect these tools to be particularly useful to LIGO and SCEC, though other VOs will likely benefit as well. We also conducted an in-person storage forum in June 2009 at Fermilab, to help us better understand the needs of our users and to directly connect them with storage experts.
•
We have begun intense efforts to provide the OSG software stack as so-called “native packages” (e.g. RPM on Red Hat Enterprise Linux). With the release of OSG 1.2, we have pushed the packaging abilities of our infrastructure (based on Pacman) as far as we can. While our established users are willing to use Pacman, there has been a steady pressure to package software in a way that is more similar to how they get software from their OS vendors. With the emergence of Tier-3s, this effort has become more important because system administrators at Tier-3s are often less experienced and have less time to devote to managing their OSG sites. We have wanted to support native packages for some time, but have not had the effort to do so, due to other priorities; but it has become clear that we must do this now. We initially focused on the needs of the LIGO experiment and in April 2010 we shipped to them a complete set of native packages for both CentOS 5 and Debian 5 (which have different packaging systems), and they are now in production. The LIGO packages are a small subset of the entire OSG software stack, and we are now phasing in complete support for native packages across the OSG software stack. We hope to have usable support for a larger subset of our software stack by Fall 2010.
•
We have added a new software component to the OSG software stack, the gLite FTS (File Transfer Service) client, which is needed by both CMS and ATLAS.
•
We have expanded our ability to do accounting across OSG by implementing mechanisms that account file transfer statistics and storage space utilization.
The VDT continues to be used by external collaborators. EGEE/WLCG uses portions of VDT (particularly Condor, Globus, UberFTP, and MyProxy). The VDT team maintains close contact with EGEE/WLCG via the OSG Software Coordinator's (Alain Roy's) weekly attendance at the 47
EGEE Engineering Management Team's phone call. EGEE is now transitioning to EGI, and we are closely monitoring this change. TeraGrid and OSG continue to maintain a base level of interoperability by sharing a code base for Globus, which is a release of Globus, patched for OSG and TeraGrid’s needs. The VDT software and storage coordinators (Alain Roy and Tanya Levshina) are members of the WLCG Technical Forum, which is addressing ongoing problems, needs and evolution of the WLCG infrastructure in the face of data taking. 3.3
Operations
The OSG Operations team provides the central point for operational support for the Open Science Grid and provides the coordination for various distributed OSG services. OSG Operations performs real time monitoring of OSG resources, supports users, developers and system administrators, maintains critical grid infrastructure services, provides incident response, and acts as a communication hub. The primary goals of the OSG Operations group are: supporting and strengthening the autonomous OSG resources, building operational relationships with peering grids, providing reliable grid infrastructure services, ensuring timely action and tracking of operational issues, and assuring quick response to security incidents. In the last year, OSG Operations continued to provide the OSG with a reliable facility infrastructure while at the same time improving services to offer more robust tools to the OSG stakeholders. OSG Operations is actively supporting the LHC re-start and we continue to refine and improve our capabilities for these stakeholders. We have supported the additional load of the LHC startup by increasing the number of support staff and implementing an ITIL based (Information Technology Infrastructure Library) change management procedure. As OSG Operations supports the LHC data-taking phase, we have set high expectations for service reliability and stability of existing and new services. During the last year, the OSG Operations continued to provide and improve tools and services for the OSG: •
Ticket Exchange mechanisms were updated with the WLCG GGUS system and the ATLAS RT system to use a more reliable web services interface. The previous email based system was unreliable and often required manual intervention to ensure correct communication was achieved. Using the new mechanisms, tickets opened by the WLCG are in the hands of the responsible ATLAS representative within 5 minutes of being reported.
•
The OSG Operations Support Desk regularly responds to ~150 OSG user tickets per month of which 94% are closed within 30 days of being reported.
•
A change management plan was developed, reviewed, and adopted to insure service stability during WLCG data taking.
•
The BDII (Berkeley Database Information Index), which is critical to CMS and ATLAS production, is now functioning with an approved Service Level Agreement (SLA) which was reviewed and approved by the affected VOs and the OSG Executive Board. The BDII performance has been at 99.89% availability and 99.99% reliability during the 8 preceding months.
•
The MyOSG system was ported to MyEGEE and MyWLCG. MyOSG allows administrative, monitoring, information, validation and accounting services to be displayed within a single user defined interface.
48
•
Using Apache Active Messaging Queue (Active MQ) we have provided WLCG with availability and reliability metrics.
•
The public ticket interface to OSG issues was continually updated to add requested features aimed at meeting the needs of the OSG users.
•
We have increased focus and effort toward completing the SLAs for all Operational services including those services distributed outside of the Open Science Grid Operations Center (GOC) at Indiana University.
And we continued our efforts to improve service availability via the completion of several hardware and service upgrades: •
The GOC services located at Indiana University were moved to a new more robust physical environment, providing much more reliable power and network stability.
•
Monitoring of OSG Resources at the CERN BDII was implemented to allow end-to-end information system data flow to be tracked and alarmed on when necessary.
•
A migration to a virtual machine environment for many services is now complete to allow flexibility in providing high availability services.
Service reliability for GOC services remains excellent and we now gather metrics that can quantify the reliability of these services with respect to the requirements provided in the Service Level Agreements (SLAs). SLAs have been finalized for the BDII and MyOSG, while SLAs for the OSG software cache and RSV are being reviewed by stakeholders. Regular release schedules for all GOC services have been implemented to enhance user testing and regularity of software release cycles for OSG Operations provided services. It is the goal of OSG Operations to provide excellent support and stable distributed core services the OSG community can continue to rely upon and to decrease the possibility of unexpected events interfering with user workflow. 3.4
Integration and Site Coordination
The OSG Integration and Sites Coordination activity continues to play a central role in helping improve the quality of grid software releases prior to deployment on the OSG and in helping sites deploy and operate OSG services thereby achieving greater success in production. For this purpose we continued to operate the Validation (VTB) and Integration Test Beds (ITB) in support of updates to the OSG software stack that include compute and storage element services. In addition to these activities there were three key areas of focus involving sites and integration undertaken in the past year: 1) provisioning infrastructure and training materials in support of two major OSG sponsored workshops in Colombia, as part of the launch of the Grid Colombia National Grid Infrastructure (NGI) program; 2) deployment of an automated workflow system for validating compute sites in the ITB; and 3) directed support for OSG sites, in particular activities targeted for the ramp-up of U.S. LHC Tier-3 centers. We also sponsored a Campus Grids workshop to take stock of challenges and best practices for connecting to scientific computing resources at the campus level. The Grid Colombia workshops – one held in October 2009 (a two-week affair), the other in March 2010 (one week) – were supported by OSG core staff and provided contributions from the OSG community at large to help that project launch its NGI. For this purpose we developed a new set of installation guides for setting up central grid (GOC-like) services including information and resource selection services. Supporting this were deployed instances of these central 49
services on OSG provided reference platforms that could be use to support workshop grid building exercises, demonstrate application workflows, and to provide instruction for building grid infrastructure to the workshop participants. We also developed training manuals for building grid sites to be used during the workshop to build a prototype, multi-site, self-contained grid. This work was later re-used in support of Tier-3 facilities. A major new initiative, begun in 2009, was launched to improve the effectiveness of OSG release validation on the ITB. The idea was to automate functional testing where possible, put immediate testing power for realistic workloads directly into the hands of ITB site administrators, and to provide for larger-scale workload generation complete with real-time and archival monitoring so that high level summaries of the validation process could be reported to various VO managers and other interested parties. The system has a suite of test jobs that can be executed through the pilot-based Panda workflow system. The test jobs can be of any type and flavor; the current set includes simple ‘hello world’ jobs, jobs that are CPU-intensive, and jobs that exercise access to/from the associated storage element of the CE. Importantly, ITB site administrators are provided a command line tool they can use to inject jobs aimed for their site into the system, and then monitor the results using the full monitoring framework (pilot and Condor-G logs, job metadata, etc) for debugging and validation at the job-level. In the future, we envision that additional workloads will be executed by the system, simulating components of VO workloads. As new Tier-3 facilities come online we are finding new challenges to supporting systems administrators. Often Tier-3 administrators are not UNIX computing professionals but postdocs or students working part time on their facility. To better support these sites we have installed a virtualized Tier-3 cluster using the same services and installation techniques that are being developed by the ATLAS and CMS communities. An example is creating very friendly instructions for deploying an Xrootd distributed storage system. Finally in terms of site support we continue to interact with the community of OSG sites using the persistent chat room (“Campfire”) which has now been in regular operation for nearly 18 months. We offer three hour sessions at least three days a week where OSG core Integration or Sites support staff are available to discuss issues, troubleshoot problems or simply “chat” regarding OSG specific issues; these sessions are archived and searchable. 3.5
Virtual Organizations Group
The Virtual Organizations (VO) Group coordinates and supports the portfolio of the “at-large” Science VOs in OSG except for the three major stakeholders (ATLAS, CMS, and LIGO) which are directly supported by the OSG Executive Team. At various times through the year, science communities were provided assistance in planning their use of OSG. Direct input was gathered from nearly 20 at-large VOs and reported to the OSG Council. This collaborative community interface (https://twiki.grid.iu.edu/bin/view /VirtualOrganizations/ Stakeholder_PlansNeedsRequirements) provided a roadmap of intended use to enable OSG to better support the strategic goals of the science stakeholders; these roadmaps covered: scope of use; VO mission; average and peak grid utilization quantifiers; extrapolative production estimates; resource provisioning scales; and plans, needs, milestones. We continued our efforts to strengthen the effective use of OSG by VOs. D0 increased to 7585% efficiency at 80K-120K hours/day and this contributed to new levels of D0 Monte Carlo production, reaching a new peak of 13.3 million events per week. CDF undertook readiness ex50
ercises to prepare for CentOS 5 and Kerberos CA transitions. The Fermilab VO with its wide array of more than 12 individual sub-VOs, continued efficient operations. SBGrid sustained an increased scale of production after a successful startup; this VO ramped up its concurrent job runs and reached peak usage at 70,000 hours/day in burst-mode and efforts are ongoing to make weekly efficiency more consistent. NanoHUB was provided assistance by NYSGrid in facilitating HUBzero Portal adoption. We conducted a range of activities to jump-start VOs that are new to OSG or looking to increase their leverage of OSG. •
The IceCube Neutrino Observatory was enabled to start grid operations using resources at GLOW and 5 remote sites: 1) their workflow was re-factored with DAGs and glide-ins to access photonic data in parallel mode; 2) introduced splitting of data, caching, and multijob cache access to reduce I/O traffic; and, 3) work is ongoing to move from prototype to production mode.
•
GLUE-X was started up as a VO and is sharing usage of its resources by 5 other VOs in production.
•
NanoHUB sustained its peak scale of 200-700 wall hours/day. Five separate categories of nanotechnology applications are now supported through the NanoHUB portal for jobs routed to OSG resources and the generic HUBzero interface now supports end-user production job execution on OSG sites.
•
GridUNESP in Brazil achieved full functionality with active support from the DOSAR community; end-user MPI applications were submitted through its full regional grid infrastructure and utilized OSG.
•
The GEANT4 Collaboration’s EGEE-based biannual regression-testing production runs were expanded onto the OSG, assisting in its toolkit’s quality releases for BaBar, MINOS, ATLAS, CMS, and LHCb.
•
Molecular dynamics simulations of mutant proteins run by CHARMM group at NHLBI/NIH and JHU were re-established on OSG using PanDA, as part of OSG-VO; and are being scaled up from 5 to 25 sites.
In addition, we continue to actively address plans for scaling up operations of additional communities including CompBioGrid, GPN, and HCC. The weekly VO forum teleconferences (https://twiki.grid.iu.edu/bin/view/VirtualOrganizations /Meetings) were continued to promote regular interaction between representatives of VOs and staff members of OSG. Besides in-depth coverage of issues, these meetings continued to function as a prime avenue for shared exchanges and community building; and the attendance by a broad mix of VOs enables stakeholders assisting each other thus leading to expeditious resolution of operational issues. We encouraged VOs to identify areas of concern, and invited discussion on issues that need improvement such as: lack of dynamic mechanisms to find and benefit from opportunistic storage availability; accounting discrepancies; exit code mismatches in Pilot-based environments; need for real-time job status monitoring; and need for more accuracy in site-level advertisement of heterogeneous sub-cluster parameters. We also recognized the need to develop an end-to-end
51
understanding of pre-emption and eviction policies by sites and VOs. In the coming year, we plan additional efforts to improve these aspects of the OSG infrastructure. 3.6
Engagement and Campus Grids
During this reporting period we have seen continued growth in usage of the Open Science Grid by new users from various science domains as a direct result of the Engagement program. We analyzed and documented our experiences in working with universities to deploy campus level CI based on OSG technology and methodologies, acquired a survey of Engage users by a third party, contributed to OSG Program Management as the area coordinator of the Engagement activities, and contributed to OSG efforts in working with large science projects such as the Southern California Earthquake Center (SCEC) and the Large Synoptic Survey Telescope (LSST).
Figure 26: Two year window of CPU hours per engaged user
The Engage VO use of OSG depicted in Figure 26 represents a number of science domains and projects including: Biochemistry (Zhao, Z.Wang, Choi, Der), theoretical physics (Bass, Peterson), Information and Library Science (Bapat), Mathematics (Betten), Systems Biology (Sun), Mechanical Engineering (Ratnaswamy), RCSB Protein Data Bank (Prlic), Wildlife Research (Kjaer), Electrical Engineering (Y.Liu), Coastal Modeling (Gamiel), and PRAGMA (Androulakis). We note that all usage by Engage staff depicted here is directly related assisting users, and not related to any computational work of the Engage staff themselves. This typically involves running jobs on behalf of users for the first time or after significant changes to test wrapper scripts and probe how the distributed infrastructure will react to the particular user codes. In February 2010, James Howison and Jim Herbsleb of Carnegie Mellon University conducted a survey of the OSG Engagement Program as part of their VOSS SciSoft research project funded by NSF grant number 0943168. The full 17 page report is available upon request, and indicates that the OSG Engagement Program is effective, helpful, and appreciated by the researchers rely52
ing on both the human relationship based assistance and the hosted infrastructure which enables their computational science. Figure 27 demonstrates Engage VO usage per facility spanning both the prior year and current reporting period. Usage this year totals roughly 8.5M CPU hours, representing a 340% increase over the prior year.
Figure 27: Engage VO usage by facility over previous two years
The Campus Grids team’s goal is to include as many US universities as possible in the national cyberinfrastructure. By helping universities understand the value of campus grids and resource sharing through the OSG national framework, this initiative aims at democratizing cyberinfrastructures by providing all resources to users and doing so in a collaborative manner. In the last year, the campus grids team concentrated on developing enabling technologies based on the cloud computing model. Working with STAR, architecture was developed to run jobs from VOs within virtual machines customized by them. Several other groups participated in this effort. The Condor VM group offered opportunistic resources to STAR using the virtual machine universe on the GLOW campus resources. Clemson provided dedicated resources on a test cluster. STAR had previously tested the Nimbus software and run jobs on the EC2 cloud. These efforts will inform the future OSG architecture and pave the way towards integration of the cloud computing model and virtualization in particular. Currently STAR is testing a new VM technology at Clemson, accessing thousands of virtual machines running on the 8,000 cores Palmetto cluster. Some detailed reports are available on the Campus Grid wiki at: https://twiki.grid.iu.edu/bin/view/CampusGrids/WebHome. The campus grid group also studied the CERNVM technology to investigate whether it could be used as a worker node on OSG sites - in particular on Tier-3 sites and campus grids. The conclu-
53
sion is that while CERNVM provides a very nicely package appliance, it is highly geared towards the user’s desktop and not for worker nodes. However the CERNVM team is hard at work to provide a batch instance image that will directly contact the pilot job frameworks. Once such an image is developed and tested, campus grids will be able to use it at worker nodes. A campus grid workshop was organized at University of Chicago with participants from UC, UW-Madison, Clemson, Nebraska, RENCI, LBNL, Buffalo and UCLA. The report is available at: https://twiki.grid.iu.edu/bin/view/CampusGrids/WorkingMeetingFermilab. The report highlights several architectures in use by the most prominent campus grids to date. It also highlights challenges and opportunities to extend the campus grid efforts. This report has been forwarded to the OSG council and shared with the NSF advisory committee on campus integration. Finally, outreach to Western Kentucky University was done with a site visit to introduce the local community to the mode of operations and user engagement strategies of large computing resources. WKU is purchasing a 4,000 core cluster and planning to join OSG. The large local community is being engaged both from an IT perspective and a user perspective. Significant outreach also took place with the South Carolina School for Science and Math (GSSM), a high school with the brightest students in SC. While not a university, this proved very productive as the students brought together a cluster of 30 laptops donated by Google and built a Condor pool. Course content from Dr. Goasguen at Clemson was used to run HTC jobs. GSSM plans to incorporate Condor in its computing curriculum as well as follow Dr. Goasguen undergraduate course on distributed computing remotely. This may prove to be a viable campus grid outreach and training mechanism. 3.7
Security
The Security team continued its multi-faceted approach to successfully meeting the primary goal of maintaining operational security, developing security policies, acquiring or developing necessary security tools and software, and disseminating security knowledge and awareness. During the past year, we focused our efforts on assessing the identity management infrastructure and the future research and technology directions in this area. We have organized three security workshops in collaboration with ESnet Authentication and Trust Fabric Team: 1) Living in an Evolving Identity World Workshop brought technical experts together to discuss security systems; 2) OSG Identity Management Requirements Gathering Workshop brought the VO security contacts together to discuss their security needs and requirements from OSG; and, 3) Security and Virtual Organizations Workshop, held during OSG All Hands Meeting, was a follow-up to the issues identified at the two previous workshops and brought technical experts and VOs together to discuss the current state of the security infrastructure and necessary improvements. We conducted a detailed survey with our VO security contacts to pinpoint the problem areas and the workshop reports and the survey results are available at https://twiki.grid.iu.edu/bin/view/Security/OsgEsnetWorkshopReport. Usability of the identity management system has surfaced as a key element needing attention. Obtaining, storing, and managing certificates by the end user have significant usability challenges and thus easy-to-use tools for the end user are a critical need. A typical end user computer lacks the native built-in support for such functions mainly because the majority of products and vendors do not heavily favor PKI as their security mechanism. For example, Google promotes OpenID and Microsoft integrates Shibboleth/SAML with their products to achieve interorganizational identity management. Moreover, the widespread adoption of the products within 54
the science community makes it inevitable that OSG adopts and integrates diverse security technologies with its existing middleware. We made solving these problems a top priority for ourselves and have started identifying both short-tem and long-term solutions. The short-term plans include quick fixes on the most urgent problems while we are working towards the long-term solutions that can restructure our security infrastructure on the basis of usability and diverse security technologies. We started working with SBGrid since they are very much affected by these problems. Our work with SBgrid since October 2009 has resulted in: a reduced number of user actions necessary to join the SBgrid and to get security credentials; finding support tools that can help with certificate management on a user desktop and browser; and, designing a new web page for certificate procurement process with a number of automated features. We sought feedback from other VOs as well as from SBGrid on these initiatives and the new certificate web site has been welcomed by all OSG VOs. We collaborate with the CILogon project team at NCSA, where they are implementing a Shibboleth-enabled certificate authority. This collaboration will allow us to test with a different technology and also provide easier methods for “InCommon” members to gain access to OSG. During the last 12 months, in the area of operational security, we did not have any incidents specifically targeting the OSG infrastructure. However, we had a few tangential incidents, where a computer in OSG became compromised due to vulnerabilities found in non-grid software, such as ssh scanning attacks. Nevertheless, in order to improve our communication with our community, we started a security blog that we use as a security bulletin board. Our long-term goal is to build a security community, where each site administrator is an active contributor. We provide ATLAS and CMS Tier-3 sites with additional help on system-level security issues, including coverage beyond the grid-specific software. So far, we have identified two high-risk root-level kernel vulnerabilities. Although these vulnerabilities are not specific to our infrastructure or to grid computing, we included them in our advisories. We contacted each Tier-3 site individually and helped them patch and test their systems as there was a clear need for this type of support for Tier-3 sites. However, individualized care for close to 20 sites was not sustainable due to our existing effort level. As a result, we searched for automated monitoring products and found one from CERN; this software checks if a site is vulnerable against a specific threat. Our sites have requested that monitoring results from a specific site should only be accessible to that site staff and thus we worked with CERN to address this concern and recently received the enhanced software. As part of our support of Tier-3 sites, we started weekly phone calls with site administrators. The summaries of our meetings can be found at https://twiki.grid.iu.edu/bin/view/Security/FY10CourtesyCallLog. In the previous year, we had conducted a risk assessment of our infrastructure and completed contingency plans. We continued our work this past year with implementing high-priority contingency plans. A particularly important risk was a breach in the DOEGrids Certificate Authority infrastructure and we built a monitoring tool that enables the OSG team to observe anomalous activities in the DOEGrids CA infrastructure (including behaviors of registration authorities, agents and user). To measure LHC Tier-2’s ability to respond to an ongoing incident, we conducted an incident drill covering all LHC Tier-2 sites and a LIGO site (19 sites in total). Each site has been tested
55
one-on-one; a security team member generated a non-malicious attack and assessed sites responses. In general the sites performed very well with only a handful of sites needing minor assistance or support in implementing the correct response actions. Our next goal is to conduct an incident drill to measure the preparedness of our user community and the VOs. With the help of Barton Miller of University of Wisconsin, we organized a “Secure Programming Workshop” in conjunction with the OGF. The tutorial taught the software providers the secure coding principles with hands-on code evaluation. A similar tutorial was also given at EGEE'09 conference to reach out to European software providers. Miller and his team have conducted a code-level security assessment of OSG Gratia probes and we coordinated the work to fix the identified vulnerabilities. Work on the Grid User Management System (GUMS) for OSG, which provides ID and authorization mapping at the sites, has entered a new stage, focusing on better testing, packaging, ease of maintenance and installation. This will result in synchronizing the upgrade cycle of this product with release cycle of OSG Virtual Data Toolkit (VDT). 3.8
Metrics and Measurements
OSG Metrics and Measurements strive to give OSG management, VOs, and external entities quantitative details of the OSG Consortium’s growth throughout its lifetime. The recent focus was maintenance and stability of existing metrics projects, as well as the new “OSG Display” built for the DOE. The OSG Display (http://display.grid.iu.edu) is a high-level, focused view of several important metrics demonstrating the highlights of the consortium. It is meant to be a communication tool that can provide scientifically-savvy members of the public a feel for what services the OSG provides. We expect to start increasing the visibility of this website in July 2010. The OSG Metrics area converted all of its internal databases and displays to the Gratia accounting system. This removes a considerable amount of legacy databases and code from the OSG’s “ownership,” and consolidates metrics upon one platform. We are currently finishing the transition of all of the metrics web applications to the OSG GOC, eliminating and freeing OSG Metrics effort for other activities. Continued report tasks include metric thumbnails, monthly eJOT reports, and the CPU normalization performance table. This year, we have produced an updated “Science Field” report classifying OSG usage by science field. We envision transitioning these personnel to more developer-oriented tasks throughout the remainder of the project. The new “science field” report was created in response a request from the OSG consortium members, specifically the owners of large sites. It categorizes the large majority of OSG CPU usage by the science domain (physics, biology, computer science, etc) of the application. While this is simple for VOs within a single domain (LHC VOs), it is difficult for community VOs containing many diverse users, such as HCC or Engage. The current solution is a semi-manual process, which we are working on automating. This analysis (Figure 28) shows a dramatic increase in the non-HEP usage over the past 12 months.
56
Figure 28: Monthly non-HEP Wall hours for different science fields
The OSG Metrics team continues to perform as the liaison to the Gratia accounting project, maintaining a close working relationship. We have worked to implement service state accounting, allowing the OSG to record the historical status of batch systems. OSG Operations deployed a new Gratia hardware setup this year, and Metrics has worked to provide feedback on different issues as they have arisen. This collaboration has been important as the LHC turn-on greatly increased the number of transfer records collected; collaborating with the storage area, we have re-written the dCache transfer probe to reduce the number of records necessary by an order of magnitude. Overall, OSG transfer accounting covers more storage element technologies compared to the previous year. For non-accounting data, OSG Metrics has delivered a new monitoring probe to verify the consistency of the OSG Information Services. The new monitoring probe is a piece of the OSG site monitoring framework (RSV). For FY10, we planned to incorporate the network performance monitoring data into our set of metrics, but this item has been delayed due to the new OSG Display deliverable. The Metrics area continues to be heavily involved with the coordination of WLCG-related reporting efforts. Items continued from last year include installed capacity reporting, upgrading of the reports to a new, WLCG-specific, benchmark, and transferring of accounting data from the OSG to the WLCG. Each month, the automated WLCG-related accounting is reviewed by OSG Metrics personnel. 3.9
Extending Science Applications
In addition to operating a facility, the OSG includes a program of work that extends the support of Science Applications both in terms of the complexity as well as the scale of the applications that can be effectively run on the infrastructure. We solicit input from the scientific user com-
57
munity both as it concerns operational experience with the deployed infrastructure, as well as extensions to the functionality of that infrastructure. We identify limitations, and address those with our stakeholders in the science community. In the last year, the high level focus has been threefold: (1) improve the scalability, reliability, and usability as well as our understanding thereof; (2) evaluate new technologies, such as GRAM5 and CREAM, for adoption by OSG; and (3) improve the usability of our Work Load Management systems to enable broader adoption by non-HEP user communities. We continued with our previously established processes designed to understand and address the needs of our primary stakeholders: ATLAS, CMS, and LIGO. The OSG has designated certain members (sometimes called senior account managers) of the executive team to handle the interface to each of these major stakeholders and meet, at least quarterly, with their senior management to go over their issues and needs. Additionally, we document the stakeholder desired worklists from OSG and crossmap these requirements to the OSG WBS; these lists are updated quarterly and serve as a communication method for tracking and reporting on progress. 3.10
Scalability, Reliability, and Usability
As the scale of the hardware that is accessible via the OSG increases, we need to continuously assure that the performance of the middleware is adequate to meet the demands. There were three major goals in this area for the last year and they were met via a close collaboration between developers, user communities, and OSG. •
At the job submission client level, the goal is 30,000 jobs running simultaneously from a single client installation, and achieving in excess of 95% success rate while doing so. The job submission client goals were met in collaboration with CMS, Condor, and DISUN, using glideinWMS. This was done in a controlled environment, using the “overlay grid” for large scale testing on top of the production infrastructure developed the year before. To achieve this goal, Condor was modified to drastically reduce the number of ports used for its operation. The glideinWMS is also used for production activities in several scientific communities, the biggest being CMS, CDF and HCC, where the job success rate has constantly been above the 95% mark.
•
At the storage level, the present goal is to have 50Hz file handling rates with hundreds of clients accessing the same storage area at the same time, and delivering at least 1Gbit/s aggregate data throughput. The BeStMan SRM with HadoopFS has shown to scale to about 100Hz with 2000 clients accessing it concurrently. It can also handle in the order of 1M files at once, with directories containing up to 50K files at once. There was no major progress on the dCache based SRM and we never exceeded 10Hz in our tests. On the throughput front, we achieved a sustained throughput of 15 Gbit/s over wide area network using BeStMan and FTS.
•
At the functionality level, this year’s goal was to evaluate new Gatekeeper technologies in order to replace the Globus preWS Gatekeeper (GRAM2) currently in use on OSG. This is particularly important due to the fact that Globus has deprecated the WS Gatekeeper (GRAM4) that was supposed to be the successor of the preWS method which had been tested in OSG over the past years. The chosen candidates to replace GRAM2 are Globus GRAM5, INFN CREAM and Nordugrid ARC. In the past year, GRAM5 and CREAM have been tested and seem to be a big step forward compared to GRAM2. OSG has taken the initial
58
steps to get these integrated into its software stack. ARC testing has also started, but no results are yet available. In addition, we have continued to work on a number of lower priority objectives: •
A package containing a framework for using Grid resources for performing consistent scalability tests against centralized services, like CEs and SRMs. The intent of this package is to quickly “certify” the performance characteristics of new middleware, a new site, or deployment on new hardware, by using thousands of clients instead of one. Using Grid resources allows us to achieve this, but requires additional synchronization mechanisms to perform in a reliable and repeatable manner.
•
A package to monitor a certain class of processes on the tested nodes. Existing tools typically only measure system wide parameters, while we often need the load due to a specific class of applications. This package offers exactly this functionality in an easy to install fashion.
•
We have been involved in tuning the performance of BeStMan SRM by performing a configuration sweep and measuring the performance at each point.
•
We have been evaluating the capabilities of a commercial tool, namely CycleServer, to both submit to and to monitor Condor pools. Given that Condor is at the base of much of OSG infrastructure, having a commercially supported product could greatly improve the usability of OSG. The evaluation is still in progress.
•
We have been working with CMS to understand the I/O characteristics of CMS analysis jobs. We helped by providing advice and expertise, Changes have been made to all layers of the software stack to improve the management of data I/O and computation. This work has resulted in improved CPU efficiencies of CMS software on OSG sites.
•
We have been involved with other OSG area coordinators in reviewing and improving the user documentation. The resulting improvements are expected to increase the usability of OSG for both the users and the resource providers.
3.11
Workload Management System
As in the previous year, the primary goal of the OSG Workload Management System (WMS) effort was to provide a flexible set of software tools and services for efficient and secure distribution of workload among OSG sites. In addition to two Condor-based suites of software previously utilized in OSG, Panda and glideinWMS, the OSG Match Maker (based directly on Condor) has reached significant usage level. The OSG Match Maker was developed to address the needs of users who need automated resource provisioning across multiple facilities managed by OSG, while using the Condor front-end for job submission. The Panda system continued to be supported by OSG as a crucial infrastructure element of the ATLAS experiment at LHC, as we entered the critically important period of data taking and processing. With more experience in continuously operating Panda itself as well as a suite of monitoring services, we gained better insight into the direction of the Panda monitoring upgrade, choice of technologies and integration options. We have created a prototype of an upgraded Panda monitor based on a modern technology platform (framework-based web service as a data source, and a rich AJAX-capable client). Migration to this application will allow us to greatly reduce the amount of application code, separate data preparation from presentation, facilitate in-
59
tegration with external systems, leverage open source for tools such as authentication and authorization mechanisms, and provide a richer and more dynamic user experience. This reporting period saw continued utilization of Panda by the CHARMM collaboration working in the field of structural biology. We have also completed Panda setup for the BNL research group active in Daya Bay and LBNE (DUSEL) neutrino experiments and have started test runs of their Monte Carlo simulation software. The ITB (Integration Testbed) activity in OSG has benefited from using Panda, which allows site administrators to automate test job submission and monitoring and have test results documented via Panda logging mechanism and transmitted to any specific location for analysis. With the glideinWMS system, we continued stable operation across global large-scale resources of the CMS experiment (with an instance hosted at FNAL), and deployed a newer version capable of serving multiple virtual organizations from a single instance (these include CMS, HCC and GLOW/IceCube) at UCSD. There have been important improvements of the glideinWMS security model, as well as added support for NorduGrid and CREAM. Work continued on improvements in documentation, installation, scalability, diagnostics and monitoring areas. During the reporting period, there was a collaborative effort with the Corral/Pegasus project (workflow management system), and glideinWMS has been deployed by SBGrid team (structural biology). We also continued the maintenance of the gLExec (user ID management software), as a project responsibility. One of the issues we had in the previous reporting period, the lack of awareness of potential entrants to OSG of capabilities and advantages of OSG Workload Managements Systems, was addressed by creation of a document which contains comparative analysis of features and characteristics of the systems used, such as depth of monitoring provided and ease of installation and maintenance. An important challenge that will need to be addressed is the impact of pilot-based systems (with Panda and glideinWMS squarely in this category) on reporting resource usage and this affects accounting and metrics. The problem lies in the pilot and its payload running either concurrently or sequentially, with potentially a few jobs tied to a pilot, and the optional switch of identity while execution takes place (such is the case with gLExec). The work on solution to this accounting problem is starting now. The WMS program will continue to be important for the science community and OSG. First, Workload Management Systems supported by OSG continue to be a key enabling factor for large science projects such as ATLAS and CMS. Second, OSG continues to draw new entrants who benefit greatly by leveraging stable and proven work load management systems for access to opportunistic resources. 3.12
Condor Collaboration
The OSG software platform includes Condor, a high throughput computing system developed by the Condor Project. Condor can manage local clusters of computers, dedicated or cycle scavenged from desktops or other resources, and can manage jobs running on both local clusters and delegated to remote sites via Condor itself, Globus, CREAM, and other systems. The Condor Team collaborates closely with OSG and provides new releases and ongoing technical support for the OSG stakeholders. In addition the Condor team provides collaboration forums for the OSG community to enable shared learning. 60
3.12.1 Release Condor This activity consisted of the ongoing work required to have regular new releases of Condor. Creating quality Condor releases at regular intervals required significant effort. New releases fixed known bugs; supported new operating system releases (porting); supported new versions of dependent system software and hardware; underwent a rigorous quality assurance and development lifecycle process (consisting of a strict source code management, release process, and regression testing); and received updates to the documentation. From July 2009 through May 2010, the Condor team made 10 releases of Condor, with at least one more release planned before the end of June 2010. During this time, the Condor team created and code-reviewed 148 publicly documented bug fixes. Condor ports are maintained and released for 5 non-Linux operating systems as well as 10 ports for different Linux distributions. We continued to invest significant effort to improve our automated test suite in order to find bugs before our users do, and continued our efforts to maximize our leverage of the NMI Build and Test facility and the Metronome framework.1 The number of automated builds we perform via NMI averages over 70 per day, and the ratio of failed builds to successful builds has improved. This allows us to better meet our release schedules by alerting us to problems in the code or a port as early as possible. We currently perform approximately 45,000 tests per day on the current Condor source code snapshot (see Figure 29).
Figure 29: Number of daily automated regression tests performed on the Condor source
In the course of performing our Condor release activities, in a typical month we:
1
•
Released a new version of Condor to the public
•
Performed over 200 commits to the codebase (see Figure 30)
•
Modified over 350 source code files See http://nmi.cs.wisc.edu/ for more about the NMI facility and Metronome. 61
•
Changed over 12,000 lines of code (Condor source code written at UW-Madison now sits about 760,000 lines of code)
•
Compiled about 2,300 builds of the code for testing purposes
•
Ran about 1.3 million regression tests (both functional and unit)
Figure 30: Number of code commits made per month to Condor source repository
3.12.2 Support Condor Users received support directly from project developers by sending email questions or bug reports directly to the
[email protected] email address. All incoming support email was tracked by an email-based ticket tracking system running on servers at UW-Madison. From July 2009 through May 2010, over 1,800 email messages were exchanged between the project and users towards resolving 1,285 support incidents (see Figure 31).
62
Figure 31: Number of tracked incidents and support emails exchanged per month
The Condor team provides on-going support through regular phone conferences and face-to-face meetings with OSG collaborations that use Condor in complex or mission-critical settings. This includes monthly meetings with USCMS, weekly teleconferences with ATLAS, and biweekly teleconferences with LIGO. The Condor team uses an email-based issue tracking system to organize longer term support work; over the last year this system has been used to manage 35 issues with LIGO, resolving 15 of them. The Condor team also uses a web page system to track ongoing issues. This web system is tracking 16 issues associated with ATLAS (of which 10 are resolved), 42 issues associated with CMS (of which 27 are resolved), 51 issues associated with LIGO (of which 21 are resolved), and 9 for other OSG users (of which 6 are resolved). 3.12.3 Condor Week 2010 Event The Condor team organized a four-day meeting “Paradyn/Condor Week 2010” at UW-Madison in April 2010. About 40 presentations, tutorials, and discussions were available to the approximately 100 registered participants. The participants came from academia, government, and industry. For example, presentations were made by attendees from Argonne National Laboratory, Brookhaven National Laboratory, Fermi National Laboratory, Aruba Networks, Bank of America, Cycle Computing, Microsoft, Red Hat, Clemson University, Indiana University, Louisiana State University, Marquette University, Northeastern University, Purdue University, University of California, University of Nebraska, and University of Notre Dame. Talks were given by Condor project members, as well as presentations from members of the Condor user community sharing how they leveraged Condor at their institutions in order to improve the quality and quantity of their computing. The agendas and many of the presentations are available at http://www.cs.wisc.edu/condor/CondorWeek2010/. Condor Week 2010 provided us an opportunity to meet users, learn what they do, and understand what they need from Condor. The event was not widely advertised or promoted, as we aimed it towards keenly engaged members of the Condor community. We did not want enrollment to surpass 120 due to facility limitations and also to keep a level of intimacy during plenary sessions. It was an invaluable experience both for us and the users.
63
3.13
High Throughput Parallel Computing
With the advent of 4- and soon 16-core CPUs packaged in commodity CPU systems, OSG stakeholders have shown an increased interest in computing that combines small scale parallel applications with large scale high throughput capabilities, i.e. ensembles of independent jobs, each using 8 to 64 tightly coupled processes. The OSG “HTPC” program is funded through a separate NSF grant to evolve the technologies, engage new users, and support the deployment and use of these applications. The work is in its early stages. However, there have been some useful application running to date and even submitted publications. The focus of the program has been to: • Bring the MPI and other specific libraries from the client to the remote executive site as part of the job – thus removing the dependence on the different libraries invariably found on different sites. •
Adapt applications to only use the number of cores available on a single CPU.
•
Extend the OSG information services to advertise support for HTPC jobs.
To date chemistry applications have been run across 6 sites – Oklahoma, Clemson, Purdue, Wisconsin, Nebraska and UCSD. The work is being watched closely by the HTC communities who are interested in taking advantage of multi-core while not adding a dependency on MPI. Challenges remain in all the above areas as well as adapting the OSG accounting, troubleshooting and monitoring systems to work well with this new job paradigm.
Figure 32: Chemistry Usage of HPTC (reported at May 2010 Condor meeting)
3.14
Internet2 Joint Activities
Internet2 collaborates with OSG to develop and support a suite of tools and services that make it easier for OSG sites to support its widely distributed user community. Identifying and resolving performance problems continues to be a major challenge for OSG site administrators. A complication in resolving these problems is that lower than expected performance can be caused by 64
problems in the network infrastructure, the host configuration, or the application behavior. Advanced tools can quickly isolate problem(s) and will go a long way toward improving the grid user experience and making grids more useful to science communities. In the past year, Internet2 has worked with OSG software developers to update the advanced network diagnostic tools already included in the VDT software package. These client applications allow VDT users to verify the network performance between end site locations and perfSONAR-based servers deployed on the Internet2 and ESnet backbones by allowing on-demand diagnostic tests to be run. The tools enable OSG site administrators and end users to test any individual compute or storage element in the OSG environment thereby reducing the time it takes to diagnose performance problems. They allow site administrators to more quickly determine if a performance problem is due to network specific problems, host configuration issues, or application behavior. In addition to deploying client tools via the VDT, Internet2 staff, working with partners in the US and internationally, have continued to support and enhance a simple live-CD distribution mechanism for the server side of these tools (perfSONAR-Performance-Toolkit). This bootable CD allows an OSG site-admin to quickly stand up a perfSONAR-based server to support the OSG users. These perfSONAR hosts automatically register their existence in a distributed global database, making it easy to find new servers as they become available. These servers provide two important functions for the OSG site-administrators. First they provide an end point for the client tools deployed via the VDT package. OSG users and siteadministrators can run on-demand tests to begin troubleshooting performance problems. The second function they perform is to host regularly scheduled tests between peer sites. This allows a site to continuously monitor the network performance between itself and the peer sites of interest. The US-ATLAS community has deployed perfSONAR hosts and is currently using them to monitor network performance between the Tier-1 and Tier-2 sites. Internet2 attends weekly USATLAS calls to provide on-going support of these deployments, and has come out with regular bug fixes. Finally, on-demand testing and regular monitoring can be performed to both peer sites and the Internet2 or ESNet backbone network using either the client tools, or the perfSONAR servers. Internet2 will continue to interact with the OSG admin community to learn ways to improve this distribution mechanism. Another key task for Internet2 is to provide training on the installation and use of these tools and services. In the past year Internet2 has participated in several OSG site-admin workshops, the annual OSG all-hands meeting, and interacted directly with the LHC community to determine how the tools are being used and what improvements are required. Internet2 has provided handson training in the use of the client tools, including the command syntax and interpreting the test results. Internet2 has also provided training in the setup and configuration of the perfSONAR server, allowing site-administrators to quickly bring up their server. Finally, Internet2 staff has participated in several troubleshooting exercises; this includes running tests, interpreting the test results and guiding the OSG site-admin through the troubleshooting process. 3.15
ESNET Joint Activities
OSG depends on ESnet for the network fabric over which data is transferred to and from the Laboratories and to/from LIGO Caltech (by specific MOU). ESnet is part of the collaboration delivering and supporting the perfSONAR tools that are now in the VDT distribution. OSG makes significant use of ESnet’s collaborative tools with telephone and video meetings. And 65
ESnet and OSG are planning the collaborative testing of the 100Gigabit network testbed as it becomes available in the future. OSG is the major user of the ESnet DOE Grids Certificate Authority for the issuing of X509 digital identity certificates for most people and services participating in OSG. Registration, renewal and revocation are done through the OSG Registration Authority (RA) and ESnet provided web interfaces. ESnet and OSG collaborate on the user interface tools needed by the OSG stakeholders for management and reporting of certificates. The numeric distribution for the currently valid DOEGrids certificates across institutions is shown below: Community All .edu .gov Other
Personal Certificates Number of certificates 2825 1369 907 549
Service Certificates 8308 3211 4749 348
.edu .gov
Number of institutions 140 15
66 10
OSG and ESnet are implementing features of COO and contingency plans to make certificates and CA/RA operations more robust and reliable by replication and monitoring. We also partner as members of the identity management accreditation bodies in America (TAGPMA) and globally (International Grid Trust Federation, IGTF). OSG and ESnet jointly organized a workshop on Identity Management in November 2009 with two complementary goals: (1) To look broadly at the identity management landscape and evolving trends regarding identity in the web arena and (2) to gather input and requirements from the OSG communities about their current issues and expected future needs. A main result of the analysis of web-based technologies is the ability to delegate responsibility, which is essential for grid computing, is just beginning to be a feature of web technologies and still just for interactive timescales. A significant result from gathering input from the users is that the communities tend to either be satisfied with the current identity management functionality or are dissatisfied with the present functionality and see a strong need to have more fully integrated identity handling across the range of collaborative services used for their scientific research. The results of this workshop and requirements gathering survey are being used to help plan the future directions for work in this area with two main thrusts being improvements to the registration process and closer integration between web and command line services. More details are included in the Security section of this report. There is currently an effort underway lead by Mike Helm of ESnet to help and encourage DOE laboratories to use the Shibboleth identity federation technology and to join the InCommon Federation as a way to provide more efficient and secure network access to scientific facilities for the widely distributed user communities located at universities as well as laboratories. Technical discussions of issues particular to DOE laboratories are carried out on the Science Federation
66
Google group as well as a demonstration collaborative web space at confluence.scifed.org. This activity is of great interest to OSG as it leads to the next stage in the evolution of secure network identity credentials.
67
4. 4.1
Training, Outreach and Dissemination Training and Content Management
Starting in August 2009, OSG began an evolution of its Education and Training area from a combination of general grid technology education and more specific training toward a focus on training in creation and use of grid technologies. This shift and reduction in scope in education and training was undertaken to reduce the staffing level and to accommodate an increase in effort needed to improve the quality and content relevancy of our documentation and training material. Consistent with the change in focus, Content Management has been added to the area with the realization that improved documentation forms the basis for improved training. A study of OSG documentation involved interviews of 29 users and providers of OSG documentation and subsequent analysis of the results to produce recommendations for improvement. These study recommendations were reviewed and analyzed in a two-day workshop of relevant experts. The implementation of those recommendations began in October, 2009 and is continuing through 2010. The Content Management project has defined a collaborative process for managing production of high-quality documentation, defined document standards and templates, reviewed and identified documents for improvement or elimination, and begun reorganization of documentation access by user role. The new process includes ownership of each document, a formal review of new and modified documents, and testing of procedural documents. Over 70% of the official documents for users, system administrators, Virtual Organization management and others are now in the new process. Documents related to storage and those targeted at scientific users have been reorganized, rewritten, and most will be reviewed and in production by the end of June. The OSG Training program brings domain scientists and computer scientists together to provide a rich training ground for the engagement of students, faculty and researchers in learning the OSG infrastructure, applying it to their discipline and contributing to its development. During 2009, OSG sponsored and conducted training events for students and faculty. Training organized and delivered by OSG in the last year is identified in the following table: Workshop
Length
Location
Month
Site Administrators Workshop
2 days
Indianapolis, IN
Aug. 2009
Grid Colombia Workshop
2 weeks
Bogota, Colombia
Oct., 2009
Grid Colombia Workshop
11 days
Bucaramanga, Colombia
Mar., 2010
The two OSG workshops in Colombia have been part of the initial steps of the Grid Columbia project in which 11 universities (hosting more than 100,000 students and 5,000 faculty members) are involved in the creation of a National Grid. The workshops provided technical training and hands-on experience in setting up and managing grids. OSG staff also participated as keynote speakers, instructors, and/or presenters at three venues this year as detailed in the following table:
68
Venue
Length
Location
Month
International Summer School on 2 weeks Grid Computing
Nice, France
July, 2009
IX DOSAR Workshop
3 days
Pilanesburg, South Africa
April, 2010
Grace Hopper Conference 09
4 days
Tucson, AZ
Oct., 2009
OSG was a co-organizer of the International Summer School on Grid Computing (ISSGC09). The OSG Education team arranged sponsorship (via NSF grant 0936102) for US-based students to attend this workshop; we selected and prepared ten US students who participated in the school. In addition, OSG staff provided direct contributions to the International Grid School by attending, presenting, and being involved in lab exercises, development, and student engagement. 4.2
Outreach Activities
We present a selection of the activities in the past year: •
Joint EGEE and OSG Workshop at the High Performance and Distributed Computing (HPDC 2009): “Workshop on Monitoring, Logging and Accounting, (MLA) in Production Grids. http://indico.fnal.gov/conferenceDisplay.py?confId=2335
•
The NSF Task Force on Campus Bridging (Miron Livny, John McGee)
•
Software sustainability (Miron Livny, Ruth Pordes)
•
HPC Best Practices Workshop (Alain Roy).
•
Member of the Network for Earthquake Engineering Simulation Project Advisory Committee (Ruth Pordes)
•
Member of the DOE Knowledge Base requirements group (Miron Livny)
In the area of international outreach, we continued activities in South America and maintaining our connection to the completed site in South Africa. OSG staff conducted a grid training school in Colombia with information about the services needed to build their own infrastructure. OSG was a co-sponsor of the International Summer School on Grid Computing in France (http://www.issgc.org/). OSG sponsored 10 students to attend the 2-week workshop, provided a key-note speaker and 3 teachers for lectures and hands-on exercises. 6 of the students have responded to our follow up queries and are continuing to use CI – local, OSG, TG. •
Continued co-editorship of the highly successful International Science Grid This Week newsletter, www.isgtw.org. OSG is very appreciative that DOE and NSF have been able to supply funds matching to the European effort starting in January 2009. A new full time editor was hired effective July 2009. Future work will include increased collaboration with TeraGrid.
•
Presentations at the online International Winter School on Grid Computing http://www.iceage-eu.org/iwsgc10/index.cfm
69
4.3
Internet dissemination
OSG co-sponsors the weekly electronic newsletter called International Science Grid This Week (http://www.isgtw.org/); this is a joint collaboration with GridTalk, a European project affiliated with Enabling Grids for EScience (EGEE). Additional contributions come from the Department of Energy’s ASCR and HEP. The newsletter has been very well received, having published 178 issues with subscribers totaling approximately 6,068, an increase of over 27% in the last year. This newsletter covers scientific research enabled through application of advanced computing, with an emphasis on distributed computing and cyber infrastructure. In the last year, due to increased support from NSF and DOE, we were able to hire a full-time US editor with the time and expertise to develop and improve the publication. This in turn improved our ability to showcase US contributions. In addition, the OSG has a web site, http://www.opensciencegrid.org, intended to inform and guide stakeholders and new users of the OSG. As a part of that website, we solicit and publish research highlights from our stakeholders; research highlights for the last year are accessible via the following links: •
Case Study - Einstein@OSG Einstein@Home, an application that uses spare cycles on volunteers' computers, is now running on the OSG. March 2010
•
PEGrid gets down to business Grid technology enables students and researchers in the petroleum industry. January 2010
•
Linking grids uncovers genetic mutations Superlink-online helps geneologists perform compute-intensive analyses to discover diseasecausing anamolies. August 2009
•
Grid helps to filter LIGO’s data Researchers must process vast amounts of data to look for signals of gravitational waves, minute cosmic ripples that carry information about the motion of objects in the universe. August 2009
•
Grid-enabled virus hunting Scientists use distributed computing to compare sequences of DNA in order to identify new viruses. June 2009
The abovementioned research highlights and those published in prior years are available at http://www.opensciencegrid.org/About/What_We%27re_Doing/Research_Highlights.
70
5. 5.1
Participants Organizations
The members of the Council and List of Project Organizations 1. Boston University 2. Brookhaven National Laboratory 3. California Institute of Technology 4. Clemson University 5. Columbia University 6. Distributed Organization for Scientific and Academic Research (DOSAR) 7. Fermi National Accelerator Laboratory 8. Harvard University (Medical School) 9. Indiana University 10. Information Sciences Institute (USC) 11. Lawrence Berkeley National Laboratory 12. Purdue University 13. Renaissance Computing Institute 14. Stanford Linear Accelerator Center (SLAC) 15. University of California San Diego 16. University of Chicago 17. University of Florida 18. University of Illinois Urbana Champaign/NCSA 19. University of Nebraska – Lincoln 20. University of Wisconsin, Madison 21. University of Buffalo (council) 22. US ATLAS 23. US CMS 24. STAR 25. LIGO 26. CDF 27. D0 28. Condor 29. Globus 5.2
Partnerships and Collaborations
The OSG continues its relaiance on external project collaborations to develop the software to be included in the VDT and deployed on OSG. These collaborations include: Community Driven Improvement of Globus Software (CDIGS), SciDAC-2 Center for Enabling Distributed Petascale Science (CEDPS), Condor, dCache collaboration, Data Intensive Science University Network (DISUN), Energy Sciences Network (ESNet), Internet2, National LambdaRail (NLR), BNL/FNAL Joint Authorization project, LIGO Physics at the Information Frontier, Fermilab Gratia Accounting, SDM project at LBNL (BeStMan), SLAC Xrootd, Pegasus at ISI, U.S. LHC software and computing.
71
OSG also has close working arrangements with “Satellite” projects, defined as independent projects contributing to the OSG roadmap, with collaboration at the leadership level. Four new satellite projects are: •
High Throughput Parallel Computing (HTPC) on OSG resources for an emerging class of applications where large ensembles (hundreds to thousands) of modestly parallel (4- to ~64way) jobs.
•
Application testing over the ESnet 100-Gigabit network prototype, using the storage and compute end-points supplied by the Magellan cloud computing at ANL and NERSC.
•
CorralWMS to enable user access to provisioned resources and “just-in-time” available resources for a single workload integrate and build on previous work on OSG's GlideinWMS and Corral, a provisioning tool used to complement the Pegasus WMS used on TeraGrid.
•
VOSS: “Delegating Organizational Work to Virtual Organization Technologies: Beyond the Communications Paradigm” (OCI funded, NSF 0838383)
Two joint proposals between members of the OSG and TeraGrid have been submitted to NSF: • •
ExTENCI: Extending Science Through Enhanced National Cyberinfrastructure CI-TEAM: Cyberinfrastructure Campus Champions (CI-CC)
The European EGEE infrastructure has transitioned to several separate projects in March 2010. We have several joint EGEE-OSG-WLCG technical meetings a year. At the last one in December 2009 the following list of existing contact points/activities was revisiting in the light of organizational changes: • Continue within EGI-InSPIRE. The EGI Helpdesk (previously GGUS) will continue being run by the same team. • Grid Operations Center ticketing systems interfaces. • Security Incident Response. • Joint Monitoring Group (MyOSG, MyEGEE). MyEGI development undertaken by CERN (on behalf of EGI-InSPIRE) will establish relationship with IU. • Interoperations (possibly including Interoperability) Testing. • Software Security Validation. [Security Vulnerability Group continues inc. EMI] • Joint Operations meetings and collaboration [May stop naturally]. • Joint Security Policy Group. [Security Policy Group] • IGTF. • Infrastructure Policy Working Group. [European e-Infrastructure Forum as well] • Middleware Security Working Group. [Software Security Group - EGI - EMI representation] • Dashboards. [HUC EGI-InSPIRE] In the context of the WLCG Collaboration • • • • •
WLCG Management Board. WLCG Grid Deployment Board. WLCG Technical Forum. Accounting Gratia-APEL interfaces. MyOSG-SAM interface [Exchange/Integration of OSG monitoring records].
72
•
Xrootd collaboration and support.
European Middleware Initiative • Storage Resource Manager (SRM) specification, interfaces and testing. • dCache.org collaboration. • Glexec • Authorization projects. • OGF Production Grid Infrastructure Working Group. • Virtual Data Toolkit support and Engineering Management Team. Joint activities with TeraGrid are summarized in Table 5. Table 5: Joint OSG – TeraGrid activities Task Mission/Goals OSG & TG Joint Activity Tracking and Reporting iSGTW continuation and joint support plan; proposal for US based contributions Resource Allocation Analysis and Recommendations Resource Accounting Inter-operation and Convergence Recommendations Explore how campus outreach and activities can be coordinated SCEC Application to use both OSG & TG Joint Middleware Distributions (Client side)
OSG Owner Chander Sehgal
TG Owner Tim Cockerill
Stage Active
Judy Jackson
Elizabeth Leake
Planning
Miron Livny, Chander Sehgal Phillip Canal, Brian Bockelman John McGee, Ruth Pordes John McGee, Mats Rynge Alain Roy
Kent Milfeld
Planning
Kent Milfeld, David Hart Kay Hunt, Scott Lathrop Dan Katz
Planning
Lee Liming, J.P. Navarro
Active
Security Incidence Response
Infrastructure Policy Group (run by Bob Jones, EGEE)
Ruth Pordes, Miron Livny
Jim Marsteller, Von Welch John Towns, Scott Lathrop Scott Lathrop, Laura McGiness J.P. Navarro, Phil Andrews
Active
Joint Student Activities (e.g. TG conference, summer schools, etc.)
Mine Altunay, Jim Barlow Ruth Pordes, Miron Livny David Ritchie, Jim Weichel
Workforce Development
73
Active Active
Planning Active Active
6.
Cooperative Agreement Performance
OSG has put in place processes and activities that meet the terms of the Cooperative Agreement and Management Plan: •
The Joint Oversight Team meets periodically, as scheduled by DOE and NSF, via phone to hear about OSG progress, status, and concerns. Follow-up items are reviewed and addressed by OSG, as needed.
•
Two intermediate progress reports were submitted to NSF in February and June of 2007.
•
The Science Advisory Group (SAG) met in June 2007. The OSG Executive Board has addressed feedback from the Advisory Group. Revised membership of the SAG was done in August 2009. Telephone discussions with each member are half complete for the end of December 2009.
•
In February 2008, a DOE annual report was submitted.
•
In July 2008, an annual report was submitted to NSF.
•
In December 2008, a DOE annual report was submitted.
•
In June 2009, an annual report was submitted to NSF.
•
In January 2010, a DOE annual report was submitted.
As requested by DOE and NSF, OSG staff provides pro-active support in workshops and collaborative efforts to help define, improve, and evolve the US national cyberinfrastructure.
74
OSG – DocDB 965 June 15, 2010 www.opensciencegrid.org
Open Science Grid Annual Report June 2010
List of Attachments Appendix 1
Production on OSG
Appendix 2
OSG Research Highlights
Appendix 3
OSG VO Publications
Production on Open Science Grid 1. OSG Facility Summary
June 2010
Appendix 1
Page 1
June 2010
Appendix 1
Page 2
2. ATLAS on the OSG
June 2010
Appendix 1
Page 3
3. CMS on the OSG
June 2010
Appendix 1
Page 4
4. CDF Usage
June 2010
Appendix 1
Page 5
5. D0 Usage
June 2010
Appendix 1
Page 6
6. LIGO on the OSG
June 2010
Appendix 1
Page 7
7. ENGAGE Usage
June 2010
Appendix 1
Page 8
8. SBGrid Usage
June 2010
Appendix 1
Page 9
9. GLOW Usage
June 2010
Appendix 1
Page 10
10. HCC Usage
June 2010
Appendix 1
Page 11
11. Non-HEP Usage
June 2010
Appendix 1
Page 12
Open Science Grid Research and Technology Highlights June 2009 – June 2010 Case Study: Einstein@OSG ............................................................................................. 2 US scientists analyze first LHC data through the Open Science Grid ..................... 5 Bringing LHC data to US Tier-3s ................................................................................... 7 PEGrid gets down to business ....................................................................................... 9 Linked grids uncover genetic mutations .................................................................... 11 Grid helps to filter LIGO’s data ................................................................................... 13 Authorization project helps grids interoperate ......................................................... 15 Data-Taking Dress Rehearsal Proves World’s Largest Computing Grid is Ready for LHC Restart ........................................................................................................ 17 Grid-enabled virus hunting .......................................................................................... 19
June 2010
Appendix 2
Page 1
Case Study: Einstein@OSG For over five years, volunteers have been lending their computers’ spare cycles to the Laser Interferometer Gravitational Wave Observatory (LIGO) and GEO-600 projects via the BOINC application Einstein@Home. Now a new application wrapper, dubbed “Einstein@OSG,” brings the application to the Open Science Grid. Today, although Einstein@OSG has been running for only six months, it is already the top contributor to Einstein@Home, processing about 10 percent of jobs.
A screenshot of the Einstein@Home screensaver. Image courtesy of Einstein@Home.
“The Grid was perfectly suitable to run an application of this type,” said Robert Engel, lead developer and production coordinator for the Einstein@OSG project. “BOINC would benefit from every single CPU that we would provide for it. Increasing the number of CPUs by 1000 really results in 1000 times more science getting done.” Getting Einstein@Home to run on a grid was not without difficulties. Normally, a volunteer would download and install the application. The application would constantly download data, analyze it, and then return the results. In short, each instance of Einstein@Home has a permanent home on a volunteer’s computer. The same process would not work on the Grid. Grid jobs cannot run indefinitely, so each instance of Einstein@OSG was given a time limit. “Once the time limit is up, the Einstein@Home application exits, followed by the Einstein@OSG application, which will save all results to an external storage location,” Engel explained. “The next time Einstein@OSG starts, it likely starts on a different cluster node which may use a different architecture.” Next, the Einstein@OSG application detects changes in the environment, such as the architecture, location, version of software, or network connectivity, and then compiles any missing software ‘on-the-fly.’ After a final check to verify that all requirements for Einstein@Home are met, it starts up. The results from the previous run are loaded from the remote storage location, and Einstein@Home picks up where it left off.
June 2010
Appendix 2
Page 2
An application on a grid will encounter software and hardware issues much more frequently than a desktop application such as Einstein@Home, according to Engel. This is because grids are much more complex, and deal with an extremely high volume of jobs. Because the average Einstein@Home user will only encounter an error every couple of months, it’s practical for her to handle the error manually. With Einstein@OSG running on up to 10,000 cores, however, there are errors every couple of minutes. Fixing these manually simply isn’t practical, so Einstein@OSG eventually automated the process. “It was only because of that mechanism that we were able to scale up,” Engel said. “A computer never gets tired looking for errors and fixing them, unlike me, who likes to sleep at night and spend time with his family.” Before Engel began work on Einstein@OSG, he was a member of a team led by Thomas Radke at the Max Planck Institute for Gravitational Physics. Radke’s team created a wrapper for Einstein@Home compatible with the German Grid Initiative (D-Grid) in 2006. Part of Engel’s contribution was the design of a user interface that allows one person to effectively monitor and control thousands of Einstein@Home applications. “Back then it consisted of a command line tool that would summarize all activities on the Grid on a single terminal page,” Engel said. Now the tool records activities and uses that historical data to create error statistics. Those and other statistics are displayed on an internal webpage. The wrapper created by Radke’s team could not simply be repurposed to run on OSG, unfortunately. “OSG and the German grid are different,” Engel said. For example, “in Germany the entire grid depends on Globus.”
The number of clusters running on Einstein@OSG is plotted on the horizontal axis; the total number of CPU cores across all clusters is plotted on the vertical axis. The rectangles each represent one week between June 2009 and February 2010. The color indicates how much work was accomplished that week, ranging from blue (the least) to red (the most). Note that the dates of three arbitrarily chosen weeks are written in white to illustrate how over time, the amount of work as well as the number of clusters and cores has increased. Image courtesy of Einstein@OSG.
June 2010
Engel and his team examined their options for getting Einstein@Home onto OSG, and concluded that the best option was Condor-G, a sort of hybrid of Condor and Globus. But implementing Condor-G would have required a great deal of work, delaying the launch of Einstein@Home on OSG. That’s why Engel’s team opted to implement Globus’ GRAM, which took only two weeks of work, before they began work on a Condor-G solution.
Appendix 2
Page 3
It’s a good thing too, because they soon discovered a serious issue with GRAM. “It doesn’t go up in scale very well,” Engel said. “If you try to run more than 100 jobs on a given resource, you’ll bring down that resource.” Still, given a chance to do things differently, Engel would have implemented GRAM, he said. “It meant that for a year, we could run jobs on OSG.” The Condor-G version went live in September 2009, and it has rapidly picked up steam. “On a typical day, we are running between 5000 and 8000 jobs at any time,” Engel said. “Before that we were running approximately 500.” Miriam Boon, iSGTW March, 2010
June 2010
Appendix 2
Page 4
US scientists analyze first LHC data through the Open Science Grid On November 30, 2009, the Large Hadron Collider became the most powerful particle accelerator in the world. Over the next month, the LHC’s four particle detectors recorded 100,000 particle collisions at record-breaking energies. Since then, scientists around the world have been continuously analyzing and re-analyzing the long-awaited collision data, and are publishing the first scientific papers . These first collisions have tested not only the LHC and experiments, but also the global and national computing systems that link scientists around the world with LHC data. The task of connecting scientists with LHC data falls to the Worldwide LHC Computing Grid, a collaboration linking computing grid infrastructures with each other and with 170 computing centers in 34 countries. In the United States, the Open Science Grid enables scientists to connect with the WLCG, and thus with data from their LHC experiments. “We’re very proud to see how the Open Science Grid assisted LHC experiments during 2009,” says Ruth Pordes, OSG executive director. “It gives us confidence that the Worldwide LHC Computing Grid will enable future physics discoveries.” The Open Science Grid allows the LHC experiments to access, maintain, and move huge quantities of data in the US. Through the OSG, experiments distribute the data taken from the detector to special computing centers called Tier-1 facilities. The two largest such facilities in the US are Fermi National Accelerator Laboratory for the CMS experiment and Brookhaven National Laboratory for ATLAS. Through the OSG, US centers make available roughly 7 petabytes of data for the ATLAS experiment and 4.4 petabytes for CMS. To give a sense of scale, one petabyte is roughly equivalent to the data stored in 10,000 laptops. From a Tier-1 facility, data are accessed by smaller Tier-2 and Tier-3 facilities such as universities, where students and scientists study the data. Even now, when the LHC is not running, the data in Tier-1 facilities are being accessed continually as scientists study 2009 data to improve their models and predictions about what the LHC may reveal. “On average right now, we are running anywhere from 6,000 to 10,000 computer processing tasks with this data at all times. It’s really amazing to see how such a large computing infrastructure spread over five Tier-2 and a Tier-1 facility can operate at this level continuously,” says US ATLAS operations coordinator, Kaushik De. Though the LHC was only running for two months, the data collected during 2009 has given experiments like ATLAS and CMS, which search for a range of new physics phenomena, an opportunity to better understand their detectors. Studies of a detector’s behavior may not make headlines like a big-name physics discovery, but they are crucial to all future research. In order for scientists to recognize a never-before-seen phenomenon, whether it is the Higgs boson or dark matter, they need to know what to expect from the detector when looking at familiar physics events.
June 2010
Appendix 2
Page 5
With this crucial study of the detector under way, scientists are even more prepared and excited for the discoveries to come. The importance of the 2009 data to the experiments was evident from the high volume of experimental collaborators accessing the LHC data through the Open Science Grid. “The activity level was driven heavily by the number of people interested,” says CMS computing coordinator Ian Fisk. “A lot of people had been waiting for this.” As the LHC begins collisions at even higher energies in the coming month, thousands of experimental collaborators worldwide will want to study the data. The successes of 2009 suggest that the OSG is fully prepared for the challenge. Daisy Yuhas Symmetry Breaking February 23, 2010
June 2010
Appendix 2
Page 6
Bringing LHC data to US Tier-3s It’s a challenge for smaller research groups to get set up on a grid, but that’s exactly what physicists at over 40 sites across the United States need to do to get access to data from the Large Hadron Collider. The new US Tier-3 centers – evenly split between the ATLAS and the Compact Muon Solenoid experiments – have each received about $30,000 in funding as part of the American Recovery and Reinvestment Act. Physicists scattered around the country will be able to use them to do their own analysis of data generated by two of the LHC experiments. To get these sites online, a great deal of expertise will be needed. And that’s where the US LHC Tier3 support group comes into the picture. “What we are trying to do is to help them get their systems set up and connected to the grid, to make it easier for them to get access to data and additional processing power,” said Dan Fraser, production coordinator for the Open Science Grid.
Computer racks at the Fermilab Grid Computer Center. Image courtesy of Fermilab.
Normally, when a new cluster gets set up on Open Science Grid, it is managed by an experienced system administrator using computers that already have networked systems, with batch schedulers running on them. The documentation, workshops, and other support offered by OSG reflect that fact. These new Tier-3 centers, on the other hand, are starting from scratch. They are generally too small to warrant the cost of an experienced system administrator. Instead, students or physicists dedicate a limited portion of their time to getting the facility up and running and then maintaining it. To make the job easier, OSG is providing Tier-3 centers with onsite support during the setup of their new systems. Fraser’s team also revised the OSG documentation, adding new sections on how to create a cluster, and expanding existing sections to include more detailed instruction specifically for the Tier-3s. Likewise, OSG’s workshops have undergone some changes. “We’ve done a lot of work on refactoring our workshops into a hands-on format so that anyone who wants to come and learn can get up and running very quickly,” Fraser said. As a result, the OSG Software Stack has evolved as well. “We have already packaged all of the key components that you need for building a grid,” Fraser said. In the process, they have also incorporated new software to help the Tier-3s manage the large datasets they will be analyzing.
June 2010
Appendix 2
Page 7
“The hope is that these changes will help not just the Tier-3s, but also other small sites that want to participate in OSG but may not have the ability to put in large amounts of administrator time,” said Fraser. “Every time we make things simpler, it lowers the barrier to more scientific groups either using the OSG or adopting the software for use on their own grids.” —Miriam Boon, iSGTW February 10, 2010
June 2010
Appendix 2
Page 8
PEGrid gets down to business Although Petroleum Engineering Grid officially started up in late 2009, it is already a classic example of how research projects can have unexpected benefits. PEGrid came into existence using techniques and tools created by TIGRE (Texas Internet Grid for Research and Education), a state-funded project based on the use of federally funded middleware such as Open Science Grid’s Virtual Data Toolkit. When TIGRE ended in 2007, it left behind a grid software stack, three completed demonstration projects, and budding relationships with contacts in the air quality, biosciences, and petroleum engineering industries. “What we did is to go to industry and ask what they needed,” said Alan Sill, a Texas Tech University researcher who led the TIGRE development team. Because the petroleum industry is well-funded, they decided to take their next step by creating PEGrid. “What we’re trying to do with PEGrid is, using the technology that we have, to allow the best practices in industry and academics to be shared,” Sill said. “What we’ve found is that this isn’t being done.”
This image depicts a simulation of the water saturation changes in a quarter of a homogeneous oil reservoir over time, as water is injected. The water increases the pressure in the reservoir, pushing the oil to the surface. Because the reservoir is symmetrical, researchers were able to save time by simulating only one quarter of the reservoir. The colors indicates the water saturation, with purple being highly saturated and red being least saturated. Each of the six slices represents a snapshot in time. Image courtesy of Shameem Siddiqui.
The most basic problem they uncovered is that the software that petroleum engineers use is extremely expensive and requires significant computational resources. Schools cannot afford to purchase licenses for recent versions of this expensive software, and so students are learning their trade using programs that are more than a decade out of date. The companies that will eventually hire these students are also losing out. Their entry-level hires can’t just hit the ground running; they need to be trained on industry-standard software first. In response to those needs, Texas Tech researchers Ravi Vadapalli, Sill, Shameem Siddiqui, and Lloyd Heinze co-founded PEGrid in collaboration with three other Texas universities, four software companies, and four oil companies. Together, the industry partners have donated $45 million in software donations and private three-year grants (the latter in the form of student internships). PEGrid came up with an elegant way of ensuring that the donated software licenses benefit as many students and researchers as possible. “You’re probably not going to use the licenses all the time,” Vadapalli pointed out. “Why not just grid-enable these applications?”
June 2010
Appendix 2
Page 9
By sharing licenses between students, faculty, and institutions, PEGrid has enough licenses to meet the educational needs of students while providing faculty access to cutting-edge software for their research. And by working closely with industry, the PEGrid team has learned more about why their partners have not implemented any grid technologies. In academia, budgetary constraints make it difficult to acquire enough computational resources. But in cash-rich industries, that’s not the problem at all. “The number of processors is never a limitation,” Sill said. “What stops most people is they are not familiar with techniques for scientific computing.” In the petroleum industry in particular, they found that data security was a major concern that was stopping companies from exploring grid technologies. “The whole idea of running their data on multiple sites is troubling to them,” Sill said. “But we discovered quite quickly that the reason it is so troubling to them is that they are one to two generations behind in some areas of security, compared to the best practices in grid computing.”
A 3D model of a hydrocarbon reservoir which incorporates geological, geophysical, and numerical simulation data. Here, the simulation is displayed at a three dimensional visualization facility. Image courtesy of Shameem Siddiqui.
Thanks to this revelation, entirely new horizons are opening up for petroleum companies. “They do run very big models. Once they get comfortable with these techniques, we expect them to start more widely implementing grid techniques,” Sill said.
In the meantime, as PEGrid gets up and running, students are learning to use the new software. Eventually, they will start running simulations and analyses that will place demands on PEGrid’s large-scale clusters. Researchers will continue to learn more about oil reservoirs using these new resources. And industry partners will continue to learn more about cutting-edge grid technology. What next? The people behind PEGrid are already pursuing opportunities in other fields. Vadapalli, for example, is interested in applying PEGrid’s tools to other areas of energy research. There have also been conversations following up on TIGRE's initial demonstration projects. “We’ve actually had a couple years of fairly deep discussions with people involved in cancer radiation therapy,” Sill said. “The dosimetry calculations for radiation treatment with proton beams, for example, are very large-scale problems.” International Science Grid This Week January, 2010
June 2010
Appendix 2
Page 10
Linked grids uncover genetic mutations Knowing which genetic mutation within a family causes a particular disease can lead to recommended lifestyle changes that may help avoid symptoms. Or it could give rise to medication to treat the disease. But the type of computing analysis needed to identify the mutations can take the equivalent of years to complete on a single computer.
Superlink-online structure.
Geneticists use a statistical method called genetic linkage analysis to determine the location of disease-provoking mutations on the chromosome. Based on a given genealogy and its members’ genetic makeup, the analysis is mapped onto a probabilistic graphical model that represents the likelihood of a genetic marker being linked to the disease. In large families with many genetic markers, these computations are extremely compute-intensive. Superlink-online, a distributed system developed at the Technion-Israel Institute of Technology, helps researchers perform their analyses in a matter of days by distributing these computations over thousands of computers worldwide. Geneticists submit their data through the web portal with a single click and get their results, ready to use. Behind the scenes, the system splits the computations into hundreds of thousands of independent jobs, invokes them on the available resources, and assembles the results back into a single data set. Superlink ran on a single computer in 2002 when it was first released by Professor Dan Geiger and his students at the Technion. By 2005, as more computer power was needed to perform increasingly complex analyses, then-doctoral student Mark Silberstein began working on a distributed version. Silberstein and his adviser, Professor Assaf Schuster, realized that the only way to satisfy exceedingly high computing demands was to enable opportunistic use of nondedicated computers. "[The data] was too complex to analyze on one CPU...,” says Silberstein. “It was impossible to provide ’service’ with this quality of service.” The opportunistic model was chosen because “with literally zero budget for purchasing and maintaining dedicated hardware, and with the actual resource demand reaching thousands of CPUs, we could not afford any other model.” In early 2006, thanks to close collaboration with the Condor team at the University of WisconsinMadison, the first version of Superlink-online was released using Wisconsin’s Condor pool and the Technion’s own home-brewed Condor pool with about 100 CPUs. Eventually, additional resources came from the Open Science Grid, EGEE, and the Superlink@Technion community grid, which uses the idle cycles on participants’ home computers. Since then, the system has enabled hundreds of geneticists worldwide to analyze much larger data sets, producing results two orders of magnitude faster than the serial version. "The analysis
June 2010
Appendix 2
Page 11
of complicated pedigrees is always painful and challenging," says Researcher Kwanghyuk (Danny) Lee of Baylor College of Medicine. "With the help of Superlink-online, however, the large complicated families can be analyzed very fast and very accurately." In fact, several raredisease-causing mutations have been found, including those causing Hereditary Motor and Sensory Neuropathy, “Uncomplicated” Hereditary Spastic Paraplegia, and Ichthyosis. During a 3 month period, over 25,000 non-dedicated hosts from all grids have been actively participating in the computations, reaching maximum effective throughput roughly equal to that of a dedicated cluster of up to 8,000 cores.
An example of a complex consanguineous pedigree, or graphic map of a family tree. The squares represent males, while the circles represent females. Individuals affected by a genetic mutation are represented with solid squares or circles. Image courtesy Kwanghyuk (Danny) Lee, Baylor College of Medicine
Silberstein and others are finalizing a version that will significantly extend Superlink-online’s power by accessing resources over all the aforementioned grids and Tokyo Institute of Technology’s Tsubame supercomputer. A new resource management system, GridBot, unifies these into a single scheduling framework. For its part, Open Science Grid has been an essential part of Superlink-online.
“Without OSG, we would not be close to where we stand now,” says Silberstein.” Recently we managed to complete one genetic analysis task in 5 days. This task comprised about 3.5 million jobs of approximately 10 minutes each—roughly 55 years of CPU time. One-third of this workforce was from OSG.” And this particular analysis was especially important—It confirmed the suspected location of a mutation which causes Age-Related Corticol Cataracts. Marcia Teckenbrock August, 2009
June 2010
Appendix 2
Page 12
Grid helps to filter LIGO’s data Tiny ripples in the fabric of space-time may provide scientists with a way to study cosmic processes that are invisible to optical telescopes, such as the collision of two black holes. In his theory of general relativity, Einstein predicted that such ripples, called gravitational waves, would be created when a mass accelerates. However, gravitational waves are so small – about one thousand times smaller than a proton – that even the relatively large ones generated by massive astrophysical events are very difficult to detect.
The suspension system is one of the physical measures LIGO takes to isolate the detector from false signals such as seismic activity. Courtesy of LIGO Lab.
The Laser Interferometer Gravitational Wave Observatory (LIGO), which has sites in Washington and Louisiana, uses lasers to search for these minute cosmic ripples that carry information about the motion of objects in the universe. Analysis of data from LIGO’s detectors is very computationally intensive, however, and researchers depend on the LIGO Data Grid and Open Science Grid to process large amounts of data to look for signals of gravitational waves. When gravitational waves pass through an object, the object’s length fluctuates by an extremely small amount. It is these tiny fluctuations that scientists search for using LIGO detectors. However, many things – a truck on the highway, an earthquake, or even someone dropping a hammer nearby – can create signatures in LIGO’s detectors that appear similar to those of gravitational waves. That’s why researchers need to weed out false signals. LIGO instruments collect roughly one terabyte of raw data each day that researchers must sift through. In one type of search, the data is broken into smaller segments, and each segment is compared to tens of thousands of computer-generated signatures to identify candidate signals – a process that requires hundreds to thousands of CPUs, said LIGO researcher Kent Blackburn. A second type of search looks at weeks of data at a time, but requires researchers to account for a slight glitch caused by the motion of the earth and the object producing the waves. Correcting for this glitch and searching for signals in the data is extremely computationally intensive.
June 2010
Appendix 2
Page 13
LIGO researchers use the grid to filter the data quickly enough to keep up with the instruments’ high rate of data production. They also run mock data challenges where they blindly introduce a fake signal into the LIGO instruments to enhance their analysis techniques and verify that their process for picking out real signals works. Although the LIGO Scientific Collaboration researchers have not yet directly detected gravitational waves, they can estimate the rate at which gravitational waves are generated based on the fact that they did not detect any with instruments of a given sensitivity over a given period of time. Courtesy of D. Shoemaker LIGO Lab
“Scientists have built telescopes that can observe the universe using infrared, x-rays, and gamma rays, but these are all types of light,” Blackburn said. “By using gravitational waves, it’s like creating a whole new set of eyes to look at the universe. We’ll be able to see processes that don’t give off light that we’ve never been able to see before.” Amelia Williamson, for iSGTW August, 2009
June 2010
Appendix 2
Page 14
Authorization project helps grids interoperate The Grid is all about resource sharing. Grid facilities make their computational and storage resources available over the internet for use by others through gateway middleware. When sharing a resource, it is crucial to give users the appropriate privileges, but this task can be daunting when multiple Grids need to interoperate. That is the hurdle over which the Authorization Interoperability Project jumped.
Authorization Interoperability Project members , left to right, Oscar Koeroo (NIKHEF), Gabriele Garzoglio (Fermi National Accelerator Lab), and Frank Siebenlist (Argonne National Laboratory).
To access the Grid, a user presents her credentials—certification that she has rights to access Grid resources—to a resource gateway. The gateway in turn talks to a local authorization system to assign the appropriate privileges to the user. Because they are so central to the operations of any Grid, authorization systems have been developed since the early days, usually independent of other Grids. This meant that if one Grid wanted to share a resource gateway developed for its environment, it could not deploy it on another Grid without major code development. The idea for the Interoperability Project began a few years ago from white board drawings during various meetings in Amsterdam. “We were all trying to figure it out. We had these tools that made resources available over the Grid and made the authorization decisions,” said Oscar Koeroo, a security middleware developer at Nikhef, the national institute for subatomic physics in the Netherlands. “By drawing how each of the grids worked and had implemented their middleware, we saw ways to expand upon what each group could do with each other’s middleware.” Around October 2007, EGEE, Open Science Grid, and Globus came together to agree on a common protocol for resource gateways to talk to authorization systems. First, a common language to express authorization information and common sets of attributes, for example, to express user identity, were agreed upon. Then a library implementing the protocol was developed. All this provided the fundamental blocks necessary to create an interoperable authorization infrastructure.
June 2010
Appendix 2
Page 15
"By speaking the same language, we no longer needed separate implementations of libraries for authorization,” said Gabriele Garzoglio of Fermilab, a leader in the Interoperability Project. “We could now share functionalities between Globus, OSG, and EGEE, basing our software on a common code base.” “In the end, you have more freedom of choice on what solution you can install,” said Koeroo. “After this project, middleware isn’t bound to be installed on the Grid infrastructure it was initially developed for. For example, in the case of dCache, parts were made by Brookhaven and others by Fermilab, and they were adjusted to work with any Grid authorization system.” The Interoperability Project provided several benefits: First, software that was developed in EGEE, for example, now can be seamlessly deployed on OSG and vice versa simply by changing software configuration. Second, it is much easier to deploy software out of the box on a production Grid because it naturally interfaces with the authorization system. Third, a standard set of libraries to implement a common protocol was developed; these can be shared, meaning less code maintenance. Frank Siebenlist, a project member whose earlier work with OASIS' XACML Technical Committee set the standards for the common authorization language used for the project, said, “We should all pat ourselves on the back. It just doesn’t happen too often that different entities work so well together.” Jen Nahn August, 2009
June 2010
Appendix 2
Page 16
Data-Taking Dress Rehearsal Proves World’s Largest Computing Grid is Ready for LHC Restart BATAVIA, IL and UPTON, NY –The world’s largest computing grid has passed its most comprehensive tests to date in anticipation of the restart of the world’s most powerful particle accelerator, the Large Hadron Collider (LHC). The successful dress rehearsal proves that the Worldwide LHC Computing Grid (WLCG) is ready to analyze and manage real data from the massive machine. The United States is a vital partner in the development and operation of the WLCG, with 15 universities and three U.S. Department of Energy (DOE) national laboratories from 11 states contributing to the project. The full-scale test, collectively called the Scale Test of the Experimental Program 2009 (STEP09), demonstrates the ability of the WLCG to efficiently navigate data collected from the LHC’s intense collisions at CERN, in Geneva, Switzerland, all the way through a multi-layered management process that culminates at laboratories and universities around the world. When the LHC resumes operations this fall, the WLCG will handle more than 15 million gigabytes of data every year. Although there have been several large-scale WLCG data-processing tests in the past, STEP09, which was completed on June 15, was the first to simultaneously test all of the key elements of the process. “Unlike previous challenges, which were dedicated testing periods, STEP09 was a production activity that closely matches the types of workload that we can expect during LHC data taking. It was a demonstration not only of the readiness of experiments, sites and services but also the operations and support procedures and infrastructures,” said CERN’s Ian Bird, leader of the WLCG project. Once LHC data have been collected at CERN, dedicated optical fiber networks distribute the data to 11 major “Tier-1” computer centers in Europe, North America and Asia, including those at DOE’s Brookhaven National Laboratory in New York and Fermi National Accelerator Laboratory in Illinois. From these, data are dispatched to more than 140 “Tier-2” centers around the world, including 12 in the United States. It will be at the Tier-2 and Tier-3 centers that physicists will analyze data from the LHC experiments – ATLAS, CMS, ALICE and LHCb – leading to new discoveries. Support for the Tier-2 and Tier-3 centers is provided by the DOE Office of Science and the National Science Foundation. “In order to really prove our readiness at close-to-real-life circumstances, we have to carry out data replication, data reprocessing, data analysis, and event simulation all at the same time and all at the expected scale for data taking,” said Michael Ernst, director of Brookhaven National Laboratory’s Tier-1 Computing Center. “That’s what made STEP09 unique.” The result was “wildly successful,” Ernst said, adding that the U.S. distributed computing facility for the ATLAS experiment completed 150,000 analysis jobs at an efficiency of 94 percent.
June 2010
Appendix 2
Page 17
A key goal of the test was gauging the analysis capabilities of the Tier 2 and Tier 3 computing centers. During STEP09’s 13-day run, seven U.S. Tier 2 centers for the CMS experiment, and four U.S. CMS Tier 3 centers, performed around 225,000 successful analysis jobs. “We knew from past tests that we wanted to improve certain areas," said Oliver Gutsche, the Fermilab physicist who led the effort for the CMS experiment. "This test was especially useful because we learned how the infrastructure behaves under heavy load from all four LHC experiments. We now know that we are ready for collisions." U.S. contributions to the WLCG are coordinated through the Open Science Grid (OSG), a national computing infrastructure for science. OSG not only contributes computing power for LHC data needs, but also for projects in many other scientific fields including biology, nanotechnology, medicine and climate science. "This is another significant step to demonstrating that shared infrastructures can be used by multiple high-throughput science communities simultaneously," said Ruth Pordes, executive director of the Open Science Grid Consortium. "ATLAS and CMS are not only proving the usability of OSG, but contributing to maturing national distributed facilities in the U.S. for other sciences." Physicists in the U.S. and around the world will sift through the LHC data in search of tiny signals that will lead to discoveries about the nature of the physical universe. Through their distributed computing infrastructures, these physicists also help other scientific researchers increase their use of computing and storage for broader discovery. Press Release – Brookhaven National Laboratory, Fermilab, Open Science Grid and CERN July 1, 2009
June 2010
Appendix 2
Page 18
Grid-enabled virus hunting DNA sequencing and sequence analysis happens daily in many biological sciences laboratories, but analyzing large sets of genetic data increasingly requires computing resources beyond the capabilities of most labs. The search for the best hardware and software led Eric Delwart, a professor of laboratory medicine at the University of California, San Francisco, and a senior investigator at the Blood Systems Research Institute, and Chunlin Wang, a research associate at the Stanford University Genome Technology Center, to the Renaissance Computing Institute's (RENCI) Engagement Team and then to the distributed computing resources of the TeraGrid and the Open Science Grid (OSG). Delwart works with Wang to identify new viruses. The team uses a technique called massively parallel pyrosequencing, which can determine sequences for millions of DNA fragments using high-throughput computing. The resulting DNA sequences are then compared to all the sequences in public sequence databases to identify viral fingerprint sequences. One single sequencing reaction generates massive volumes of data that can take months, even years, to analyze on a small-scale computing cluster.
3D replica of senecavirus, a pathogen discovered several years ago by researchers in Pennsylvania. UC San Francisco researcher Eric Delwart and his colleague Chunlin Wang of Stanford University use the RENCI-developed TeraGrid Science Gateway and the Open Science Grid to access grid computing resources in their search for new viruses.
Image courtesy of the Institute for Animal Health, UK
In an effort to analyze more data more efficiently, Delwart and Wang turned to the RENCI Engagement Team, which participates in the TeraGrid Science Gateways program and leads the OSG Engagement program. TeraGrid’s Science Gateways aim to bring new communities of users to TeraGrid resources by providing easy access to the TeraGrid’s distributed computing resources. The OSG Engagement program recruits new users from a wide range of disciplines and helps them become users of the distributed computing systems operated and maintained by OSG members. The effort to find more computing power paid off for the Delwart and Wang: In early May, they used 70,000 CPU hours on TeraGrid and OSG resources to complete in a week a DNA sequence analysis that would have taken over three months on their own lab cluster. The team submitted its jobs to the national resources using a RENCI-developed, Web services-based computational science platform (link to PDF).
“We created an application that communicates with RENCI's TeraGrid Science Gateway,” said Jason Reilly, a RENCI senior research software developer. “For the user, it’s very simple — just log in and the application maps the data input to specific tasks to be done. The beauty is you
June 2010
Appendix 2
Page 19
don’t have to submit commands over and over again. You can run hundreds or even thousands of operations and you only have to submit the command once.” The custom application created by Reilly was dubbed BLASTMaster because it builds on the Basic Local Alignment Search Tool (BLAST) used to search sequence databases. BLASTMaster divides commands into tasks and pushes the work to RENCI's TeraGrid Science Gateway, which submits, monitors, and manages the compute workload on systems that are part of TeraGrid’s nationwide network of high performance machines, and to OSG machines. After entering the initial commands, the researchers merely had to wait for their results. “Large computer farms that we might use are often composed of heterogeneous smaller clusters,” said Wang. “The BLASTMaster tool and a Web services environment is particularly useful to those of us without much experience using compute clusters. It gives us a uniform interface to submit jobs, which greatly enhances our productiveness.”
Distribution of BLAST jobs from one Pyrosequencing run (96,866 jobs total, 29 April to 7 May 2009) with the glide-in factory configured for: three TeraGrid resources, one OSG resource, one RENCI resource, and one UNC-CH resource. Image courtesy of John McGee, Jason Reilly, and Mats Rynge (RENCI)
The sequence analysis work used TeraGrid resources at Purdue University (West Layfayette, IN), OSG resources at RENCI (Chapel Hill, NC) and a cluster in the University of North Carolina at Chapel Hill computer science department supported by the National Institutes of Health.
The work has real-world value taken straight from recent headlines about the H1N1 virus. “Knowing the genomic sequence of a human virus allows for quicker diagnostics to identify infections,” said Delwart. “Quicker diagnostics can lead to more informed decisions on how an emerging virus is spread and how to control it. Knowing the sequence can also help make vaccines or anti-virals against that virus.” Karen Green, RENCI June, 2009
June 2010
Appendix 2
Page 20
Publications Enabled by Open Science Grid July 2009 - June 2010
June 2010
Appendix 3
Page 1
Contents Accelerator Physics Virtual Organization..................................................................................................... 3 ATLAS Virtual Organization ....................................................................................................................... 3 CDF Virtual Organization........................................................................................................................... 13 CDMS Virtual Organization ....................................................................................................................... 19 CIGI Virtual Organization .......................................................................................................................... 20 CMS Virtual Organization .......................................................................................................................... 20 DZero Virtual Organization ........................................................................................................................ 27 ENGAGE Virtual Organization .................................................................................................................. 30 GLOW Virtual Organization ...................................................................................................................... 33 HCC Virtual Organization .......................................................................................................................... 38 LIGO Collaboration .................................................................................................................................... 38 MiniBooNE Virtual Organization............................................................................................................... 41 MINOS Virtual Organization ..................................................................................................................... 42 NanoHUB Virtual Organization ................................................................................................................. 43 NYSgrid Virtual Organization .................................................................................................................... 43 SBgrid Virtual Organization ....................................................................................................................... 44 STAR Virtual Organization ........................................................................................................................ 44
June 2010
Appendix 3
Page 2
Accelerator Physics Virtual Organization 1) Beam physics of the 8-GeV H- linac, Jean-Paul Carneiro, B. Mustapha, and P.N. Ostroumov, Nuclear Instruments and Methods in Physics Research A, 606 (2009) 271 – 280. 2) Numerical simulations of stripping effects in high-intensity hydrogen ion linacs, J.-P. Cameiro, B. Mustapha, and P.N. Ostroumov, Phys. Rev. Special Topics - Accelerators and Beams 040102 (2009).
ATLAS Virtual Organization 1) Relative luminosity measurement of the LHC with the ATLAS forward calorimeter. By A. Afonin, et al., JINST 5:P05005,2010,. [arXiv:1005.1784] 2) The ATLAS discovery reach for SUSY models with early data. By Janet Dietrich & for the ATLAS collaboration. [arXiv:1005.2034] (May 2010) 4p. 3) Commissioning of the ATLAS Liquid Argon Calorimeter. By S. Laplace. Nucl.Instrum.Meth. A617:30-34,2010,. [arXiv:1005.2935] 4) Collective flow in (anti)proton-proton collision at Tevatron and LHC. By Tanguy Pierog, Sarah Porteboeuf, Iurii Karpenko, Klaus Werner. [arXiv:1005.4526] (May 2010) 4p. 5) The ATLAS Simulation Infrastructure. By The ATLAS Collaboration. [arXiv:1005.4568] (May 2010) 6) Performance of the ATLAS Detector using First Collision Data. By The ATLAS Collaboration. [arXiv:1005.5254] (May 2010) 7) Readiness of the ATLAS detector: Performance with the first beam and cosmic data. By Francesca Pastore. Nucl.Instrum.Meth.A617:48-51,2010,. 8) Prospects for Observing the Standard Model Higgs Boson Decaying into b\bar{b} Final States Produced in Weak Boson Fusion with an Associated Photon at the LHC. By D.M. Asner, M. Cunningham, S. Dejong, K. Randrianarivony, C. Santamarina, M. Schram. [arXiv:1004.0535] (Apr 2010) 15p.
June 2010
Appendix 3
Page 3
9) The ATLAS Inner Detector commissioning and calibration. By ATLAS Collaboration [arXiv:1004.5293] (Apr 2010) 34p. 10) Identification of muons in ATLAS. By Zdenko van Kesteren. (Mar 2010) 159p. 11) LHC Detectors and Early Physics. By Guenther Dissertori. [arXiv:1003.2222] (Mar 2010) 12) Charged-particle multiplicities in pp interactions at sqrt(s) = 900 GeV measured with the ATLAS detector at the LHC. By ATLAS Collaboration Phys Lett B 688, 2010, Issue 1, 21-42, Phys.Lett.B688:21-42,2010,. [arXiv:1003.3124] 13) Higgs Boson Search Sensitivity in the $H \to WW$ Dilepton Decay Mode at $\sqrt s = 7$ and 10 TeV. By Edmond L. Berger, Qing-Hong Cao, C.B. Jackson, Tao Liu, Gabe Shaughnessy. [arXiv:1003.3875] ANL-HEP-PR-10-8 (Mar 2010) 14) Topics in the measurement of top quark events with ATLAS: Pixel detector optoelectronics, track impact parameter calibration, acceptance correction methods. By Stephan A. Sandvoss. (Mar 2010) 170p. 15) Expected Performance of the ATLAS Detector in GMSB Models with Tau Final States. By Dorthe Ludwig & for the ATLAS collaboration. PoS HCP2009:073,2009,. [arXiv:1002.0944] 16) Commissioning and early physics analysis with the ATLAS and CMS experiments. By Andreas Hoecker. [arXiv:1002.2891] (Feb 2010) 84p. 17) QCD and Electroweak Physics at LHC. By Klaus Rabbertz. PoS RADCOR2009:016,2009,. [arXiv:1002.3628] 18) Drift Time Measurement in the ATLAS Liquid Argon Electromagnetic Calorimeter using Cosmic Muons. By The ATLAS Collaboration. [arXiv:1002.4189] (Feb 2010) 30p. 19) Search for Supersymmetry in Trilepton Final States with the ATLAS Detector and the Alignment of the ATLAS Silicon Tracker. June 2010
Appendix 3
Page 4
By Oleg Brandt. [arXiv:1001.1365] (Jan 2010) 236p. 20) The Hunt for New Physics at the Large Hadron Collider. By P. Nath, et al., Nucl.Phys.Proc.Suppl.200-202:185-417,2010,. [arXiv:1001.2693] 21) B Physics and Quarkonia studies with early ATLAS data. By for the ATLAS Collaboration [arXiv:1001.3806] ATL-PHYS-PROC-2010-006 (Jan 2010) 22) NLO QCD corrections to top anti-top bottom anti-bottom production at the LHC: 2. full hadronic results. By Axel Bredenstein, Ansgar Denner, Stefan Dittmaier, Stefano Pozzorini. 23) The construction and testing of the silicon position sensitive modules for the LHCf experiment at CERN. By O. Adriani, et al., JINST 5:P01012,2010,. 24) Micromegas study for the sLHC environment. By T. Alexopoulos, et al., JINST 5:P02003,2010,. 25) Atlas TRT: Research & design B-type module mass production. By Yu.V. Gusakov, et al., Phys.Part.Nucl.41:1-26,2010,. 26) New developments in data-driven background determinations for SUSY searches in ATLAS. By ATLAS Collaboration AIP Conf.Proc.1200:297-300,2010,. 27) Prospects for the discovery of supersymmetry based on inclusive searches in ATLAS. By ATLAS Collaboration AIP Conf.Proc.1200:301-304,2010,. 28) Multi-lepton SUSY searches with the ATLAS detector. By ATLAS Collaboration AIP Conf.Proc.1200:305-308,2010,. 29) Search for dilepton and lepton + E(T)**miss resonances at high mass with ATLAS. By ATLAS Collaboration AIP Conf.Proc.1200:309-312,2010,.
June 2010
Appendix 3
Page 5
30) Reconstruction of tau leptons and prospects for SUSY in ATLAS. By ATLAS Collaboration AIP Conf.Proc.1200:341-344,2010,. 31) Exclusive measurements for SUSY events with the ATLAS detector at the LHC. By ATLAS Collaboration AIP Conf.Proc.1200:345-348,2010,. 32) R-parity violation with jet signatures at the ATLAS detector. By ATLAS Collaboration AIP Conf.Proc.1200:354-357,2010,. 33) Signatures of GMSB with non-pointing photons at the ATLAS detector. By ATLAS Collaboration AIP Conf.Proc.1200:362-365,2010,. 34) Associated standard model Higgs boson search with ATLAS. By ATLAS Collaboration AIP Conf.Proc.1200:378-381,2010,. 35) Inclusive standard model Higgs boson searches in ATLAS. By ATLAS Collaboration AIP Conf.Proc.1200:382-385,2010,. 36) Beyond the standard model Higgs searches at ATLAS. By ATLAS Collaboration AIP Conf.Proc.1200:394-397,2010,. 37) SM Higgs properties measurement at ATLAS. By ATLAS Collaboration AIP Conf.Proc.1200:402-405,2010,. 38) Search for resonances decaying into top quark pairs with ATLAS. By ATLAS Collaboration AIP Conf.Proc.1200:678-681,2010,. 39) Electron reconstruction and identification with the ATLAS detector. By ATLAS Collaboration AIP Conf.Proc.1200:693-696,2010,. 40) Photon reconstruction and identification with the ATLAS detector. By ATLAS Collaboration AIP Conf.Proc.1200:697-700,2010,.
June 2010
Appendix 3
Page 6
41) In-situ determination of the ATLAS muon performance. By ATLAS Collaboration AIP Conf.Proc.1200:705-708,2010,. 42) Search for new physics with long-lived particles. By ATLAS Collaboration AIP Conf.Proc.1200:742-745,2010,. 43) Reconstruction and identification of heavy long-lived particles at the ATLAS detector at the LHC. By ATLAS Collaboration AIP Conf.Proc.1200:762-765,2010,. 44) Prospects for early discoveries at the LHC with dileptons, jets and no missing energy with the ATLAS detector. By ATLAS Collaboration AIP Conf.Proc.1200:802-805,2010,. 45) Study of a micromegas chamber in a neutron beam. By T. Alexopoulos, et al., JINST 5:P02005,2010,. 46) Measurement of the response of the ATLAS liquid argon barrel calorimeter to electrons at the 2004 combined test-beam. By M. Aharrouche, et al., Nucl.Instrum.Meth.A614:400-432,2010,. 47) The ATLAS muon spectrometer. By Giora Mikenberg. Mod.Phys.Lett.A25:649-667,2010,. 48) How physics defines the LHC environment and detectors. By D. Green. Int.J.Mod.Phys.A25:1279-1313,2010,. 49) Higgs physics with ATLAS. By ATLAS Collaboration Nucl.Phys.Proc.Suppl.198:149-156,2010,. 50) The hybrid tracking system of ATLAS. By Leonardo Rossi. Int.J.Mod.Phys.A25:1519-1540,2010,. 51) The ATLAS electromagnetic calorimeters: Features and performance. By Luciano Mandelli. Int.J.Mod.Phys.A25:1739-1760,2010,. June 2010
Appendix 3
Page 7
52) ATLAS electronics: An overview. By Philippe Farthouat. Int.J.Mod.Phys.A25:1761-1784,2010,. 53) TileCal: The hadronic section of the central ATLAS calorimeter. By K. Anderson, T. Del Prete, E. Fullana, J. Huston, C. Roda, R. Stanek. Int.J.Mod.Phys.A25:1981-2003,2010,. 54) Readiness of the ATLAS Liquid Argon Calorimeter for LHC Collisions By ATLAS Collaboration [arXiv:0912.2642] (Dec 2009) 31p. 55) Neutron Production and ZDC Acceptance. By Sebastian White. [arXiv:0912.4320] (Dec 2009) 11p. 56) Prospects for R-Parity Conserving SUSY searches at the LHC. By CMS Collaboration [arXiv:0912.4378] (Dec 2009) 5p. 57) The ATLAS transition radiation detector (TRT) Fast-OR trigger. By S. Fratina, et al., ATL-INDET-PUB-2009-002 (Dec 2009) 21p. 58) ATLAS Pixel Radiation Monitoring with HVPP4 System. By Igor Gorelov, Martin Hoeferkamp, Sally Seidel, Konstantin Toms. [arXiv:0911.0128] (Nov 2009) 8p. 59) Forward jets physics in ATLAS, CMS and LHCb. By David d'Enterria. [arXiv:0911.1273] (Nov 2009) 6p. 60) Charmonium production at the LHC. By Magdalena Malek. [arXiv:0911.1522] (Nov 2009) 8p. 61) A Layer Correlation Technique for Pion Energy Calibration at the 2004 ATLAS Combined Beam Test. By ATLAS Liquid Argon Collaboration [arXiv:0911.2639] (Nov 2009) 7p. 62) Optical Link ASICs for the LHC Upgrade. By K.K. Gan, H.P. Kagan, R.D. Kass, J.R. Moore, D.S. Smith. [arXiv:0911.4305] (Nov 2009) 3p.
June 2010
Appendix 3
Page 8
63) Characterization of 3D-DDTC detectors on p-type substrates. By Gian-Franco Dalla Betta, et al., [arXiv:0911.4864] (Nov 2009) 8p. 64) The Data acquisition and calibration system for the ATLAS Semiconductor Tracker. By Tom Barber. Conference Record of the 16th IEEE NPSS Real Time Conference 2009 (RT 2009), pp 465-471. 65) ATLAS silicon microstrip tracker operation. By P. Vankov. Proceedings of the Topical Workshop on Electronics for Particle Physics (TWEPP-09), pp 553556. 66) Scalable Database Access Technologies for ATLAS Distributed Computing. By ATLAS Collaboration [arXiv:0910.0097] ANL-HEP-CP-09-085 (Oct 2009) 6p. 67) Discovery Potential of the Standard Model Higgs Boson Through H -> WW Decay Mode with the ATLAS Detector at LHC. By ATLAS Collaboration [arXiv:0910.0193] (Oct 2009) 8p. 68) Diamond Prototypes for the ATLAS SLHC Pixel Detector. By RD42 Collaboration [arXiv:0910.0347] (Oct 2009) 3p. 69) Searches for SUSY with the ATLAS detector. By ATLAS Collaboration AIP Conf.Proc.1200:32-40,2010,. [arXiv:0910.0559] 70) Results from the Commissioning of the ATLAS Pixel Detector with Cosmic data. By ATLAS Collaboration [arXiv:0910.0847] (Oct 2009) 9p. 71) Measurement of the Z boson transverse momentum spectrum on ATLAS with early data. By Lashkar Kashif. [arXiv:0910.0910] (Oct 2009) 6p. 72) Development of FTK architecture: a fast hardware track trigger for the ATLAS detector. By A. Annovi, et al., [arXiv:0910.1126] (Oct 2009) 73) Standard Model $H \gamma \gamma$ discovery potential with ATLAS. By Yaquan Fang & for the ATLAS Collaboration. [arXiv:0910.2149] (Oct 2009) 5p.
June 2010
Appendix 3
Page 9
74) Performance of the ATLAS Detector on First Single Beam and Cosmic Ray Data. By Martin Aleksa. AIP Conf.Proc.1200:689-692,2010,. [arXiv:0910.2198] 75) ATLAS Sensitivity to Leptoquarks, W(R) and Heavy Majorana Neutrinos in Final States with High-p(T) Dileptons and Jets with Early LHC Data at 14-TeV proton-proton collisions. By Vikas Bansal. [arXiv:0910.2215] (Oct 2009) 6p. 76) Track and vertex reconstruction in the ATLAS inner detector. By M. Limper. Amsterdam, Netherlands: Eigenverl. (2009) 164 p Amsterdam, Netherlands: Eigenverl. (2009) 164 p. 77) ATLAS Muon Detector Commissioning. By ATLAS Muon Collaboration [arXiv:0910.2767] (Oct 2009) 6p. 78) ATLAS Great Lakes Tier-2 Computing and Muon Calibration Center Commissioning. By Shawn McKee. [arXiv:0910.2878] (Oct 2009) 6p. 79) Search for Supersymmetry Signatures at the LHC. By ATLAS Collaboration [arXiv:0910.2964] (Oct 2009) 10p. 80) Commissioning of the ATLAS Liquid Argon Calorimeters. By ATLAS Liquid Argon Collaboration AIP Conf.Proc.1182:184-187,2009,. [arXiv:0910.2991] 81) The ATLAS Detector: Status and Results from Cosmic Rays. By James T. Shank. [arXiv:0910.3081] (Oct 2009) 6p. 82) Discovery Potential for Di-lepton and Lepton+Etmiss Resonances at High Mass with ATLAS. By ATLAS Collaboration [arXiv:0910.3378] (Oct 2009) 6p. 83) Measurement of the W + jets and Z + jets Cross Section with the ATLAS detector. By ATLAS Collaboration [arXiv:0910.3382] (Oct 2009) 4p. 84) Search for Contact Interactions in the Dimuon Final State at ATLAS. By E.N. Thompson, S. Willocq, K.M. Black. [arXiv:0910.3384] (Oct 2009) 4p. June 2010
Appendix 3
Page 10
85) Prospects for measuring t anti-t production cross-section at s**(1/2) =10-TeV using the likelihood method with the ATLAS detector. By Dilip Jana. [arXiv:0910.3431] (Oct 2009) 6p. 86) Development of 3D-DDTC pixel detectors for the ATLAS upgrade. By Gian-Franco Dalla Betta, et al., [arXiv:0910.3629] (Oct 2009) 20p. 87) Preliminary results of 3D-DDTC pixel detectors for the ATLAS upgrade. By Alessandro La Rosa, et al., [arXiv:0910.3788] (Oct 2009) 8p. 88) Single-Top Cross Section Measurements at ATLAS. By ATLAS Collaboration [arXiv:0910.3824] ATL-COM-PHYS-2009-468 (Oct 2009) 2p. 89) Top quark pair cross section prospects in ATLAS. By ATLAS Collaboration [arXiv:0910.3930] (Oct 2009) 5p. 90) Discovery Potential for GMSB Supersymmetry in ATLAS using the Z gamma + not E(T) Final State at a center of mass energy of s**(1/2) = 10-TeV. By ATLAS Collaboration [arXiv:0910.4062] (Oct 2009) 6p. 91) Tau physics at the LHC with ATLAS. By ATLAS Collaboration [arXiv:0910.4727] ATL-PHYS-PROC-2009-126 (Oct 2009) 6p. 92) Alignment of the ATLAS Inner Detector Tracking System. By ATLAS Collaboration [arXiv:0910.5156] (Oct 2009) 7p. 93) Measuring Central Exclusive Processes at LHC. By Marek Tasevsky. [arXiv:0910.5205] (Oct 2009) 8p. 94) ScotGrid: Providing an Effective Distributed Tier-2 in the LHC Era. By Sam Skipsey, David Ambrose-Griffith, Greig Cowan, Mike Kenyon, Orlando Richards, Phil Roffe, Graeme Stewart. J.Phys.Conf.Ser.219:052014,2010,. [arXiv:0910.4507] 95) The reach of the ATLAS experiment in SUSY parameter space. By for the ATLAS collaboration Balk.Phys.Lett.16:86-89,2009,. [arXiv:0910.5602] June 2010
Appendix 3
Page 11
96) ATLAS SUSY search prospects at 10-TeV. By ATLAS Collaboration [arXiv:0910.5653] (Oct 2009) 4p. 97) The ATLAS muon spectrometer: Commissioning and tracking. By Jochem Snuverink. (Oct 2009) 125p. 98) Study of GMSB models with photon final states using the ATLAS detector. By Mark Terwort. DESY-THESIS-2009-033 (Oct 2009) 145p. 99) Measurements from supersymmetric events. By ATLAS Collaboration ATL-PHYS-PUB-2009-067 (Oct 2009) 27p. 100) Design and performance of the ABCN-25 readout chip for ATLAS inner detector upgrade. By W. Dabrowski, et al., IEEE Nucl.Sci.Symp.Conf.Rec.2009:373-380, 2009,. 101) Searches for the Higgs boson at the LHC. By Marco Delmastro. [arXiv:0909.0493] (Sep 2009) 4p. 102) Discovery Potential of R-hadrons with the ATLAS Detector at the LHC. By ATLAS Collaboration AIP Conf.Proc.1200:750-753,2010,. [arXiv:0909.1911] 103) On the possibility to use ATLAS and CMS detectors for neutrino physics. By A. Guskov. [arXiv:0909.2513] (Sep 2009) 4p. 104) Fake E-slash(T) from Calorimeter Effects. By ATLAS Collaboration [arXiv:0909.4152] ATL-PHYS-PROC-2009-088 (Sep 2009) 2p. 105) Determination of QCD Backgrounds in ATLAS: A Challenge for SUSY searches. By Bernhard Meirose. AIP Conf.Proc.1200:717-720,2010,. [arXiv:0909.4427] 106) The Software of the ATLAS beam pick-up based LHC monitoring system. By C. Ohm & T. Pauly. J.Phys.Conf.Ser.219:022040,2010,. [arXiv:0909.5378]
June 2010
Appendix 3
Page 12
107) An HV power supply system for the transition radiation tracker of the ATLAS experiment. By K.I. Zhukov, V.A. Kantserov, S.V. Muravev, A.P. Shmeleva. Instrum.Exp.Tech.52:673-677,2009, Prib.Tekh.Eksp.5:66-71,2009,. 108) Temperature Studies for ATLAS MDT BOS Chambers. By A. Engl, R. Hertenberger, O. Biebel, R. Mameghani, D. Merkl, F. Rauscher, D. Schaile, R. Strohmer. NSS Conf. Rec. 2008, 2274-2277. [arXiv:0908.1541] 109) ATLAS Monitored Drift Tube Chambers in E = 11 MeV Neutron Background. By T. Muller, A. Mlynek, O. Biebel, R. Hertenberger, T. Nunnemann, D. Merkl, F. Rauscher, D. Schaile, R. Strohmer. [arXiv:0908.1562] (Aug 2009) 4p. 110) ATLAS monitored drift tube chambers for super-LHC. By Albert Engl, Otmar Biebel, Ralf Hertenberger, Alexander Mlynek, Thomas A. Mueller, Felix Rauscher. [arXiv:0908.2507] (Aug 2009) 3p. 111) Study of the response of the ATLAS central calorimeter to pions of energies from 3 to 9 GeV. By E. Abat, et al., Nucl.Instrum.Meth.A607:372-386,2009,. 112) Upgrading the ATLAS barrel tracker for the super-LHC. By R.L. Bates. Nucl.Instrum.Meth.A607:24-26,2009,. 113) Commissioning the ATLAS silicon microstrip tracker. By C. Escobar. Nucl.Instrum.Meth.A607:21-23,2009,.
CDF Virtual Organization 1. Search for Single Top Quark Production in p anti-p Collisions at 1.96 TeV in the Missing Transverse Energy Plus Jets Topology. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. D81, 072003 (2010). arXiv: 1001.4577. 2. Measurement of Z_gamma Production in p anti-p Collisions at s**(1/2) 1.96 TeV. T. Aaltonen et al., The CDF Collaboration, submitted to Phys. Rev. D April 7, 2010. FermilabPub-10-073-E. arXiv: 1004.1140.
June 2010
Appendix 3
Page 13
3. A Search for the Higgs Boson Using Neural Networks in Events with Missing Energy and bquark Jets in p anti-p Collisions at s**(1/2) \ufffd.96 TeV. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. Lett. 104, 141801 (2010). arXiv: 0911.3935. 4. Measurement of the B- Lifetime Using a Simulation Free Approach for Trigger Bias Correction. T. Aaltonen et al., The CDF Collaboration, submitted to Phys. Rev. D April 27, 2010. Fermilab-Pub-10-095-E. arXiv: 1044.4855. 5. Measurement of the b-jet Cross Section in Events with a W Boson in p anti-p Collisions at s**(1/2) \ufffd.96 TeV. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. Lett. 104, 131801 (2010). arXiv: 0909.1505. 6. Observation of Single Top Quark Production and Measurement of |V(tb)| with CDF. T. Aaltonen et al., The CDF Collaboration, submitted to Phys. Rev. D April 7, 2010. FermilabPub-10-063-E. arXiv: 1004.1181. 7. First Measurement of the Ratio sigma_tt-bar/sigma_Z/gamma* --> ell ell and Precise Extraction of the tt-bar Cross Section. T. Aaltonen et al., The CDF Collaboration, submitted to Phys. Rev. Lett. April 19, 2010. Fermilab-Pub-10-084-E. arXiv: 1004.3224. 8. Search for R-parity Violating Decays of tau Sneutrinos to e mu, mu tau, and e tau Pairs in p anti-p Collisions at s**(1/2) \ufffd.96 TeV. T. Aaltonen et al., The CDF Collaboration, submitted to Phys. Rev. Lett. April 19, 2010. Fermilab-Pub-10-077-E. arXiv: 1004.3042. 9. Measurement of W-Boson Polarization in Top-quark Decay in p anti-p Collisions at sqrt 1.96 TeV. T. Aaltonen et al., The CDF Collaboration, submitted to Phys. Rev. Lett. March 1, 2010. Fermilab-Pub-10-041-E. arXiv: 1003.0224. 10. Measurement of the Top Quark Mass and p anti-p --> tt-bar Cross Section in the AllHadronic Mode with the CDF II Detector. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. D81, 052001 (2010). arXiv: 1002.0365. 11. Measurement of the WW+WZ Production Cross Section using Lepton+Jets Final State at CDF II. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. Lett. 104, 101801 (2010). arXiv: 0911.4449. 12. Search for New Bottomlike Quark Pair Decays QQ- --> (t W-/+)(t-bar W+/-) in SameCharge Dilepton Events. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. Lett. 104, 091801 (2010). arXiv: 0912.1057. 13. Search for Technicolor Particles Produced in Association with a W Boson at CDF. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. Lett. 104, 111802 (2010). arXiv: 0912.2059.
June 2010
Appendix 3
Page 14
14. Study of the Associated Production of Photons and b-quark Jets in p anti-p Collisions at s**(1/2) \ufffd.96 TeV. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. D81, 052006 (2010). arXiv: 0912.3453. 15. Measurement of the Lambda0(b) Lifetime in Lambda0(b) --> Lambda+_c pi+- Decays in p anti-p Collisions at s**(1/2) \ufffd.96 TeV. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. Lett. 104, 102002 (2010). arXiv: 0912.3566. 16. Measurements of the Branching Fraction Ratios and CP Asymmetries in B --> D0_CP K+/Decays in Hadron Collisions. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. D81, 031105 (2010). arXiv: 0911.0425. 17. Studying the Underlying Event in Drell-Yan and High Transverse Momentum Jet Production at the Tevatron. T. Aaltonen et al., The CDF Collaboration, submitted to Phys. Rev. D March 17, 2010. Fermilab-Pub-10-053-E. arXiv: 1003.3146. 18. Measurement of d sigma/dy of Drell-Yan e+e- Pairs in the Z Mass Region from p anti-p Collisions at s**(1/2) \ufffd.96 TeV. T. Aaltonen et al., The CDF Collaboration, submitted to Physics Letters B March 29, 2010. Fermilab-Pub-10-057-E. arXiv: 0908.3914. 19. Measurement of the Top Pair Production Cross Section in the Dilepton Decay Channel in p anti-p Collisions at s**(1/2) \ufffd.96 TeV. T. Aaltonen et al., The CDF Collaboration, submitted to Phys. Rev. D February 15, 2010. Fermilab-Pub-10-032-E. arXiv: 1002.2919. 20. Measurement of the Top Quark Mass in the Dilepton Channel Using m_t2 at CDF. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. D81, 031102 (2010). arXiv: 0911.2956. 21. Inclusive Search for Standard Model Higgs Boson Production in the WW Decay Channel Using the CDF II Detector. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. Lett. 104, 061803 (2010). arXiv: 1001.4468. 22. Measurement of the tt-bar Production Cross Section in p anti-p Collisions at s**(1/2) \ufffd.96 TeV using Soft Electron b-Tagging. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. D81, 092002 (2010). arXiv: 1002.3783. 23. Search for Supersymmetry with Gauge-Mediated Breaking in Diphoton Events with Missing Transverse Energy at CDF II. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. Lett. 104, 011801 (2010). arXiv: 0910.3606. 24. Combination of Tevatron Searches for the Standard Model Higgs Boson in the W+W- Decay Mode. T. Aaltonen et al., The CDF Collaboration, submitted to Phys. Rev. Lett. January 25, 2010. Fermilab-Pub-10-017-E. arXiv: 1001.4162. 25. Measurement of the W+W- Production Cross Section and Search for Anomalous WWgamma and WWZ Couplings in p anti-p Collisions at s**(1/2) \ufffd.96 TeV. T. Aaltonen et al., The June 2010
Appendix 3
Page 15
CDF Collaboration, submitted to Phys. Rev. Lett. December 22, 2009. Fermilab-Pub-09-635E. arXiv: 0912.4691 26. Search for Pair Production of Supersymmetric Top Quarks in Dilepton Events from p anti-p Collisions at s**(1/2) \ufffd.96 TeV. T. Aaltonen et al., The CDF Collaboration, submitted to Phys. Rev. Lett. December 7, 2009. Fermilab-Pub-09-614-E. 27. Measurement of the Inclusive Isolated Prompt Photon Cross Section in p anti-p Collisions at s**(1/2) \ufffd.96 TeV using the CDF Detector. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. D80, 11106 (2009). arXiv: 0910.3623. 28. A Search for the Associated Production of the Standard Model Higgs Boson in the AllHadronic Channel. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. Lett. 103, 221801 (2009). arXiv: 0907.0810. 29. Search for New Physics with a Dijet plus Missing E(t) Signature in p anti-p Collisions at s**(1/2) \ufffd.96 TeV. T. Aaltonen et al., The CDF Collaboration, submitted to Phys. Rev. Lett. December 23, 2009. Fermilab-Pub-09-635-E. arXiv: 0912.4691. 30. Search for New Color-Octet Vector Particle Decaying to t anti-t in p anti-p Collisions at s**(1/2) \ufffd.96 TeV. T. Aaltonen et al., The CDF Collaboration, submitted to Phys. Lett. B November 16, 2009. Fermilab-Pub-09-578-E. arXiv: 0911.3112. 31. Search for Higgs Bosons Predicted in Two-Higgs-Doublet Models Via Decays to Tau lepton Pairs in 1.96 TeV p anti-p Collisions. A. Aaltonen et al., The CDF Collaboration, Phys. Rev. Lett. 103, 201801 (2009). 32. Measurement of the tt-bar Production Cross Section in 2 fb-1 of p anti-p Collisions at s**(1/2) \ufffd.96 TeV Using Lepton Plus Jets Events with Soft Muon Tagging. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. D79, 052007 (2009). arXiv: 0901.4142. 33. First Observation of B-bar0_s --> D+/-_s K-/+ and Measurement of the Ratio of Branching Fractions B(B-bar0_s --> D+/-_s K-/+)/B(B-bar0_s -> D+_s pi+-). T. Aaltonen et al., The CDF Collaboration, Phys. Rev. Lett. 103, 191802 (2009) 34. Search for Anomalous Production of Events with Two Photons and Additional Energetic Objects at CDF. T. Aaltonen et al., The CDF Collaboration, Fermilab-Pub-09-535-E. arXiv: 0910.5170. Submitted to Phys. Rev. D October 27, 2009. 35. Measurements of the Top-Quark Mass Using Charged Particle Tracking. T. Aaltonen et al., The CDF Collaboration, submitted to Phys. Rev. D October 7, 2009. Fermilab-Pub-09-464E. arXiv: 0910.0969.
June 2010
Appendix 3
Page 16
36. A Search for the Higgs Boson Produced in Association with Z --> ell+ ell- Using the Matrix Element Method at CDF II. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. D80, 071101 (2009). arXiv: 0908.3534. 37. Observation of the Omega_b and Measurement of the Properties of the Xi-_b and Omega-_b Baryons. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. D80, 072003 (2009). arXiv: 0905.3123. 38. Precision Measurement of the X(3872) Mass in J/psi pi+_ pi+- Decays. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. Lett. 103, 152001 (2009). 39. First Observation of Vector Boson Pairs in a Hadronic Final State at the Tevatron Collider. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. Lett. 103, 091803 (2009). arXiv: 0905.4714. 40. Measurement of the Mass of the Top Quark Using the Invariant Mass of Lepton Pairs in Soft Muon b-tagged Events. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. D80, 051104 (2009). arXiv: 0906.5371. 41. Search for a Standard Model Higgs Boson in WH --> l nu bb-bar in p anti-p Collisions at s**(1/2) \ufffd.96 TeV. Phys. Rev. Lett. 103, 101802 (2009). arXiv: 0906.5613. 42. Search for Charged Higgs Bosons in Decays of Top Quarks in p anti-p Collisions at s**(1/2) \ufffd.96 TeV. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. Lett. 103, 101803 (2009). arXiv: 0907.1269. 43. Search for the Neutral Current Top Quark Decay t --> Zc Using Ratio of Z-Boson + 4 Jets to W-Boson + 4 Jets Production. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. D80, 052001 (2009). arXiv: 0905.0277. 44. Search for Anomalous Production of Events with a Photon, Jet, b-quark Jet, and Missing Transverse Energy. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. D80, 052003 (2009). arXiv: 0905.0231. 45. Search for Hadronic Decays of W and Z Bosons in Photon Events in p anti-p Collisions at s**(1/2) \ufffd.96 TeV. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. D80, 052011 (2009). arXiv: 0803.4264. 46. Measurement of d sigma/dy of Drell-Yan e+e- Pairs in the Z Mass Region from p anti-p Collisions at s**(1/2) \ufffd.96 TeV. T. Aaltonen et al., The CDF Collaboration, FermilabPub-09-402-E. arXiv: 0908.3914. Submitted to Phys. Rev. Lett. August 27, 2009. 47. Observation of Electroweak Single Top Quark Production. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. Lett. 103, 092002 (2009). arXiv: 0903.0885. June 2010
Appendix 3
Page 17
48. Search for a Fermiophobic Higgs Boson Decaying into Diphotons in p anti-p Collisions at s**(1/2) \ufffd.96 TeV. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. Lett. 103, 061803 (2009). arXiv: 0905.0413. 49. Search for the Production of Narrow tb-bar Resonances in 1.9 fb-1 of p anti-p Collisions at s**(1/2) \ufffd.96 TeV. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. Lett. 103, 041801 (2009). arXiv: 0902.3276. 50. Production of psi(2S) Mesons in p anti-p Collisions at 1.96 TeV. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. D80, 031103 (2009). arXiv: 0905.1982. 51. Search for Hadronic Decays of W and Z in Photon Events in p anti-p Collisions at s**(1/2) \ufffd.96 TeV. T. Aaltonen et al., The CDF Collaboration, Eur. Phys. J. C62: 319:326 (2009). arXiv: 0903.2060. 52. Search for a Standard Model Higgs Boson Production in Association with a W Boson using a Neural Network Discriminant at CDF. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. D80, 012002 (2009) 53. Search for Long-Lived Massive Charged Particles in 1.96 TeV p anti-p Collisions. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. Lett. 103, 021802 (2009). arXiv: 0902.1266. 54. Observation of New Charmless Decays of Bottom Hadrons. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. Lett. 103, 031801 (2009) 55. Precision Measurement of the X(3872) Mass in J/psi pi++ pi+- Decays. T. Aaltonen et al., The CDF Collaboration, Fermilab-Pub-09-330-E. arXiv: 0906.5218. Submitted to Phys. Rev. Lett. June 29, 2009. 56. Evidence for a Narrow Near-Threshold Structure in the J/psi phi Mass Spectrum in B+ --> J/psi phi K+ Decays. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. Lett. 102, 242002 (2009). arXiv: 0903.2229. 57. Measurement of the Particle Production and Inclusive Differential Cross Sections in p anti-p Collisions at s**(1/2) \ufffd.96 TeV. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. D79, 112005 (2009). 58. Search for WW and WZ Production in Lepton Plus Jets Final States at CDF. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. D79, 112011 (2009). arXiv: 0903.0814. 59. A Measurement of the t anti-t Cross Section in p anti-p Collisions at sqrt s \ufffd.96 TeV using Dilepton Events with a Lepton plus Track Selection. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. D79, 112007 (2009). arXiv: 0903.5263. June 2010
Appendix 3
Page 18
60. Observation of Exclusive Charmonium Production and gamma gamma -> mu+ mu- in p antip Collisions at s**(1/2) \ufffd.96 TeV. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. Lett. 102, 242001 (2009). arXiv: 0902.1271. 61. Measurement of the k_T Distribution of Particles in Jets Produced in p anti-p Collisions at s**(1/2) \ufffd.96 TeV. T. Aaltonen et al., The CDF Collaboration, Phys. Rev. Lett. 102, 232002 (2009). arXiv: 0811.2820.
CDMS Virtual Organization 1. Analysis of the low-energy electron-recoil spectrum of the CDMS experiment. By CDMS Collaboration (Z. Ahmed et al.). FERMILAB-PUB-09-340-E, Jul 2009. (Published Feb 15, 2010). 4pp. Published in Phys.Rev.D81:042002, 2010. e-Print: arXiv:0907.1438 [astro-ph.GA] 2. Search for Axions with the CDMS Experiment. By CDMS Collaboration (Z. Ahmed et al.). FERMILAB-PUB-09-053-E, Feb 2009. (Published Oct 2, 2009). 5pp. Published in Phys.Rev.Lett.103:141802,2009. e-Print: arXiv:0902.4693 [hep-ex] 3. Search for Weakly Interacting Massive Particles with the First Five-Tower Data from the Cryogenic Dark Matter Search at the Soudan Underground Laboratory By CDMS Collaboration (Z. Ahmed et al.). FERMILAB-PUB-08-055-A-E, Feb 2008. (Published Jan 9, 2009). 5pp. Published in Phys.Rev.Lett.102:011301,2009. e-Print: arXiv:0802.3530 [astro-ph] 4. Dark Matter Search Results from the CDMSII Experiment Science The CDMS II Collaboration 327 (5973): 1619 (published March 2010)
June 2010
Appendix 3
Page 19
CIGI Virtual Organization 1. Padmanabhan, A., Ghosh, S., and Wang, S. 2010. “A Self-Organized Grouping (SOG) Method for Efficient Grid Resource Discovery.” Journal of Grid Computing, In press. DOI 10.1007/s10723-009-9145-0.
CMS Virtual Organization 1. Transverse momentum and pseudorapidity distributions of charged hadrons in pp collisions at = 0.9 and 2.36 TeV Measurements of inclusive charged-hadron transverse-momentum and pseudorapidity distributions are presented for proton-proton collisions at sqrt(s) = 0.9 and 2.36 TeV. The data were collected with the CMS detector during the LHC commissioning in December 2009. [...] arXiv:1002.0621; CMS-QCD-09-010; CERN-PH-EP-2010-003.- Geneva : CERN, 2010 - 33 p. - Published in : J. High Energy Phys. 02 (2010) 041 Preprint ; Supplemental material ; Figure_001a ; Figure_001b ; Figure_002a ; Figure_002b ; Figure_003a ; Figure_003b ; Figure_004a ; Figure_004b ; Figure_005a ; Figure_005b ; Figure_006a ; Figure_006b ; Figure_007a ; Figure_007b ; Springer Open Access article ; 2. Commissioning and Performance of the CMS Pixel Tracker with Cosmic Ray Muons / CMS Collaboration The pixel detector of the Compact Muon Solenoid experiment consists of three barrel layers and two disks for each endcap. The detector was installed in summer 2008, commissioned with charge injections, and operated in the 3.8 T magnetic field during cosmic ray data taking. [...] arXiv:0911.5434; CMS-CFT-09-001.- Geneva : CERN, 2010 - 37 p. - Published in : J. Instrum. 5 (2010) T03007 Preprint ; SISSA/IOP Open Access article 3. Measurement of the Muon Stopping Power in Lead Tungstate / CMS Collaboration A large sample of cosmic ray events collected by the CMS detector is exploited to measure the specific energy loss of muons in the lead tungstate of the electromagnetic calorimeter. The measurement spans a momentum range from 5 GeV/c to 1 TeV/c. [...] June 2010
Appendix 3
Page 20
arXiv:0911.5397; CMS-CFT-09-005.- 2010 - 31 p. - Published in : J. Instrum. 5 (2010) P03007 Preprint ; SISSA/IOP Open Access article 4. Performance of the CMS Level-1 Trigger during Commissioning with Cosmic Ray Muons and LHC beams / CMS Collaboration The CMS Level-1 trigger was used to select cosmic ray muons and LHC beam events during data-taking runs in 2008, and to estimate the level of detector noise. This paper describes the trigger components used, the algorithms that were executed, and the trigger synchronisation. [...] arXiv:0911.5422; CMS-CFT-09-013.- 2010 - 49 p. - Published in : J. Instrum. 5 (2010) T03002 Preprint ; SISSA/IOP Open Access article 5. Performance of CMS Muon Reconstruction in Cosmic-Ray Events / CMS Collaboration The performance of muon reconstruction in CMS is evaluated using a large data sample of cosmic-ray muons recorded in 2008. Efficiencies of various high-level trigger, identification, and reconstruction algorithms have been measured for a broad range of muon momenta, and were found to be in good agreement with expectations from Monte Carlo simulation. [...] arXiv:0911.4994; CMS-CFT-09-014.- 2010 - 47 p. - Published in : J. Instrum. 5 (2010) T03022 Preprint ; SISSA/IOP Open Access article 6. Commissioning and Performance of the CMS Silicon Strip Tracker with Cosmic Ray Muons / CMS Collaboration During autumn 2008, the Silicon Strip Tracker was operated with the full CMS experiment in a comprehensive test, in the presence of the 3.8 T magnetic field produced by the CMS superconducting solenoid. Cosmic ray muons were detected in the muon chambers and used to trigger the readout of all CMS sub-detectors. [...] arXiv:0911.4996; CMS-CFT-09-002.- 2010 - 45 p. - Published in : J. Instrum. 5 (2010) T03008 Preprint ; SISSA/IOP Open Access article 7. Performance of the CMS Cathode Strip Chambers with Cosmic Rays / CMS Collaboration The Cathode Strip Chambers (CSCs) constitute the primary muon tracking device in the CMS endcaps. Their performance has been evaluated using data taken during a cosmic ray June 2010
Appendix 3
Page 21
run in fall 2008. [...] arXiv:0911.4992; CMS-CFT-09-011.- 2010 - 39 p. - Published in : J. Instrum. 5 (2010) T03018 Preprint ; SISSA/IOP Open Access article 8. Performance of the CMS Hadron Calorimeter with Cosmic Ray Muons and LHC Beam Data / CMS Collaboration The CMS Hadron Calorimeter in the barrel, endcap and forward regions is fully commissioned. Cosmic ray data were taken with and without magnetic field at the surface hall and after installation in the experimental hall, hundred meters underground. [...] arXiv:0911.4991; CMS-CFT-09-009.- 2010 - 35 p. - Published in : J. Instrum. 5 (2010) T03012 Preprint ; SISSA/IOP Open Access article 9. CMS Data Processing Workflows during an Extended Cosmic Ray Run / CMS Collaboration The CMS Collaboration conducted a month-long data taking exercise, the Cosmic Run At Four Tesla, during October-November 2008, with the goal of commissioning the experiment for extended operation. With all installed detector systems participating, CMS recorded 270 million cosmic ray events with the solenoid at a magnetic field strength of 3.8 T. [...] arXiv:0911.4842; CMS-CFT-09-007.- 2010 - 43 p. - Published in : J. Instrum. 5 (2010) T03006 Preprint ; SISSA/IOP Open Access article 10. Performance of the CMS Drift Tube Chambers with Cosmic Rays / CMS Collaboration Studies of the performance of the CMS drift tube barrel muon system are described, with results based on data collected during the CMS Cosmic Run at Four Tesla. For most of these data, the solenoidal magnet was operated with a central field of 3.8 T. [...] arXiv:0911.4855; CMS-CFT-09-012.- 2010 - 47 p. - Published in : J. Instrum. 5 (2010) T03015 Preprint ; SISSA/IOP Open Access article 11. Calibration of the CMS Drift Tube Chambers and Measurement of the Drift Velocity with Cosmic Rays / CMS Collaboration This paper describes the calibration procedure for the drift tubes of the CMS barrel muon system and reports the main results obtained with data collected during a high statistics cosmic ray data-taking period. The main goal of the calibration is to determine, for each drift June 2010
Appendix 3
Page 22
cell, the minimum time delay for signals relative to the trigger, accounting for the drift velocity within the cell. [...] arXiv:0911.4895; CMS-CFT-09-023.- 2010 - 39 p. - Published in : J. Instrum. 5 (2010) T03016 Preprint ; SISSA/IOP Open Access article 12. Commissioning of the CMS Experiment and the Cosmic Run at Four Tesla / CMS Collaboration The CMS Collaboration conducted a month-long data-taking exercise known as the Cosmic Run At Four Tesla in late 2008 in order to complete the commissioning of the experiment for extended operation. The operational lessons resulting from this exercise were addressed in the subsequent shutdown to better prepare CMS for LHC beams in 2009. [...] arXiv:0911.4845; CMS-CFT-09-008.- 2010 - 37 p. - Published in : J. Instrum. 5 (2010) T03001 Preprint ; SISSA/IOP Open Access article 13. Identification and Filtering of Uncharacteristic Noise in the CMS Hadron Calorimeter / CMS Collaboration Commissioning studies of the CMS hadron calorimeter have identified sporadic uncharacteristic noise and a small number of malfunctioning calorimeter channels. Algorithms have been developed to identify and address these problems in the data. [...] arXiv:0911.4881; CMS-CFT-09-019.- 2010 - 31 p. - Published in : J. Instrum. 5 (2010) T03014 Preprint ; SISSA/IOP Open Access article 14. Commissioning of the CMS High-Level Trigger with Cosmic Rays / CMS Collaboration The CMS High-Level Trigger (HLT) is responsible for ensuring that data samples with potentially interesting events are recorded with high efficiency and good quality. This paper gives an overview of the HLT and focuses on its commissioning using cosmic rays. [...] arXiv:0911.4889; CMS-CFT-09-020.- 2010 - 31 p. - Published in : J. Instrum. 5 (2010) T03005 Preprint ; SISSA/IOP Open Access article 15. Aligning the CMS Muon Chambers with the Muon Alignment System during an Extended Cosmic Ray Run / CMS Collaboration
June 2010
Appendix 3
Page 23
The alignment system for the muon spectrometer of the CMS detector comprises three independent subsystems of optical and analog position sensors. It aligns muon chambers with respect to each other and to the central silicon tracker. [...] arXiv:0911.4770; CMS-CFT-09-017.- 2010 - 35 p. - Published in : J. Instrum. 5 (2010) T03019 Preprint ; SISSA/IOP Open Access article 16. Performance of the CMS drift-tube chamber local trigger with cosmic rays / CMS Collaboration The performance of the Local Trigger based on the drift-tube system of the CMS experiment has been studied using muons from cosmic ray events collected during the commissioning of the detector in 2008. The properties of the system are extensively tested and compared with the simulation. [...] arXiv:0911.4893; CMS-CFT-09-022.- 2010 - 33 p. - Published in : J. Instrum. 5 (2010) T03003 Preprint ; SISSA/IOP Open Access article 17. Fine Synchronization of the CMS Muon Drift-Tube Local Trigger using Cosmic Rays / CMS Collaboration The CMS experiment uses self-triggering arrays of drift tubes in the barrel muon trigger to perform the identification of the correct bunch crossing. The identification is unique only if the trigger chain is correctly synchronized. [...] arXiv:0911.4904; CMS-CFT-09-025.- 2010 - 33 p. - Published in : J. Instrum. 5 (2010) T03004 Preprint ; SISSA/IOP Open Access article 18. Performance of CMS hadron calorimeter timing and synchronization using test beam, cosmic ray, and LHC beam data / CMS Collaboration This paper discusses the design and performance of the time measurement technique and of the synchronization systems of the CMS hadron calorimeter. Timing performance results are presented from the Cosmic Run At Four Tesla and LHC beam runs taken in the Autumn of 2008. [...] arXiv:0911.4877; CMS-CFT-09-018.- 2010 - 33 p. - Published in : J. Instrum. 5 (2010) T03013 Preprint ; SISSA/IOP Open Access article
June 2010
Appendix 3
Page 24
19. Alignment of the CMS Muon System with Cosmic-Ray and Beam-Halo Muons / CMS Collaboration The CMS muon system has been aligned using cosmic-ray muons collected in 2008 and beam-halo muons from the 2008 LHC circulating beam tests. After alignment, the resolution of the most sensitive coordinate is 80 microns for the relative positions of superlayers in the same barrel chamber and 270 microns for the relative positions ofendcap chambers in the same ring structure. [...] arXiv:0911.4022; CMS-CFT-09-016.- 2010 - 41 p. - Published in : J. Instrum. 5 (2010) T03020 Preprint ; SISSA/IOP Open Access article 20. Performance Study of the CMS Barrel Resistive Plate Chambers with Cosmic Rays / CMS Collaboration In October and November 2008, the CMS collaboration conducted a programme of cosmic ray data taking, which has recorded about 270 million events. The Resistive Plate Chamber system, which is part of the CMS muon detection system, was successfully operated in the full barrel. [...] arXiv:0911.4045; CMS-CFT-09-010.- 2010 - 33 p. - Published in : J. Instrum. 5 (2010) T03017 Preprint ; SISSA/IOP Open Access article 21. Time Reconstruction and Performance of the CMS Electromagnetic Calorimeter / CMS Collaboration The resolution and the linearity of time measurements made with the CMS electromagnetic calorimeter are studied with samples of data from test beam electrons, cosmic rays, and beam-produced muons. The resulting time resolution measured by lead tungstate crystals is better than 100 ps for energy deposits larger than 10 GeV. [...] arXiv:0911.4044; CMS-CFT-09-006.- 2010 - 27 p. - Published in : J. Instrum. 5 (2010) T03011 Preprint ; SISSA/IOP Open Access article 22. Precise Mapping of the Magnetic Field in the CMS Barrel Yoke using Cosmic Rays / CMS Collaboration Task physics.acc-ph The CMS detector is designed around a large 4 T superconducting solenoid, enclosed in a 12000-tonne steel return yoke. A detailed map of the magnetic field is required for the accurate simulation and reconstruction of physics events in the CMS detector, not only in the June 2010
Appendix 3
Page 25
inner tracking region inside the solenoid but also in the large and complex structure of the steel yoke, which is instrumented with muon chambers. [...] arXiv:0910.5530; CMS-CFT-09-015.- 2010 - 37 p. - Published in : J. Instrum. 5 (2010) T03021 Preprint ; SISSA/IOP Open Access article 23. Performance and Operation of the CMS Electromagnetic Calorimeter / CMS Collaboration Task physics.acc-ph The operation and general performance of the CMS electromagnetic calorimeter using cosmic-ray muons are described. These muons were recorded after the closure of the CMS detector in late 2008. [...] arXiv:0910.3423; CMS-CFT-09-004.- 2010 - 39 p. - Published in : J. Instrum. 5 (2010) T03010 Preprint ; SISSA/IOP Open Access article 24. Alignment of the CMS Silicon Tracker during Commissioning with Cosmic Rays / CMS Collaboration Task physics.acc-ph The CMS silicon tracker, consisting of 1440 silicon pixel and 15148 silicon strip detector modules, has been aligned using more than three million cosmic ray charged particles, with additional information from optical surveys. The positions of the modules were determined with respect to cosmic ray trajectories to a precision of 3-4 microns RMS in the barrel and 314 microns RMS in the endcap in the most sensitive coordinate. [...] arXiv:0910.2505; CMS-CFT-09-003.- 2010 - 41 p. - Published in : J. Instrum. 5 (2010) T03009 Preprint ; SISSA/IOP Open Access article 25. The CMS experiment at the CERN LHC / CMS Collaboration 2008 - Published in : J. Instrum. 3 (2008) S08004 SISSA/IOP Open Access article Presented at : The CERN Large Hadron Collider 26. CMS technical design report, volume II: Physics performance 2007 - Published in : J. Phys. G 34 (2007) 995-1579 27. CMS Physics : Technical Design Report v.2: Addendum on High Density QCD with Heavy Ions / d'Enterria, D (ed.) (CERN) ; Ballintijn, M (ed.) (MIT) ; Bedjidian, M (ed.) (Lyon, IPN) ; Hoffman, D (ed.) (Illinois U., Chicago) ; Kodolova, Olga (ed.) (Moscow State U.) ; Loizides, C (ed.) (MIT) ; Lokhtin, I P (ed.) (Moscow State U.) ; Lourenco, C (ed.) (CERN) ; June 2010
Appendix 3
Page 26
Mironov, C (ed.) (Illinois U., Chicago) ; Petrushanko, S V (ed.) (Moscow State U.) et al. CMS-TDR-008.2-add-1; CERN-LHCC-2007-009.- Geneva : CERN, 2007 - 172 p. Technical Design Report CMS, 8.2-add.1 - Published in : J. Phys. G 34 (2007) 2307-2455
DZero Virtual Organization Submitted: 1. Search for CP Violation in Semileptonic Bs0 Decays, Submitted 4/29/09: Phys. Rev. Lett., arXiv.org:0904.3907 2. Measurement of the tt Cross Section using High-Multiplicity Jet Events, Submitted 11/23/09: Phys. Rev. D, arXiv.org:0911.4286 3. Dependence of the tt Production Cross Section on the Transverse Momentum of the Top Quark, Submitted 1/12/10: Phys. Lett. B, arXiv.org:1001.1900 4. Measurement of the Dijet Invariant Mass Cross Section in pbarp Collisions at ¥V 7H9, Submitted 2/25/10: Phys. Lett. B, arXiv.org:1002.4594 5. Search for Randall-Sundrum Gravitons in the Dielectron and Diphoton Final States with 5.4 fb-1 of Data from pp Collisions at ¥V 7H9, Submitted 4/12/10: Phys. Rev. Lett., arXiv.org:1004.1826 6. Search for Scalar Bottom Quarks and Third-Generation Leptoquarks in pbarp Collisions at ¥V = 1.96 TeV, Submitted 5/13/10: Phys. Lett. B, arXiv.org:1005.2222 7. Evidence for an Anomalous Like-Sign Dimuon Charge Asymmetry, Submitted 5/16/10: Phys. Rev. D, arXiv.org:1005.2757 8. Combined Tevatron Upper Limit on ggĺ+ĺ::DQG Constraints on the Higgs Boson Mass in Fourth-Generation Fermion Models, Co-authors: DØ and CDF collaborations. Submitted 5/18/10: Phys. Rev. D, arXiv.org:1005.xxxx Accepted: 1. b-Jet Identification in the DØ Experiment, Accepted 3/14/10: Nucl. Instrum. Methods A, arXiv.org:1002.4224
June 2010
Appendix 3
Page 27
2. Search for Single Top Quarks in the Tau+Jets Channel using 4.8 fb-1 of pbarp Collision Data, Accepted 5/3/10: Phys. Lett. B, arXiv.org:0912.1066 3. Measurement of Direct Photon Pair Production Cross Sections in pp Collisions at ¥V TeV, Accepted 5/10/10: Phys. Lett. B, arXiv.org:1002.4917
Published: 1. Search for the Associated Production of a b Quark and a Neutral Supersymmetric +LJJV%RVRQWKDW'HFD\VLQWRIJ3DLUV, Phys. Rev. Lett. 104, 151801 (2010) 2. 'RXEOH3DUWRQ,QWHUDFWLRQVLQȖ-HW(YHQWVLQSEDUS&ROOLVLRQVDW¥V 7H9 Phys. Rev. D 81, 052012 (2010) 3. Search for the Standard Model Higgs Boson in the ZHĺȞȞEE&KDQQHOLQIE -1 of pbarp Collisions at ¥V 7H9, Phys. Rev. Lett. 104, 071801 (2010) 4. Search for Higgs Boson Production in Dilepton and Missing Energy Final States with 5.4 fb-1 of pbarp Collisions at ¥V 7H9, Phys. Rev. Lett. 104, 061804 (2010) 5. Combination of Tevatron Searches for the Standard Model Higgs Boson in the W+W- Decay Mode, Phys. Rev. Lett. 104, 061802 (2010) 6. Search for a Resonance Decaying into WZ Boson Pairs in pp Collisions Phys. Rev. Lett. 104, 061801 (2010) 7. Determination of the Strong Coupling Constant from the Inclusive Jet Cross Section in pbarp Collisions at ¥V 7H9, Phys. Rev. D 80, 111107 (2009) 8. Direct Measurement of the W Boson Width, Phys. Rev. Lett. 103, 231802 (2009) 9. Measurement of the Top Quark Mass in Final States with Two Leptons, Phys. Rev. D 80, 092006 (2009)
June 2010
Appendix 3
Page 28
10. Measurement of the t-Channel Single Top Quark Production Cross Section Phys. Lett. B 682, 363 (2010) 11. Measurement of Z/Ȗ MHW;$QJXODU'LVWULEXWLRQVLQSEDUS&ROOLVLRQVDW¥V 7H9 Phys. Lett. B 682, 370 (2010) 12. Search for Charged Higgs Bosons in Top Quark Decays, Phys. Lett. B 682, 278 (2009) 13. Measurement of Dijet Angular Distributions at ¥V 7H9Dnd Searches for Quark Compositeness and Extra Spatial Dimensions, Phys. Rev. Lett. 103, 191803 (2009) 14. Measurement of the WW Production Cross Section with Dilepton Final States in pbarp Collisions at ¥V 7H9DQG/LPLWVRQ$QRPDORXV7ULOLQHDU*DXJH&RXSOLQJV Phys. Rev. Lett. 103, 191801 (2009) 15. Combination of tt Cross Section Measurements and Constraints on the Mass of the Top Quark and its Decay into Charged Higgs Bosons, Phys. Rev. D 80, 071102 (2009) 16. Search for Pair Production of First-Generation Leptoquarks in pbarp Collisions at ¥V TeV, Phys. Lett. B 681, 224 (2009) 17. Measurement of the W Boson Mass, Phys. Rev. Lett. 103, 141801 (2009) 18. Search for Charged Higgs Bosons in Decays of Top Quarks, Phys. Rev. D 80, 051107 (2009) 19. Measurement of Trilinear Gauge Boson Couplings from WW + WZ ĺOȞMM(YHQWV in pbarp Collisions at ¥V 7H9, Phys. Rev. D 80, 053012 (2009) 20. Direct Measurement of the Mass Difference Between Top and Antitop Quarks Phys. Rev. Lett. 103, 132001 (2009)
June 2010
Appendix 3
Page 29
21. A Novel Method for Modeling the Recoil in W Boson Events at Hadron Colliders Nucl. Instrum. Methods in Phys. Res. A 609, 250 (2009) 22. Observation of Single Top-Quark Production, Phys. Rev. Lett. 103, 092001 (2009) 23. Search for Dark Photons from Supersymmetric Hidden Valleys Phys. Rev. Lett. 103, 081802 (2009) 24. Search for Associated Production of Charginos and Neutralinos in the Trilepton Final State using 2.3 fb-1 of Data Phys. Lett. B 680, 34 (2009) 25. Search for Resonant Pair Production of Neutral Long-Lived Particles Decaying to bbarb in pbarp Collisions at ¥V 7H9, Phys. Rev. Lett. 103, 071801 (2009) 26. Search for Squark Production in Events with Jets, Hadronically Decaying Tau Leptons and Missing Transverse Energy at ¥V = 1.96 TeV, Phys. Lett. B 680, 24 (2009) 27. Search for Next-to-Minimal Supersymmetric Higgs Bosons in the h ĺDDĺȝȝȝȝȝȝIJIJ Channels using pp Collisions at ¥V 7H9, Phys. Rev. Lett. 103, 061801 (2009) 28. Measurement of the tt Production Cross Section and Top Quark Mass Extraction Using Dilepton Events in pp Collisions, Phys. Lett. B 679, 177 (2009)
ENGAGE Virtual Organization 1.
Feng Zhao, ShuaiCheng Li, Beckett W. Sterner and Jinbo Xu. Discriminative learning for protein conformation sampling. PROTEINS: Structure, Function and Bioinformatics, 2008 Oct; 73(1):228-40.
2.
A. D. Rosato, V. Ratnaswamy, D. J. Horntrop, L. Kondic, “Discrete element modeling of tapped density relaxation”, IUTAM-ISIMM Symposium on Mathematical Modeling and Physical Instances of Granular Flows, Sept. 14-18, 2009, Reggio-Calabria, Italy. (Presentations/Posters)
June 2010
Appendix 3
Page 30
3.
M. A. Murphy and S. Goasguen “Virtual Organization Clusters: Self-Provisioned Clouds on the Grid” submitted to Elsevier Journal of Future Generation Computer Systems special issue on Cloud Computing, (Accepted Oct 2009 with revisions)
4.
L. Stout, M. Murphy and S. Goasguen. “Kestrel: An XMPP based Framework for Many-Task Computing Applications” 2nd Workshop on Many-Task Computing on Grids and Supercomputers, Portland, OR, November 16th 2009 (MTAGS'09)
5.
Rosato, A.D., Ratnaswamy, V., Horntrop, D.J., Dybenko, O., Kondic, L., "A Concise Review of Tapped Density Relaxation and Recent Discrete Element Results", Proceedings of the IUTAM Symposium \Mathematical Modeling and Physical Instances of Granular Flows, ed. J. Goddard, Calabria, Italy (2009).
6.
M. Fenn, J. Lauret and S. Goasguen “Contextualization in Practice: The Clemson Experience” ACAT , Jaipur, India February 2010.
7.
M. A. Murphy, L. Abraham, M. Fenn and S. Goasguen “Autonomic Clouds on the Grid” Journal of Grid Computing Volume 8, Number 1 (March 2010), pages 1-18.
8.
V. Patel, J. McGregor and S. Goasguen “SysFlow: A Workflow Engine for Service Based Infrastructures” IEEE Sysconf 2010, May 2010, CA
9.
A. D. Rosato, O. Dybenko, V. Ratnaswamy, D. Horntrop, Nathan Andrysco, Xavier Tricoche, Lou Kondic, "Microstructure Evolution in Density Relaxation by Tapping", Frontiers in Applied and Computational Mathematics, New Jersey Institute of Technology, Newark, NJ, USA, May 2010.
June 2010
Appendix 3
Page 31
10.
L. Stout, M. Fenn, M. Murphy and S. Goasguen "Scaling Virtual Organization Clusters over a Wide Area Network using the Kestrel Workload Management System" VTDC, June 2010, Chicago
11.
L. Abraham, M. Fenn, M. Murphy and S. Goasguen “Self-Provisioned Hybrid Clouds” 7th IEEE International Conference on Autonomic Computing, June 7-11, 2010, D.C, USA
12.
A. D. Rosato, O. Dybenko, V. Ratnaswamy, D. Horntrop, Nathan Andrysco, Xavier Tricoche, Lou Kondic, "Microstructure Evolution in Density Relaxation by Tapping", Gordon Research Conference: Granular & Granular Fluid Flow, Colby College, Waterville, Maine, USA, June 2010
13.
A. D. Rosato, O. Dybenko, D. J. Horntrop, V. Ratnaswamy, L. Kondic, “Microstructure Evolution in Density Relaxation by Tapping”, to appear in Phys. Rev. E
14.
J.McGee, M.Rynge, J.Reilly. Building Solutions for Researchers with a Science Gateway. TeraGrid 09 conference. http://archive.teragrid.org/tg09/files/tg09_submission_75.pdf
15.
A. Prlic UCSD, C. Bizon RENCI, Calculating all Pairwise Similarities from the RCSB Protein Data Bank: Client/Server Work Distribution on the Open Science Grid. http://www.renci.org/publications/techreports/TR-09-03.pdf
16.
Tao Wu, X. Sheldon Wang, Barry Cohen and Hongya Ge. Molecular Modeling of Normal and Sickle Hemoglobins, IJMCE. Accepted.
17.
Feng Zhao, Jian Peng and Jinbo Xu. Fragment-free Approach to Protein Folding Using Conditional Neural Fields. Bioinformatics (Proceedings of ISMB 2010), 2010.
June 2010
Appendix 3
Page 32
18.
Feng Zhao, Jian Peng, Joe DeBartolo, Karl F. Freed, Tobin R. Sosnick and Jinbo Xu. A probabilistic and continuous model of protein conformational space for templatefree modeling. Journal of Computational Biology, 2010. In Press.
GLOW Virtual Organization 1. Methods for point-source analysis with temporal structure in high-energy neutrino telescopes, J. Braun, et al., Astropart.Phys.33:175-181,2010. e-Print: arXiv:0912.1572
2. Limits on a muon flux from neutralino annihilations in the Sun with the IceCube 22-string detector, R. Abbasi, et al. (IceCube collaboration), Physical Review Letters 102 (2009) 201302,
3. Determination of the Atmospheric Neutrino Flux and Searches for New Physics with AMANDA-II, A. Achterberg et al. (IceCube collaboration), Phys.Rev.D79:102005,2009
4. Search for point sources of high-energy neutrinos with final data from AMANDA-II, IceCube collaboration, Physical Review D79 (2009) 062001; astro-ph/08091646.
5. Teague, B., Waterman, M.S., Goldstein, S., Potamousis, K., Zhou, S., Reslewic, S., Sarkar, D., Valouev, A., Churas, C., Kidd, J., Kohn, S., Runnhein, R., Lamers, C., Forrest, D., Newton, M.A., Eichler, E.E., Kent-First, M., Surti, U., Livny, M., and Schwartz, D.C. Highresolution human genome structure by single molecule analysis. Proc. Natl. Acad. Sci. USA, epub May 31, 2010 doi:10.1073/pnas.0914638107.
6. Ma, L., van der Doe, H. C., Borkovich, K.A., Coleman, J.C., Daboussi, M.-J., Di Pietro, A., Dufresne, M., Freitag, M., Grabherr, M., Henrissat, B., Houterman, P.M., Kang, S., Shim, W.-B., Woloshuk, C., Xie, X., Xu, J.-R., Antoniw, J., Baker, S.E., Bluhm, B.H., Breakspear, A., Brown, D.W., Butchko, R.A.E., Chapman, S., Coulson, R., Coutinho, P.M., Danchin, E.G.J., Diener, A., Gale, L.R., Gardiner, D.M., Goff, S., Hammond-Kosack, K.E., Hilburn, K., Hua-Van, A., Jonkers, W., Kazan, K., Kodira, C.D., Michael Koehrsen, M., Kumar, L., Lee, Y.-H., Li , L. Manners, J.M., Miranda-Saavedra, D, Mukherjee, M., Park, G., Park, J., Park, S.-Y., Robert H. Proctor , Regev, A., Ruiz-Roldan, M.C., Sain, D., Sakthikumar, S., Sykes, S., Schwartz, D.C., Turgeon, B.G., Wapinski, I., Yoder, O., Young, S., Zeng, Q., June 2010
Appendix 3
Page 33
Zhou, S., Galagan, J., Cuomo, C.A., Kistler, H.C and Rep, M.. Comparative genomics reveals mobile pathogenicity chromosomes in Fusarium. Nature, epubl March 18, 2010.
7. Schnable, P.S. et al. (~100 authors). The B73 maize genome: complexity, diversity and dynamics. Science 326: 1112-1115, 2009.
8. Zhou, S., Wei, F., Nguyen, J., Bechner, M., Potamousis, K., Goldstein, S., Pape, L., Mehan, M., Churas, C., Pasternak, S., Forrest, D.K., Wise, R., Ware, D., Wing, R., Waterman, M.S., Livny, M., and Schwartz, D.C. A single molecule scaffold for the maize genome. PLoS Genetics 5: e1000711, 2009.
9. Wei, F., Zhang, J., Zhou, S., He, R., Schaeffer, M., Collura, K., Kudrna, D., Faga, B.P., Wissotski, M., Golser, W., Rock, S.M., Graves, T.A., Fulton, R.S., Coe, E., Schnable, P.S., Schwartz, D.C., Ware, D. Clifton, S.W., Wilson, R.K., and Wing, R.W. The physical and genetics framework of the Maize B73 genome. PLoS Genetics 5: e1000715, 2009.
10. Wei, F., Stein, J.C., 2, Liang, C., Zhang, J., Fulton, R.S., Baucom, R.S., De Paoli, E., Zhou, S., 6, Yang, L., Han, Y., Pasternak, S., Narechania, A., Zhang, L., Yeh, C.-T., Ying, K., Nagel, D.H., Collura, K., Kudrna, D., Currie, J., Lin, J., Kim, H.R., Angelove, A., Scara, G., Wissotski, M., Golser, W., Courtney, L., Kruchowski, S., Graves, T., Rock, S., Adams, S., Fulton, L., Fronick, C., Courtney, W., Kramer, M., Spiegel, L., Nascimento, L., Kalyanaraman, A., Chaparro, C., Deragon, J.-M., SanMiguel, P., 11, Ning Jiang, N., Wessler, S.R., Green, P.J., Yeisoo Yu, Y., Schwartz, D.C., Meyers, B.C., Bennetzen, J., Martienssen, R., McCombie, W.R., Srinivas Aluru, S., Clifton, S.W., Schnable, P.S., Ware, D., Wilson, R.K., Wing, R.A., Detailed analysis of a contiguous 22-Mb region of the maize genome, PLoS Genetics 5: e1000728, 2009.
11. Sambriski, E.J., Schwartz, D.C., and de Pablo, J.J. Uncovering pathways in DNA oligonucleotide hybridization via transition state analysis. Proc Natl Acad Sci U S A 106(43):18125-30. Epub Oct 8, 2009.
12. Coleman, J.J., Rounsley, S.D., Rodriguez-Carres, M., Kuo, A., Wasmann, C.C., Grimwood, J., Schmutz, J. , Taga, M., White, G.J., Zhou, S., Schwartz, D.C., Freitag, M., Ma, L., Danchin, E.G.J., Henrissat, B., Coutinho, P.M., Nelson, D.R., Straney, D., Napoli, C.A., June 2010
Appendix 3
Page 34
Gribskov, M., Rep, M., Kroken, S., Molnár, I., Rensing, C., Kennell, J.C., Zamora, J., Farman, M.L., Selker, E.U., Salamov, A., Shapiro, H., Pangilinan, J., Lindquist, E., Lamers, C., Grigoriev, I., Geiser, D.M., Covert, S.F., Temporini, E., and VanEtten, H.D. The genome of Nectria haematococca: contribution of supernumerary chromosomes to gene expansion. PLoS Genetics 5(8): e1000618, 2009; Epub Aug 28, 2009.
13. Haas, B., Kamoun, S., Zody, M.C., Jiang R.H.Y., Handsaker, R.E., Cano, L.M., Grabherr, M., Kodira, C.D., Raffaele, S., Torto-Alalibo, T., Bozkurt, T.O., O'Neill, K., Ah-Fong, A.M.V., Alvarado, L., Anderson, V.L., Armstrong, M.R., Avrova, A., Baxter, L., Beynon, J., Boevink, P.C., Bos, J.I.B., Broad Institute Genome Sequencing Platform, Bulone, V., Cai, G., Cakir. C., Carrington, J.C., Chawner, M., Costanzo, S., Fahlgren, N., Fugelstad, J., Gilroy, E.M., Gnerre. S., Green, P.J., Grenville-Briggs, L.J., Griffith, J., Gupta, S., Horn, K., Horner, N.R., Hu, C.H., Huitema, E., Jeong, D.H., Jones, A., Jones, J.D.G., Jones, R., Karlsson, E., Lamour, K., Liu, Z., Ma, L., MacLean, D., Marcus, C., McDonald, H., McWalters, J., Meijer, H.J.G., Morgan, W., Morris, P.F., Munro, C.A., Ospina-Giraldo, M., Pinzón, A., Pritchard, L., Ramsahoye, B., Ren, Q., Restrepo, S., Roy. S., Sadanandom, A., Savidor, A., Schornack, S., Schwartz, D.C., Schumann, U.D., Schwessinger, B., Seyer, L., Sharpe, T., Silvar, C., Song, J., Studholme, D.J., Sykes, S., van de Vondervoort, P.J.I., Vipaporn, P., Wawra, S., Weide, R., Win, J., Young, C., Zhou, S., Fry, W., Meyers, B.C., van West, P., Ristaino, J., Govers, F., Birch, P.R.J., Whisson, S., Judelson, H.S., and Nusbaum, C. Genome sequence and analysis of the Irish potato famine pathogen Phytophthora infestans. Nature 461 (7262):393-8, 2009. Epub Sep 9, 2009. (Featured in Nature’s Editor’s Summary and cover).
14. Interpolation in the Directed Assembly of Block Copolymers on Nanopatterned Substrates: Simulation and Experiments. Franois A. Detcheverry, Guoliang Liu, Paul F. Nealey, Juan J. de Pablo. Macromolecules 2010 43 (7), 3446-3454
15. Remediation of Line Edge Roughness in Chemical Nanopatterns by the Directed Assembly of Overlying Block Copolymer Films. Mark P. Stoykovich, Kostas Ch. Daoulas, Marcus Mller, Huiman Kang, Juan J. de Pablo, Paul F. Nealey. Macromolecules 2010 43 (5), 23342342
June 2010
Appendix 3
Page 35
16. 2D IR Line Shapes Probe Ovispirin Peptide Conformation and Depth in Lipid Bilayers. Ann Marie Woys, Yu-Shan Lin, Allam S. Reddy, Wei Xiong, Juan J. de Pablo, James L. Skinner, Martin T. Zanni. Journal of the American Chemical Society 2010 132 (8), 2832-2838
17. Characterization of the Reversible Interaction of Pairs of Nanoparticles Dispersed in Nematic Liquid Crystals. Gary M. Koenig Jr., Juan J. de Pablo, Nicholas L. Abbott, Langmuir 2009 25 (23), 13318-13321
18. Nonlinear Effects in the Nanophase Segregation of Polyelectrolyte Gels. Prateek K. Jha, Francisco J. Solis, Juan J. de Pablo, Monica Olvera de la Cruz. Macromolecules 2009 42 (16), 6284-6289
19. Characterization of Adsorbate-Induced Ordering Transitions of Liquid Crystals within Monodisperse Droplets. Jugal K. Gupta, Jacob S. Zimmerman, Juan J. de Pablo, Frank Caruso, Nicholas L. Abbott, Langmuir. Macromolecules 2009 25 (16), 9016-9024
20. Fluorinated Quaternary Ammonium Salts as Dissolution Aids for Polar Polymers in Environmentally Benign Supercritical Carbon Dioxide. Manabu Tanaka, Abhinav Rastogi, Gregory N. Toepperwein, Robert A. Riggleman, Nelson M. Felix, Juan J. de Pablo, Christopher K. Ober. Chemistry of Materials 2009 21 (14), 3125-3135
21. Single Nanoparticle Tracking Reveals Influence of Chemical Functionality of Nanoparticles on Local Ordering of Liquid Crystals and Nanoparticle Diffusion Coefficients. Gary M. Koenig Jr., Rizal Ong, Angel D. Cortes, J. Antonio Moreno-Razo, Juan J. de Pablo, Nicholas L. Abbott. Nano Letters 2009 9 (7), 2794-2801
22. "Heterogeneous dynamics during deformation of a polymer glass". Robert A. Riggleman, Hau-Nan Lee, Mark D. Ediger, and Juan J. de Pablo. Soft Matter 6(2):287-291, January 2010. [doi:10.1039/b912288e ]
June 2010
Appendix 3
Page 36
23. "Antiplastization and the elastic properties of glass-forming polymer liquids". Robert A. Riggleman, Jack F. Douglas, and Juan J. de Pablo. Soft Matter 6(2):292-304, January 2010. [doi:10.1039/b915592a ]
24. "Human embryonic stem cell-derived keratinocytes exhibit an epidermal transcription program and undergo epithelial morphogenesis in engineered tissue contacts". Christian M. Metallo, Samira M. Azarin, Laurel E. Moses, Lin Ji, Juan J. de Pablo, and Sean P. Palecek. Tissue Engineering Part A 16(1):213-223, January 2010. [doi:10.1089/ten.tea.2009.0325 ]
25. "Simulations of theoretically informed coarse grain models polymeric systems". Francois A. Detcheverry, Darin Q. Pike, Paul F. Nealey, Marcus Meuller, and Juan J. de Pablo. Faraday Discussions 144():111-125, January 2010. [doi:10.1039/b902283j ]
26. GNNQQNY - Investigation of Early Steps during Amyloid Formation. Allam S. Reddy, Manan Chopra, Juan J. de Pablo. Biophysical Journal - 17 March 2010 (Vol. 98, Issue 6, pp. 1038-1045) [doi:10.1016/j.bpj.2009.10.057]
27. Solution Structures of Rat Amylin Peptide: Simulation, Theory, and Experiment. Allam S. Reddy, Lu Wang, Yu-Shan Lin, Yun Ling, Manan Chopra, Martin T. Zanni, James L. Skinner, Juan J. De Pablo. Biophysical Journal - 3 February 2010 (Vol. 98, Issue 3, pp. 443451) [doi:10.1016/j.bpj.2009.10.029]
28. Disruption and formation of surface salt bridges are coupled to DNA binding in integration host factor (IHF): acomputational analysis, L. Ma, M. T. Record, Jr., N. Sundlass, R. T. Raines and Q. Cui, {\it J. Mol. Biol.}, Submitted
29. An implicit solvent model for SCC-DFTB with Charge-Dependent Radii, G. Hou, X. Zhu and Q. Cui, {\it J. Chem. Theo. Comp.}, Submitted
30. Sequence-dependent interaction of $\beta$-peptides with membranes, J. Mondal, X. Zhu, Q. Cui and A. Yethiraj, {\it J. Am. Chem. Soc.}, Submitted June 2010
Appendix 3
Page 37
31. A new coarse-grained model for water: The importance of electrostatic interactions, Z. Wu, Q. Cui and A. Yethiraj, {\it J. Phys. Chem. B} Submitted
32. How does bone sialoprotein promote the nucleation of hydroxyapatite? A molecular dynamics study using model peptides of different conformations, Y. Yang, Q. Cui, and N. Sahai, {\it Langmuir}, Submitted
33. Preferential interactions between small solutes and the protein backbone: A computational analysis, L. Ma, L. Pegram, M. T. Record, Jr., Q. Cui, {\it Biochem.}, 49, 1954-1962 (2010)
34. Establishing effective simulation protocols for $\beta$- and$\alpha/\beta$-peptides. III. Molecular Mechanical (MM) model for a non-cyclic $\beta$-residue, X. Zhu, P. K\"onig, M. Hoffman, A. Yethiraj and Q. Cui, {\it J. Comp. Chem.}, In press (DOI: 10.1002/jcc.21493)
35. Curvature Generation and Pressure Profile in Membrane with lysolipids: Insights from coarse-grained simulations, J. Yoo and Q. Cui, {\it Biophys. J.} 97, 2267-2276 (2009)
HCC Virtual Organization 1. "PROFESS: A PROtein Function, Evolution, Structure and Sequence database"Thomas Triplet, Matthew D. Shortridge, Mark A. Griep, Jaime L. Stark, Robert Powers and Peter Revesz. It has been accepted with revisions to the journal Database.
LIGO Collaboration 17 S3 Bursts LSC, AURIGA Class. Quantum Grav. 25 (2008) 095004 arXiv:0710.0497 200710 A Joint Search for Gravitational Wave Bursts with AURIGA and LIGO. 18
S3 Inspiral LSC Phys. Rev. D 78 (2008) 042002 200712 Search of S3 LIGO data for gravitational wave signals from spinning June 2010
Appendix 3
arXiv:0712.2050
Page 38
black hole and neutron star binary inspirals. 20 S4 Bursts LSC Phys. Rev. D 76 (2007) 062003 astro-ph/0703419 Search for gravitational wave radiation associated with the pulsating tail of the SGR 1806-20 hyperflare of December 27, 2004 using LIGO.
200703
21 S4 Bursts LSC Class. Quantum Grav. 24 (2007) 5343-5369 arXiv:0704.0943 200704 Search for gravitational-wave bursts in LIGO data from the fourth science run. 22 S4/S3/S2 Bursts LSC Phys. Rev. D 77 (2008) 062004 arXiv:0709.0766 200709 Search for Gravitational Waves Associated with 39 Gamma-Ray Bursts Using data from the Second, Third, and Fourth LIGO Runs. 23 S4 Bursts LSC Class. Quantum Grav. 25 (2008) 245008 arXiv:0807.2834 200807 First joint search for gravitational-wave bursts in LIGO and GEO600 data. 24 S4/S3 Inspiral LSC Phys. Rev. D 77 (2008) 062002 arXiv:0704.3368 200704 Search for gravitational waves from binary inspirals in S3 and S4 LIGO data. 25 S4/S3 CW LSC, Kramer, Lyne Phys. Rev. D 76 (2007) 042001 gr-qc/0702039 200702 Upper Limits on Gravitational Wave Emission from 78 Radio Pulsars. 26 S4 CW LSC Phys. Rev. D 77 (2008) 022001 All-sky search for periodic gravitational waves in LIGO S4 data.
arXiv:0708.3818
200708
S4 CW LSC Phys. Rev. D 79 (2009) 022001 arXiv:0804.1747 The Einstein@Home search for periodic gravitational waves in LIGO S4 data.
200804
27
28 S4 200608
Stochastic LSC Astrophys. J. 659 (2007) 918 astro-ph/0608606 Searching for Stochastic Background of Gravitational Waves with LIGO.
29 S4 200703
Stochastic LSC Phys. Rev. D 76 (2007) 082003 Upper limit map of a background of gravitational waves.
astro-ph/0703234
30 S4 Stochastic LSC, ALLEGRO Phys. Rev. D 76 (2007) 022001 gr-qc/0703068 200703 First Cross-Correlation Analysis of Interferometric and Resonant-Bar Gravitational-Wave Data for Stochastic Backgrounds. 31 S5 Bursts/Inspiral LSC, Hurley Astrophys. J. 681 (2008) 1419 arXiv:0711.1163 200711 Implications for the Origin of GRB 070201 from LIGO Observations. June 2010
Appendix 3
Page 39
32 S5 CW LSC Astrophys. J. Lett. 683 (2008) 45 arXiv:0805.4758 Beating the spin-down limit on gravitational wave emission from the Crab pulsar.
200805
33 S5 Bursts LSC, Barthelmy, Gehrels, Hurley, Palmer Phys. Rev. Lett. 101 (2008) 211102 arXiv:0808.2050 200808 Search for Gravitational Wave Bursts from Soft Gamma Repeaters. 35 S5 CW LSC Phys. Rev. Lett.102 (2009) 111102 arXiv:0810.0283 All-sky LIGO Search for Periodic Gravitational Waves in the Early S5 Data.
200810
36 S5 CBC LSC Phys. Rev. D 79 (2009) 122001 arXiv:0901.0302 Search for Gravitational Waves from Low Mass Binary Coalescences in the First Year of LIGO's S5 Data.
200901
37 S4 Bursts LSC Phys Rev D 80 (2009) 062002 arXiv:0904.4718 First LIGO search for gravitational wave bursts from cosmic (super)strings.
200904
38 S5 Bursts LSC Phys Rev D 80 (2009) 102002 arXiv:0904.4910 Search for High Frequency Gravitational Wave Bursts in the First Calendar Year of LIGO's Fifth Science Run.
200905
39 S5 200905 Storm.
Bursts LSC Astrophys. J. 701 (2009) L68-L74 arXiv:0905.0005 Stacked Search for Gravitational Waves from the 2006 SGR 1900+14
40 S5 Bursts LSC Phys Rev D 80 (2009) 102001 arXiv:0905.0020 Search for gravitational-wave bursts in the first year of the fifth LIGO science run.
200905
41 S4 CBC LSC Phys. Rev. D 80 (2009) 062001 arXiv:0905.1654 Search for gravitational wave ringdowns from perturbed black holes in LIGO S4 data.
200905
42
S5 CW LSC, Anderson Phys. Rev. D 80 (2009) 042003 arXiv:0905.1705 200905 Einstein@Home search for periodic gravitational waves in early S5 LIGO data. 43 S5 CBC LSC Phys. Rev. D 80 (2009) 047101 arXiv:0905.3710 Search for Gravitational Waves from Low Mass Compact Binary Coalescence in 186 Days of LIGO's fifth Science Run 45 S5/VSR1 arXiv:0910:5772 June 2010
Stochastic 200908
200905
LSC, Virgo Nature 460 (2009) 990 An upper limit on the stochastic Appendix 3
Page 40
gravitational-wave background of cosmological origin. 46 S5/VSR1 Bursts LSC, Virgo Astrophys. J. 715 (2010) 1438 arXiv:0908.3824 200908 Search for gravitational-wave bursts associated with gamma-ray bursts using data from LIGO Science Run 5 and Virgo Science Run 1. 47 S5/VSR1 CW LSC, Virgo Astrophys. J. 713 (2010) 671 arXiv:0909.3583 200909 Searches for gravitational waves from known pulsars with S5 LIGO data. 48 S5/VSR1 CBC LSC, Virgo Astrophys. J. 715 (2010) 1453 arXiv:1001.0165 201001 Search for gravitational-wave inspiral signals associated with short Gamma-Ray Bursts during LIGO's fifth and Virgo's first science run. 49 S5/VSR1 Bursts LSC, Virgo Phys. Rev. D 81 (2010) 102001 arXiv:1002.1036 201001 All-sky search for gravitational-wave bursts in the first joint LIGO-GEO-Virgo run.
MiniBooNE Virtual Organization 1.
A.A. Aguilar-Arevalo et al., First Measurement of the Muon Neutrino Charged Current Quasielastic Double Differential Cross Section, arXiv:1002:2680 [hep-ex], Phys. Rev. D81, 092005 (2010), Data release.
2.
A.A. Aguilar-Arevalo et al., "MeasurHPHQWRIȞȝ ȝ induced neutral current single 0 ʌ production cross sections on mineral oil at EȞ~O(1 GeV)", arXiv:0911.2063 [hep-ex], Phys. Rev. D81, 013005 (2010) Data release.
3.
A.A. Aguilar-Arevalo et al., "A Search for Core-Collapse Supernovae using the MiniBooNE Neutrino Detector", arXiv:0910.3182 [hep-ex], Phys. Rev. D81, 032001 (2010)
4.
A.A. Aguilar-Arevalo et al., 0HDVXUHPHQWRIWKHȞȝ CC pi+/QE Cross Section Ratio on Mineral Oil in a 0.8 GeV Neutrino Beam", arXiv:0904.3159 [hep-ex], Phys. Rev. Lett. 103, 081801 (2009)
June 2010
Appendix 3
Page 41
5.
A.A. Aguilar-Arevalo et al., "A Search for Electron Anti-Neutrino Appearance at the ǻP2 ~1 eV2 Scale", arXiv:0904.1958 [hep-ex], Phys. Rev. Lett. 103, 111801 (2009) Data release.
6.
A.A. Aguilar-Arevalo et al., "A Search for Muon Neutrino and Anti-Neutrino Disappearance in MiniBooNE", arXiv:0903.2465 [hep-ex], Phys. Rev. Lett. 103, 061802 (2009) Data release.
MINOS Virtual Organization 1. P. Adamson et al.(152 authors), "Neutrino and antineutrino inclusive charged-current cross section measurements with the MINOS near detector, "Fermilab-Pub-09-468-E, Phy.Rev.D81:072002(2010) (Vol.81, No.7, 8 April 2010); in arXiv:0910.2201 and MINOSdoc-6252. 2. E.W. Grashorn, J.K. de Jong, M.C. Goodman, A. Habig, M.L. Marshak, S. Mufson, S. Osprey, P.Schreiner, "The atmospheric charged kaon/pion ratio using seasonal variation methods," Fermilab-Pub-10-058-E, Astroparticle Physics 33,140-145 (2010) (Issue 3, 6 March 2010); in arXiv:0909.5382 and MINOS-doc-6236. 3. P. Adamson et al.(149 authors), "Search for sterile neutrino mixing in the MINOS longbaseline experiment," Fermilab-Pub-09-650-E, Phy.Rev.D81:052004 (2010)(11 March 2010); also in arXiv:1001.0336 and MINOS-doc-6028. 4. P. Adamson et al. (163 authors), "Search for muon-neutrino to electron-neutrino transitions in MINOS," Fermilab-Pub-09-443-E, Phys.Rev.Lett. 103, 261802 (2009), Issue 26, 31 December 2009; in hep-ex, arXiv:0909.4996 and MINOS-doc-5908. 5. P. Adamson et al. (154 authors), "Observation of muon intensity variations by season with the MINOS far detector," Fermilab-Pub-09-427-E, Phys.Rev.D81:012001 (2010), 6 January 2010; in arXiv:0909.4012 and MINOS-doc-3719. 6. Cabrera, P. Adamson, M. Barker, A. Belias, S. Boyd, G. Crone, G. Drake, E. Falk, P.G. Harris, J. Hartnell, L. Jenner, M. Kordosky, K. Lang, R.P. Litchfield, D. Michael, P.S. Miyagawa, R. Morse, S. Murgia, R. Nichol, T. Nicholls. G.F. Pearce, D. Petyt, D. Reyna, R. Saakyan, P. Shanahan, C. Smith, P. Symes, N. Tagg, J. Thomas, P. Vahle, A. Weber, "Comparisons of the MINOS near and far detector readout systems at a test beam," FermilabPub-09-050-E, Nucl.Instrum.Meth. A (in press, doi.10.1016 / j.nima. 2009.07.016); in arXiv:0902.1116 and MINOS-doc-5425.
June 2010
Appendix 3
Page 42
7. P.A. Schreiner, J. Reichenbacher, M.C. Goodman, "Interpretation of the underground muon charge ratio," Fermilab-Pub-09-511-PPD, Astroparticle Physics 32, pp. 61-71 (2009); in arXiv:0906.3726 and MINOS-doc-5437.
NanoHUB Virtual Organization 1. Michael McLennan, Rick Kennell, "HUBzero: A Platform for Dissemination and Collaboration in Computational Science and Engineering," Computing in Science and Engineering, vol. 12, no. 2, pp. 48-53, Mar./Apr. 2010, doi:10.1109/MCSE.2010.41 (URL: http://www.computer.org/portal/web/csdl/doi/10.1109/MCSE.2010.41) 2. Alejandro Strachan, Gerhard Klimeck, Mark Lundstrom, "Cyber-Enabled Simulations in Nanoscale Science and Engineering," Computing in Science and Engineering, vol. 12, no. 2, pp. 12-17, Mar./Apr. 2010, doi:10.1109/MCSE.2010.38 (URL: http://www.computer.org/portal/web/csdl/doi/10.1109/MCSE.2010.38) 3. Gerhard Klimeck, Mathieu Luisier, "Atomistic Modeling of Realistically Extended Semiconductor Devices with NEMO and OMEN," Computing in Science and Engineering, vol. 12, no. 2, pp. 28-35, Mar./Apr. 2010, doi:10.1109/MCSE.2010.32 (URL: http://www.computer.org/portal/web/csdl/doi/10.1109/MCSE.2010.32)
NYSgrid Virtual Organization Publications 1. A.J. Schultz and D.A. Kofke, “Virial coefficients of model alkanes”, J. Chem. Phys (submitted). 2. A.J. Schultz and D.A. Kofke, “Sixth, seventh and eighth virial coefficients of the LennardJones model”, Mol. Phys., 107(21), 2309-2318 (2009). 3. K.M. Benjamin, A.J. Schultz, and D.A. Kofke, “Fourth and fifth virial coefficients of polarizable water”, J Phys. Chem. B 113(22), 7810-7815 (2009). Dissertation 1. Ercan Dumlupinar –Title :" CFD Studies of Dynamic Stall for Rotor Applications" Dec/2009. June 2010
Appendix 3
Page 43
SBgrid Virtual Organization 1. Aoki et al. Structure of rotavirus outer-layer protein VP7 bound with a neutralizing Fab. Science (2009) vol. 324 (5933) pp. 1444-7
STAR Virtual Organization Reference: Jerome Lauret (
[email protected]), http://drupal.star.bnl.gov/STAR/publications 1. Balance Functions from Au+Au, d+Au, and p+p Collisions at $\sqrt{s_{NN}}$ = 200 GeV Submitted May. 13, 2010 e-Print Archives (1005.2307) : Abstract | PS | PDF 2. Higher Moments of Net-proton Multiplicity Distributions at RHIC Submitted Apr. 29, 2010 e-Print Archives (1004.4959) : Abstract | PS | PDF Data and figures: click here 3. Azimuthal di-hadron correlations in d+Au and Au+Au collisions at $\sqrt{s_{NN}}=200$ GeV from STAR Submitted Apr. 14, 2010 e-Print Archives (1004.2377) : Abstract | PS | PDF Data and figures: click here 4. Pion femtoscopy in p+p collisions at sqrt(s)=200 GeV Submitted Apr. 6, 2010 e-Print Archives (1004.0925) : Abstract | PS | PDF Data and figures: click here 5. Longitudinal scaling property of the charge balance function in Au + Au collisions at 200 GeV Submitted Feb. 12, 2010 e-Print Archives (arXiv:1002.1641) : Abstract | PS | PDF Data and figures: click here 6. Charged and strange hadron elliptic flow in Cu+Cu collisions at $\sqrt{s_{NN}}$ = 62.4 and 200 GeV Submitted Jan. 28, 2010 , published Apr. 9, 2010 June 2010
Appendix 3
Page 44
Phys. Rev. C 81 (2010) 44902 e-Print Archives (arXiv:1001.5052) : Abstract | PS | PDF SLAC-Spires HEP: Entry|Cited by|Citebase Journal article: Phys. Rev. C server Data and figures: click here 7. Upsilon cross section in p+p collisions at sqrt(s) = 200 GeV Submitted Jan. 15, 2010 e-Print Archives (1001.2745) : Abstract | PS | PDF Data and figures: click here 8. Three-particle coincidence of the long range pseudorapidity correlation in high energy nucleus-nucleus collisions Submitted Jan. 8, 2010 e-Print Archives (0912.3977) : Abstract | PS | PDF Data and figures: click here 9. Inclusive pi^0, eta, and direct photon production in p+p and d+Au collisions at sqrt(s_NN) = 200 GeV Submitted Dec. 18, 2009 e-Print Archives (0912.3838) : Abstract | PS | PDF 10. Studying Parton Energy Loss in Heavy-Ion Collisions via Direct-Photon and ChargedParticle Azimuthal Correlations Submitted Dec. 9, 2009 e-Print Archives (0912.1871) : Abstract | PS | PDF Data and figures: click here 11. Observation of pi^+pi^-pi^+pi^- photoproduction in ultraperipheral heavy-ion collisions at sqrt(s_NN) = 200 GeV at the STAR detector Submitted Dec. 4, 2009 , published Apr. 2, 2010 Phys. Rev. C 81 (2010) 44901 e-Print Archives (0912.0604) : Abstract | PS | PDF SLAC-Spires HEP: Entry|Cited by|Citebase Journal article: Phys. Rev. C server Data and figures: click here 12. Identified high-$p_T$ spectra in Cu+Cu collisions at $\sqrt{s_{NN}}$=200 GeV Submitted Nov. 16, 2009 e-Print Archives (0911.3130) : Abstract | PS | PDF June 2010
Appendix 3
Page 45
Data and figures: click here 13. Longitudinal double-spin asymmetry and cross section for inclusive neutral pion production at midrapidity in polarized proton collisions at sqrt(s) = 200 GeV Submitted Nov. 14, 2009 , published Dec. 30, 2009 Phys. Rev. D 80 (2009) 111108 e-Print Archives (0911.2773) : Abstract | PS | PDF SLAC-Spires HEP: Entry|Cited by|Citebase Journal article: Phys. Rev. D server Data and figures: click here 14. Observation of an Antimatter Hypernucleus Submitted Oct. 29, 2009 , published Mar. 4, 2010 Science 328 (2010) 58 e-Print Archives (1003.2030) : Abstract | PS | PDF SLAC-Spires HEP: Entry|Cited by|Citebase Journal article: Science server Data and figures: click here 15. Longitudinal Spin Transfer to $\Lambda$ and $\bar{\Lambda}$ Hyperons in Polarized Proton-Proton Collisions at $\sqrt{s} = 200 GeV$ Submitted Oct. 8, 2009 , published Dec. 8, 2009 Phys. Rev. D 80 (2009) 111102 e-Print Archives (0910.1428) : Abstract | PS | PDF SLAC-Spires HEP: Entry|Cited by|Citebase Journal article: Phys. Rev. D server Data and figures: click here 16. Identified particle production, azimuthal anisotropy, and interferometry measurements in Au+Au collisions at $\sqrt{\bm {s_{NN}}} =$ 9.2 GeV Submitted Sep. 23, 2009 , published Feb. 26, 2010 Phys. Rev. C 81 (2010) 24911 e-Print Archives (0909.4131) : Abstract | PS | PDF SLAC-Spires HEP: Entry|Cited by|Citebase Journal article: Phys. Rev. C server Data and figures: click here 17. Observation of charge-dependent azimuthal correlations and possible local strong parity violation in heavy ion collisions Submitted Sep. 9, 2009 June 2010
Appendix 3
Page 46
e-Print Archives (0909.1717) : Abstract | PS | PDF Data and figures: click here 18. Azimuthal Charged-Particle Correlations and Possible Local Strong Parity Violation Submitted Sep. 9, 2009 , published Dec. 14, 2009 Phys. Rev. Lett. 103 (2009) 251601 e-Print Archives (0909.1739) : Abstract | PS | PDF SLAC-Spires HEP: Entry|Cited by|Citebase Journal article: Phys. Rev. Lett. server Data and figures: click here 19. Yields and elliptic flow of $d(\overline{d})$ and $^{3}He(\overline{^{3}He})$ in Au+Au collisions at $\sqrt{s_{_{NN}}} =$ 200 GeV Submitted Sep. 3, 2009 e-Print Archives (0909.0566) : Abstract | PS | PDF Data and figures: click here 20. Long range rapidity correlations and jet production in high energy nuclear collisions Submitted Sep. 1, 2009 , published Dec. 29, 2009 Phys. Rev. C 80 (2009) 64912 e-Print Archives (arXiv:0909.0191) : Abstract | PS | PDF SLAC-Spires HEP: Entry|Cited by|Citebase Journal article: Phys. Rev. C server Data and figures: click here 21. Neutral Pion Production in Au+Au Collisions at $\sqrt{s_{NN}}$ = 200 GeV Submitted Jul. 16, 2009 , published Oct. 23, 2009 Phys. Rev. C 80 (2009) 44905 e-Print Archives (0907.2721) : Abstract | PS | PDF SLAC-Spires HEP: Entry|Cited by|Citebase Journal article: Phys. Rev. C server Data and figures: click here 22. Center of mass energy and system-size dependence of photon production at forward rapidity at RHIC Submitted Jun. 12, 2009 , published Nov. 26, 2009 Nucl. Phys. A 832 (2009) 134 e-Print Archives (arXiv:0906.2260v1) : Abstract | PS | PDF SLAC-Spires HEP: Entry|Cited by|Citebase Journal article: Nucl. Phys. A server June 2010
Appendix 3
Page 47
Data and figures: click here 23. Growth of Long Range Forward-Backward Multiplicity Correlations with Centrality in Au+Au Collisions at $\sqrt{s_{NN}}$ = 200 GeV Submitted May. 3, 2009 , published Oct. 21, 2009 Phys. Rev. Lett. 103 (2009) 172301 e-Print Archives (0905.0237) : Abstract | PS | PDF SLAC-Spires HEP: Entry|Cited by|Citebase Journal article: Phys. Rev. Lett. server Data and figures: click here 24. Perspectives of a Midrapidity Dimuon Program at RHIC: A Novel and Compact Muon Telescope Detector Submitted Apr. 30, 2009 , published Jul. 17, 2009 J. Phys. G 36 (2009) 95001 e-Print Archives (0904.3774) : Abstract | PS | PDF SLAC-Spires HEP: Entry|Cited by|Citebase Journal article: J. Phys. G server 25. System size dependence of associated yields in hadron-triggered jets Submitted Apr. 13, 2009 , published Dec. 16, 2009 Phys. Lett. B 683 (2010) 123 e-Print Archives (0904.1722) : Abstract | PS | PDF SLAC-Spires HEP: Entry|Cited by|Citebase Journal article: Phys. Lett. B server Data and figures: click here 26. J/psi production at high transverse momentum in p+p and Cu+Cu collisions at $\sqrt{s_{NN}}$ = 200 GeV Submitted Apr. 2, 2009 , published Oct. 27, 2009 Phys. Rev. C 80 (2009) 41902 e-Print Archives (arXiv:0904.0439) : Abstract | PS | PDF SLAC-Spires HEP: Entry|Cited by|Citebase Journal article: Phys. Rev. C server Data and figures: click here 27. Pion Interferometry in Au+Au and Cu+Cu Collisions at RHIC Submitted Mar. 6, 2009 , published Aug. 24, 2009 Phys. Rev. C 80 (2009) 24905 e-Print Archives (0903.1296) : Abstract | PS | PDF June 2010
Appendix 3
Page 48
SLAC-Spires HEP: Entry|Cited by|Citebase Journal article: Phys. Rev. C server Data and figures: click here 28. K/pi Fluctuations at Relativistic Energies Submitted Jan. 12, 2009 , published Aug. 24, 2009 Phys. Rev. Lett. 103 (2009) 92301 e-Print Archives (arXiv:0901.1795v1) : Abstract | PS | PDF SLAC-Spires HEP: Entry|Cited by|Citebase Journal article: Phys. Rev. Lett. server Data and figures: click here 29. Measurement of D* Mesons in Jets from p+p Collisions at sqrt(s) = 200 GeV Submitted Jan. 6, 2009 , published Jul. 1, 2009 Phys. Rev. D 79 (2009) 112006 e-Print Archives (0901.0740) : Abstract | PS | PDF SLAC-Spires HEP: Entry|Cited by|Citebase Journal article: Phys. Rev. D server Data and figures: click here
June 2010
Appendix 3
Page 49
