Challenges in Data Acquisition at the Beginning of the New Millennium Johannes Gutleber e-mail:
[email protected] CERN (European Organization for Nuclear Research), Division EP, 1211 Geneva 23, Switzerland
Abstract Data acquisition systems (DAQ) are a well-explored research domain. With increasing measurement resolution in time and space they grow and traditional approaches do not satisfy the needs anymore. The amounts of data become too big to be processed by adhoc systems. Eventually, a real-time approach is necessary when it comes to guarantee 24/7 availability service. Distribution is a common means to exploit local processing and data buffering in order to let the system grow gracefully. Such approach however introduces all the problems that go along with distributed architectures into the domain of high efficiency real-time systems. This paper presents the challenges of DAQ systems as we find them in high-energy physics experiments at CERN. We discuss potential technology attacks to address the problems of such systems. The applicability of the proposed solutions is not limited to particle physics. It fits also well industry applications like medical imaging, video-streaming or scalable WEB search engines.
1. Introduction In a high-energy physics experiment, subatomic particles are accelerated and brought to collision. The results of these collisions are recorded in order to find an answer to the fundamental question in nature: what is matter made of? A new era of particle physics is about to begin with the construction of the Large Hadron Collider at CERN. Hadrons, such as protons are not elementary particles and thus their collisions are not “clean”. The advantage of having higher energetic collisions and the increased probability of producing new particles is paid with increased background noise that has to be subtracted before the interesting event can be studied. The collision products are recorded by using a collection of detectors that are placed around the interaction point. Layers of detectors surround the collision center like an onion-skin (see also Figure 1). A strong magnetic field bends the tracks of emerging charged particles. From the reconstructed tracks, we can derive their properties. Calorimeters record particle energies. Particles to which the detectors are blind have to be computed indirectly.
Figure 1: The Compact Muon Solenoid experiment As such experiments comprise billions of channels, they produce large amounts of data that have to be read, processed and stored persistently for later analysis [1]. These are the tasks of a data acquisition system. As an example we mention the current general-purpose experiments under planning for the Large Hadron Collider. For each collision of two proton bunches will produce data in the order of 1 MByte. The LHC collider has an inter-particle bunch gap of 25ns, thus resulting in a collision rate of 40 MHz. A potential data acquisition system for the experiment must be capable of digesting this enormous data stream of roughly 40 TBytes/sec. Not enough, the installation that goes into operation in 2005 has a projected lifetime of 10 years. Hence a high abstraction solution for a hybrid hardware/software system has to be found that is capable of bridging several technology generations. This paper describes some of the challenges that we find in the construction of such systems and discusses promising approaches.
2. Related Applications Distributed real-time data acquisition can be found in various application domains other than high-energy physics. Many systems encounter only strong requirements on a single efficiency dimension; either the amounts of data are big, or the data rate is high, or the size of the system is large. Space-flight launch control systems [2] for example exhibit a high degree of distribution and have to handle large amounts of data. Environmental monitoring systems [3] on the other hand comprise a large number of measurement units. Only few applications need to address all of these properties at once. Control systems for nuclear fusion experiments [4] for example come close to the requirements of our domain. Another field that exhibits high requirements on data amounts, that has to have short and predictable response times and tends to have large system sizes is medical imaging [8]. Even current high-energy physics experiments [5,6,7] tend to have efficiency requirements that are an order of magnitude lower than the ones of the next decade. The natural and uncontrolled growth of the network infrastructure and the necessity to include legacy software puts even more burden on the successful installation of this kind of system.
3. Challenges 3.1. Data Dissemination The raw data avalanche in LHC experiments of 40 TBytes/sec originating from 100.000.000 channels cannot be handled without prior data reduction. Therefore the data acquisition system is split into several levels, each of which aims at a certain degree of data reduction. A first level entirely implemented in hardware and pipelined to apply threshold cut-off and pattern recognition algorithms. It achieves a data rate reduction to 100 kHz. This layer is comprised of proprietary processing devices placed closely to the detectors. The output is connected to a global trigger processing system that provides a summary description of the detected event to the subsequent levels. This information can later be used to decide if the data from this collision should be transferred to general-purpose computers for further processing. Analysis of the data is computational too extensive to be performed on a singled processor. However, particle collisions are independent of each other and hence they represent a source of natural parallelism. Each event can be studied concurrently with other events in a large processing cluster called the filter farm. Its task is the analysis of the physics process for further data reduction. The processing has to take place in real-time; 100 000 particle collisions must be checked per second. Only
those events that might contain an interesting physics process to a sufficient high probability will be made persistent. So called filter units act as clients to real-time data servers known as readout units that buffer the data from the detector elements. The amount of captured data requires multiple readout units. Transferring the whole event data set at the first level reduction rate of 100 kHz to the filter farm would require a network that is capable of operating at a sustained level of 100 GBytes/sec. As the full event data set consists of the combined data fragments from all the readout units, point-to-point communication infrastructure has to be provided for each combination of readout unit and filter unit. This does not only impose a high data transfer capacity requirement on the interconnection technology, but also on the systems themselves. The processors in both the readout-unit and the filter-unit must be powerful enough to keep up with the data stream. Memory bandwidth must be sufficient high to cope with the transport of the data from the network interface to the applications. We do not expect an off-the-shelf network with the required throughput and number of ports to be available in time, so further reduction of the data is done. The event is not processed as a whole, but in pieces. For this purpose an intelligent caching device is introduced between the readout- and the filter units. It is called the builder unit. Its task is to cache event fragment from the readout part, put them into a contiguous data structures and provide them to the filter units on demand. An algorithm starts requesting part of an event from the builder unit. If it satisfies certain criteria, more event fragments are requested. If the event seems not interesting, it can be overwritten in the readout units and the data do not have to be transferred. If the event is sufficiently important, all remaining data are requested and made persistent such that a reduction to an output data stream of 100 MBytes/sec can be achieved. What concerns transfer to persistent data store, it has been shown that the output data rate of 100 Hz at a data item size of 1 MByte can be handled by an object-oriented database management system on a HP Exemplar supercomputer with a custom interconnection network[9]. For performing the computational intensive eventfiltering task, the cluster must achieve a total performance of 5 TeraOps, 500 to 1000 commercial-off-the-shelf computers should be capable of reaching this goal in 5 years. Achieving the processing power with a network of workstations is not a challenging task anymore. ASCI (Advanced Strategic Computing Initiative by Department of Energy) installations reach already in the order of 2 TeraOPS (e.g. Option Red) and are projected to achieve a 100 TeraOps by 2004. Massive parallel systems however have different requirements on communication. Due to the nature of the fine-grained parallel algorithms, it is important to have high bandwidth links to neighbouring processing nodes. Thus, the aggregate bandwidth that is
given for these machines might be misleading. For highenergy physics experiments, direct n-to-n communication is needed between the data servers and the processing nodes. The bandwidth that is available in today’s supercomputers would be sufficient for our tasks if it Level 1 Trigger
Detector Frontend
Manager
Readout Units Readout Builder
Builder Networks Builder Units
Filter Units
were truly available between all participating processors. Unfortunately this is not the case.
Figure 2: outline of the data acquisition architecture. The capacity that is required for each network port of a non-blocking, full interconnection network will be in the order of 2 Gb/sec when applying multiple stage event building. In practice, full bandwidth is not seen when performing n-to-m data communication. So further realtime traffic shaping has to take place [10,11] in order to exploit the switches and meet the timeliness requirements. Furthermore, large monolithic switches are not available, and therefore a composite switch has to be built from smaller blocks. Still, the per-switch port cost will be high in a large composite switch. Attaching several processing nodes to one port is not only advantageous from the economical point of view, but also for achieving good throughput performance. This goal can also be reached through the caching of the builder unit. The ultimate goal and thus biggest challenge in data dissemination is to sustain the receiver rate that is the input parts in readout-, builder- and filter units. The builder unit has to assemble the data fragments that it requests from the readout units to contiguous data structures that the filter units can process. This task has to be performed at a rate of 100 kHz. So, even if the data network offers us the required throughput and latency, the transfer to memory and the communication software
becomes a major bottleneck. This will be the topic of the next section. 3.2. The Hardware-Software Gap The growing gap between hardware and software becomes most visible in the efficiency of communication middleware. Middleware resides between applications and the underlying operating systems, protocol stacks, and hardware. It aims at reducing the effort required to develop high quality systems by composing applications out of reusable software component services [33]. Building long living systems without the support of middleware is not imaginable anymore. They decouple the application from the diverse low-level services. Although link speeds are increasing fast and processor cycle times dwindle, the time to actually handle an incoming message in software seems to decrease only slowly. The reason is that middleware does neither belong to the low level services, nor to the application, but does not have its own processing resources. It has to borrow them from the application (main processor, slow application buffers) and thus gives rise to additional congestions in the communication path. Since data ingestion within specified amounts of time without loosing samples is essential for data acquisition systems, communication software becomes a mission-critical issue. In the previous section we outlined the physical requirements of DAQ systems for the beginning of the new millennium. We can look with optimism into the future, that there will be technologies to achieve the goals. For efficient middleware approaches however, we may not rely on the industry. The need to follow often complex standards, like OMG’s CORBA [34], Java RMI [35] prohibit to break the current speed barriers. On the other side however, heterogeneous platforms, long development and upgrade cycles as well as the need to remain open for future technologies do not allow us to apply lowest level, custom solutions. There is a clear demand for lightweight and easy to use communication middleware. It must be easy to adapt to different communication means and shall be customizable to the needs of the users. If a service is not needed, it shall be possible to exclude it from the middleware layer so that it does not present a potential performance limitation. Currently, we do not see such tools. Message passing libraries are often the only means to achieve a certain level of abstraction. However, they have way too many limitations, for what concerns extensibility and adaptability. 3.3. System Engineering A distributed data acquisition system as we find it in high-energy physics represents a challenge to system engineering. In order to achieve high efficiency, we must
reflect the capabilities of the communication and processing hardware at the lowest level. This tight and puzzling interaction with the environment makes it difficult to provide a high abstraction software framework that is capable of surviving several technology generations and be still efficient. Such abstraction is vital, as ever evolving needs and discontinued products call for upgrades of the data acquisition system several times. Development of a data acquisition system in high-energy physics is a distributed collaborative process. Institutions from three different continents participate in this project. Although controlled documentation helps to synchronise the efforts, the difficulty of integrating the subsystems after each development cycle remains. We adopted therefore the component based software engineering (CBSE) paradigm [13,14,15,16]. It allows by means of clean and narrow interfaces self-contained building blocks to ease of integration and testing. Traditional research on CBSE has however up to date mainly focused on functional requirements. Hence to apply this principle to highly efficient systems, we have to put emphasis on capturing components performance properties, compare them and establish a catalog of vital components for realtime applications. 3.4. Simulation It would be hard to build a scalable distributed realtime data acquisition system that is capable of meeting the requirements imposed by high-energy physics experiments without thorough investigation of feasibility. For this purpose the system has to be simulated at architectural level. The results of the simulation make it possible to set configuration parameters such as maximum service intervals and minimum buffer sizes such, that the mission goal can be accomplished. Keeping the dead-time small is a mission critical aspect since beam time, electrical energy for the magnet or cryogenics supplies are too costly to be wasted. Performance models enable us to represent the probabilistic nature of the system and help to predict its availability [37]. Analysis from a pure performance viewpoint tends to be optimistic since it ignores the failure-recovery behaviour of the system. If system performance is modeled separately from dependability, the assumption is being made that the system has only two performance levels: fully functional or failed. A large part of our system is however able to survive the failure of one or more components and can still provide the service at a reduced service level. The degradation can have several causes. Fault-caused errors, the need for additional processing resources for error handling, reconfiguration and repair due to fault-related actions [38] are just some examples. Therefore, a system like ours cannot be adequately modeled through separate
dependability and performance models. There is the need for a combined methodology.
4. Technology Attacks 4.1. Hardware The readout units have been prototyped for the software test-bed using PCI based Motorola PPC MV2034 boards [17]. They are interconnected using the VME backplane for system management purposes only. Data is served to clients over Myrinet [18,19], a 1 Gb/sec wormhole routing based switching network. One feature of Myrinet is that intelligence is moved from the switches to the communicating hosts as opposed to common store-and-forward switch architectures. Each Network Interface Card (NIC) hosts a processor, memory and DMA engines. The so called Master Control Program (MCP) running on the NIC processor can take over tasks, that normally have to be performed by device drivers, protocol stack and middleware on the host. This gives the host processor more time to perform its main task: processing the application. This feature is becoming more popular. IO processors (IOP) that have been used in supercomputers in the past, find their way into commodity network computing. The intelligent IO standard [30] defines a full software architecture for operating such hardware in a standardized way. Eventually this could be the long awaited space where critical paths of middleware could be placed. Then there is a chance to achieve both: high-speed data transfer and high abstraction communication services. The current prototype of the processing farm consists of Sun UltraSparc workstations. We chose VxWorks [20] as operating system for the readout units and builder units. The Sun workstations run the Solaris operating system with real-time extensions enabled. For configuration and slow control an additional switched 100 Mbps Ethernet is available. With this configuration an 8 by 8 mock-up was demonstrated. We showed the proper scaling of the system and witch acceptable results [12]. The configuration described is only used as a learning tool and to calibrate the simulation of the system correctly. Much more important is the general architecture. The subsystem building blocks shall eventually be implemented with whatever technology will deliver an appropriate cost/performance ratio by the time the experiment has to be operational. The only way to achieve this technology independence is a software layer that smoothes out the differences in the lower levels. This leads us to the next section that describes the software that operates the hardware. 4.2. Software
Software in data acquisition systems does not only comprise the application, but also the means to glue together the hardware devices. With traditional tools this is not an easy task to accomplish. The device driver approach affects program configurability. Each device needs different configuration settings to achieve good performance. Portability of data transmission can at least be guaranteed with this approach. Message passing libraries, such as MPI [21] provide communication abstraction and are even available in optimised versions for specific communication hardware [22]. They lack, however, two properties, which are important for data acquisition: first it is not possible to mix different communication devices, and second a presentation layer is missing. With object-oriented programming languages becoming ubiquitous, we are able to address the abovementioned deficiencies. Software components are a further abstraction that comprises closely collaborating objects and present functionality through a narrow interface to the outside world [23,24,25]. By encapsulating differences in the access of hardware into specialised software components, we are able to satisfy two demands at the same time: (i) high performance by optimised operation of particular devices and (ii) communication abstraction by having a well defined interface for different kinds of data transport architectures. Using this approach we can connect components through which data streams flow [26]. On top, richer paradigms can be placed. Figure 3 shows a view of the prototyped readout unit as an example for a component based hardware/software hybrid system. The top layer consists of the application components in which the systems functionality is implemented. The I/O streams package [27] on which it is based decouples the application from the system specific functions and provides a single high-level interface to data communication devices. The readout unit is a reactive soft-real time system and has to perform a number of tasks within assigned time slots. The readout unit input (RUI), for example continuously reads data from the detector front-end. The readout unit output (RUO) serves data fragment requests from the filter units. The readout unit supervisor (RUS) listens to system control commands and monitors the operation of the Readout Unit for performance calculations fault prediction. A portable operating system library comprises all foundation classes needed to operate the hardware. New components can be added to support new hardware. Performance loss due to the application of object oriented software stays in the range of 1s per device access [27]. Compared to the degrees of portability, extensibility and thus transparency that are gained with this approach, this overhead can just be tolerated. Application designers prefer to be presented an easy to use software development kit to produce reasonable fast software
rather than bare iron programming with which one achieves only slightly better performance.
RUI
Application components
RUS RUO POS components Input/Output Streams
PCI
Ethernet
SCI
POS foundation classes VME
Myrinet
OS and Device Drivers
Figure 3: Software component layering of a prototype readout unit.
4.3. Performability
The prediction of performance degradations in our system is addressed with a performability study. Performability is a composite measure for a system’s performance and its dependability [39]. It considers the effect of structural changes and their impact on the overall performance of the system. Such measure is a vital evaluation method for degradable systems that must be able to offer continued operation at lower performance level in the presence of faults. A degradable system can have several reduced operational states that range between fully operational and failed [37]. The performability measurement aims to answer the following questions: What is the expected performance of the system a time t, including the effects of failure, recovery, repair and contention for resources? What is the time-averaged performance of the system over the interval (0,t)? What is the probability that n units of work are completed by the time t?
With the complexity of industry delivered networking and computing devices it is a challenging task to simulate the behaviour of a large distributed real-time system to the granularity of single events in order to answer the above questions. It is, however, not the precision of the simulation compared to the isolated real device that counts, but the match to the overall behaviour of the system. Describing the whole system correctly and limiting the number of free parameters are challenges that must not be underestimated. Unfortunately there exist no general tools to carry out such a performability study. We are bound to simulation toolkits that are fed with paper and pencil calculations. So although a vital task for a mission-critical system, it is the area that is least covered by established methodologies and supporting tools.
5. Work in Progress Currently the streams mechanism that allows connecting different devices for building data acquisition subsystems is completed with a lean remote method invocation toolkit. With such a toolkit, the component location becomes completely transparent to the distributed application. We investigated industry standard solutions for this task. Although considerable amounts of work have been invested in making ORBs real-time aware [28], we found this kind of architecture for component middleware not suitable for meeting the high efficiency requirements of distributed real-time data acquisition [29]. We have however shown that the principle of remote method invocation can be realised with high efficiency [36]. With the recent advent of intelligent I/O (I2O) [30] we are for the first time faced with a technology paradigm shift. It is a promising candidate to replace traditional means of networking and device interconnection in our application domain. This concept allows offloading the burden of data transfer from the main processor, thus making it free for its most important task: the application. Currently, we study the possibility of including this principle into the architecture under investigation.
6. Conclusions Distributed real-time data acquisition is a grand challenge at the dawn of the upcoming millennium. We have seen that the requirements on such systems do not only call for a push in data communication technology, but also for novel engineering approaches. Component based software engineering with the support of objectoriented programming is a well-suited means for addressing the problems encountered at high abstraction level. It also enables work in a distributed collaborative environment. There remains however a strong claim for lean value-adding middleware that is configurable to the
needs of the different environments. The presented requirements and challenges are not only found in pure scientific environments. Indeed, we already found first evidence when looking for systems that have architectures similar to DAQ systems outlined in this paper. A widely used WEB search engine exhibits the same splitting in front-end and back-end clusters [31] to achieve scalability. Very likely, challenges similar to the ones in DAQ will soon be encountered in other areas, such as real-time mobile phone call centers or video-on-demand centers. The approach taken for high-energy physics DAQ design might provide some insight and potential solutions to the other problem domains.
7. Acknowledgements I would like to thank S. Cittolin, S. Erhan, E. Meschi, L. Orsini, D. Samyn, P Sphicas and W. Schleifer for their valuable comments and contributions.
8. References [1] A.Kruse, CMS Online Event Filter Software, Computer Physics Communications 110, Elsevier Science Publishing, 1998, pp. 102-106 [2] R.Hart, Blue Book, Checkout and Launch Control System Project, NASA, Kennedy Space Center, FL, USA, September 16,1996. http://clcs.ksc.nasa.gov/bluebook.html [3] J.Gutleber, The ARCS Data Point Processor, Proceedings th of the 7 International Workshop on Database and Expert Systems Applications, Sept. 9-10, 1996, Zurich, CH, pp. 40-47. IEEE Computer Society Press [4] A.Luchetta and G.Manduchi, General Purpose Architecture for Real-Time Feedback Control in Nuclear Fusion th Experiments, Proceedings of the 5 IEEE Real-Time Technology and Applications Symposium (RTAS99), Vancouver, British Columbia, Canada, June 2-4, 1999. [5] M.Dam et al., Higher level trigger systems for the HERA-B experiment, IEEE Transactions on Nuclear Science 45(4), pp.1787-1792, August 1997. [6] R.T.Hamilton, I.Scott, R.Claus, P.Grosso, M.E.Huffer, C.O’Grady and J.J.Russel, The BaBar Data Acquisition th System. In Proceedings of the 14 IEEE NPSS Real Time Conference, Santa Fe, New Mexico, USA, June 1999. [7] C.M.E.Paus, Event Builder and Level-3 Trigger at the CDF th Experiment. In Proceedings of the 14 IEEE NPSS Real Time Conference, Santa Fe, New Mexico, USA, June 1999. http://strider.lansce.lanl.gov/rt99 [8] S.Banerjee, Multimedia traffic analysis and control in a high-speed medical communications environment. In Proceedings of the Southcon 94 conference, March 29-31, 1994, pp. 416-420. IEEE Press. [9] K.Holtman and J.Bunn. Scalability to hundreds of clients in HEP object databases, In Proceedings of the International Conference on Computing in High Energy and Nuclear Physics, Hotel Inter-Continental, Chicago, Illinois, USA, Aug. 31 – Sept. 4, 1998. http://www.hep.net/chep98.
[10] T.M.Galla, R.Pallierer, Cluster simulation-support for distributed development of hard real-time systems using th TDMA-based communication, In Proceedings of the 11 Euromicro Conference on Real-Time Systems, pp. 150 – 157, IEEE Press. [11] H.Kopetz, The time-triggered architecture, In Proceedings st of the 1 international symposium on object-oriented realtime distributed computing, April 1988, pp. 22-29, IEEE Press.
Proceedings of the Conference on Object-Oriented Technologies, USENIX, June 1995, Monterey, pp. 21-38. [26] D.Ritchie, A Stream Input-Output System. AT&T Bell Labs Technical Journal 63(8):311-324, October 1984. [27] L.Orsini et al.,A Software Approach for Readout and Data th Acquisition in CMS. In Proceedings of the 14 IEEE NPSS Real Time Conference, Santa Fe, New Mexico, USA, June 1999. http://strider.lansce.lanl.gov/rt99
[12] F.Meijers et al. The CMS Event Builder Demonstrator th based on Myrinet, In Proceedings of the 14 IEEE NPSS Real Time Conference, Santa Fe, New Mexico, USA, June 1999.
[28] F.Kuhns, D.C.Schmidt, D.Levine, R.Bector, The Design and Performance of RIO – a Real-Time I/O Subsystem for th ORB Endsystems, Proceedings of the 5 IEEE Real-Time Technology and Applications Symposium (RTAS99), Vancouver, British Columbia, Canada, June 2-4, 1999.
[13] D.Batory and S.O’Malley, The Design and Implementation of Hierarchical Software Systems with Reusable Components, ACM Transactions on Software Engineering and Methodology, 1(4), October 1992, pp. 355-398.
[29] J.Gutleber, High Performance Distributed Objects in Large Hadron Collider Experiments, Ph.D. Dissertation. Technical University Vienna, Austria. November 1999.
[14] D.McIlroy, Mass-produced software components, in: P. Naur et. al. (eds.), Software Engineering Concepts and Techniques (in Petrocelli/Charter, 1976, pp. 88-98) [15] D.L.Parnas, Designing Software for Ease of Extension and Contraction, in IEEE Transactions on Software Engineering, SE-5(2), March 1979, pp. 128-137. [16] O.Nierstrasz, S.Gibbs and D.Tsichritzis, ComponentOriented Software Development Communications of the ACM 35(9): 160-164, September 1992 [17] Motorola Computer Group, Corporate Head Quarters, 2900 S. Diable Wy, Tempe, AZ 85281. [18] N.J.Boden, D.Cohen, R.E.Feldmann, A.E.Kulawik, C.L.Seitz, J.N.Seizovic and W-K.Su: MYRINET: A Gigabit per Second Local Area Network, IEEE Micro 15(1):19-35, February 1995 [19] Myricom, Inc. High-Speed Computers and Comm., 325 N. Santa Anita Ave. Arcadia, CA 91006. [20] Wind River Systems, Corporate Head Quarters, 500 Wind River May, Alameda, CA 94501. [21] M.Lauria and A.Chien, MPI-FM: High Performance MPI on Workstation Clusters, J Par Dist Comp 40(1): 4-18, Jan 1997. [22] F.O’Carroll, H.Tezuka, A.Hori and Y.Ishikawa, The Design and Implementation of Zero Copy MPI Using Commodity Hardware with a High Performance Network, in Proceedings of the 1998 International Conference on Supercomputing (ICS 98), pages 243-250, ACM Press, Melbourne, Australia, 1998. [23] C.Pfister and C.Szyperski, Why Objects Are Not Enough, st in Proceedings of the 1 International Component Users Conference (SIGS Publishers), Munich, Ger., July 1996. [24] I.Foster and C.Kesselman, The Grid: Blueprint for a New Computing Infrastructure (Morgan Kaufmann Publishers, Inc. San Francisco,CA,USA) 1999, pp 259-278. [25] D.C.Schmidt, T.Harrison and E.Al-Shaer, Object-Oriented Components for High-speed Network Programming,
[30] I2O SIG, Intelligent I/O (I2O) Architecture Specification, The I2O Interest Group, Specification V1.5, 404 Balboa Street, San Francisco, CA, 9418, USA, March 1997, http://www.i2osig.org [31] A.Fox, S.D.Gribble, Y.Chawathe, E.A.Brewer and P. Gauthier, Cluster-Based Scalable Network Services, in Proceedings of the 16th ACM Symposium on Operating System Principles, October 1997, Saint-Malo, France, M. Banatre and H. Levy (eds.) pp. 78-91, ACM Press. [32] www.top500.org [33] C.D.Gill, F.Kuhns, D.L.Levine, D.C.Schmidt, B.S.Doerr, R.E.Schantz, Applying Adaptive Real-Time Middleware to Address Grand Challenges of COTS-based MissionCritical Real-Time Systems. This conference. [34] Object Management Group, The Common Object Request Broker: Architecture and Specification, 2.3 ed., June 1999 [35] A.Wollrath, R.Riggs and J.Waldo, A Distributed Object Model for the Java System, USENIX Computing Systems, vol. 9, November/December 1996. [36] J.Gutleber, High Performance Distributed Object Communication for the CMS Experiment, International Conference on Computing in High Energy and Nuclear Physics, August 31 – September 4, 1998, Hotel InterContinental, Chicago, Illinois, USA, Argonne National Laboratory, http://www.hep.net/chep98/ [37] J.F.Meyer, Performability Evaluation: Where it is and what lies ahead, in Proceedings of the IEEE International Computer Performance and Dependability Symposium, Erlangen, Germany, April 1995, IEEE Press. [38] K.J.Muppala, S.Woolet, K.S.Trivedi, On Modeling Performance of Real-Time Systems in the Presence of Failures, in Readings in Real-Time Systems, Y.-H. Lee and C. M. Krishna (eds.), IEEE Press, pp. 219-239, 1993. [39] J.F.Meyer, Performability: A restrospective and some pointers to the future, Performance Evaluation, vol. 14, pp. 139-156, February 1992
