Using Condor glideins for distributed testing of networkfacing services Igor Sfiligoi, Frank Wuerthwein, Christopher Theissen Physics department University of California San Diego La Jolla, CA, USA email:
[email protected],
[email protected],
[email protected] Abstract—Many compute environments rely on centralized, networkfacing services in order to work properly; examples are Web servers, Grid Compute Elements and Grid Storage Elements. Testing such services requires a large amount of geographically dispersed compute resources all interacting concurrently with the service to test. Grid resources provide an obvious candidate for such testing, but to the best of our knowledge there is no readily available tool for such testing. In this paper we present one such tool, based on Condor glideins. KeywordsTesting, Distributed, Grid, Glidein, glideTester
I.
INTRODUCTION
Many compute environments rely on centralized, networkfacing services in order to work properly. In the commercial world an obvious example are Web servers. In the Grid world, there are Compute Elements (CE), Storage Elements (SE) and information systems. All of them receive requests from a high number of geographically distributed clients. In order to test the scalability and reliability of such services, the test environment must mimic the expected access pattern; this means we need test clients distributed over a large geographical area all concurrently talking to the tested service. For the services the authors are interested in, the required parallelism is in the O(1k) to O(10k) range. As such, the tests would need to be spread across 100s of compute nodes. Many services also behave differently when accessed from a local, reliable and low latency network, and when accessed from clients connected over a high latency, possibly unreliable wide area network. Having the test clients running over a widely geographically distributed area is thus a must. The reason for scalability and reliability testing is to assess how these services scale as we add more clients. Does the total transaction rate scale linearly with the number of clients? Is there a limit where the service gets saturated? How is this limit approached? Is it linear, exponential, or something else? What happens when we reach it? Does it stabilize, degrade, fail completely or something in between? Answering such questions helps both in choosing the right product for the situation, to get the right amount of hardware for a specific product, as well as a feedback to the software developers so they can improve their products.
PrePrint of an IEEE Article
In order to get a meaningful data set, the tests should ideally be performed at many different concurrency points and using various access patterns; and at each point the same kind of tests should be executed every time. Moreover, in order to get an accurate measurement each test should be run for an extended period of time; and to minimize interference the service should be completely idle for some time before the start of the tests at the next concurrency point. An automated framework is thus a must. In this paper we analyze the use of Grid computing resources for distributed scalability and reliability testing of centralized, networkfacing services, and describe a framework based on Condor glideins. While not described in any detail in this paper, the glideinbased solution could be used to leverage Cloud computing resources as well. II.
GRID RESOURCES
A. Overview of Grid resources Largescale Grids, like the Open Science Grid (OSG)[1], are composed of tens of thousands of compute nodes distributed over a wide geographical area, and accessed through a workload management system. This makes them a very attractive platform for scalability and reliability testing of centralized, networkfacing services. The compute nodes are however partitioned into compute pools located at various institutions, each with its own administrative domain. The administrator of each compute pool is required to install a thin Grid software layer, but is else free to manage the compute resources in the way (s)he prefers; this organization of resources gives a lot of flexibility to the resource providers, making them much more willing to grant access to external users. Using Grid resources thus requires a bit of flexibility to account for the differences between compute nodes. In order to accommodate a wide variety of resource management practices, the Grid software layer provides only a few basic methods to monitor and control the user jobs. Our Grid jobs will thus need to provide their own mechanisms to obtain a finer grained monitoring and control channel. Finally, Grid compute resources are a shared good, so we will often have to compete with other users for the same job slots. Job slots will thus almost never be available immediately when we want them; instead we will likely get only a random subset at a time, requiring our own interjob
1/5
coordination to handle concurrency levels. We may also need to submit Grid jobs speculatively and ahead of time, in order to reach the desired concurrency levels in reasonable amounts of time. B. Grid resources dedicated to testing Some groups are providing Grid resources dedicated to testing purposes, at scales comparable to the production resources[2]. This is achieved by running test jobs on the same compute nodes as regular Grid jobs. This is possible because of markedly different resources used by regular and a large class of test jobs; regular jobs are heavy CPU and possibly disk users, while jobs testing remote services are mostly network users, and thus use very little CPU and disk. Given the strict limits on CPU use, these test Grid resources are typically much less contended, so the desired concurrency level can be achieved in a much shorter time. One must however be careful to use only suitable test jobs and stay within the limits, or risk being refused access to those resources. III.
CONDOR GLIDEINS
A. Introduction to Condor glideins Condor glideins[3] are a mechanism for creating a virtualprivate compute pool out of Grid resources, using Condor as a private workload management system. One can set up ones own personal Condor pool, but without dedicated compute resources; instead, the compute resources will be harvested from the Grid. The Condor architecture[4] lends itself very well to this task, as it is based on a marketplace paradigm; on one side, scheduler daemons accept and manage user jobs, while on the other, job starter daemons manage compute resources. Each daemon publishes its availability to a central daemon, who serves both as a bookkeeper and as a resource broker. Once a match between a user job and a compute resource is found, the scheduler daemon delivers the user job to the starter daemon who launches the user job, monitors it until termination, and finally returns the job output to the scheduler daemon. To harvest Grid resources, one thus only needs to submit Grid jobs that start a properly configured starter daemon. Once started on a Grid resource, the starter daemon will contact the user central manager, advertising its availability, and the matchmaking process will happen as expected. Fig. 1 shows a schematic overview of the architecture. B. Condor glideins for scalability testing Using a Condor glidein virtualprivate compute pool for scalability and reliability testing of centralized, network facing services has a few advantages over the direct use of Grid resources. First and foremost, it separates the handling of job concurrency in a separate task. The acquisition of Grid resources is now a dedicated process that submits Condor glideins, which are independent of the test jobs that will be PrePrint of an IEEE Article
Glidein
Central Manager
1a
3
Scheduler
1b
2
Grid resource 4
Job Starter 5
Job
Figure 1. Condor glidein overview
run. Once the desired number of glideins is running, we can submit the test jobs to our virtualprivate compute pool, and we are guaranteed that they will all start at the same time. Job monitoring is also greatly improved. Since the compute resources are handled by a Condor starter, we can leverage its capabilities for this task. Condor provides a basic set of monitoring tools out of the box, like memory, disk and CPU usage, but can also be configured to provide much more detailed and personalized monitoring information as well[5]. And since we own the glidein being sent to the Grid resource, we do not need any help from the Grid administrators to achieve the desired functionality. Job control also benefits by having the Condor starter handling the jobs; authorized users can run arbitrary commands alongside the test job at any time[6]. Moreover, the detailed monitoring can also be used to detect and promptly kill misbehaving jobs. Finally, the Condor glideins can even partially uniform the compute environment for the test jobs. The starter can be configured to properly configure the shell environment, provide the missing shared libraries, or even start a full customized virtual machine, if supported by the Grid compute resource. All of the above thus gives us more time to write test jobs that are optimized for the test at hand, or to reuse existing test code that is not Gridaware. The major disadvantage of the Condor glidein approach lies in its centralized architecture; compared to the distributed nature of Grid services it naturally scales much less. However, Condor glideins have been tested up to 20k jobs across a global grid infrastructure[7], and this is enough for our current needs. IV.
LEVERAGING THE GLIDEINWMS
A. Overview of glideinWMS In order to have a fully automated framework for scalability and reliability testing of centralized, network facing services using Condor glideins, we need an easy way to automate the configuration and submission of the glideins. We decided to use glideinWMS[8,9] for this task. The glideinWMS package is composed of two components. The first element is the glidein factory that knows about the existence and configuration of the Grid resources, and will submit glideins to them. The second one
2/5
Dashboard 3
Frontend
Need glidein on X
4 1
Get frontend ads
Glidein Factory 5
Grid resource
Submit glidein
Figure 2. glideinWMS overview
is the frontend that gives orders to the factory, deciding which Grid resources to submit to and how many of them. A well defined interface, based on the use of Condor ClassAds, is used for the communication between them. Fig. 2 shows a schematic overview of the architecture. The architecture is designed to be manytomany; i.e. a single glidein factory can receive orders from many frontends, and any frontend can drive multiple glidein factories. As a result, the glidein factory is relatively generic and can serve multiple communities, while the frontend is tailored to the needs of a specific community. In this work, we will be using the glideinWMS glidein factory without any change, possibly shared with other, non testing users. We combine this with a fully custom frontend. B. The need for a dedicated frontend The frontend provided with the glideinWMS is designed to be reactive; its modus operandi is to monitor a Condor scheduler for waiting jobs and to request glideins if there are any. While this works fine for opportunistic use of Grid compute resources, it does not suit our use case in which we want to submit the test jobs only after the required number of glideins have started. Moreover, the frontend provided by the glideinWMS is not explicitly handling glidein termination. The glideins are expected to terminate by themselves when there are no more jobs to be served. In our use case glideins may need to stay idle for extended periods of time while we wait for the needed concurrency level to be reached; only when all the tests have been performed should the glideins terminate. Fig. 3 provides a visual comparison between the two approaches. The closest to what we want to achieve is provided by the explicit provisioning of Corral[10], which has recently started an integration with the glideinWMS. However, Corral is mostly interested in manual provisioning of resources, while we want a fully automated process. For all the above reasons, we decided to implement a dedicated frontend, that is tightly integrated with our testing framework. The glideinWMS provides a good communication API, allowing us to concentrate on the core functionality. V.
THE TESTING FRAMEWORK
A. High level overview The testing framework we propose thus relies on a virtualprivate Condor glidein compute pool, with a
PrePrint of an IEEE Article
glideinWMS frontend logic Jobs Glideins sub. run. sub. run. sub. sub.
Can submit to Grid resource X
Desired logic Glideins Jobs sub. run. sub. run. sub. sub.
Time
Get factory ads 2
Figure 3. A comparison between the glideinWMS frontend logic and the desired glidein submission logic
glideinWMS glidein factory in charge of submitting jobs to the Grid. On top of that, we developed a daemon that implements the glideinWMS frontend interface and is able to submit condor jobs. The testing daemon is configuration driven. The tester provides a test executable, a list of requirements for the selection of Grid resources, and a list of concurrency levels, together with the desired minimum wait time between them. At startup, the testing daemon starts requesting glideins to the glidein factory, and continues to do so until enough glideins have started to perform the tests at the first concurrency level. At this point, a number of testing jobs equal to the desired concurrency level is submitted. The daemon now monitors the scheduler until all testing jobs have terminated. Now the daemon starts requesting glideins until the next level is reached. If not enough time has passed since the last job has ended, the daemon will simply sleep for a while, else the next round of jobs is submitted and the loop restarts as above. Once the test jobs of the last concurrency level have terminated, the daemon terminates the glideins. Fig 4. gives an overview of the framework.
Get factory ads 2
Testing daemon Submit test 8 job
7 Count resources
Dashboard 3 Need glidein on X
4 1
Get frontend ads
Can submit to Grid resource X
Glidein Factory
Match
Scheduler
9
6
Submit glidein
5
Central Manager Running on X
Grid resource 10
Deliver test job
Job Starter
11
Test job
Figure 4. Overview of the testing framework
3/5
Queries per hour
20.0k 15.0k 10.0k 5.0k 0.0k 0 Fermilab
Figure 5. Grid resources used during the test
Success fraction
150 100 50 0 0 Fermilab UCSD
Seconds between first and last
UCSD
250 200 150 100 50 0 0
PrePrint of an IEEE Article
50 100 150 200 250 300 350 400 450 500 Concurrency Figure 8. Max job startup delay
Concurrency
B. Implementation details The Condor daemons and the glideinWMS glidein factory daemon are used unmodified, and configured and operated as suggested by the glideinWMS manuals. All the Condor monitoring and debugging features are thus available. The tester daemon itself, called the glideTester[11], is written in Python[12], the same language used by the glideinWMS. This allowed us to use the same libraries the glideinWMS frontend uses internally for both the communication with the glideinWMS factory and for the interaction with the Condor daemons. All interaction with the tester is based on configuration files. Simple ASCII files, composed of key = value lines are used. Comments are of course supported. Our daemon does not assume anything about the test jobs; it only submits them and saves the output. In order to prevent data loss, a separate subdirectory is created for each instance of the test jobs. Moreover, to keep bookkeeping manageable, the job directories are grouped by the concurrency level. Analysis of the output of the tests is beyond the current scope of the framework, although may be implemented in the future. However, the tester daemon does provide the exit status of the test jobs for each completed concurrency level.
Concurrency Figure 7. RESS queries per hour
50 100 150 200 250 300 350 400 450 500
Figure 6. RESS query success rate
50 100 150 200 250 300 350 400 450 500
VI.
THE TESTING FRAMEWORK IN PRACTICE
We used the testing framework to characterize the scalability and reliability of a OSG information system, namely RESS[13], located at the Fermi National Laboratory (Fermilab) near Chicago, IL, USA. The test Grid resources were provided by two sites, one at Fermilab and one at the US University of California San Diego (UCSD); both provide test Grid resources as described in one of the previous sections. The test job provided to the framework was a shell script containing a tight loop querying the RESS service. The queries are run for an hour, counting the number of successful or failed responses from the information system; assuming that all start at the same time, this guarantees that they all run in the same timeframe, too. We ran the test jobs at concurrencies of 1, 10, 50, 100, 200, 300, 400 and 500. Getting the required Grid resources was a fast and reliable operation, as showed in Fig. 5. We could have also gone higher than that, but at the concurrency level of 500 the query failure rates were already significant enough that we deemed it not worth the effort, as shown in Fig. 6. The failures were calculated with scripts tailored to our test jobs, since the framework itself does not provide any tools for test result analysis. While we do think having standardized tools would be useful, it was outside our current scope and the customized tools were easy to produce. The main aim of the test was to determine the performance of the RESS service both when queried from nodes on the Fermilab LAN and from nodes communication via WAN. We attained that goal, as shown in Fig. 6 and Fig. 7.
4/5
We also collected information on how close the jobs started to each other; this is important to assure that the desired concurrency level was indeed attained. As shown in Fig. 8, the jobs started within minutes from one another, which was more than acceptable for out tests. However, the difference in startup between the first and the last job was proportional with the concurrency level; Condor configuration optimizations should probably be attempted to avoid this to become a problem at really high concurrency levels. VII. RELATED RESEARCH A very similar product is DiPerF[14], that has also approached the use of Grid resources for scalability testing of centralized, networkfacing service. It is much larger in scope than our framework, but also more restrictive; while our goal is to allow for arbitrary tests to be run, DiPerF has a much narrower goal of a single RPClike command per test job. Additionally, DiPerF seems to use mostly code implemented inhouse, while our framework leverages as much as possible offtheshelf products, such as Condor and glideinWMS. This may partially explain why DiPerF is now abandoned and not even the original source code seems to be available from the project site anymore[15]. There are many nonGrid tools being developed, an example being MRNet[16], which do an excellent job in well controlled environments. However, their use on Grid resources is much more difficult, if at all possible. Finally, Web server performance testing, a subset of our target goal, has also been a topic of much research. Many of the results apply to our more general environment as well, but will not be discussed in detail here. VIII. CONCLUSIONS Scalability and reliability testing of centralized, network facing services requires the use of a large number of geographically distributed resources, in order to produce reliable results. Grid deployments, like OSG, are an obvious candidate for providing such resources, with slots dedicated for testing purposes being the ideal target. However, in order to use such resources a dedicated testing framework is needed. The testing framework described in this paper provides an easy to use tool to obtain a complete set of test results obtained at various concurrency levels. The use of Condor glideins provides a powerful foundation that grants us great flexibility, while still providing all the functionality we need. The framework was recently used to test the scalability and reliability of an OSG information system service. The experience was very positive as it allowed us to easily test its scalability using up to 500 concurrent clients located in two different geographical regions. The test data was gathered withing 24 hours and completely autonomously. The gathered data did however require customized tools for data analysis, so we plan to focus our future work on providing standard data output formats and some analysis tools.
PrePrint of an IEEE Article
ACKNOWLEDGMENT This research was supported in part by the US Department of Energy under Grants No. DEFC02 06ER41436 subcontract No. 647F290 (OSG), and the US National Science Foundation under Grants No. PHY0533280 (DISUN) and PHY0612805 (CMS Maintenance & Operations). REFERENCES [1] [2]
[3]
[4]
[5] [6] [7] [8] [9] [10] [11] [12] [13]
[14]
[15] [16]
“Open Science Grid home page”, http://www.opensciencegrid.org/, accessed Jan 2010 I. Sfiligoi and B. Holzman, “An objective comparison test of workload management systems”, Proceedings of CHEP 2007, J. Phys.: Conf. Ser. 119 062043 (6pp) doi: 10.1088/17426596/119/ 6/062043 J. Frey, T. Tannenbaum, M. Livny, I. Foster, and S. Tuecke, “CondorG: A computation management agent for multiinstitutional grids”, Cluster Computing, v.5 n.3, p.237246, July 2002, doi: 10.1023/A:1015617019423 Rajesh Raman, Miron Livny, and Marvin Solomon, "Matchmaking: distributed resource management for High Throughput Computing", Proceedings of the Seventh IEEE International Symposium on High Performance Distributed Computing, July 2831, 1998, Chicago, IL “Condor cron jobs”, http://www.cs.wisc.edu/condor/manual/v7.4/ 3_3Configuration.html#param:StartdCronJobs, accessed Jan 2010 I. Sfiligoi, D. Bradley, and M. Livny, “Pseudointeractive monitoring, in distributed computing”, Proceedings of CHEP 2009, in press D. Bradley, I. Sfiligoi, S. Padhi, J. Frey, and T. Tannenbaum, “Interoperability and scalability within glideinWMS”. Proceedings of CHEP 2009, in press “glideinWMS home page”, http://www.uscms.org/ SoftwareComputing/Grid/WMS/glideinWMS/, accessed Jan 2010 I. Sfiligoi, “glideinWMS—a generic pilotbased workload management system”, Proceedings of CHEP 2007, J. Phys.: Conf. Ser. 119 062044 (9pp), doi: 10.1088/17426596/119/6/062044 “Corral home page”, http://pegasus.isi.edu/corral/latest/, accessed Jan 2010 “glideTester at OSG Scalability Area Tools”, http://sourceforge.net/ projects/osgscal/, accessed Jan 2010 “Python Programming Language”, http://www.python.org/, accessed Jan 2010 G. Garzoglio, T. Levshina, P. Mhashilkar, and S. Timm, “ReSS: A Resource Selection Service for the Open Science Grid “, in Grid Computing, Springer US, 2008, pp. 8998, doi: 10.1007/9780387 784175 I. Raicu, C. Dumitrescu, M. Ripeanu, and I. Foster, “The design, performance, and use of DiPerF: an automated DIstributed PERformance testing Framework”, Special Issue of Journal on Grid Computing “DiPerF home page”, http://diperf.cs.uchicago.edu/, accessed Jan 2010 “MRNet: a Multicast/Reduction Network”, http://www.paradyn.org/mrnet/, accessed Jan 2010
5/5
![Distributed Testing with Selenium Grid [PDF]](https://m.moam.info/img/260x300/distributed-testing-with-selenium-grid-pdf_6496e706098a9e6c108b456c.jpg)
