Mining for Availability Models in Large-Scale Distributed Systems: A

Mining for Availability Models in Large-Scale Distributed Systems: A Case Study of SETI@home Bahman Javadi1 , Derrick Kondo1 , Jean-Marc Vincent2 , David P. Anderson3 INRIA, France, 2 University of Joseph Fourier, France, 3 UC Berkeley, USA

1

inria-00375624, version 1 - 15 Apr 2009

Abstract In the age of cloud, Grid, P2P, and volunteer distributed computing, large-scale systems with tens of thousands of unreliable hosts are increasingly common. Invariably, these systems are composed of heterogeneous hosts whose individual availability often exhibit different statistical properties (for example stationary versus non-stationary behaviour) and fit different models (for example Exponential, Weibull, or Pareto probability distributions). In this paper, we describe an effective method for discovering subsets of hosts whose availability have similar statistical properties and can be modelled with similar probability distributions. We apply this method with about 230,000 host availability traces obtained from a real large-scale Internet-distributed system, namely SETI@home. We find that about 34% of hosts exhibit availability that is a truly random process, and that these hosts can often be modelled accurately with a few distinct distributions from different families. We believe that this characterization is fundamental in the design of stochastic scheduling algorithms across large-scale systems where host availability is uncertain.

1

Introduction

With rapid advances in networking technology and dramatic decreases in the cost of commodity computing components, large-scale distributed computing platforms with tens or hundreds of thousands of unreliable and heterogeneous hosts are common. The uncertainty of host availability in P2P, cloud, or Grid systems can be due to the host usage patterns of users, or faulty hardware or software. Clearly, the dynamics of usage, and hardware and software stacks are often heterogeneous, spanning a wide spectrum of patterns and configurations. At same time, within this spectrum, subsets of hosts with homogeneous properties can exist. So one could also expect that the statistical properties (stationary versus non-stationary behavior for example) and models of host availability (Exponential, Weibull, or Pareto

for example) to be heterogeneous in a similar way. That is, host subsets with common statistical properties and availability models can exist, but differ greatly in comparison to other subsets. The goal of our work is to be able to discover host subsets with similar statistical properties and availability models within a large-scale distributed system. In particular, the main contributions are as follows: • Methodology. Our approach is to use tests for randomness to identify hosts whose availability is independent and identically distributed (iid). For these hosts, we use clustering methods with distance metrics that compare probability distributions to identify host subsets with similar availability models. We then apply parameter estimation for each host subset to identify the model parameters. • Modelling. We apply this method on one of the largest Internet-distributed systems in the world, namely SETI@home. We take availability traces from about 230,000 hosts in SETI@home, and identify host subsets with matching statistical properties and availability models. We find that a significant fraction (34%) of hosts exhibit iid availability, and a few distributions from several distinct families (in particular Gamma, Weibull, and Log-Normal) can accurately model the availability of host subsets. These results are essential for the design of stochastic scheduling algorithms for large-scale systems where availability is uncertain. In the next section, we describe the application context that is used to guide our modelling. In Section 3, we define availability and describe how our availability measurements were gathered. In Section 4, we contrast this measurement method and our modelling approach to related work. In Section 6, we determine which hosts exhibit random, independent, and stationary availability. In Section 7, we describe the clustering of hosts by their availability distribution. In Section 8, we describe the implications of our results for scheduling, and in the last section, we summarize our findings.


2

Application Context

3

We describe here the context of our application, which is used to guide what we model. Our modelling is conducted in the context of volunteer distributed computing, which uses the free resources of Internet-distributed hosts for large-scale computation and storage. Currently this platform is limited to embarrassingly parallel applications where the main performance metric is throughput. One of the main research goals in this area is to broaden the types of applications that can effectively leverage this platform. We focus on the problem of scheduling applications needing fast response time (instead of high throughput). This class of applications includes those with batches of compute-intensive tasks that must be returned as soon as possible, and also those whose tasks are structured as a directed acyclic graph (DAG). For these applications, the temporal structure of availability and resource selection according to this structure is critical for their performance. The temporal structure of availability is important because many applications cannot easily or frequently checkpoint due to complexities of its state or checkpointing overheads. Resource selection is important because in our experience and discussions with application scientists the maximum number of tasks in a batch for fast turnaround is needed is much lower than the number active and available hosts at a given time point (hundreds of thousands). This same is true for the maximum number of tasks available in a DAG application during its execution.

CPU on host i

A1

A2

An

Time Figure 1. CPU availability intervals on one host

Thus our modelling focuses on the interval lengths of continuous periods of CPU availability for individual hosts. We term this continuous interval to be an availability interval (see Figure 1). The lengths of the availability intervals of a particular host over time is the process that we model. Using these models, a scheduler should then be able to compute accurately the probability of successful task execution in some time frame, given a task’s execution time. Moreover, the scheduler could compute the probability of success when task replication is used.

Measurement Method for CPU Availability

BOINC [5, 2] is a middleware for volunteer distributed computing. It is the underlying software infrastructure for projects such as SETI@home, and runs across over 1 million hosts over the Internet. We instrumented the BOINC client to collect CPU availability traces from about 230,000 hosts over the Internet between April 1, 2007 to January 1, 2009. We define CPU availability to be a binary value indicating whether the CPU was free or not. The traces record the time when CPU starts to be free and stops. Our traces were collected using the BOINC server for SETI@home. The traces are not application dependent nor specific to SETI@home because they are recorded at the level of the BOINC client (versus the application it executes). In total, our traces capture about 57,800 years of CPU time and 102,416,434 continuous intervals of CPU availability. Artifacts due to a measurement method or middleware itself are almost inevitable. In our case, artifacts resulted from a benchmark run periodically every five days by the BOINC client. As a result, availability intervals longer than five days were truncated prematurely and separated by about a one minute period of unavailability before the next availability interval. Histograms and mass-count graphs showed that these 5-day intervals were outliers and significantly skewed the availability interval mass distribution. As such, we preprocessed the traces to remove these gaps after five day intervals artificially introduced by our middleware. This minimized the effect of these anomalies in the modelling process.

4

Related Work

This work differs from related in terms of what is measured and what is modelled. In terms of measurements, this work differs from others by the types of resources measured (home versus only within an enterprise or university), the scale and duration (hundreds of thousands over 1.5 years versus hundreds over weeks), and the type of measurement (CPU availability versus host availability). Specifically, the studies [12, 6] focus on host in the enterprise or university setting and may have different dynamics compare to hosts primarily on residential broadband networks. Our study includes both enterprise and university hosts, in addition to hosts at home. Also, other studies focus on hundreds of hosts over a limited time span [12, 7, 15, 1]. Thus their measurements are of limited breadth and could be biased to a particular platform. The limited number of samples per host prevents accurate modelling at the resource level.


Furthermore, while many (P2P) studies of availability exist [16, 4, 14], these studies focus on host availability, i.e., a binary value indicating whether a host is reachable. By contrast, we measure CPU availability as defined in Section 3. This is different as a host can clearly be available but not its CPU. Nevertheless, host availability is subsumed by CPU availability. In terms of modelling, most related works [7, 18, 11] focus on modelling the system as a whole instead of individual resources, or the modelling does not capture the temporal structure of availability [3, 11, 13]. Yet this is essential for effective resource selection and scheduling. For instance, in [7], the authors find that availability in wide-area distributed systems can be best modelled with a Weibull or hyperexponential distribution. However, these models consider the system as a whole and are not accurate for individual resources. Even the authors note that their best fitted global distribution is not adequate for modelling individual hosts. For another instance, in [11], the author models host availability as a fraction of time. However, the temporal structure of availability, which is critical to computational tasks for example, is not captured in the model. A host with that is 99% available but in intervals of 1 millisecond may be of little use to distributed applications. For a final instance, in our own past work in [13], we conducted a general characterization of SETI@home. However, we ignored availability intervals by taking averages to represent each host’s availability. So we used a different data representation with different distance metrics when clustering. Moreover, the purpose was different because we intentionally focused on identifying correlated patterns instead of random ones. As such, we did not build statistical models of availability intervals, and did not address issues such as partitioning hosts according to randomness and probability distributions.

5

Experimental Setup

We conduct all of our statistical analysis below using Matlab 2009a on a 32-bit on a Xeon 1.6GHz server with about 8.3GB of RAM. We use when possible standard tools provided by the Statistical Toolbox. Otherwise, we implement or modify statistical functions ourselves.

6

Randomness Testing

A fraction of hosts are likely to exhibit availability intervals that contain trends, periodicity, autocorrelation, or nonstationarity. For these hosts, modelling their availability using a probability distribution is difficult given the change of its statistical properties over time.

Therefore, as the preliminary phase before data analysis and modeling we apply randomness tests to determine which hosts have truly random availability intervals. There are several randomness tests which are divided in two general categories: parametric and nonparametric. Parametric tests are usually utilized when there is an assumption about the distribution of data. As we cannot make any assumption for the underlying distribution of the data set, we adopted non-parametric randomness tests. We conducted four well-known non-parametric tests, namely the runs test, runs up/down test, autocorrelation test and Kendall-tau test [17, 10]. For all tests, the availability intervals of each hosts was imported as a given sequence or time series. We describe intuitively what they measure. The runs test compares each interval to the mean. The runs up/down test compare each interval to the previous interval to determine trends. The autocorrelation test discovers repeated patterns that differ by some lag in time. The Kendall-tau test compares a interval with all previous ones. Since the hypothesis for all these tests are based on the normal distribution, we must make sure that there are enough samples for each host, i.e., at least 30 samples, according to [8] and personal communication with a statistician. About 20% of hosts do not have enough samples because of a limited duration of trace measurement. (Measurement for a host began only after the user downloaded and installed the instrumented BOINC client. So some hosts may have only a few measurements because they began using the instrumented BOINC client only moments before we collected and ended the trace measurements.) So we ignored these hosts in our statistical analysis. Finally, we applied all four test on 168751 hosts with a significance level of 0.05. As there is no perfect test for randomness, we decide to apply all tests and to consider only those hosts that pass all four tests to be conservative. Table 1 shows the fraction of hosts that pass each and all four tests. While iid hosts may not be in the majority (i.e., 34%), together they still form a platform with significant computing power. In total, volunteer computing systems, which now include GPU’s and Cell processors of the Sony PS3, provide a sustained 6.5 PetaFLOPS. The hosts with iid availability thus contribute about 2.2 PetaFLOPS, which is significant.

7

Clustering by Probability Distribution

For hosts whose availability is truly random, our approach was to first inspect the distribution using histograms and the mass-count disparity (see Figures 2(a) and 2(b)). We observe significant right skew. For example, the shortest 80% of the availability intervals contribute only about 10% of the total fraction of availability. The remaining longest 20% of intervals contribute about 90% of the total fraction

Test # of hosts Fraction

Runs std 101649 0.602

Runs up/down 144656 0.857

ACF 109138 0.647

Kendall 101462 0.601

All 57757 0.342

Table 1. Result of Randomness Tests of availability. The implication for modelling is that we should focus on the larger availability intervals. But which hosts can be modelled by which distributions with which parameters?


7.0.1

Clustering Background

We use two standard clustering methods, in particular kmeans and hierarchical clustering, to cluster hosts by their distribution. K-means randomly selects k cluster centers, groups each point to the nearest cluster center, and repeats until convergence. The advantage is that it is extremely fast. The disadvantages are that it requires k to be specified a priori, the clustering results depend in part on the cluster centers chosen initially, and the algorithm may not converge. Hierarchical clustering iteratively joins together the two sub-clusters that are closest together in terms of the average all-pairs distance. It starts with the individual data points working its way up to a single cluster. The advantage of this method is that one can visualize the distance of each sub-cluster at different levels in the resulting dendrogram, and that it is guaranteed to converge. The disadvantage is that we found the method to be 150 times slower than kmeans. 7.0.2

Distance Metrics for Clustering

We tested several distance metrics for clustering, each metric measures the distance between two CDFs [9]. Intuitively, Kolmogorov-Smirnov determines the maximum absolute distance between the two curves. Kuiper computes the maximum distance above and below of two CDFs. Cramer-von Mises finds the difference between the area of under the two CDFs. Anderson-Darling is similar to Cramer-von Mises, but has more weight on the tail. Using these distance metrics is challenging when the number of samples is too low or too high. In the former case, we do not have enough data to have confidence in the result. In the latter case, the metric will be too sensitive. A model by definition is only an approximation. When the number of samples is exceedingly high, the distance reported by those metrics will be exceedingly high as well. This is because the distance is often a multiplicative factor of the number of samples. Our situation is complicated even further by the fact that we have a different number of samples per host. Ideally,

when computing each distance metric we should have the same number of samples. Otherwise, the distance between two hosts with 1000 samples will be incomparable to the distance between two hosts with 100 samples. Our solution is to select a fixed number of intervals from each host at random. This method was used also in [7], where the authors had the same issue. We also discussed this issues with a statistician, who confirmed that this is an acceptable approach. However, if we choose the fixed number to be too high, it will excluded too large of a portion of hosts from our clustering. If we choose the number to be too low, we will not have statistical confidence in the result. We choose the fixed number to be 30 intervals as discussed in Section 6. The test statistics corresponding to each distance metric is normally distributed, and so with 30 intervals, one can compute pvalues with enough accuracy. We performed extensive experiments to compare the distance metrics for several cases including for different subsets of iid hosts. We also conducted positive and negative control experiments where the cluster distributions were generated and known a priori. The following conclusions were been found to be consistent across all the cases considered. We found out that Kolmogorov-Smirnov and Kuiper are not very sensitive in terms of distance as they are only concerned with extreme bounds. Cramer-von Mises and Anderson-Darling revealed that they are good candidates of clustering, but Anderson-Darling is engaged with high time and memory complexity. Moreover, Cramer-von Mises is advantageous for right-skewed distributions [9]. Thus we used the Cramer-von Mises as the distance metric when clustering. 7.0.3

Cluster Results and Justification

We found that optimal number cluster was 6. We justify this number of clusters through several means. First, we observed the dendrogram as a result of hierarchical clustering for a random subset of hosts (due to memory consumption and scaling issues). As it can be seen in Figure 3(a) the tree is an unbalanced tree where the height of tree shows the distance between different (sub)clusters. The good separation of hosts in this figure confirmed the advantage of Cramervon Mises as the distance metric (in comparison to the result with other metrics). The number of distinct groups in this dendrogram where the distance threshold is set to one

6

x 10

1

7

0.9

6

0.8 Cumulative fraction

Frequency

8

5 4 3 2

0.7

mean: 12.6971

0.6

std: 56.757

0.5

median: 1.0547

0.4

min: 2.7778e−07

0.3

max: 13215.9303

0.2

1

50

100 Availability (hours)

150

200

0 0

200

400 600 Length of interval (hours)

(a) Histogram

800

1000

(b) Mass-count

Figure 2. Distribution of Availability Intervals reveals that number of clusters should be between 5 to 10. Second, using the result of hierarchical clustering as a bootstrap, we run k-means clustering for all iid hosts and then compute the within-group and between-group distances for various values of k (see Figure 3(b)). We observe that the maximum ratio of inter-cluster over intra-cluster distance is with six clusters. Specifically, for k of 6, the inter-cluster distance between all clusters was maximized relative to other values of k. Also, for k of 6, the distance between all clusters is roughly equal (about 5). Third, we plotted the EDF corresponding to each cluster for a range of k. In this way, we can observe convergence or or divergence of clusters during their formation. Fourth, we plotted the EDF corresponding to each cluster. Figure 4 shows good separation of these plots.

7.1

Parameter Estimation for Each Cluster

1 6

0.9 0.8 Cumulative fraction


0 0

Mass Count

0.1

3

5 2

1 4

0.7 0.6 0.5 0.4 Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Cluster 6

0.3 0.2

After cluster discovery, we conduct parameter fitting for various distributions, including the Exponential, Weibull, Log-normal, Gamma, and Pareto. Parameter fitting was conducted using maximum likelihood estimation (MLE). Intuitively, MLE maximizes the log likelihood function that the samples resulted from a distribution with certain parameters. We also considered using moment matching instead of MLE. However, because the overall distribution is heavily right-skewed, we were concerned that moment matching, which is relatively much faster then MLE, would not produce the most accurate results. Moment matching is also

0.1 0 0

200

400 600 Availability (hours)

Figure 4. EDF of clusters

800

1000

n a−C us e max n e −C us e m n Ra o 6 5

D s ance

D s ance

4 3 2 1 0 10

10955 15255 21063 18179 20328 12505 19657 16978 16629 18775 19876 11538 24026 19524 18492 14128 21349 16890 23701 11486 20048 10127 16644 15915 17152 17328 20578 12616 11841 12347 24305 22407 13161 13806 17117 14066 12753 23375 10162 22260 12215 24356 22266 11633 12630 24667 22033 15875 24495 21657 19533 16074 19497 13795 22668 20321 18099 11272 19420 23568 18476 18434 23501 23440 12898 14255 15025 13699 19613 14418 24227 12411 19108 16310 21186 14194 10190 19716 17247 13734 23039 15657 13859 24742 12590 12444 23609 13702 18006 19516 22566 12912 11585 12736 24581 16984 11569 24007 19484 16909 13693 12404 24207 19761 10366 23919 14560 11782 19941 23206 12594 10404 18583 18009 17540 11442 12978 21260 18394 13668 16656 23935 13613 16928 14200 14061 12137 23014 18454 22809 17643 18417 13874 16920 14676 19459 12160 18176 4597 16489 16465 15116 18069 3704 2208 20636 17020 11375 17438 21122 4968 6169 23882 16333 13105 11042 19343 12017 9548 18337 12219 20278 1980 9019 5578 15417 22280 18140 3644 2419 5286 7934 15096 11735 22896 5502 20181 12063 18817 5748 17207 20220 7794 1015 20930 13703 24606 13274 10744 1039 24284 21485 15771 23917 10310 5598 4974 20180 14567 5337 5829 14352 16383 19033 6278 4805 19112 24946 13466 10378 21739 14004 1902 13838 18931 18446 12472 5742 22003 11411 17039 2456 5385 3446 16332 19726 5815 20345 10154 16623 6589 2474 1077 24283 19570 23126 5034 9353 1898 19398 11972 19750 24024 23980 9326 1945 17702 12118 12604 2181 3108 20133 24682 15740 16066 8820 9525 4874 19559 11601 4619 24058 18027 16530 22670 9067 21934 12213 7706 16139 1575 5802 21560 10492 23549 23751 13055 9300 3396 16442 24422 18249 9422 8947 8782 6819 16796 13665 11263 7018 9116 3605 16680 15455 12733 11401 23942 23120 19087 11455 14497 5312 19164 13256 13470 8963 17569 18968 22034 6049 24036 22269 22769 1578 3974 20034 5168 16371 24993 11428 7609 24901 13423 3192 1369 20349 22113 12129 14867 7775 10822 24539 3542 1921 17243 12567 24566 14141 6223 6748 19012 10330 24402 6501 8459 2226 23312 20007 17080 19807 8519 8802 10094 19572 12929 10376 2814 5611 15616 19075 19146 21689 11156 15805 4857 3926 22190 14290 8747 4465 224 23890 19254 18800 7200 19865 18920 17504 1596 6706 8529 20343 4880 719 771 10342 23490 9253 6832 20309 20037 12170 6309 385 18756 10395 20248 2818 24189 9662 7006 19240 8436 3084 11144 10480 4907 431 14996 16461 21390 6694 1279 1208 13610 21524 17832 1325 4303 2719 13304 19298 15050 6368 6592 4228 24683 22436 17748 19569 17735 7055 869 17321 14978 11829 7236 1682 23296 12538 18642 10963 6765 1963 15316 11204 16457 19042 7160 9642 5377 24620 24100 7742 9857 938 10185 17944 9437 4170 15160 10857 23855 9555 2268 16093 14187 14896 16142 2593 22558 11886 23898 10262 17241 11291 2608 14922 4265 12254 16570 3919 7980 23873 23799 3141 15127 13599 7943 9740 20933 15273 15313 11169 14466 8650 1864 1153 20939 23771 23511 3150 8117 7790 1388 13003 16143 15040 4841 7056 13472 10301 19260 18477 15905 18060 16754 16744 6365 20154 17845 24599 2661 20493 12982 10649 8888 4621 17295 14011 11463 24916 21166 12138 8077 4236 4778 2403 17385 1856 17790 12974 5577 8510 6106 820 20507 11963 10841 2040 21495 19334 13127 5906 7619 17025 17142 10780 14086 21668 18629 14639 8353 159 18705 14134 8714 23131 17610 17111 10673 11622 19405 15264 12267 8581 4962 15572 22864 17046 12188 4594 3147 23257 14930 22187 7545 5889 11029 11217 21178 8564 19937 19724 14738 14334 7557 6233 2161 1352 18047 17596 17136 6877 1357 11317 22153 22836 11764 2346 7536 14181 22879 20108 14495 14136 9985 7995 1877 131 20740 5825 5380 3870 1367 15070 11022 3467 631 16525 16673 14030 2434 2341 4733 18379 18680 13468 13218 1897 6529 4646 22813 23705 9013 4601 10977 7438 16488 22265 24014 21395 45 2648 10758 22452 15300 2957 20599 24783 24636 3684 19319 15501 23957 15100 5323 8125 13963 18378 23833 7842 5236 21239 21034 10934 2043 20619 17922 5438 22316 1853 9449 13743 10710 1939 10729 18878 9412 4852 2485 5835 22851 24940 23416 9197 22990 11801 11360 22225 7138 9493 4491 12183 10087 9877 5537 15850 10518 16394 15144 16174 10107 12049 3639 2213 23456 20897 20084 13760 5329 4776 7662 16937 14593 24269 14947 14297 11660 14941 3075 5868 8701 19053 16224 4536 184 24196 17949 12242 3653 4356 14109 8161 5975 24252 23434 3803 21157 17133 65 2076 1249 24158 20440 92 4172 925 23435 20729 20437 17416 7242 6742 6249 16202 16778 6997 1558 6565 10181 17839 17175 12605 8827 23488 21638 4603 6130 17558 8333 12840 18134 18014 1990 10321 22783 19052 24650 12359 17476 6470 7956 12827 18612 9889 8868 4599 12808 24265 8122 563 19927 17305 5746 234 22364 10980 9557 8281 23169 18307 22412 16921 12048 10580 21020 13299 20450 5418 1240 19139 19426 10891 17869 15036 1115 3745 14370 13263 13933 22354 3970 2725 9344 23926 17901 4533 9115 3350 673 19318 7387 6733 7932 10277 17914 8674 868 14082 15182 10418 5092 13503 23386 20747 12495 2891 5127 12416 11721 10813 16888 9686 3816 1160 3850 16689 23758 3510 5715 1975 20961 18067 15798 10933 8150 2016 11652 10970 12933 1929 9542 13067 16192 10695 10771 4476 14574 19278 1716 13302 9443 22694 21888 20863 17331 9342 3614 732 10114 2174 24736 18193 11891 5301 3130 1918 3727 22079 20737 5778 4025 2776 14094 18652 23684 8202 24858 13794 12268 9836 2865 16145 15863 120 15153 13812 4280 4042 3090 19567 13900 23626 19775 14857 13133 5571 23557 20514 19 3669 765 16102 12867 12294 5328 17063 6987 452 23945 18528 7395 19007 17158 17542 5893 8922 24775 13573 20891 9294 1924 8931 12652 15883 15538 9485 5476 6886 23757 19259 21421 7205 7271 1144 20498 16896 16549 6984 6781 17403 20936 24508 1336 22957 2001 5561 808 24272 13935 8041 21902 19344 14727 5039 1549 24247 22169 12145 2498 5410 8394 5040 10238 24109 16138 4339 4185 2992 14646 15267 18264 7379 9880 2700 252 20918 20497 11265 70 7963 9933 762 13614 20669 3881 14781 15729 17366 13793 4318 8355 3788 24417 23742 14648 13945 22179 4359 754 15433 13884 5610 7849 1993 22844 16529 23944 5110 8396 10414 12783 14893 11696 23768 15242 4714 12876 21712 17547 17492 1649 23414 14966 8349 3744 8191 10394 21322 21394 14231 17029 11240 5781 4530 7040 22099 15965 19123 91 7582 5519 4912 16672 18610 8019 8253 8854 15062 13701 9385 24320 19762 23396 17664 22721 23344 7274 10919 8856 8577 11787 2411 18856 10983 17432 7751 11046 13726 7926 10840 22344 7355 4766 23604 24791 24907 20874 7910 6113 8112 6003 21538 14174 11052 15105 23631 7411 6752 8959 22906 16353 18891 3149 668 21552 17613 17155 15379 14835 7957 18559 24518 4071 2605 2873 727 12688 22531 3302 7448 7659 24691 22290 7905 9922 5285 15360 23902 12954 10178 2743 16484 18263 20856 8740 24048 16717 21410 16533 1309 6265 364 21505 16265 13846 8814 20423 17427 6540 5540 4727 17534 10838 11516 9920 1840 18575 23536 10241 18991 1732 818 22645 24162 19801 20903 6531 7083 22320 10563 21089 3309 13644 16168 2223 13716 12348 1304 1893 10897 18749 5992 15347 10621 3360 2747 4576 7865 13372 3386 7136 10936 10546 23643 3838 7793 21286 13946 2909 2131 16843 23144 18050 6526 20246 16342 18036 9502 8487 21128 21877 14929 8918 17691 8123 6159 3214 19204 15940 10263 21885 21559 11395 20382 10269 9241 629 21281 19126 21482 21675 6558 24344 22409 21721 11418 9244 1505 7559 11555 17740 16991 18159 11259 24055 12402 5125 11392 2073 3263 14026 2157 9468 17274 16741 20824 20582 7257 3277 20800 4891 1232 5516 15454 15554 23525 6593 8489 20604 3355 2159 17678 21645 15307 11400 20269 4517 958 23639 12908 20215 19088 5895 8231 7967 9324 15676 13824 5228 4494 3920 18482 21843 19336 17704 23650 4522 5289 11105 24083 7422 7239 20087 17129 14268 9289 8785 5399 24206 3612 18531 1622 10566 3832 6367 6433 19419 6416 16336 13244 14862 13440 179 10253 5910 966 11501 19863 10511 10150 5731 4538 1053 7000 24761 13286 4814 6414 18117 15262 2672 6371 16531 11839 11682 3092 20254 19084 7488 23856 22483 22305 7189 24860 24846 22112 9957 21617 14783 21425 5026 2108 468 13410 21292 12986 6354 5061 12110 18647 24117 557 18210 14077 6525 10672 9578 11290 21363 18329 5706 2484 21870 12250 22181 10821 2452 10062 16821 11394 3650 2348 23851 14663 11623 4401 19369 13970 8896 6908 9499 2784 15855 11632 6031 4654 20471 14596 10314 23709 8004 21749 20015 9947 23546 10868 22930 6550 24309 21285 17167 3573 8924 19350 10711 17058 7400 6521 6517 13192 22701 17886 15835 18058 9085 2760 20293 19916 16870 8975 1624 16514 20267 10197 21778 1885 24712 15841 387 17975 20407 20104 18713 2666 24942 23329 10910 19161 7572 2425 5171 16354 11163 11103 23820 7700 4921 4553 18659 13309 11250 1783 3755 2269 14135 16776 15627 12106 4586 2222 15221 15014 22665 5987 5144 7739 22998 12926 17919 2558 2848 18554 17618 9515 4190 6350 13030 10828 13894 15532 14120 21237 12075 20415 7320 6672 6155 20965 18460 21901 12223 18194 18877 1837 23991 20290 20211 23046 7303 2696 16176 12793 4836 2914 1453 14081 23998 5403 9297 3691 8376 15799 21022 10932 8295 3981 6799 12655 16751 14474 23634 11882 3419 5030 3802 4988 18619 20341 12665 14 21512 11299 23512 7954 7277 6165 18802 17106 13804 1030 127 10662 15435 2299 1495 479 12577 17068 12045 9356 6822 4382 891 22357 19788 16171 21994 5314 16277 20380 8449 3478 5445 22613 14459 16421 6835 24628 12248 11689 47 9078 16850 16018 1282 1498 10315 12274 13430 2063 737 20301 2317 6800 19976 24909 8974 8933 18508 18887 8478 9858 2763 8958 13917 16510 8055 13112 12474 22076 20066 1405 19900 12849 12835 41 1860 8908 415 23608 20000 16472 15601 411 21960 14883 9263 2718 831 15176 24830 24023 19797 1569 5137 11618 22752 4632 21518 15481 20904 14722 20061 8530 3119 763 24937 12239 11318 7174 4983 354 20303 20331 10291 7581 10842 18685 6209 6370 20499 7371 22881 13913 7600 3589 5761 263 16494 14315 18303 7120 3406 271 19523 24072 15006 12198 13291 20082 15764 1047 8104 9973 23575 24862 99 9702 1550 12890 24796 10429 15482 3528 3247 2588 11312 14920 17431 6771 800 16153 12870 903 15888 20731 23288 7403 2305 12226 13940 18841 428 19563 19812 20213 6623 4404 12854 90 7409 14556 21548 21297 14069 2036 5313 5189 15722 19022 9196 3430 15155 23424 12018 15177 8381 8675 19966 21329 1426 16971 15355 15374 21294 1224 23454 15464 20201 7964 9868 18940 14214 21118 13650 739 21802 19169 10678 12580 2901 4027 8403 15817 20577 10278 11483 4276 9189 10503 14262 24183 12168 6232 2475 19366 19639 18072 17371 21913 10341 6487 21036 20634 10675 21490 7944 6936 822 14948 11101 13670 15605 2798 18653 11752 6180 5266 2247 20107 4683 16812 23137 21069 24174 17577 8735 22104 17653 6203 2827 7783 12425 11313 3922 6004 7825 11527 14643 7543 7051 282 23881 14149 10532 9484 7219 14218 21766 13713 13629 24859 6608 3286 24717 10381 20931 7648 5616 8756 14702 14566 9242 23258 6746 1484 6896 21247 23931 7682 18862 10083 7252 10925 97 4487 15549 10012 10498 12936 7064 464 19211 16580 12955 15359 4269 933 24523 22637 18504 16284 7770 9770 20574 22593 15443 16697 15977 2238 8779 19892 20079 11957 23421 2404 14953 12919 18023 7084 3714 1254 18091 21414 19373 3679 1197 4423 13311 10063 5475 17396 19719 21842 3778 22712 13095 16217 6045 23618 23580 18276 2350 11800 18301 13989 1125 2804 4704 15656 12769 9143 24731 10811 12300 13195 3244 11443 12569 24664 7243 4365 2424 18606 17437 5963 10979 15575 19821 5520 5405 15269 11500 21177 18326 9018 18370 16316 14863 15134 14805 3274 8365 6413 5084 15252 14083 14563 3556 14282 10247 7640 8776 21963 20421 5299 16303 10033 9850 1996 22877 20022 2759 827 24214 14622 21602 10312 7176 20427 13972 7947 16365 16073 21291 2894 18347 23704 14705 18594 24087 13039 17273 6098 12770 19909 16403 14665 8661 1482 5378 20768 14383 19418 11021 1136 21821 20137 15015 2699 2920 22211 14068 11063 2691 1464 3385 9149 10068 21546 20388 18132 15317 21185 20226 3313 19362 8994 5648 2952 19992 12475 10823 24198 24268 1631 8457 5462 8290 23971 23400 22829 21046 6669 3233 2879 14406 24197 3717 23151 10389 21347 12760 3122 13533 19606 13620 9181 2205 23172 7613 6574 3594 1731 225 12718 11164 5859 2311 16962 8976 9354 5024 19258 22921 13327 3022 2035 1133 22495 13374 24363 9374 17896 244 22746 20649 16844 15653 4180 12421 24393 5165 5572 19627 10478 17626 9014 9274 14635 17553 24217 7177 16684 19451 22528 14338 4298 10088 23134 1283 7374 19462 13122 11330 3495 23996 15076 7815 474 293 12476 23510 2802 24137 23745 19773 19248 10882 8080 4147 13012 14016 5081 7677 15237 15376 3036 19603 18526 11831 11986 3560 14803 21708 8377 8124 5460 10397 9875 8934 20586 11885 14824 22927 20746 22568 19548 16886 22454 5215 22029 3912 5986 9579 14586 11073 10032 21434 14330 9349 1118 14157 8641 1322 16136 16335 5247 5698 12148 11043 21907 19274 8689 13517 7164 8746 7657 16610 24131 13452 14133 9724 14618 19899 17286 21847 7520 15857 22608 12023 8229 19940 17126 12202 10600 5636 9332 23812 19598 14453 3033 3921 14590 12094 1806 1673 11861 7693 2370 1142 21951 2457 20893 18787 13396 1559 4779 15604 10095 16439 3009 12706 12833 3280 14343 20025 4350 7184 2167 211 18114 13457 15487 24275 22419 23287 20286 4486 7327 12156 19720 10954 10039 4827 7608 1842 21404 17147 11745 15859 21687 3522 8350 15165 11869 18858 9544 2714 863 13864 11976 10371 82 7256 2246 6605 16182 20745 7331 3441 2519 22275 15154 5547 3261 6496 18551 11035 10111 22693 7929 5204 21051 13259 7335 23836 14056 14697 9409 9256 14373 17044 19064 6149 23880 19825 2938 4614 13497 12829 4933 5831 10199 21190 11908 3316 23377 18121 22635 19458 8928 23128 18439 1257 3041 22287 22014 15748 17377 3123 4215 22195 17462 24755 6764 6243 7328 425 15797 20401 24377 3034 9572 699 21400 10000 14715 5496 3194 638 24366 10158 1597 3982 746 17386 20762 15749 8343 1271 3120 24759 24512 6100 5913 16583 13840 14217 8101 14269 17354 13965 24573 7571 6683 8685 11534 14410 10700 2282 7087 1693 20324 4549 4719 21525 17837 15159 2435 18054 16555 6381 2721 13215 16852 8663 9534 9620 22062 19472 24972 16824 13361 8904 7555 2265 9335 13548 18147 14759 2336 4395 20977 6058 13506 21701 12576 15498 14934 4792 3862 4319 16832 14289 23515 5662 21939 10288 17834 15401 24576 10657 3200 9382 10971 17451 716 20535 19063 2003 7319 4595 12104 10460 1950 1635 3765 17125 14554 18455 9428 9453 6117 1653 183 24338 17918 8287 22801 19925 14043 8315 4268 24577 17769 22687 10457 4887 5869 24507 7396 9731 1013 19464 17190 13662 6584 9273 2874 23197 12434 1545 7272 24089 13219 1944 17532 18912 21663 743 16772 11707 22761 6811 23055 13132 11124 1343 1521 24179 14706 23280 3347 9593 20857 20375 2669 3568 6202 11799 15618 23725 13738 5633 9312 18227 17045 6788 1347 24828 11192 24618 23642 1350 974 21609 21889 14795 8192 3989 8993 1329 23420 19145 6229 5979 14908 21094 7867 15742 24253 15646 6193 13765 24723 4697 19400 18691 15421 12417 22651 10129 7722 4658 11749 4101 20363 8240 3878 4628 19351 14729 20889 17244 4096 8380 19858 11407 19492 9844 19301 20688 14307 8798 8509 18458 15578 22177 22944 4072 1722 19888 21991 18622 24094 9821 3631 5870 5925 7005 10026 24983 17520 4578 5031 22053 21935 21072 22862 7771 8016 7784 10602 6183 23057 9024 6215 24608 13191 10195 6638 1861 18293 10227 12635 18045 3578 6642 4151 11793 20674 22656 13394 19015 18592 24224 5259 16499 15931 5318 20185 11296 2104 13996 12176 16661 19583 9137 6478 2363 12127 16060 8720 2828 1960 16124 14988 24469 13344 2940 4605 157 235 17359 21141 3289 2287 1632 12695 11202 15548 8152 15284 19869 21840 13565 4954 8321 1496 24788 16903 16768 8864 24012 12984 15258 12948 12442 1721 16871 9037 7223 5293 16769 13770 6783 12884 10041 3907 15815 21244 16972 22467 4710 4252 12317 6393 6030 111 21989 19620 24423 4454 24815 10889 18846 12642 9964 1444 1233 13998 16411 8858 4996 22591 19890 1915 15148 19526 13493 3602 3873 3577 21211 12677 9595 6017 23567 13985 8755 7486 2539 134 22368 19923 3042 7667 10667 22028 20306 15292 14098 7034 18654 14223 14375 24462 63 4648 13733 19549 7917 2508 6686 22704 14504 11704 6780 5918 7175 566 13685 12289 8476 3753 18295 19707 13995 5019 23893 16236 5121 9093 265 10551 15786 14467 18682 8825 9169 24767 10237 12918 1036 486 20675 15611 20285 12995 19893 1855 4762 3553 3307 20186 16695 3939 6889 5466 2600 11480 24823 23212 16611 23616 21007 12672 2234 7664 19908 20880 11208 8143 524 13849 18115 9757 2753 10959 4139 6700 12751 10272 10133 11473 1298 22575 12994 18721 15304 2416 8118 4495 17662 15225 5018 12943 3210 21959 18789 14035 7091 3812 6244 8234 654 19297 15681 24181 4178 22505 17930 21681 16794 2523 21107 22789 7446 3 18119 14440 2982 19964 14968 24986 5461 9746 16897 20313 1613 6971 15184 13546 5239 291 10345 14189 8753 2959 5872 212 12024 21864 10699 13907 2337 8873 23668 24832 10824 8103 7999 22134 14341 19209 2698 13316 24575 13798 5842 18763 14568 18086 15920 10002 9150 3196 22743 12779 2955 7811 844 14829 20430 15161 21921 4064 2557 22537 21272 8206 267 712 19901 9734 2758 1179 17004 11545 12095 19566 1703 5354 20677 849 333 11770 22092 16349 13326 3512 12811 19124 23566 7346 5162 12433 24405 8378 23247 21777 21625 16498 3359 4297 13064 13942 13635 24088 4653 5513 23276 15444 20157 16292 8772 8433 5787 17779 11497 15746 16761 11213 22542 13478 13038 8634 6622 14028 24997 11550 15414 6914 193 14222 18265 7863 5077 19413 12603 19602 17563 13009 20156 14243 8909 1492 1037 167 11533 1795 15068 10832 20797 2466 20261 15058 14208 11283 7415 4973 11184 21276 21014 11655 6011 17196 16882 14270 8518 11519 21114 5120 21087 7146 23376 16746 5693 7848 3603 3415 13684 23888 7116 11203 20088 18480 20099 4840 7707 1070 332 14788 13129 23633 10196 1733 1805 1137 8532 959 24360 12325 4946 8503 5346 5431 19290 20283 20136 7025 601 300 18704 17035 13166 14888 5305 7515 20991 1377 10783 18845 10169 6510 4700 15698 21803 7877 7641 1820 17581 24701 18152 22046 7762 19554 18257 17398 2585 9100 5003 470 20174 22388 17760 9997 7612 18421 16944 16428 23477 57 1494 6207 3020 17708 11434 8302 2048 4400 16469 22556 10989 22441 23116 19006 10893 4629 24288 10581 14837 12600 1038 4260 13974 10597 1773 14154 24919 20854 10365 9112 6859 1969 14078 11088 13229 6527 6236 389 15872 11293 9967 679 16402 14736 9106 4006 11813 6133 1189 15010 22969 10550 2628 14300 16660 11200 18304 1991 9318 6335 14814 13272 14592 13923 22830 11684 23291 15396 11000 20003 6032 7278 7123 489 24959 20255 18095 1274 24222 14266 898 19614 17440 12097 13427 20448 21054 1754 23067 15259 9776 8750 9448 11172 15027 12452 24902 21683 16765 14761 16047 210 13870 17026 16 9372 4784 14046 4343 619 22940 20128 11854 2051 6134 268 15440 10424 14469 7836 5472 4099 19437 17417 14339 2304 2057 21967 13820 23178 4712 4582 7803 15686 17659 19152 6251 8072 19477 16039 13170 15463 13757 18174 15502 12991 5258 3834 14407 21052 3282 18664 20454 6262 1226 832 24406 1846 7162 7618 850 17338 23748 17408 1809 3463 14031 16417 14771 3208 6938 4944 21514 23586 20751 10106 58 7898 9930 17263 13351 12701 16057 20279 23674 89 8207 5819 18911 19611 16063 1661 2359 10939 17506 2203 23265 22192 5768 2423 5106 23458 13333 18467 3540 7614 12410 11195 4189 16042 8968 19102 22403 24825 13602 8549 17658 12252 14980 21435 11926 3830 1441 18409 6943 16159 14403 10437 2566 8397 16184 19481 12563 8572 110 12888 7685 6880 1656 823 19180 16106 19585 9553 1235 3754 152 745 22274 15516 23335 17812 16915 22559 14176 17524 1516 3336 6014 21202 15851 2525 15213 3833 20588 14686 5700 7353 4999 19193 13487 12716 12388 7919 9320 16687 6171 8777 2980 1440 939 13975 10827 2954 199 16713 20664 11154 11033 3956 21561 6201 2219 11182 23599 22594 22416 21906 3061 582 13509 21201 71 6629 181 22776 12562 22331 83 3786 24721 9962 8745 3895 23111 10792 24923 14369 4791 15462 7134 2473 726 24364 21575 17372 4225 3240 18092 12260 8494 7773 22766 125 18377 24865 6785 8490 295 21316 14295 2885 2173 2716 2398 18402 19100 15690 20720 3696 20928 18996 12381 19520 4780 6904 11252 13937 2693 21449 14106 10818 19249 6980 5149 19394 13677 3851 15170 19764 6489 5664 23357 10964 13158 4558 1876 22324 19978 18068 2056 1938 8023 201 375 18889 12515 18348 4524 3904 5161 698 24239 14977 23218 3497 8210 11627 14523 12650 24488 11493 18971 4831 16158 11943 2811 19774 10949 4758 7416 23460 14924 4183 22995 15662 8127 2204 19534 13225 17415 12286 4864 1982 14283 22197 20681 9481 19505 11579 2358 2316 1090 11069 18764 15215 21754 3715 2640 2264 180 17293 12763 15066 10046 18759 9806 7093 23844 21375 19664 1522 9712 15235 18131 1958 12731 22918 15365 7247 8370 6989 4866 11565 10222 24950 14854 3479 23526 17086 12194 23373 5594 9283 24899 15101 12341 18346 12438 4499 9704 18011 12517 11218 5809 9719 8631 1765 17275 10385 7760 8806 20233 4237 21159 19970 16758 15594 7282 5811 1281 15382 2050 5148 6116 19983 17758 18706 9890 22373 18024 15711 10393 2053 7074 4652 14391 24215 18890 14541 11603 5484 2542 19618 17619 14320 9929 7075 21667 12699 22227 1277 5503 7719 21251 10500 23800 3215 23181 20559 15288 6101 17637 16140 12227 10322 9260 23517 22259 23041 21298 3097 9461 16259 10474 5356 2084 3468 19353 16282 2152 7221 4333 15370 13371 6548 1250 19772 10588 4031 12040 11220 11797 12494 13068 23332 21269 11136 6351 3087 4060 16834 19067 13721 13469 6060 19013 21194 16859 18850 9498 11588 4010 9569 2549 10463 15958 12143 18299 17410 5068 4705 5921 10876 11210 24154 6087 6968 8367 20270 22246 21437 6327 18466 17365 11478 7365 21811 22628 21810 17915 16670 12100 23401 22986 8736 1672 2366 328 15333 6611 9435 24603 5534 4016 18278 23104 15126 3203 2831 13020 15842 11183 21 5118 8630 19880 17210 10542 14227 3444 9760 691 24821 18985 18261 9902 7975 7866 16595 19848 24999 15075 6905 6362 12609 17252 3154 9982 5401 1697 721 24696 16270 3649 9524 9771 24685 3017 826 12711 20478 11543 18957 8331 5583 5242 19116 12863 13385 9064 4742 16271 13042 23692 23159 20458 17497 20190 23612 3186 18798 1535 11050 16258 6662 9399 7181 17061 23245 22206 11461 18839 19646 14735 6343 3642 1634 22016 24804 10363 7974 7547 7333 12336 12510 3537 21416 18572 12564 2486 9849 9847 6488 23630 10553 20144 4039 17807 17458 16475 8522 23813 16363 9830 22073 23198 20943 6253 6307 23484 17689 16963 11539 24351 22993 23099 5430 2115 23937 20217 19840 3820 7796 19221 10543 1678 11010 12956 19579 10647 7874 106 573 18161 21662 11941 20476 531 22937 16121 11402 9179 4832 6259 656 24974 21673 12523 1438 14962 10732 19800 21771 11301 3099 1108 9360 23271 21200 5679 3777 20524 23746 10538 6227 7736 11107 12887 16703 13951 12076 20984 22312 15867 4577 11742 22924 13671 10615 10270 5426 9881 7914 16317 12721 13443 15001 17645 15302 4849 12520 17442 19556 10902 10159 5764 3383 14630 20480 13692 5367 9540 22081 20256 7142 3014 7245 369 22751 9528 8887 914 19273 11234 18974 16648 16930 17639 2793 5073 4769 177 19777 14034 6801 3300 22614 14249 1648 1166 540 21793 14614 23010 19867 6723 23960 16784 14782 8184 7297 8301 1899 16502 12885 12459 98 9470 3393 19824 6689 22101 24820 13519 1797 7923 3435 22956 23096 12231 6205 5254 10470 5623 5152 10906 9974 3533 21012 12454 14422 5928 2309 8937 7337 21704 3676 20631 16949 13143 20332 20063 23720 17198 15733 15986 23259 1003 23286 24818 21943 21100 6821 8727 2690 4843 22421 16996 14003 7639 3809 21282 15956 18753 4890 19304 18474 8793 6006 594 20859 22127 10683 4830 12041 11365 23478 5264 9033 5585 11746 11494 1063 20978 19355 14022 3453 7548 22472 22035 14913 10193 3940 17825 19385 18320 2652 3764 19508 13778 7853 660 19135 10245 8039 10931 3874 5383 937 24669 21503 18205 9864 21809 23732 4871 1770 23760 16994 22508 12953 17118 24105 19828 3330 15061 21423 12308 20276 3962 13959 18495 108 24460 14479 12913 10808 1726 2518 21767 10994 5434 6221 15179 18907 9775 9736 22664 20550 24782 23659 9329 23194 19215 6713 20162 15339 20277 15959 10708 17811 16390 20075 73 2418 1763 5956 18532 11379 17466 24644 16705 18712 3847 17096 16620 23035 8795 6673 4715 16912 11436 10053 20539 1246 7698 7513 23809 17960 1362 19187 11117 6560 8686 15483 20616 15745 1746 9910 2864 520 14903 10775 17772 17364 5524 9823 1607 626 12990 10787 24203 14576 11429 16051 5302 7802 20078 18070 19859 7213 13639 14719 80 8579 5873 5536 3397 14520 23363 23587 6929 23064 1008 19702 15425 9793 9885 4767 22590 13460 4127 6999 6187 20607 8788 6852 20819 17430 18342 10960 8632 7038 7143 12550 12034 18223 3372 1307 12787 15436 13562 5493 414 22634 13131 14258 21523 15809 16309 1471 2614 14323 10343 3412 3807 1229 24781 24766 14860 3509 7480 16550 10598 23075 21574 21099 2210 15994 14278 22109 6942 3793 4729 16779 17745 12149 5036 22096 15884 16924 8670 23986 11502 1365 4796 19838 18973 24226 1951 1485 422 22241 15747 10054 7429 2862 8024 11085 9958 6342 9407 10901 6485 21136 24882 10685 5885 1087 14074 21558 9147 3592 4427 774 22256 17635 15623 14461 7817 1064 18596 10743 4563 5857 481 15954 13484 1681 15324 15265 3841 2421 885 10629 7378 18314 15325 20832 22355 1934 5751 19727 16564 23681 18298 5542 4033 4687 14051 23614 22400 880 11438 8742 22469 22418 9316 1404 24369 20940 21539 10123 21000 16838 19917 4357 5012 1617 9590 23180 14869 1857 17498 23282 23719 5045 5425 22496 24470 2362 24277 15620 23829 2541 21606 20358 11178 20469 9705 6828 5176 15721 1127 1652 23625 15372 12173 10593 4935 6699 22313 14600 11242 10988 18734 20289 1870 5719 19014 16223 11285 7827 16900 24895 12593 18558 16408 5697 7728 13340 19731 18413 9235 8932 16891 15017 3416 15823 12992 5624 2580 8673 4301 12165 13682 4090 3333 261 13298 21808 8424 2011 778 19306 22715 15190 5251 3732 9198 15191 13431 11953 11453 59 5010 14785 16496 24814 15728 4529 16973 13679 14229 6219 2522 10305 15528 13356 9741 3500 7071 10231 17677 24273 13357 6357 1552 932 18673 18188 11346 10556 6068 3555 11450 20697 24692 8454 5588 24006 17661 15688 21467 4192 3237 5840 3114 6152 862 14441 5699 14660 12981 11222 17448 11094 10289 6599 24982 15906 21740 5196 4372 15301 11769 18208 7620 23430 24474 14505 12077 17538 3484 5609 21819 20119 8129 8021 11778 18579 15450 7617 24633 24797 3389 2730 3340 3459 18589 23858 19448 4649 6324 14447 22644 10313 15781 22019 5424 1048 674 21893 13609 14511 11413 5592 8385 6637 23523 15865 14353 12464 11804 1882 2420 2331 12782 18137 10680 7261 6074 24768 19047 18285 2469 2570 2209 22471 13821 13581 24869 23297 4373 6078 17245 18167 17171 9062 16150 12262 11853 11194 13343 23285 12407 4097 6966 10347 18649 20951 1276 22553 22154 6441 4026 809 370 20772 12204 15879 3319 17343 18772 17902 2004 20053 12303 12054 24080 16714 3100 2292 5621 17891 14827 22630 13223 20813 16554 6804 3866 9567 15477 18624 4285 17001 11369 18195 16817 2445 4645 2683 3201 14609 22061 21882 22772 14815 17474 14493 3366 3554 22214 18071 19441 15033 5408 13155 12826 11206 14070 3906 21839 12337 23081 13782 11305 10836 15973 9545 22447 16396 22972 5482 3934 3600 16211 11396 16527 5290 5933 377 16699 21899 13121 3338 8826 19924 18986 17831 16008 7795 7532 21795 10616 7906 922 22502 18357 23876 17301 14204 16007 4168 16874 10926 22243 8832 12911 23693 12653 10149 1920 13445 24540 596 19083 16480 23284 9601 3958 24659 6334 8996 5058 5017 20890 19068 14680 7607 5415 18818 23979 16913 16686 23531 6408 24772 21522 3871 1110 22393 18503 8515 23205 22806 16024 986 24190 21169 10186 17270 1725 22923 20487 24319 16046 8093 6901 11599 19795 9490 6845 19104 7805 8314 16219 14672 434 22898 17923 11964 24325 17159 5075 8911 20320 17681 15232 13210 9580 9629 23489 16477 4194 2924 20369 17892 23819 3613 15807 21229 22450 12096 7294 4221 615 24910 13367 17047 1242 21361 19244 21220 10058 7432 23805 10915 18724 2748 8423 1609 996 156 15744 18025 8691 15387 13194 17710 22762 2290 6860 20644 16685 10214 15530 22063 20275 17228 23079 19143 3923 9829 7880 305 19744 23479 20431 17145 5033 3225 13910 5689 5307 149 12462 22175 12003 13281 7231 1789 1366 16399 19503 24725 1454 2255 11439 12002 21368 20625 14322 11018 1514 23920 21648 20840 2237 22449 13499 6111 359 21488 20996 17276 3967 21745 21174 11097 12573 5297 12728 24177 7322 8317 20852 23238 15271 5586 14108 17238 6166 101 23049 13597 13858 8526 3523 14390 11708 24193 14487 886 13861 22284 18693 1822 8309 4951 18111 21386 13608 14961 4472 2834 709 21748 10271 3037 6260 957 23027 18104 16064 23061 19375 22841 9101 6684 1106 22428 19781 24719 5272 19408 12124 11529 4952 4102 3747 10040 19722 5734 3448 7154 2978 16113 15730 19277 11723 8427 19212 24071 17100 16979 19791 17531 5982 1685 19235 13317 4424 9729 9482 16221 24675 20020 6885 22477 9781 18646 15999 11884 4910 176 19969 22763 15812 1926 8600 12739 15385 19862 1079 9661 5145 1175 11477 23328 8278 9979 4900 4678 13159 17268 18603 12480 8345 1345 17216 10082 1001 14365 1859 1294 9158 21372 17330 17601 5661 18827 2816 15557 24025 21489 6346 3257 2534 23988 23961 12709 5238 4789 5806 19949 13382 9429 9843 4740 23147 21596 8629 894 2530 18360 17112 9414 6887 3817 22067 14267 18 2766 22252 21227 13618 23695 13490 10047 2027 4036 20394 23040 8141 7960 13045 19662 8082 10655 11082 7941 1833 17460 13886 11862 12102 13962 20043 6612 8105 21677 18141 6869 3587 10883 23955 13786 10491 2741 12332 24335 23002 11911 3571 20935 10302 24170 9072 5672 3670 13993 17838 3165 13094 14894 19796 6121 23700 13320 8438 17192 19069 22574 15849 1971 4414 18635 11012 14234 8392 2397 5914 20257 17181 16810 9879 19592 10595 12818 14755 7481 976 16835 14882 13163 2779 19649 8973 23195 19870 18737 15870 4503 4458 884 24149 10917 22587 21630 19850 2544 8583 5639 24301 23256 24773 1403 15342 2124 2155 20038 13154 14192 18463 8702 9725 553 15769 13797 11677 5629 700 19500 20633 6803 17867 13698 7010 8812 17518 15533 22974 8421 24140 22404 12191 23506 7094 1868 18441 11357 10898 8304 7666 2399 3893 23003 18525 3362 15432 13048 20009 20516 5090 8481 13137 15211 16344 4375 17703 17065 8114 5758 8288 12053 7218 4238 9762 3932 16137 19982 21955 15964 7251 15507 12033 6385 3694 1069 19434 14843 20901 8550 6657 3250 15428 10812 7096 4583 19721 10778 3776 4011 8180 20950 1093 3096 554 15939 21884 22144 3211 10415 14443 12548 7023 5530 19574 20613 8170 2259 5582 18865 19197 15049 1900 9363 4735 8961 17097 11699 3599 7220 9533 22729 17765 14264 18883 4984 5879 12031 13605 14371 9096 18795 10682 5999 2092 17733 18509 15731 2788 8662 20622 17978 14636 12283 8263 1020 18741 24317 12396 2790 1651 6770 22361 14872 21635 13220 21335 16731 2052 9766 3308 383 22767 22196 2511 5020 2437 12059 16203 22595 21665 17699 11850 2191 20337 17763 21613 2547 17904 18244 13197 2989 4610 7156 17754 12383 10028 6947 4924 20645 23065 7574 3426 19415 24387 8562 491 14902 22119 6829 16035 10101 18807 2937 2377 10297 11045 20376 11819 10908 2970 17718 24340 19637 9141 2117 1565 10693 20815 20 22057 24477 21330 8095 7858 23093 20356 24108 22384 1007 8444 4819 1209 188 18690 14131 2099 9092 5865 18540 24547 24500 23877 3859 12181 3829 22223 16450 23472 5021 6463 5780 13570 23192 6758 6567 7339 20547 15928 13063 16559 22744 1557 23912 5950 628 24178 21268 15911 15122 8011 22802 12813 9113 4868 8624 22463 13819 18150 23780 19445 10318 12159 3559 3586 22056 13211 18410 2535 10064 15475 20057 5568 2148 22996 5129 10990 10769 13147 6188 5349 15541 6443 6557 13638 13530 13069 6424 22180 16524 18354 18805 5894 1793 9032 2195 3018 1024 12643 12851 4163 6797 18423 12377 18300 5680 8144 17934 20601 9477 502 17586 8607 21381 16557 11020 10878 4686 1327 19185 10831 24874 12516 11628 1734 21716 14470 20413 17876 1875 5365 1709 19596 10420 5994 5139 17676 23343 16581 11581 825 14708 4202 12306 24894 24827 12450 12335 2561 22438 20882 13328 18412 7901 23457 22027 62 1303 22815 12784 7484 4287 21975 13952 8277 13749 18444 23875 2823 23127 11743 7463 22244 17575 9786 12872 14013 10987 1701 4304 3395 24690 12477 9351 8060 23588 16608 15885 8774 7913 3378 18089 19684 24251 6635 22572 24316 12949 6326 8569 2884 14147 15679 9586 222 16538 24022 14855 15606 9990 9531 5327 14489 2062 494 24960 24888 24279 3161 10825 11657 10287 5553 8527 8142 4327 11387 20799 7041 2641 9317 16628 2704 13288 15393 21797 13061 10781 7090 8066 4306 14186 18371 16256 4764 23790 20374 13807 11320 2298 1293 20475 23353 14095 11583 16430 21182 8017 4847 2049 13471 12153 6450 13626 7163 3007 14152 13936 23358 4392 5783 22596 4177 23884 20111 14386 9361 15864 20077 9164 6358 144 19818 12232 6596 2087 605 23547 12065 15570 15459 9521 22285 14426 22258 5391 9691 3404 17283 14864 15335 4876 7228 15985 15759 22411 4798 455 19271 10194 9769 9047 1562 1161 9694 5749 3931 13813 23268 6514 21641 5132 3622 2428 15496 9401 7868 8955 12925 21030 10802 12052 1455 567 16752 4345 3431 12723 21452 17755 10208 2948 5414 15525 13301 1590 13558 14224 21637 5836 6553 17333 13279 23485 3746 10973 24979 24306 18251 3913 6407 8611 23842 23691 14607 653 19635 16710 18979 794 22413 16490 10642 18522 21516 3757 2705 17450 4580 5185 109 19898 18980 16381 10993 17535 6257 6131 19291 17527 17471 2123 9127 21189 23227 15992 6191 3019 165 15540 22856 14661 9079 6934 2281 647 18998 24153 20112 1076 15912 23939 7287 7958 6033 17033 15588 11223 16960 4349 21021 19501 24403 13919 8431 8450 22666 23153 15103 7445 2872 14866 14409 12212 1215 1570 307 13621 11322 23305 2112 2338 1691 625 18670 22425 14798 16645 18905 5436 1107 6411 13779 20656 3310 16988 17951 3394 8334 19648 5544 6586 6263 24698 16449 19479 3473 7045 3460 13501 15531 22649 17829 12068 17048 9050 22903 11587 9783 220 12549 14207 9753 23680 22733 23229 18781 1644 22005 16111 14526 9086 8696 21542 11039 5847 3798 3361 24311 2225 16728 21038 21755 4283 7323 7497 24241 19347 18567 10210 4600 2240 13688 11083 23782 17911 12141 9232 7637 20319 22523 17617 21139 4482 8862 6097 19552 16392 13801 43 686 15826 11424 10010 10004 2974 8400 279 20115 6831 5245 17920 12931 17028 4676 21597 19845 13601 9527 3860 23321 17698 17910 9343 7068 7237 6718 19294 23807 4906 4557 5117 17792 23570 11032 15375 17239 2018 8965 5630 14992 13141 21228 11879 2510 4452 3283 3296 19920 14549 21651 2177 24318 14647 49 7394 3708 9107 24322 23781 20492 4994 10592 9688 9692 11666 69 1141 1946 16651 16065 15089 14427 8567 22301 14773 9258 18695 15094 18868 9359 17817 23260 7390 4201 16289 20646 16003 21240 21527 22690 9216 7456 8109 337 13276 18895 945 16907 19018 5828 15584 1620 7475 17871 24323 11937 18751 8811 19038 17072 16072 7296 2910 24688 12589 4396 7720 4673 20777 14288 12206 13578 24141 20116 1173 19689 15562 5504 18754 24571 9156 8414 24870 10084 20460 2662 168 618 17251 9187 3905 16329 10895 2028 4014 21197 24588 8254 4544 317 21660 24503 10845 2371 2701 3610 19605 17775 5210 9267 3703 23860 24951 2477 11122 18914 18703 9730 3975 15130 19575 15652 10359 22221 23359 14348 2450 6749 23009 17114 23765 21931 8709 8239 9124 11844 19591 14677 22091 6725 439 23310 20120 18043 13188 2293 20836 22548 20932 5198 5339 5525 443 16075 5708 4295 8612 15438 3063 3434 587 16863 11001 17468 14188 11643 22778 22580 10815 11145 22011 15011 18697 22928 16167 18665 18660 22385 24263 12447 7725 2907 8078 6458 7354 17370 10166 21958 14145 9961 24808 16195 14914 14169 20734 6588 859 17406 23852 9887 12449 12961 21135 23423 19089 18312 17062 17109 5226 21583 13839 15230 21610 3390 3373 4493 23699 17549 19045 4624 12556 13262 10130 20206 6115 5344 3766 6293 21049 9129 8590 19261 17844 15173 11476 20791 21422 14073 9446 3321 1459 22455 19125 4004 21731 17908 20929 5570 4970 9440 17503 17477 22371 2632 2430 20130 11444 10506 22854 23839 15388 20472 17076 4674 3185 22004 21250 2538 9371 2275 17056 14937 12558 5799 7464 8528 20621 16607 14877 18221 3738 24262 16813 23916 8443 7267 23292 22060 17865 7324 4009 16797 5818 24586 10485 22345 7807 17323 17990 10105 15597 4813 5271 21831 20228 10284 4242 6086 10871 11626 18793 22299 1194 7114 21650 19683 15650 8371 4270 9665 24343 14298 12756 4929 7765 15571 13502 5359 1066 9279 379 19225 4378 23733 19905 16536 2126 1749 6456 18533 24468 8386 8833 23082 13235 8226 2972 416 21163 13019 8058 16478 3209 759 12883 12207 6082 21576 14492 18934 12296 16539 24929 5052 23374 10368 13001 17429 10945 3656 24627 23871 10455 7826 1236 8083 4627 10856 13428 17798 9752 6963 7591 16954 13759 1000 3515 21639 22129 13842 15029 23921 17771 23464 21879 9186 6284 4247 190 21829 22662 17383 17187 682 23253 24498 13706 6373 612 13461 20477 3831 2571 2172 5953 17966 14092 12946 12121 4783 10819 10611 11503 14905 24208 20383 4138 5888 213 15694 5827 6392 3452 449 13500 20234 22381 1890 10522 21253 21382 5527 3867 552 18953 11277 17629 12552 12598 22636 12772 1658 6591 19449 14259 12329 14858 15072 22250 14361 13495 7908 6579 7459 3552 10913 19333 6204 717 12980 18937 6084 955 11409 1711 12700 13348 17651 9358 22145 19111 22929 8337 6995 20438 17254 19245 16092 22867 21484 1563 401 15167 6333 7625 3021 22812 13689 11197 8534 21203 23140 20618 4920 22030 12781 5202 15340 17801 2322 1826 20252 10080 16787 7885 14725 18200 14589 4123 7879 20628 14641 17954 430 22683 22008 19227 14126 2023 607 18498 12529 3345 20397 12067 11619 5043 20910 19834 22479 8614 492 18885 11918 7030 4997 20453 24670 16330 8969 9759 8360 1889 10559 2812 17487 4909 8846 3026 17842 20709 16020 24496 5105 8052 6269 24631 12676 15738 10473 21713 8319 5182 12263 11216 6849 2448 8971 2312 24984 19974 14333 6925 2677 18234 23735 13829 20829 18442 18358 10362 8320 3892 4985 10875 10776 21993 13107 22380 17350 20110 6220 17348 22123 15088 14500 4140 7065 2216 20251 16804 14717 5095 9004 8769 16373 16228 5515 8179 10348 14121 12570 16285 22433 15971 7834 138 11386 19690 12190 1659 255 24212 21199 6892 4271 2932 22705 14332 5186 11658 14346 10922 11645 1669 9151 18717 12448 1804 10887 10468 23167 20479 16641 2068 13140 17067 18678 10086 9703 1278 17948 13310 12375 21195 4325 11727 18425 8088 10183 7036 7886 20779 15368 17260 19179 18566 19226 12246 9054 5916 23408 22391 20623 10224 9456 2272 20466 19486 10142 21487 3811 7953 3408 21172 17996 19203 15916 13814 1094 4671 310 112 21858 11629 10476 6112 7632 14544 24287 12803 3930 6163 1965 23801 22724 22010 5409 9108 5428 13329 2074 9897 21120 23777 23645 3050 22675 19283 22198 8300 6261 5104 21225 15356 13124 3538 18168 13941 11678 5140 3933 20828 20385 8999 6652 588 155 21180 18219 13097 12824 9970 24886 15580 236 21354 10483 11625 17952 1172 1206 22827 24663 19219 11351 3015 2479 21290 14707 18935 1724 19990 24939 24745 14612 9223 4291 4122 907 13532 6921 7528 10016 11388 18904 6922 2029 9373 24805 8744 5876 3066 1953 21438 15637 17597 10858 24526 22696 24449 10694 9280 1760 23148 23168 15603 17387 6235 5897 11580 10976 15576 13370 11667 22549 4284 14906 13577 13098 8690 4420 11350 8308 4989 11598 15286 14483 7986 5993 806 19887 18489 23087 13442 11189 16827 2618 14807 17965 1553 3815 23636 15399 8897 3048 2676 13878 15938 8907 12281 18408 11261 16636 46 3162 19117 14830 9411 3492 21256 20982 20953 16077 4728 9726 12821 10038 3315 21257 18658 12028 10102 3179 15917 21376 13050 12657 17998 17516 15853 10089 18305 314 782 24645 23928 10360 23646 21813 5770 19789 16584 14281 8824 7385 2850 11374 10890 15210 8221 3032 9154 23341 18306 4299 3089 21666 10419 9459 6329 23050 16662 18331 21459 15839 6218 4353 4245 363 23938 15026 15569 20525 12465 3687 574 14292 12233 10731 24135 22317 21254 5851 21179 12561 24314 24128 9535 7266 7224 23204 16983 15354 22539 21794 6085 3241 18736 10283 3076 4408 24328 6680 3731 3160 12277 22792 22234 10268 13892 5007 5006 17752 3388 4602 17486 4807 15381 14428 24383 22308 7443 5716 367 20457 16122 11809 9172 7358 22465 20481 16815 6304 20150 22525 12116 5912 8603 18162 14402 19792 17409 7500 403 13071 16961 18585 13230 20801 1995 24596 14514 22491 13568 19099 4721 2568 11150 3994 4137 24840 13748 21654 2960 2922 20807 23433 15422 15023 4059 1383 3374 21405 1302 4690 19699 20966 23706 12792 5316 13085 1719 12084 12435 6460 23845 18026 13758 8633 7756 10219 19237 11244 12856 10487 9494 141 22839 20513 12842 9840 4622 9462 6562 21082 18158 15334 9790 786 17472 22961 16050 19332 16006 18605 11121 52 8574 7212 6911 15003 14617 11912 21908 2782 936 23072 1435 6926 5066 17514 14166 23383 9450 5934 20739 17573 19352 2080 21378 19842 8563 6741 14573 21137 11830 16777 8758 147 20040 24497 19770 8849 2308 4120 24542 19234 15067 847 14139 22561 18760 10431 12483 22340 22748 19450 12259 7413 16114 2923 15119 17069 6135 8086 15397 16170 17685 12618 1478 5464 3791 20085 16306 16029 14429 20585 15825 14506 1657 9053 18842 24337 2603 2936 21904 16262 23278 7259 6426 24295 7108 13875 13075 15976 16708 12993 13496 17317 13337 508 16938 12243 23105 1574 22536 22334 21656 8034 3643 1706 16864 20820 18361 12755 4035 3297 6067 20523 14999 4651 8637 6466 16263 21023 15426 16603 15586 16802 5491 20662 11389 22916 9772 14238 1129 4991 23107 15703 6610 18644 22064 20640 10610 21234 6420 4955 9040 21964 15596 15174 17672 18625 7316 2942 622 16201 17309 4548 3493 814 13834 11333 11175 14451 6817 5763 9941 678 13411 24545 24157 2351 9118 20184 15046 192 16397 10564 18262 17346 8780 5896 1584 1306 19542 10692 7990 14024 12789 1284 6410 2187 15998 21786 7568 7644 5230 21068 13099 23789 15327 13731 11982 14237 1148 15684 24584 20755 4160 5838 21643 24434 10279 17545 11670 12525 6879 5123 15128 15832 9803 24187 13433 6735 8513 404 551 14225 6891 2537 18896 12659 16325 4967 21543 14695 18037 8204 1687 7168 20194 11757 7080 24715 19409 12554 17687 14603 21377 13199 2756 1169 2739 3968 12581 18661 22431 13156 2263 16634 4966 7752 23480 18093 15338 22497 17627 4803 23824 18834 18825 3519 19631 15944 3780 6602 3947 19073 17288 24673 12093 4129 2636 7872 953 22461 14246 23783 21653 1751 23369 23326 11070 14959 6837 2562 12858 11846 8900 8671 12614 19855 11984 4421 5133 21785 17641 20068 22779 6944 8241 8155 20155 18483 16248 12199 9822 8917 21824 21160 20445 24085 11795 22466 2239 24580 24123 13623 13566 7818 250 22859 20042 1222 4773 9063 980 15982 12620 10182 6681 1045 19175 12932 16646 24647 4520 9333 17488 6161 8479 6945 672 11761 17191 11881 9608 9264 436 18494 21782 1507 8919 3344 18641 16320 4012 2795 15600 21535 23743 11391 4176 6570 5574 18636 10774 14265 4513 528 11053 11168 24289 21586 13185 22475 6158 13774 24092 19341 6702 18066 21246 15654 7562 7263 13292 11932 14213 11638 4716 17093 19615 12694 6154 24705 21515 4511 1372 21984 21876 16818 24042 2630 2706 7300 22303 14613 3894 7873 20690 19823 18716 16308 2395 8700 2639 11031 12542 19066 6104 5360 447 22435 17625 18291 7635 7822 23647 14047 6096 8598 19383 24063 23708 14658 10727 11868 7441 7767 8352 21978 21828 16340 12832 8561 9785 23150 15705 24124 5701 2098 18613 17453 15780 2449 5056 597 23345 16041 10413 2658 24525 20518 9247 2170 14468 19945 10510 6738 5156 13630 22474 6349 23970 22774 6428 740 19997 7924 837 17983 17423 15550 11340 1052 5455 3348 22526 23721 15494 829 18784 17568 21491 9325 2010 11193 7454 9750 13065 24903 22089 7704 22626 24640 13046 12708 12497 1699 5282 13576 20845 18722 23865 9103 6755 1704 24027 10997 9948 9637 7082 24521 15529 22233 6719 19101 23179 1447 8163 12527 19539 23108 1756 19184 13058 21604 6153 3101 22615 18743 12109 8666 9532 9600 534 24893 14742 11611 3006 21492 12482 8830 18936 19642 2387 6072 5509 1394 24753 20808 24519 23073 7915 4562 2497 23442 13617 10392 9084 2755 9426 24129 19712 8049 2318 15111 18930 18726 3723 5388 7330 562 24345 21387 11947 23724 15465 23336 14674 8002 7912 17297 13413 20224 11139 1461 21741 22980 9314 4878 23519 13101 10304 9680 10766 16101 18363 10120 13355 16878 16386 3890 10643 7712 5858 11639 24065 15655 12866 1908 7238 3749 23333 21134 23520 7611 14349 15524 8514 16700 13580 14438 5209 1988 24817 13249 12014 2797 6731 19361 17160 23308 3794 3902 14294 22248 11432 24678 22342 23554 18871 2189 10260 10636 18972 9754 5936 13924 13336 15622 12837 500 23551 18055 1371 9396 521 22695 14097 23313 15078 13359 8224 3049 751 24741 23959 6285 11253 11700 17608 7552 4181 20271 18209 17805 8483 2950 24375 11405 4555 4821 6430 18984 13387 5834 11174 19198 17821 9693 4112 13117 15411 17436 9424 14513 11087 22094 23602 6577 1830 22286 20081 19302 14582 13476 4573 22492 20853 18309 4502 8085 1156 10049 15181 18926 5931 24147 16086 10916 23897 3181 23793 21321 19827 8106 8037 16416 7417 13123 23327 5792 5421 8493 23835 23728 5200 6230 5773 3174 12950 21497 15802 6480 4346 23553 14201 20089 19387 5400 21956 14972 16305 13592 5590 2012 7004 24282 20661 8984 3352 7727 18747 24078 8715 11196 12114 16481 12039 7293 8545 1931 18744 15109 12376 15642 7735 5538 21206 19786 6497 18627 13236 24849 16929 13799 19407 2919 1199 8456 22935 15047 12345 14480 21972 7147 21644 21353 23471 6099 16189 19989 9102 19396 17955 17422 5308 8511 4171 19735 21957 8946 5844 697 294 12571 11676 15044 16532 4925 1796 23791 21055 11589 24834 6323 24505 20817 17489 28 12112 16825 9991 7372 312 19416 15112 11526 2820 21340 24195 13134 24598 81 9087 4321 6439 373 23562 18618 6597 2809 7098 22987 22975 24209 24031 4175 18848 21396 8425 9285 5229 24120 24118 17137 6820 15234 11573 17567 8791 8261 23541 17709 8712 15260 22663 9788 8497 24163 11304 784 13512 15364 20546 8501 24332 14000 19745 1450 21104 16249 23837 3739 2846 602 23135 19593 16955 4829 9357 24176 22184 4182 779 13362 22055 15272 6665 7284 14653 24799 10717 9605 388 22831 21987 12967 11324 1445 5169 4241 24225 18284 11002 14488 2034 22047 9543 4574 12391 14038 22977 22722 12899 687 811 15754 10479 17550 1580 15983 16164 9520 4077 10627 15208 21618 6757 22007 20544 15277 18395 10719 3823 21310 22059 9088 7561 7382 1255 280 3077 20877 21370 7126 3059 3601 15125 23924 4802 17003 22834 4449 13084 6747 21693 5322 3751 24948 4402 6310 1490 15499 21399 13335 302 11888 10935 9310 21385 17059 24424 9226 12764 10782 20505 11610 8025 2844 10689 15367 10029 4117 23103 11426 21946 24544 9815 18230 12831 10051 14088 6981 5666 21162 11979 8557 9710 833 18292 16451 2044 3525 19356 15519 9497 16147 15145 23320 15361 18549 4390 24654 14130 1027 4434 6226 22289 18813 23900 2355 15568 22484 4566 441 805 12924 18032 22228 2857 6464 15858 17640 6507 11005 16981 24172 8271 4708 6348 7730 12155 12019 23354 23110 8729 21397 19868 15784 11255 4539 2934 19746 12144 22873 4720 9658 2964 20432 18519 1814 2347 13920 17444 15892 5750 3422 129 18081 17014 16877 9895 22370 18333 17986 24771 14789 5733 7057 22919 14816 5961 3273 16500 11753 17694 20653 6728 5604 1851 22068 9240 14640 12532 1728 158 17499 12836 9471 6002 16692 16056 15439 215 21366 11748 24770 13953 24093 23628 23078 1974 7357 2323 16569 15539 24980 2573 7201 17329 15024 22833 9701 7691 18511 10797 17490 3664 21865 17587 13642 3735 7988 5222 10296 20322 17413 8796 21600 20847 16885 19077 17092 6520 16025 10861 10471 19968 3039 5046 9105 2897 1430 19861 6874 3647 879 18982 22678 9568 8964 8053 22609 21896 9295 4531 7747 16718 17947 16974 16269 9341 9302 8935 12644 12972 11106 7122 2137 1871 12326 10575 17624 1396 22042 3287 1154 22963 15585 4369 3712 18127 10257 15930 9509 6199 14190 11772 16805 1163 164 11760 11559 24186 3278 4519 5695 12401 23114 18385 64 3588 3402 16588 20968 7717 420 22041 11057 18701 5175 21815 23617 22456 8778 3864 3701 20008 17376 14718 13091 4747 7366 2702 22443 24336 14895 12026 17368 22151 2368 5055 21833 20887 7564 6228 548 15666 10751 87 3950 8872 14421 17227 13208 19113 8054 16585 23226 3199 9799 12848 7689 7845 20418 11671 12395 19252 17116 1524 5333 24084 14984 14435 5141 9017 7987 2956 2646 14801 10790 11714 21995 10135 19470 9348 315 21869 13632 11348 16242 4883 17041 14550 19431 7774 5955 9215 24258 18302 16100 22674 8458 20426 19090 17695 10209 8592 4926 24201 16552 20051 24254 15970 18241 5205 21280 17272 13404 4693 13092 11956 3678 3955 2855 171 5044 4518 4869 269 620 20064 24411 11890 17322 9797 1683 23352 20086 19284 8335 9327 13375 1744 15707 20199 2854 463 24538 20702 22444 7119 5786 23711 22910 8121 496 21629 13741 12399 2422 2761 19057 15196 14693 13053 16133 2601 5243 14385 18819 18201 23869 4706 16582 12555 1561 1420 1715 13246 16606 7982 200 14379 19814 19251 5131 3759 12855 10545 10677 21725 5154 6177 7246 18597 14871 22434 8156 7471 23123 23703 23995 18844 12253 1594 13057 21655 16959 22571 4018 2118 961 15723 13584 13598 4187 9051 4152 741 306 21965 24657 15896 7993 5694 5601 20121 22756 22348 7420 7697 24891 20663 11270 23426 8645 10060 12004 2711 7121 22346 10646 22948 6330 3948 4870 22058 18198 23550 5187 7059 9906 6542 683 20787 23413 19421 7437 13224 10442 23958 3872 2302 998 20798 22894 15141 10031 3213 6506 17921 18080 14072 7809 150 23623 10202 6992 4509 5416 928 10216 21312 8422 6897 19629 14142 8175 19097 16503 24962 9679 1192 2838 943 19885 5517 4246 6541 12440 21924 17799 24020 15254 14522 10203 3475 14936 14442 19504 19538 5696 8886 8597 16551 12985 12645 24165 8638 12690 24234 19275 16560 12963 23337 4030 22015 18546 14090 5511 12437 20483 12196 6198 20123 6703 15663 24826 187 21398 17463 22842 5023 3677 7878 24326 15753 7492 5552 5450 4980 11903 20299 6173 7665 23325 14119 18061 4797 4224 1354 17289 17404 14536 2991 3681 3356 19182 19138 20340 15390 11945 3888 22353 23516 6708 4885 18863 22868 18479 18129 9330 15405 21950 12139 17663 3427 11237 21726 24952 18544 3455 9028 8467 843 23367 10333 5769 24626 15156 16016 4317 7029 3752 15445 13449 19785 23795 10570 4504 3418 15418 20532 4115 3855 24922 16889 15942 231 14452 12413 18373 14151 1104 9980 20610 20151 13489 4514 8361 22765 14080 15228 1130 789 6056 20314 22657 21045 9030 24652 19214 7526 1131 23071 10499 17156 7935 14449 10666 10454 12285 6931 23899 13119 15979 19530 10445 19307 5454 1009 21154 17931 16795 10251 5263 8134 331 12938 9518 3626 3346 1120 22298 17211 4114 6047 22039 21986 23233 10103 493 19659 11215 6711 4547 3685 21071 12514 16441 11621 8258 16793 15952 4243 17961 15447 14952 1391 2365 22229 19609 17325 15069 5498 4063 3079 21441 11238 9813 5942 9211 24811 15133 15178 15331 2679 21531 14631 19507 4156 3882 6041 7391 10723 21446 14396 23062 19044 11448 19148 16104 24535 18587 22351 7606 3230 11744 21184 16518 13395 2951 2232 7211 12663 11466 24450 24052 17938 24707 8057 5652 13239 2227 130 24119 12544 1311 22116 10517 1645 8151 12810 12634 15792 14747 24242 1940 2620 2313 24819 17513 16081 6940 6029 4936 617 19196 20292 12372 8375 292 19264 13293 16367 6516 6979 5529 23929 18325 21348 6162 5531 5170 19747 11352 20485 5875 7508 4565 10467 8710 1637 421 18170 22173 17742 15256 2602 13960 24091 24267 20602 10494 4527 6609 23339 19313 16129 20942 9383 665 17887 14839 23410 23115 11282 17138 21080 2182 7896 6319 22338 6956 16563 22686 13768 19316 24582 17794 3771 9417 16233 14111 11078 19287 17024 7997 6868 22569 21841 9676 15193 18473 22282 8556 14579 10701 24837 6434 18218 13221 23214 18282 3584 3195 5184 1178 873 12881 21110 2046 9873 22950 22000 22876 4347 8230 13072 23450 24376 7465 2060 22567 21217 16157 17593 10332 8113 8837 16516 13606 11806 1867 4899 408 964 12288 24976 10952 19830 23977 22414 4002 5887 1280 15645 22460 10879 22617 6836 4922 10344 9951 21308 8040 2409 239 15668 23468 14731 246 20134 9855 5995 23886 16799 16678 21102 14557 798 22573 16760 23351 14158 11336 10521 6933 4672 24493 22992 17525 23665 14049 21941 24331 15810 5281 18742 20227 1016 16314 20919 17040 16819 23826 24906 8369 8415 909 641 18216 11803 23362 8990 21140 17716 21887 15138 10486 1288 18771 21961 18762 22727 6394 2431 948 16740 15673 12528 16675 2634 4948 3224 14918 20351 10519 18513 1747 5804 4323 23482 20771 12220 4050 6102 19357 10447 21188 2125 1432 18171 11822 14669 11236 2591 23663 11952 18990 6467 8899 22760 24159 22773 6958 13844 13240 4942 3376 13377 23685 17732 4795 9909 11637 19633 16257 3666 18130 16558 2859 6482 17778 14325 15969 7766 4675 15989 24090 16749 5065 15467 18430 19879 14853 8898 7460 23441 20377 4790 9777 637 13332 16627 13746 21476 11383 5071 5784 10331 17231 17108 9551 9381 8157 5257 12222 21690 18268 8768 20172 13034 9177 9136 14690 22045 17836 16809 12111 7336 1029 20735 9039 9155 17680 10198 1210 14253 19062 8262 580 23787 15163 17250 1601 5807 14076 13516 11214 9801 2442 16723 21868 4023 1593 21942 19903 2813 24400 17937 4787 8408 8030 14444 13233 24958 2746 19946 4957 2751 13776 20557 11211 16594 14591 13014 13727 22437 1256 366 20241 13622 16096 18514 7511 4415 1448 23007 23818 13740 22117 10493 1265 7897 7726 1906 14308 17456 8001 1101 23683 21235 17654 15045 7768 2138 2496 466 10024 19299 764 24487 18403 8332 6866 949 20883 23863 16567 8863 676 12182 10300 24384 17898 14345 23670 18472 9487 752 16235 14381 8578 2778 5911 10502 11326 12902 9068 5777 20712 19345 16738 2985 22945 13753 4195 18672 16911 21062 18547 17360 23033 24453 12940 17265 15504 21554 13544 16155 4145 5483 16034 16017 24941 9792 8051 3505 9869 6769 19647 12806 10737 18345 11978 7081 13697 19337 21925 7026 23074 23803 2047 11230 17134 7875 2495 18113 10544 11462 21278 3291 5255 19806 14491 5796 2671 19499 16571 14183 14221 3496 2717 867 15287 7342 5197 22321 17956 5721 4590 13540 11605 11248 18375 23592 8665 18992 22535 15150 23405 7904 2440 19167 14515 11080 21861 2833 7222 16185 13435 17953 7629 3234 14331 24342 20540 13984 4448 9173 13551 21130 20573 3918 4447 22785 11560 11512 5347 2349 15002 18773 23209 11992 7369 6366 8762 21911 14678 2697 9152 13649 21905 5022 4200 24192 23660 12261 8818 8553 14748 15921 16350 1835 15363 13543 16486 3451 6238 21871 19799 23529 10985 9355 5315 17726 2551 1884 21153 10148 5214 6065 9436 15808 21305 6389 3951 21587 19456 651 17054 15484 18640 21032 19960 15329 19587 11047 3826 22784 14911 19338 8648 14302 20212 21142 9082 9942 16943 13594 12184 16040 21331 2680 7740 5235 24614 3268 16826 20318 15820 10488 6902 7042 11663 13289 22209 8195 3146 11247 11798 2771 5614 15189 17319 14561 15692 11989 4546 10686 22389 19265 4860 4938 3148 11820 20788 16044 15770 23754 9376 8789 9954 14629 14206 18569 9338 6937 20650 13146 20835 6453 7439 962 24003 11920 13492 9926 7360 20558 21695 17793 4473 23048 15727 10577 22387 2803 19314 21270 15609 10184 18256 9250 6957 20855 4379 4679 14018 11783 18847 22402 13930 8981 3118 24590 13173 15682 10750 3792 2438 1 10131 22100 7852 4149 20754 11562 17312 20915 6484 5813 18667 16919 9464 7724 799 22648 11909 18148 1583 21273 20501 18518 7504 9375 5972 4045 13832 23191 13417 14272 11410 8276 22024 13420 16048 3056 1252 9684 5227 946 13869 5951 16117 16283 6286 14623 15793 17800 2583 19443 6724 3107 14793 19987 5573 126 24492 17652 23675 8587 9222 1571 5702 12245 16868 18399 8018 10187 19678 14481 8187 6881 22650 19655 3976 11873 12172 10484 4475 20864 12343 12743 5569 4289 2990 17656 15030 17881 6005 2067 21711 19506 9321 3909 1227 23571 10801 17684 4982 1834 24285 2929 3142 20080 14553 11151 5644 14519 12005 6920 1476 12719 16941 22136 4894 204 21470 3135 8723 16726 11894 16380 21738 20609 14036 20046 20960 3176 1690 8174 19080 22464 20775 14634 3827 8036 19805 17554 23621 9319 2888 6634 13822 13980 18776 6268 2026 688 20826 5335 22207 16977 22372 24354 18049 2578 9950 1395 13636 7660 4635 14530 12046 18242 4682 12010 24851 11316 4634 3563 21898 12909 9666 21509 22709 19882 9398 1831 941 14844 13393 9756 8480 23432 14193 16998 8455 4119 8440 24057 23590 21796 18287 16144 10817 20851 1049 2206 7509 128 13201 19802 11796 10953 8533 3206 639 20428 20617 23859 7308 1823 18837 13459 24704 14239 8838 7573 8672 182 21722 19525 17127 23100 7930 2988 23672 16052 11724 11722 5363 5011 4352 722 813 19961 10568 6093 15735 19528 15408 7102 5195 23889 22423 11688 17809 2441 2446 20598 15456 23447 11504 4960 19453 11100 23453 16814 15099 14384 22723 2592 25000 16274 23474 3449 4817 982 13781 18352 14625 11808 1668 2520 10674 3652 6805 22318 13526 21209 18182 66 8358 7675 3973 24611 15312 13171 1801 5824 12323 24261 2166 4656 15874 18938 16304 9139 6379 4076 406 10854 19207 10408 10317 2729 7016 15206 10679 12221 13152 1803 24594 17011 10475 4560 6073 11695 23775 19856 17240 8441 6091 21471 23677 23239 20463 74 12247 14619 78 2943 1540 12500 24443 21900 8372 9516 621 23739 15878 15394 1873 5861 21567 20756 19509 21462 17473 3606 3271 8236 5671 3835 4918 9142 4467 6128 733 6206 6038 7570 7002 9286 6080 3487 9392 7948 4162 5142 3844 6930 5134 9655 526 3914 4971 9860 1587 991 3051 9602 7792 9265 5417 5368 3437 3899 9779 3490 4399 7847 9848 5996 4637 6842 2856 9257 4685 4810 3354 6190 1183 3861 6854 3880 257 600 2343 1205 2458 7718 7178 4460 681 9891 3068 3733 1122 611 9672 7602 3157 874 1781 4136 4342 911 9025 9894 5369 1943 318 346 775 362 392 465

8

nr a-00375624, vers on 1 - 15 Apr 2009

Hos s

a Dend og am o h e a ch ca c us e ng o a andom subse o hos s

7 Numbe o C us e s

6

5

b Compa son o d s ances n c us e s

F gure 3 C uster Compar son very sens ve o ou ers [10] We measured he goodness of fi (GOF) of he resu ng d s r bu ons us ng s andard probab y-probab y (PP) p o s as a v sua me hod and a so quan a ve me r cs e Ko mogorov-Sm rnov (KS) and Anderson-Dar ng (AD) es s The graph ca resu s wh ch are dep c ed n F gure 5 revea ed ha n he mos cases he Gamma d s r bu on s a good fi A so Exponen a d s r bu on has some c ose fi s spec a y n c us er 4 These resu s cou d a ow an approx ma on ha wou d resu n a s mp e ana y ca mode Based on he h rd co umn of PP-p o s s obv ous ha he Pare o s comp e e y far from our under y ng d s r bu on so we do no have a heavy- a ed d s r bu on However as n some p o s Log-norma and We bu have a c ose ma ch some c us er d s r bu ons are ke y o be ong- a ed To be more quan a ve we a so repor ed he p-va ues of wo goodness-of-fi es s We random y se ec a subsamp e of 30 of each da a se and compu e he p-va ues era ve y for 1000 mes and fina y ob a n he average p-va ue Th s me hod s s m ar o he one used by he au hors n [7] and was sugges ed o us by a s a s c an The resu s of GOF es s are s ed n Tab e 2 where n he each row he bes fi s h gh gh ed These quan a ve resu s s rong y confirm he graph ca resu of he PP-p o s (In he PP-p o s he c oser he p o s are o he ne y = x he be er he fi ) We find ha he bes -fi ng d s r bu ons of each c us er o d ffer s gn fican y from he overa d s r bu on over a d hos s Thus c us er ng was essen a for accura e hos ava ab y mode ng To be more prec se Tab e 3 s s some proper es of d

hos s and c us ers as we The firs po n s he ex s ng he erogene y n he c us ers n erms of members and percen age of o a ava ab y The b gges c us er ( e c us er 6) ha nc udes more han 50% of o a d hos s on y conr bu e 20% of he o a ava ab y and a so c us er 3 ha s beh nd c us er 6 and 3 n number of hos s has he h ghes ava ab y con r bu on One poss b e ques on abou he d s r bu on fi ng resu s s why four c us ers ha fo ow he Gamma d s r buons are no n one c us er The answer s n he as co umn of Tab e 3 where he shape and sca e parame ers of fi ng are repor ed (For Log-norma d s r bu on he numbers are µ and σ respec ve y) These va ues revea ed ha a four c us ers have a mos same shape (see F gure 4) bu a a d fferen sca e wh ch are very far from each o her Ano her ques on ha may be ra sed here s whe her or no we are ab e o mode a hos s as a s ng e d s r bu on e Gamma d s r bu on w h fixed parame ers The answer cou d s no he shape and sca e parame ers for he Gamma d s r bu on fi ng for a d hos s are 0 2442 and 51 9864 respec ve y wh ch does no reflec he proper es of d fferen d s r bu ons s ed n Tab e 3 However based on he graph ca and quan a ve p-va ue resu s we cou d say ha he Gamma s a good fi for a c us ers so we can use a un que d s r bu on bu w h d fferen parame ers (a eas d fferen sca es) o mode he ava ab y of d fferen hos s I s wor h no h ng ha he Gamma d s r bu on has very n eres ng proper es n erms of flexb y and genera za on and can be eas y used o propose an ana y ca Markov mode as we [10]

1 0.9

0.8

0.8

0.7

0.7

0.7

0.7

0.7

0.6 0.5 0.4

0.6 0.5 0.4

0.6 0.5 0.4

Fitted Quantiles

1 0.9

0.8

Fitted Quantiles

1 0.9

0.8

Fitted Quantiles

1 0.9

0.8

Fitted Quantiles

Fitted Quantiles

1 0.9

0.6 0.5 0.4

0.6 0.5 0.4

0.3

0.3

0.3

0.3

0.3

0.2

0.2

0.2

0.2

0.2

0.1

0.1

0.1

0.1

0 0

0.2

0.4 0.6 Emprical Quantiles

0.8

0 0

1

0.2


0.8

0 0

1

0.2


0.8

0 0

1

0.1 0.2


0.8

0 0

1

0.2


0.8

1

0.2


0.8

1

0.2


0.8

1

0.2


0.8

1

0.2


0.8

1

0.2


0.8

1

1

1

0.9

0.9

0.8

0.8

0.8

0.7

0.7

0.7

0.7

0.7

0.6 0.5 0.4

0.5 0.4

0.6 0.5 0.4

0.6 0.5 0.4

0.6 0.5 0.4

0.3

0.3

0.3

0.3

0.3

0.2

0.2

0.2

0.2

0.2

0.1

0.1

0.1

0.1

0 0

0.2


0.8

0 0

1

0.2


0.8

0 0

1

0.2


0.8

0 0

1

0.1 0.2


0.8

0 0

1

1

1

0.9

0.9

0.8

0.8

0.8

0.7

0.7

0.7

0.7

0.7

0.6 0.5 0.4

0.6 0.5 0.4

0.6 0.5 0.4

Fitted Quantiles

1 0.9

0.8

Fitted Quantiles

1 0.9

0.8

Fitted Quantiles

1 0.9

Fitted Quantiles

Fitted Quantiles

(b) Cluster 2

0.6 0.5 0.4

0.6 0.5 0.4

0.3

0.3

0.3

0.3

0.3

0.2

0.2

0.2

0.2

0.2

0.1

0.1

0.1

0.1

0 0

0.2


0.8

0 0

1

0.2


0.8

0 0

1

0.2


0.8

0 0

1

0.1 0.2


0.8

0 0

1

1

1

0.9

0.9

0.8

0.8

0.8

0.7

0.7

0.7

0.7

0.7

0.6 0.5 0.4

0.6 0.5 0.4

0.6 0.5 0.4

Fitted Quantiles

1 0.9

0.8

Fitted Quantiles

1 0.9

0.8

Fitted Quantiles

1 0.9

Fitted Quantiles

Fitted Quantiles

(c) Cluster 3

0.6 0.5 0.4

0.6 0.5 0.4

0.3

0.3

0.3

0.3

0.3

0.2

0.2

0.2

0.2

0.2

0.1

0.1

0.1

0.1

0 0

0.2


0.8

0 0

1

0.2


0.8

0 0

1

0.2


0.8

0 0

1

0.1 0.2


0.8

0 0

1

1

1

0.9

0.9

0.8

0.8

0.8

0.8

0.8

0.7

0.7

0.7

0.7

0.7

0.6 0.5 0.4

0.6 0.5 0.4

0.6 0.5 0.4

Fitted Quantiles

1 0.9

Fitted Quantiles

1 0.9

Fitted Quantiles

1 0.9

Fitted Quantiles

Fitted Quantiles

(d) Cluster 4

0.6 0.5 0.4

0.6 0.5 0.4

0.3

0.3

0.3

0.3

0.3

0.2

0.2

0.2

0.2

0.2

0.1

0.1

0.1

0.1

0 0

0.2


0.8

0 0

1

0.2


0.8

0 0

1

0.2


0.8

0 0

1

0.1 0.2


0.8

0 0

1

(e) Cluster 5

1

1

0.9

0.9

0.8

0.8

0.8

0.8

0.8

0.7

0.7

0.7

0.7

0.7

0.6 0.5 0.4

0.6 0.5 0.4

0.6 0.5 0.4

Fitted Quantiles

1 0.9

Fitted Quantiles

1 0.9

Fitted Quantiles

1 0.9

Fitted Quantiles

Fitted Quantiles


0.6

Fitted Quantiles

1 0.9

0.8

Fitted Quantiles

1 0.9

0.8

Fitted Quantiles

1 0.9

Fitted Quantiles

Fitted Quantiles

(a) Cluster 1

0.6 0.5 0.4

0.6 0.5 0.4

0.3

0.3

0.3

0.3

0.3

0.2

0.2

0.2

0.2

0.2

0.1

0.1

0.1

0.1

0 0

0.2


0.8

1

0 0

0.2


0.8

1

0 0

0.2


0.8

1

0 0

0.1 0.2


0.8

1

0 0

Data sets All iid hosts Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Cluster 6

Exponential AD KS 0.004 0.000 0.155 0.071 0.188 0.091 0.002 0.000 0.264 0.163 0.204 0.098 0.059 0.016

Pareto AD KS 0.061 0.013 0.029 0.008 0.020 0.004 0.068 0.023 0.002 0.000 0.013 0.002 0.033 0.009

Weibull AD KS 0.581 0.494 0.466 0.243 0.471 0.259 0.485 0.380 0.484 0.242 0.498 0.296 0.570 0.439

Log-Normal AD KS 0.568 0.397 0.275 0.116 0.299 0.128 0.556 0.409 0.224 0.075 0.314 0.153 0.485 0.328

Gamma AD KS 0.431 0.359 0.548 0.336 0.565 0.384 0.372 0.241 0.514 0.276 0.563 0.389 0.538 0.467


Table 2. P-value results from GOF tests for all iid hosts and six clusters

Clusters

# of hosts

% of total avail.

Best fit

All iid hosts Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Cluster 6

57757 3606 9321 13256 275 1753 29546

1.0 0.16 0.35 0.22 0.01 0.05 0.20

Weibull Gamma Gamma Log-Normal Gamma Gamma Weibull

Parameters shape scale 0.3787 3.0932 0.3131 289.9017 0.3372 161.8350 -0.8937 3.2098 0.3739 329.6922 0.3624 95.6827 0.4651 1.8461

Table 3. Some properties of clusters and iid hosts

9

6

5

x 10

10

5 CPU speed (FLOPS)

Expected number per cluster

4

10

3

10

2

10

4

3

2

1

10

0

10 0 10

1

Venue Timezone Reference 1

10

2

3

10 10 Actual number per cluster

4

10

5

10

0

1

2

(a) By venue or timezone

Figure 6. Clustering by other criteria

3 4 Cluster index

(b) By CPU speed

5

6


7.2

Significance of Clustering Criteria

Our results in the previous section beg the following question: could the same clusters have been found using some other static criteria? In this section we considering cluster formation by static criteria, namely host venue, timezone, and clock rates. (Note that a small fraction of hosts did not have this information specified, and were thus excluded from this analysis.) We define host venue to be whether a host is used at home, school, or work. This is specified by the user to the BOINC client (though this is not required). Roughly, 23900 are at home, 5500 are at work, and 772 are at school. If our clustering results correspond to those categories, we would expect the distribution of host venue across clusters to be even more skewed. To measure this, we computed the expected number of hosts of a particular venue for each one of the six clusters identified in Section 7.0.3. The expected number is computed using the global percentage of home, work, and school hosts. The we counted the actual number of hosts of each venue in each cluster. Figure 6(a) shows the expected number versus the actual number for each venue type over each cluster. The results for larger cluster sizes are more significant. If the clusters correspond to host venues, we would expect large deviations from the y = x line. However, this is not the case as the expected and actual values are similar. Thus, the same clusters would not have resulted from the host venue. We conducted the same comparison using host timezones. We took counted the number of hosts in each cluster for each of the six largest timezones (in terms of hosts). These timezones in corresponded to Central Europe (17,000 hosts), Eastern North America (11,003 hosts), Central North America (6,077 hosts)), Western North America (4,900 hosts), Western Europe (4280 hosts), and Eastern Asia (2396 hosts). We then compared this with the expected number, given the number of hosts in total for each of the six timezones. We find again that the corresponding points in Figure 6(a) are close the the line y = x. This indicates that the clusters are not a direct result of timezones alone. We also investigated the relationship between CPU speeds (in FLOPS) and the clusters. Figure 6(b) shows a box-and-whisker plot of the CPU speeds for each cluster. The box represents the inter-quartile range. As this box for each cluster appears in similar ranges, we conclude that the clusters could not have been formed using CPU speeds alone, and that there is little correlation between CPU speeds and the length of intervals. Nonetheless, one would like an intuition to explain why there was a formation of six clusters. We do not have a precise answer to this question. It could depend very much on

the behavior of the user, which is hard to measure. For future work, we could like to conduct a survey of volunteers or to monitor their host processes to classify them by how they use their computer (for example, for gaming, word processing, web surfing). This might shed light on the cause of these clusters.

8

Scheduling Implications

Our study has pinpointed the exact mixture of iid versus non-iid hosts, and when iid, the distribution that best models it and its parameters. The difference between the global distribution for all iid hosts, and the distributions for each individual cluster impacts scheduling accuracy greatly. For example, if one uses the Weibull distribution as the global model, the probability of a host to successfully complete a 24 hour task is less than 20%. By contrast, for hosts in cluster 4, the probability is about 70%. These differences are magnified multiplicatively when computing the probability of task completion with replication or for a series of tasks with dependencies. Moreover, in terms of multi-job scheduling, an online scheduler could create a multi-level priority queue and channel jobs to hosts depending on their model requirements. For example, in typical volunteer computing systems, there are a mixture of high-throughput and lowlatency jobs. In practice, we have observed that the total sum of availability across hosts in large numbers to be relatively constant and tends towards a Normal distribution. High-throughput jobs could then be sent to non-iid hosts as their lack of randomness likely will not affect the aggregate system throughput as a whole. On the other hand, the jobs that need low response time, such as batches of low-latency tasks or DAG’s, can be scheduled on hosts that exhibit iid availability, and the probability of successful task completion can be computed with the discovered model.

9

Conclusions

We considered the problem of discovering availability models for host subsets from a large-scale distributed system. Our specific contributions were the following: • With respect to methodology. – We detected and determined the cause of outliers in the measurements. We minimize their effect during the preprocessing phase of the measurements. This is important for others to use the trace data set effectively. – We described a new method for partitioning hosts into subsets that facilitates the design of stochas-


tic scheduling algorithms. We use randomness tests to identify hosts with iid availability. – For iid hosts, we use clustering based on distance metrics that measure the difference between two probability distributions. We find that the Cramer-von Mises metric is the most sensitive and computationally efficient compared to others. – In the process, we describe how we deal with with large and variable sample sizes and the hypersensitivity of statistical tests by taking a fixed number (30) of random samples from each host. • With respect to modelling. We apply the methodology for the one of the largest Internet-distributed platforms on the planet, namely SETI@home. – We find that about 34% of hosts have truly random availability intervals. – These iid hosts have availability that can be modelled using 6 distributions, in particular, the Gamma, Weibull, and Log-Normal distributions. About 51% of the total availability comes from hosts whose availability follows the Gamma distribution. This is useful as this distribution is short-tailed and can be used easily to construct analytical Markov models. – The family of Gamma distributions (with different scales) could be used to model all iid hosts. – The discovered distributions from the same family differ greatly by scale, which explains their separation after clustering. The distribution for all hosts with iid availability also differs significant in scale from any of the distributions corresponding to each cluster.

Acknowledgements We thank Eric Heien for useful discussions. This work has been supported in part by the ANR project SPADES (08-ANR-SEGI-025).

References [1] A. Acharya, G. Edjlali, and J. Saltz. The Utility of Exploiting Idle Workstations for Parallel Computation. In Proceedings of the 1997 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, pages 225–234, 1997. [2] D. Anderson. Boinc: A system for public-resource computing and storage. In Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing, Pittsburgh, USA, 2004.

[3] D. Anderson and G. Fedak. The Computational and Storage Potential of Volunteer Computing. In Proceedings of the IEEE International Symposium on Cluster Computing and the Grid (CCGRID’06), 2006. [4] R. Bhagwan, S. Savage, and G. Voelker. Understanding Availability. In Proceedings of IPTPS’03, 2003. [5] The berkeley open infrastructure for network computing. http://boinc.berkeley.edu/. [6] W. Bolosky, J. Douceur, D. Ely, and M. Theimer. Feasibility of a Serverless Distributed file System Deployed on an Existing Set of Desktop PCs. In Proceedings of SIGMETRICS, 2000. [7] J. Brevik, D. Nurmi, and R. Wolski. Quantifying Machine Availability in Networked and Desktop Grid Systems. Technical Report CS2003-37, Dept. of Computer Science and Engineering, University of California at Santa Barbara, November 2003. [8] G. Casella and R. Berger. Duxbury, 2002.

Statistical Inference.

[9] G Cirrone et al. A goodness-of-fit statistical toolkit. IEEE Transctions on Nuclear Science, 51(5):2056– 2063, 2004. [10] Feitelson. D. Workload Modeling for Computer Systems Performance Evaluation. 2009. [11] John R. Douceur. Is remote host availability governed by a universal law? SIGMETRICS Performance Evaluation Review, 31(3):25–29, 2003. [12] D. Kondo, M. Taufer, C. Brooks, H. Casanova, and A. Chien. Characterizing and Evaluating Desktop Grids: An Empirical Study. In Proceedings of the International Parallel and Distributed Processing Symposium (IPDPS’04), April 2004. [13] Derrick Kondo, Artur Andrzejak, and David P. Anderson. On correlated availability in internet distributed systems. In IEEE/ACM International Conference on Grid Computing (Grid), Tsukuba,Japan, 2008. [14] D. Long, A. Muir, and R. Golding. A Longitudinal Survey of Internet Host Reliability. In 14th Symposium on Reliable Distributed Systems, pages 2–9, 1995. [15] M. Mutka and M. Livny. The available capacity of a privately owned workstation environment . Performance Evaluation, 4(12), July 1991. [16] S. Saroiu, P.K. Gummadi, and S.D. Gribble. A measurement study of peer-to-peer file sharing systems. In Proceedinsg of MMCN, January 2002.

[17] Ying Wang. Nonparametric tests for randomness. Research report, UIUC, May 2003.


[18] J. Wingstrom and H. Casanova. Probabilistic allocation of tasks on desktop grids. In IPDPS’08, pages 1–8, 2008.