getting speedup with accuracy in simulation ... - Semantic Scholar

GETTING SPEEDUP WITH ACCURACY IN SIMULATION EXPERIMENTS OF COMMUNICATION NETWORKS Edjair Mota, Adam Wolisz Technical University of Berlin, Germany Krzysztof Pawlikowski University of Canterbury, Christchurch, New Zealand e-mail : [email protected] Keywords : Ecient simulation, multiple replications, parallelization, computer networks, MAC protocols

ABSTRACT With the advent of new communication technologies, modern networks tend to be more complex and there is a necessity of ecient tools for enhancing the computational eort of their simulations. Multiple Replications in Parallel (MRIP ) is an attractive approach for this purpose, since it can decrease the simulation cost and increase the accuracy of the results. Akaroa.2, a user-friendly package is an MRIP implementation, and has been found easy to install, use and extend. We present some issues concerning Akaroa's statistical properties, which indicate that it is a good candidate for simulating communication networks with eciency.

1 INTRODUCTION Nowadays we witness a great challenge in the computing area. As the communication technologies advance and computer processing power increases, new kind of applications arise, e.g. integrated services. As a consequence, the interest of new users is aroused, which implies that new requirements and applications, and thus other users, will certainly appear. Dynamic increasing of the complexity of the networks and the growth of the number of users require ecient tools for analysing them and improving their performance. Straightforward simulation of such complex systems as communications networks  The rst author was supported by Conselho Nacional de Desenvolvimento Cient co e Tecnologico, from Brazil, under process number 29.000394.6.

takes frequently a prohibitive amount of time despite the increasing of processing speed of modern computers. It's not unusual a simulation experiment to take hours, even days, for yielding reasonable information. Reason for that, among others, is the statistical nature of the simulation experiments. Most simulation models contain stochastic input variables and, thereby, stochastic output variables are generated, the last ones being used for estimating the characteristics of the real system. In order to obtain an accurate estimation it is necessary to collect a substantial amount of simulation output data, and this can require very long simulation run length. This paper addresses the necessity of tools for enhancing the computational eciency of computer communication network models, and particularly presents our works on enhancing functionality of Akaroa.2, a user-friendly implementation of Multiple Replications in Parallel | MRIP, which launches ordinary sequential simulation in parallel and averages the results (Pawlikowski et al. 1994). As it is known, the MRIP approach can decrease the simulation cost and automatically control the accuracy of the results. Section 2 overviews main problems associated with simulation experiments of communication networks and their possible solutions. Section 3 presents an MRIP implementation in Akaroa.2. Section 4 deals with the problem of accuracy of the results and discusses some con dence interval procedures (CIPs) proposed for this purpose. Section 5 compares the performance of two CIPs and comments

on what can be expected when applying them in sequential stochastic simulation together with automatic parallelizationin MRIP scenario. Section 6 contains nal remarks and directions to further research.

Despite of its simplicity, MRIP approach answers satisfactorily the rst two problems listed at the beginning of this section, and, moreover, oers a quite simple solution for the third and last problem, concerned with the credibility of the nal simulation results.

2 PROBLEM OVERVIEW

3 AKAROA.2

Suppose we are conducting a simulation experiment to investigate the behavior of a new medium access control (MAC) protocol and we come across one of the following problems :

Akaroa.2 is an MRIP implementation designed at the Department of Computer Science of the University of Canterbury in Christchurch, New Zealand, to run stochastic discrete-event simulation on Unix multiprocesor systems or networks of heterogeneous Unix workstations. It makes uses of an ecient interprocess communication (IPC), which facilitates the exchange of messages among the simulation engines and the central process. Parallelization process is completely transparent from the point of view of the analyst, who does not even need to be aware of the existence of such parallel structure. She/he is just required to add one extra line to code to her/his sequential simulation model defore Akaroa.2 takes care of its parallel execution. Akaroa.2 provides its own sources of randomness and ensures that the simulation engines are using non-overlapping sequences of random numbers, thus cross-observations are independent and identically distributed. The question of bias produced by the initial transient, pointed out by (Heidelberger 1986) as the greatest challenge for this kind of approach, is taken into account by applying an algorithm conceived by Schruben (Schruben 1982). For a revision of this and other problems related to steadystate simulation readers are refered to survey (Pawlikowski 1990). Crash of a simulation engine, or a too slow behavior of some processors imposes no obstacle to Akaroa.2's task, since it demands no order of arrivals for the intermediate results yielded by dierent simulation engines. Just the overall simulation run length takes a little longer, proportional to the number of lost simulation engines. It works as if the pool of processors had fewer processors. On the other hand, it is obvious that MRIP works better in an homogeneous environment. Akaroa.2 is also equipped with a library of routines suitable for modeling communication networks, but it can also be used in an integration with other simulation tools written in C or C++, or at least capable of calling routines in those languages. Akaroa.2 has been linked with such simula-

1. The experiment takes very long time to conclude. 2. We need to analyse several variants of this protocol. 3. We want to have higher con dence in the credibility of the results. A possible solution to the rst two issues would be to increase the statistical eciency of our simulation experiment by means of (i ) applying procedures which can reduce the variability of estimators being analysed or (ii ) making use of more computational resources. Alternative (i ) is generally known as Variance Reduction Techniques |VRT, a set of experimental design and analysis techniques used to achieve a desired precision in shorter simulation time (Law and Kelton (Law and Kelton 1991). Although their ef cacy is by all means recognized (e.g. Frost 1988; Izydorczyk et al. 1984), they can sometimes require great additional eort from the analyst in terms of statistical background. Alternative (ii ) has been commonly seen as a pool of cooperative hosts, each one processing a part of the same model (submodel) and, therefore, synchronization among these parts is a relevant issue. One can also take advantage of the available computational resources of computer networks by applying an approach called Multiple Replications in Parallel (see Pawlikowski e al. 1994 for a discussion of its properties in the context of steady-state simulation). Here, each processor runs an instance of the entire simulation model (hereafter called simulation engine ), and the intermediate results are analysed by a central process. A faster processor can run more than one independent simulation engine.

M/H2/1

M/M/1

During analysis of simulation output data we can make use of one of several procedures found in the literature, which construct con dence intervals of mean value at a con dence level assigned by the analyst, in order to assess the accuracy of the results. These procedures are hereafter called Con dence Interval Procedures, or CIPs. Actually, Akaroa.2 implements two CIPs, based on Nonoverlapping Batch Means (NOBM) (Schmeiser 1982) and on a version of Spectral Analysis proposed by Heildelberger and Welch (SA/HW) (Heidelberger and Welch 1981). Performance of these CIPs under MRIP were addressed in (McNickle et al. 1996, Mota 1998 et al., Ewing et al. 1998). The main criterion for comparing the performance of CIPs is the coverage, the frequency with which the nal con dence intervals contain the true value of the performance parameter being simulated. Theoretically, a con dence interval constructed at a (1 ? ) con dence level should contain the simulated parameter with the probability of (1 ? ), but since the underlying assumptions are rarely completely full led, the nal coverage is frequently smaller than the nominal. Other not less important criteria are the number of observations required to achieve the stopping rule, and the coecient of variation of the half-length of the nal con dence interval. The former is directly related to the duration of the simulation experiment and the latter refers to the variability of the nal con dence intervals. These CIPs were originally conceived for a single processor environment, and it's interesting to investigate their behavior under a parallel computing environment created by Akaroa.2, by applying them on models with known analytical solution. Preliminary investigations showed us that adding more processors doesn't involve exactly proportional decrease in the number of observations, but the framework implemented by Akaroa.2 can collect more observations during a shorter interval of time, which implies a reasonable speedup. In other words, Akaroa.2 is a natural variance reductor, without any

M/E4/1

4 ACCURACY OF THE RESULTS

additional eort from the analyst. Considering the popularity of Batch Means approach due to its simplicity of comprehension and implementation, we considered as relevant implementing sequential versions of other variants of Batch Means, for instance Overlapping Batch Means (OBM), proposed by Meketon and Schmeiser (Meketon 1984), and Spaced Batch Means (SBM), proposed by Fox et al. (Fox91). It is out of the scope of this work to discuss them in detail, but, just to remember, OBM is an approach in which it is more important to have a larger number of batches than to have these batches independent. Thus, some observations from previous batches are reused for generating more batch means, using the same number of observations. In that case, the only additional information passed by the analyst is the degree of overlap. SBM takes into account the fact that the usual strong correlation among the observations in the stationary phase can be reduced by discarding some observations between batches. M/D/1

tion tools as PTOLEMY Ewing et al. 1998). In the Telecommunication Networks Group at the Technical University of Berlin, we have linked Akaroa.2 with SMURPH.

Cx 0

0.25

1

1.414

Fig. 1: Queueing models with increasing coecient of variation

5 CIPs COMPARISON UNDER MRIP

Con dence interval procedures (CIPs) usually make various assumptions in order to generate valid con dence intervals, and deviations from these assumptions might aect their validity. It is important to investigate the correctness of the assumptions taken by a CIP being studied, by means of a detailed coverage analysis. One should investigate how well a CIP behaves for a range of con dence levels (Schruben 1980). Suppose that during simulation of a telecommunication network we are investigating a MAC protocol performance parameter  through the estimator X (n). Let us assume that a CIP collects a sequence of iid observations xi fi=1,2, . . . g, calculates X (n) and constructs a con dence interval for  at the con dence level (1 ? ). If the underlying assumptions

Observed Confidence Level

P=2

P=6

P = 10

1.0

1.0

1.0

.9

.9

.9

8

8

8

.7

.7

.7

.6

.6

.6

.5

.5

.5

.4

.4

.4

.3

.3

.3

.2

.2

.2

.1

.1

.1

.1

.2

.3

.4 .5 .6 .7 Desired Confidence Level

8

.9

1.0

.1

.2

.3

.4 .5 .6 .7 Desired Confidence Level

8

.9

1.0

.1

.2

.3

.4 .5 .6 .7 Desired Confidence Level

8

.9

1.0

Fig. 2: Coverage function under MRIP, for a sequential version of the method OBM using P processors (M=M=1,  = 0:9). are true, then P (X (n) ? H    X (n) + H ) = 1 ?  where X is the meanPfor the n-sized sample, de ned as X (n) = n1 n1 xi , and H is the half-length of the con dence interval estimator of  de ned as H = tn?1;1?  ^ [X ], where ^ [X (n)] is the standard deviation of X (n), and tn?1;1?  is the upper (1 ? 2 ) critical point of the t distribution with n-1 degrees of freedom. 2

2

When the assumptions are satis ed, the coverage

 has a uniform distribution. (Schruben 1980) sug-

gests the construction of an empirical distribution function G () with the observed values of con dence level  for a sample size n. This empirical function presents more information than a single value of coverage, e.g. at  = 0:9, a common practice to summarize empirical results in coverage analysis. Ideally, G () = , but usually two other situations can arise :

 G () < , occurs when positive serial correlation has been ignored, which means that the nal  was overestimated.

 G () > , occurs when negative serial correla-

tion has been ignored, which can lead to wasting the time in sequential procedures, since it may result in collecting more observations than necessary for achieving a desired precision. We have applied the sequential coverage analysis proposed in (Pawlikowski et al. 1998), since it gives us a more exact information than that obtained from xed-sample versions, to investigate the coverage function proposed by (Schruben 1980). Our analysis stopped when the relative precision of the mean value of the coverage compared to the halflength of its con dence interval was less than 5%. To assess the performance of the CIPs we used queueing models with known analytical solution, 90% loaded. As a matter of fact, we chose queueing models with increasing coecient of variation of its service times p (See Figure 1) , ranging from 0 (for M=D=1) to 2 (for M=H2 =1). Table 1 summarizes the results of such sequential analysis by using 2, 4, 6, 8 and 10 processors, at a 90% of con dence level. After nding sequentially the optimal batch size M, which guarantees an acceptably low correlation among the batch means at a signi cance level , the sequential procedure based on OBM divides each batch into four sub-batches, each of which initiates an overlapping batch of size M. Originally, (Meketon 1984) suggested that each observation should initiate an overlapping batch, but we found that satis-

Observed Confidence Level

P=2

P=6

P = 10

1.0

1.0

1.0

.9

.9

.9

8

8

8

.7

.7

.7

.6

.6

.6

.5

.5

.5

.4

.4

.4

.3

.3

.3

.2

.2

.2

.1

.1

.1

.1

.2

.3

.4 .5 .6 .7 Desired Confidence Level

8

.9

1.0

.1

.2

.3

.4 .5 .6 .7 Desired Confidence Level

8

.9

1.0

.1

.2

.3

.4 .5 .6 .7 Desired Confidence Level

8

.9

1.0

Fig. 3: Coverage function under MRIP, for a sequential version of the method SBM using P processors (M=M=1,  = 0:9) factory variance reduction can be achieved with this computationally impler overlapping scheme. Since the sequential determination of the optimal batch size, SBM discards an amount s of observations between consecutive batches. Although greater values of s should improve a little bit the nal coverage, we adopted a parsimonious approach and xed s equal 20% of the initial batch size provided by the user. Recent investigations, however, showed us that SBM seems to perform somewhat better when s is chosen dynamically, based on the correlation structure of the underlying process being simulated. Both CIPs produced acceptable coverages, except in 1 of 40 experiments, when the coverage was much bellow than 80%, but it seems to be a statistical variation that does not invalidate our conclusions. EfTg stands for the average duration (in seconds) of each experiment. Here, Akaroa.2 shows an attractive time reduction in boths CIPs. For OBM this reduction increased with the increasing coecient of variation of the service times. EfOg stands for the average total number of observations (x106), required for achieving the desired precision of the nal results. Following this criterion, OBM overcame SBM (at least in the actual implementation, with no attempt of tunning), and required a little bit more observations. Figures 2 and 3 give some examples of the coverage function for the CIPs investigated. With little

exception, both procedures performed closely to the ideal case, but OBM seems to present less uctuations. Better results should be obtainable after tuning each of these CIPs, and this is a subject of our current research.

6 FINAL REMARKS Multiple Replications in Parallel is a very promissing methodology for enhancing the computational eciency of computer communication network simulation. The provided speedup is limited by number of processors involved, and no additional eort is required from the analyst for launching parallel simulation engines, if one uses such package as Akaroa.2. Interesting issues for further research includes the implementation of other CIPs and the analysis of the asymptotic behavior of these CIPs to know how they will perform as the run length of a real simulation model increases.

CIP

O B M S B M

Queue M=D=1 M=E4 =1 M=M=1 M=H2 =1 M=D=1 M=E4 =1 M=M=1 M=H2 =1

2

0.875 0.845 0.841 0.866 0.884 0.801 0.778 0.842

4

0.845 0.895 0.843 0.865 0.848 0.853 0.822 0.866

G (0:9) 6

0.824 0.867 0.856 0.875 0.835 0.844 0.815 0.822

8

0.847 0.831 0.853 0.832 0.813 0.828 0.815 0.812

10

0.835 0.868 0.848 0.847 0.912 0.870 0.848 0.867

2

27.80 21.71 34.03 89.42 22.95 11.96 37.97 50.30

4

8.80 15.73 30.62 34.80 14.08 13.58 18.79 18.47

EfT g 6

7.59 15.49 11.01 17.07 6.52 7.59 12.75 12.27

8

9.18 7.89 10.13 13.15 5.30 7.66 13.68 13.31

10

5.23 8.74 8.93 12.47 4.13 6.63 6.53 10.95

2

0.37 0.55 0.84 1.28 0.33 0.43 0.43 0.68

4

1.51 2.00 1.71 2.14 1.34 1.88 3.13 5.17

EfOg

6

3.14 4.27 3.03 7.65 1.32 3.39 4.04 8.35

8

2.60 3.64 4.79 13.32 2.54 3.61 5.45 12.44

10

4.13 5.48 6.59 11.83 4.13 3.81 8.15 12.03

Table 1: Performance comparison of OBM and SBM under MRIP, for P=2,4,6,8 and 10 processors

References [1] Ewing, G., Pawlikowski, K., and McNickle, D., Akaroa.2 : Exploiting network computing by distributing stochastic simulation, Tech. Rep. TRCOSC 08/98, Department of ComputerScience, University of Canterbury, Christchurch, New Zealand, November 1998. [2] Fox, B.L., Goldsman, D. and Swain, J. Spaced bstch Means Operations Research Letters 10 (1991), 255{263, 1991. [3] Frost, V.S., Larue Jr. and Shanmugan, K.S., Ecient techniques for the simulation of computer communications networks, IEEE Journal of Selected Areas in Communications, vol.6, no.1, pp. 146{157, January 1988. [4] Heidelberger, P. and Welch, P.D., A spectral method for con dence interval generation and run length control in simulations, Communications of the ACM, Vol.24, No.4, 233{245, April 1981. [5] Heildelberger,P., Statistical analysis of parallel simulations, Proc. 1986 Winter Simulation Conference, 290{295, 1986. [6] Heidelberger, P., Discrete event simulations and parallel processing: statistical properties, SIAM Journal Stat. Comput., vol.9, pp.1114{ 1132, November 1988. [7] Izydorczyk, J., Mierzwa, J. and Wolisz, A., A control variable method for increasing the eciency of multiple access protocol simulation. In Perfomance of Computer-Communication Systems, H. Rudin and W. Bux, North-Holland, Amsterdam, 95{108, 1984. [8] Law, A.M. and Kelton, W.D., Simulation Modeling and Analysis, 2nd Ed. McGraw-Hill, New York, 1991. [9] McNickle, D., Pawlikowski, K. and Ewing, G., Experimental evaluation of con dence interval

[10]

[11]

[12]

[13]

[14]

[15] [16] [17]

procedures in sequential steady-state simulation, Proc. 1996 Winter Simulation Conference, pp. 382|389, 1996. Meketon, M.S. and Schmeiser, B., Overlapping batch means: something for nothing ?, Proc. 1984 Winter Simulation Conference, 227{230, 1984. Mota, E., Wolisz, A. and Pawlikowski. K., Experience with Akaroa: Speeding up stochastic simulation by using multiple replications in parallel, Proc. Advanced Simulation Technologies Conference, Boston, pp. 296{301, 1998. Pawlikowski, K., Steady-state simulation of queueing processes: a survey of problems and solutions, ACM Computing Surveys, vol.22, pp. 123{170, 1990. Pawlikowski, K., Yau, V. and McNickle, D., Distributed stochastic discrete-event simulation in parallel time streams, Proc. of the 1994 Winter Simulation Conference, pp. 723{730, 1994. Pawlikowski, K., McNickle, D. and Ewing. G., Coverage of con dence intervals in steady-state simulation, Journal Simulation Practice and Theory, 6(1998), pp. 255{267. Schmeiser, B., Batch size eects in the analysis of simulation output, Operations Research, Vol.30, no.3, 556{598, May-June 1982. Schruben, L., A coverage function for interval estimators of simulation response, Management Science 26, 18{27, 1980. Schruben, L.W., Detecting initialization bias in simulation output, Oper. Res., Vol.30, pp. 569{ 590, 1982.