... A Framework for Statistically Rigorous. Simulation-based Network Research. E. Millman, D. Arora, S. W. Neville. Dept. of Electrical and Computer Engineering.
2011 Workshops of International Conference on Advanced Information Networking and Applications
STARS: A Framework for Statistically Rigorous Simulation-based Network Research E. Millman, D. Arora, S. W. Neville Dept. of Electrical and Computer Engineering University of Victoria Victoria, BC, Canada Email: {emillman, darora, sneville}@ece.uvic.ca [6], [7]). But, as shown within this work, these approaches do not guarantee that the effect of transients is indeed removed in its entirety, which in turn can engender misleading results. In MANET simulations, averaging across multiple experiment runs is typically done. But, such averaging presumes ergodic data, (i.e., that the underlying distributions producing the data are statistically similar from run to run). If averaging is performed over non-ergodic data sets then the potential exists for biases to be introduced into the reported results. As a trivial example, assume a given experiment is repeated twice with the first experiment producing a Gaussian (Normal) distributed feature where N (µ 1 = 0, σ1 = 1), while in the second experiment the same feature is distributed according to N (µ1 = 6, σ1 = 1). In such a case, averaging across the data will produce an averaged mean of µ = 3 even though the value x ≥ 3 for experiment #1 and x ≤ 3 for experiment #2 only occur 0.05% of the time. The innate problem in this case is that the ensemble of data across the combined runs is non-ergodic, (i.e., the experiment runs produce statistically distinct distributions for different random seeds). Finally, in many cases, network simulation results are only reported in terms of their estimated means and variances. Such reporting, of course, assumes that the underlying data has a Gaussian distribution, (i.e., given that the mean and variance are the sufficient statistics for Gaussian). Gaussian distributions appear commonly in physical systems, a result explained via the Central Limit theorem’s tenets [8]. Within network traffic, though, heavy-tailed, self-similar, and longrange dependent behaviors have all been observed [9], [10]. Hence, for network traffic it has been observed that the Central Limit theorem’s tenets do not hold. Moreover, in many cases, it may be difficult to argue that the given simulation’s features indeed arise as a sum of random variables, a necessary prerequisite of the Central Limit theorem. Overall, therefore, without applying goodness-of-fit tests it becomes problematic to claim that the estimated means and variances are indeed sufficient, (i.e., that they are all that needs to be reported) for many of the network simulation features of interest. Hence, again, solely reporting these quantities can potentially produce misleading results. The above observations argue for the need for statistically rigorous testing within network simulation-based research.
Abstract—Simulation has become one of the dominant tools in wired and wireless network research. With the advent of cloud, grid, and cluster computing it has become feasible to use parallelization to perform richer larger-scale simulations. Moreover, the computing resources needed to perform statistically rigorous simulations are now easily obtainable. Although a number of parallel network simulation frameworks exists, the issue of statistical rigorous testing has largely not been addressed. This work presents a parallel MPI-aware network simulation framework that is specifically designed to provide automated support for statistically rigorous experimentation, thereby offloading this significant researcher burden. Unlike prior frameworks, the proposed framework includes a distribution-free statistical analysis feedback loop that automatically deduces the next set of experiments that need to be run. The value of this new framework is highlighted by exploring the well known issue of assessing the true duration of start-up transients within mobile ad hoc networks (MANETs) simulations. Index Terms—Simulation; statistics; parallelization; network engineering; OMNeT++; MANET
I. I NTRODUCTION Simulation has become a primary tool to support both wired and wireless network research, (e.g., via tools such as ns-2 [1], OPNET [2], OMNeT++ [3]), as it provides a low-cost means for assessing solution approaches prior to moving to higher fidelity emulation-based testing, (i.e., via EMUlab [4]), or Internet-scale testing environments, (i.e., via PlanetLab [5]). The generally used network simulators, (e.g., ns-2, OPNET, OMNeT++) perform event-based stochastic simulations; hence, data (features) measured from simulation runs, (i.e., experiments) exist formally as stochastic processes. Therefore, all of the standard random data issues come into play. Moreover, it becomes important to formally assess how the underlying distributions, (i.e., those that give rise to the measured features) change both over time and, potentially, across experiment runs, (i.e., for runs employing differing random seeds). Within statistics the former issue denotes the data’s stationarity whereas the latter denotes its ergodicity. In mobile ad hoc network (MANET) simulations, for example, start-up transients are the obvious cause of non-stationary periods within the measured data. Numerous, fixed threshold approaches have been proposed and applied to remove these transients, (i.e., via discarding the initial 40 to 50 seconds of simulation time data 978-0-7695-4338-3/11 $26.00 © 2011 IEEE DOI 10.1109/WAINA.2011.147
733
But, traditionally this has not been done. With the advent of cloud [11], grid [12], and cluster computing [13] facilities the parallelization of network simulators, such as OPNET [14], OMNeT++ [15], [16], [17], and ns-2 [18], has been explored, but not from the perspective of supporting statistically rigorous research. To the authors’ knowledge this work presents the first parallelization framework that focuses on exploiting parallelization to provide automated support for statistically rigorous network simulations. The main contributions of this work are therefore: • The development of a parallelized testing framework that integrates an automated distribution-free statistical testing feedback loop, (i.e., to support stationarity and ergodicity testing). • The use of this framework to perform statistically rigorous testing for MANETs. • Highlighting the value of performing rigorous testing via assessing the true duration of MANET start-up transients. The outline for the remainder of this work is as follows. Section II discusses the related work. Section III then provides a high-level overview of the constructed framework focusing on its use of MPI as a middleware, the automated initiation of experiments, and its use of a feedback loop to control the next set of experiments to be run. Section IV focuses on the specifics of how the statistical testing for stationarity and ergodicity is performed in a distribution-free manner. Section V presents the results of using the framework to perform statistical testing in the MANET domain and, specifically, to assess the actual distributions of start-up transients within MANET simulations. Section VI discusses the limitations of the current tools and directions along which it is being expanded. Section VII concludes the work.
Akaroa [16]: Akaroa is designed for running quantitative stochastic discrete event simulations on Unix multiprocessor systems or a network of Unix workstations, (i.e., the running of multiple instances of a stochastic experiment concurrently on different processors). Results from these runs are collated and the overall mean for each parameter of interest is computed. If the results obtained satisfy the desired confidence level and precision then further simulation runs are stopped. The core limitation is that testing for precision is a manual task. Akaroa does not provide testing for stationary/ergodic behaviors. It also does not support using other software for visualization. • Xgrid [17]: Xgrid is a Apple OS grid framework that has been applied to parallelize OMNeT++ based simulations. This is achieved by adding additional features to OMNeT++ such that it can be used in conjunction with Apple’s Mac OS X, which comes with Xgrid support. Xgrid allows multiple simulations to be run in parallel either on multiple CPU cores or on multiple machines. However, Xgrid lacks any direct support for statistically rigorous testing. Xgrid currently cannot be used with any non-Apple OS. Xgrid does not support the use of other software tools for visualization. These frameworks provide the ability to run network simulations in a parallel manner over multiple CPUs and/or machines but are limited in a number of ways. Their inability to statistically test output data for stationarity/ergodicity, initiating simulations based on results of previous simulation and lack of support for visualization of results has led to the development of the framework presented in this paper.
II. R ELATED W ORK Several frameworks currently exist that support the parallelization of multiple simulation runs. Apart from OMNeT++ [19] itself, other available frameworks are, SimProcTC [15], Akaroa [16], and Xgrid [17]. A brief description of the parallel frameworks that are most similar to the framework developed in this work, along with their limitations, are presented below. A more detailed description and limitations of simulation frameworks in general can be found in [17]. • SimProcTC [15]: SimProTC is an open source simulation tool-chain for OMNeT++ designed to efficiently perform common simulation tasks, such as defining simulation parameters, number of runs desired, processing simulations in parallel using a distributed framework, and visualizing the results. Simulations are run in parallel using Reliable Server Pooling (RSerPool) [20] and the results are displayed using GNU R [21]. SimProcTC employs a simpler parallelization methodology than Akaroa and Xgrid, though SimProcTC does allow other softwares to be used for visualization, (e.g., Microsoft Excel, GNU plot, and GNU Octave). In SimProcTC the number of simulation runs is pre-configured and no direct support exists for statistical rigor.
The Statistically Rigorous Simulation (STARS) framework presented in this paper was developed to leverage an existing cluster of 42 IBM dual-processor blade computers for network simulations. The core framework is implemented using the Python language and runs within the MPI-aware interpreter pyMPI. The core of the framework is based on the manager/worker distributed design paradigm and is used to parallelize the processing of user defined tasks in order to achieve practical runtimes for statistically rigorous experiments. Figure 1 presents the overall architecture of STARS as implemented. It is composed of a single manager node, a configurable set of one or more worker nodes, and a centralized storage volume. The manager is responsible for scheduling the processing of user defined experiments and maintaining a recovery record in case of system failure. Each worker processes the tasks issued to it by the manager and reports the results back once completed. Simulation and analysis results are persisted to or retrieved from the centralized storage volume to minimize the disk space required by each node. This generic distributed task processing framework follows standard cluster task management frameworks and is driven by user-defined workloads. These take the form of an experiment process which describes the sequence of tasks to
•
III. STARS OVERVIEW
734
[general] name= resources= modulepath= modulename= results=
Manager
[process] simulation_config=Custom analysis_script=Custom maxruns=100
Data Store
[Script Custom] % MATLAB Analysis Config
Worker 1
Worker N
[Config Custom] % #OMNeT++ Simulator Config
IV. S TATISTICAL T ESTING Fig. 1: STARS Framework Architecture
The core statistical testing currently supported via the STARS framework is to test for stationary and ergodic periods within the simulator produced data, (i.e., the framework automates the production of sufficient sets of stationary or ergodic data). It is assumed that a researcher then performs subsequent analysis on these statistically assessed data sets, (i.e., a researcher’s reported research conclusions, whatever these may be, are based on the analysis of statistically stable periods of network behavior). Moreover, the framework uses a feedback process in which the next set of experiments to be run are automatically deduced via performing statistical tests on the set of prior experiment runs. A researcher can, therefore, task the framework to automatically generate a data set for the selected features X that contains N records of ergodic data each of which have stationary periods of at least tstat. min seconds. The framework then automatically performs a sufficient number of experiments to either: a) produce the required data, or b) validate that the data cannot be produced within the upper bound of the number of experiments that a researcher has specified. It should be clearly noted, that the framework, at this time, does not seek to re-tune any of the experimental parameters such that the desired data becomes producible. Instead, it merely seeks to statistically assess the behaviors of the measurement features of interest defined by a researcher experiment’s configurations. The nature of the stationarity and ergodicity testing is best explained by considering a particular generic feature of interest, denoted X, that the researcher has instrumented the simulator to measure. Denote the ordered sequence of data regarding X produced during the nth run of the given simulation experiment by X n = {#xj , tj $ |j = 1, . . . , Jn }, where tj is the time step and tj+1 > tj for all j ∈ {1, . . . , Jn − 1}. Denote the ensemble of data produced by the N experiments as X = {Xn |n = 1, . . . , N }. For simplicity, the xj will be assumed to be scalar, though the subsequent discussions can be trivially extended to the vector case. For each n ∈ [1, N ] it is assumed that different random seeds are used, (i.e., the goal is not to exactly reproduce the same experiment, which is uninteresting statistically, but instead to assess the statistical nature of the experiments). Standard network simulators perform event-based simulations. Hence, the sample time τ n = tj+1 − tj associated with
Workfile
Meta Experiment Control
Next Exp. Configuration Information
Network Simulation(s) Statistical Analysis Results
Measured Features
Fig. 2: Block diagram of the STARS experiment Process, inclusive of automatically producing the next set of experiments depending on the results of the analysis stage.
be issued to the framework’s manager node and a workfile which contains the experiment’s specific configuration. The experimental process presented in this paper is depicted in Figure 2, where for the purpose of this work a set of MANET simulations (OMNeT++) are controlled and analysis of feature data is done via integration of the framework with Matlab. The key extension of this framework over traditional cluster management systems is that the analysis components assesses whether or not additional experiments need to be run to collect a set of ergodic data of the size the researcher has specified. If the current experiment runs have failed to produce a sufficient set of ergodic data, then the next set of experiments are automatically initiated by the framework without researcher intervention. This process then continues until either: a) a sufficient set of ergodic data is produced or b) a researcher set upper limit on experiment runs is reached. The general form of a researcher-provided experiment configuration file defining a particular MANET simulation is shown below, where the details specific to the OMNeT++ simulation itself and the features to be tested by Matlab for stationarity and ergodicity have been left out for clarity, (i.e., both involve simulator-level details that are not pertinent to the STARS framework itself).
735
any recorded X n sequence is itself a random variable drawn from, in general, an unknown distribution. The stochastic nature of the simulations, implies that exactly when interesting areas within an experiment occur will vary from experiment run to experiment run. This creates the fundamental signal processing problem that the X n records cannot be directly compared as their sampling times τ n are independent random variables. If a researcher only cares about X’s behavior over very long intervals, (i.e., over periods T where T >> max[τ n |n = 1, . . . , N ]), then the randomly sampled X n records can be converted to constantly sampled X n (kT ) records as follows: Xn (kT ) = G[{#xj , tj $ ∈ Xn |tj ∈ [kT, (k + 1)T )}],
Experiment Ensemble
{
x1(t) x2(t)
xN(t) 0 sec
Texp
t
(a) The ensemble of records produced by N experiment runs for measurement feature X.
Direction of KS-testing
5 4
(1)
p 3
where G[.] can be any researcher chosen function that maps the set {#xj , tj $ ∈ Xn |tj ∈ [kT, (k + 1)T )} to a point in X’s domain. In general, a typical example for such a function would be just replacing the !Nset of tuples with its statistical average, (i.e., G[.] = N1 i=1 xi ). Alternatively, G[.] could denote xj ’s maximum value within the given interval. Hence, the Xn records can then be treated following the signal processing standard assumptions. This approach though explicitly cannot be used if a researcher is interested in behaviors that are exhibited over short time intervals, (i.e., where T ≈ max[τ n |n = 1, . . . , N ] exist as an upper bound on the time periods of interest). Unfortunately, this tends to be the case for many of the MANET features generally of interest; hence, the above simplification cannot be used and the X n must be treated as stochastically sampled data records. Hence, for the following discussions regarding the stationarity and ergodicity testing we will presume X n = {#xj , tj $ |j = 1, . . . , Jn }, and that τn is a random variable of unknown distribution and, moreover, that this distribution varies with n.
2 1 0 0
q
time
}W }W }W }W }W }W
(b) The stationarity tests Wqp window tiling.
Fig. 3: Illustration of the ensemble of records produced via rerunning a given experiment N times, showing X n ’s random sampling intervals and the window tilings used for stationarity testing.
resolution level and is$ given by W p = 2p ∗ max[τn ] for # T 1 p = 0, . . . , log2 [ exp 2 ] , (i.e., Wn is Xn cut up into windows 2 of width max[τn ] and Wn having windows twice as large , max[p] subdivides Xn into no etc., up to the point where W n less than 2 windows). The Kolmogorov-Smirnov test (KS-test) [22] is then used to test the sequence of windows in W np in reverse order to check whether the sample cumulative distribution function (CDF) for p window has been drawn from the same underlying each Wq−1 probability distribution as the sample CDF from the preceding Wqp window and lies within a 95% confidence interval. The KS-test is used as it allows the comparison of similarity of statistical distributions without needing to presume the analytical form taken by the underlying distribution, (i.e., a requirement, for example, if a Chi-squared goodness-of-fit test was to be used instead). To improve the statistical rigour of the stationarity testing, if p window of data passes the KS-test then the next a given Wq−1 p Wq−2 window is tested against the CDF formed as the average CDF from all of the preceding windows that iteratively passed the KS-test. As is well known in statistical analysis [23], this has the effect of reducing the uncertainty with respect to the estimated CDF to which the next window of data is compared. Hence, iteratively applying this process leads to the testing for stationarity becoming increasingly rigorous as one moves from right to left through the windows of data. This testing process is begun with p = 0 and proceeds iteratively, until a sufficiently large set of X n ’s recorded
A. Testing for Stationarity The tests for stationarity focuses on a single X n record and fundamentally, with respect to network simulations, ask: “When do the start-up transients end?” Prior network simulation-based research has provided evidence that network simulations, particularly MANET simulations, initially have transient behaviors that settle out as the experiment runs, culminating in the experiment entering (or reaching) steady∈ [0, Texp ], state statistical behaviors at some point t stat. n where the range [0, T exp ] denotes the in-simulation run-time of the experiment in seconds. This observation provides the basis for the following stationstatistically arity test, the goal of which is to determine t stat. n for each of the N experiment runs. The overall stationarity testing process explained below is illustrated in Figure 3. Given the above it can be assumed that the tail of the X n records has a high likelihood of being stationary, whereas the head likely is not. Hence, the record"X n is iteratively up # cut $% Texp into a set of windows of data W np = Wq |q = 1, . . . , W p where Wp denotes the employed window width at the p
736
5 4 3 2 1 0
data, (e.g., ≈ 200 seconds) passes the above KS-testing is set of the left boundary of period, at which point t stat. n the window with the lowest q value that passed the KStest. No testing is performed for higher p values as it is assumed that, given the characteristics of the data, if a smaller window width passes stationarity test then larger windows will also pass and, thereby, these will generate similar t stat. n values. Additionally, the tests becomes less rigorous as the window widths increases, given that the uncertainty in the CDF estimates innately increase with increasing window widths, values (i.e., with increasing p), the accuracy of the t stat. n increases with decreasing p. If for all p no windows pass the KS-test over a sufficiently large set of samples then the X n record is marked as being non-stationary over its entire period.
commonly observed MANET features, namely: packet delivery ratio (PDR), normalized routing overhead, and data delivery duration. The reported measured features describe the operation of a 50 host MANET simulated within a 300m by 600m area for a period of 30 minutes (1800 seconds) using the Dynamic MANET On-Demand (DYMO) routing protocol. Each host within the network sends data based on a Pareto distributed on/off application-level traffic model configured to result in a mean total network load of 13.6KBps. The motion of each host is determined by the random walk mobility model [24] with a constant speed of 5m/s and a 15m mobility parameter. The data collected is generated through the use of the STARS framework and consists of 500 repetitions of the simulation configuration, where the only difference between runs is the use of distinct random seeds. The information presented is based upon the results of the stationarity test performed on each of the three measured features to determine for each experiment run, (i.e., the tests performed define tstat. n the transient period as [0, t stat. ) and the stationary period n , T = 1800]). Feature samples which were not as [tstat. exp n observed to exhibit stationary behavior for the given feature anywhere in [0, 1800] were excluded from the results, (i.e., denoted as being non-stationary in their entirety). The PDR feature of the simulated MANET describes the fraction of routeable data packets which successfully reach their destinations. Data packets for which no route is discovered by DYMO are not counted in this feature. Figure 4a gives the CDF of start times for the observed stationary periods based on 412 of the 500 repetitions, (i.e., 88 (or 17.6%) of the runs were non-stationary). From these results it can be observed that out of the 82.4% of samples which passed the stationarity test almost none did so before 200 seconds of simulation time. The mean start time for the stationary interval was in fact over 370 seconds with some exceeding 900 seconds. Hence, this provides strong evidence that even within relatively simple MANET simulations, transient PDF behaviors are more complex then generally presumed. The normalized routing overhead feature describes the ratio of DYMO control packets sent into the MANET to the data packets received by the MANET hosts over the same time period. These two measurements are performed at the networklayer of the simulation as data flows to and from the link-layer. Figure 4b shows the CDF of the observed stationary periods for the normalized overhead feature for the 394 (78.8%) of the 500 experiment runs that exhibited stationarity, (i.e., 106 (or 21.2%) of the runs were non-stationary). Again virtually none of the samples were observed to become stationary at less than 200 seconds. The mean beginning of steady-state statistical behaviors was over 380 seconds with several runs taking over 700 seconds. Again, this provides strong evidence of the complexity of transient behaviors and, moreover, shows that these behaviors can change from measurement feature to measurement feature even within the same ensemble of experiment runs. The data delivery duration feature describes the time taken
B. Testing for Ergodicity The ergodicity testing proceeds after the stationarity testing has been performed; hence for every X n an associated tstat. ∈ [0, Texp ] has been determined for a given set of n windows Wnp . Hence, the KS-test can then be applied down the ensemble on each of the N estimated CDFs produced from the stationary periods identified for each X n record. In general, these per X n stationary CDF should be produced by averaging the CDF’s produced from the data held within each window within the X n record’s given W np stationarity window set. If the stationary data from two X n records is shown, at a 95% confidence interval, to have been drawn from the same underlying distribution by the KS-test then the two X n records are denoted as being ergodic with each other but only within , Texp ] periods. their [tstat. n The set of N ensemble X n records in X are then tested in a pair by pair manner to determine all sets of stationary data which are pair-wise ergodic. It is fully possible that more than one ergodic set of records may exist, (i.e., there may exist distinct modes of steady-state statistical behaviors within a given type of experiment leading to more than one ergodic region within the ensemble X). Currently, the testing framework reports the largest ergodic region found in X as X’s ergodic data. It should be noted that the ergodic (or stationarity) testing must be performed on a feature by feature basis, as ergodicity (or stationarity) for one feature, in general, provides no information about the ergodicity (or stationarity) of another feature. Moreover, if a composite feature is produced by a mathematical operation on two other stochastic features then it becomes important that this composite feature is only computed within the ergodic regions common to the underlying features (or stationary regions if the composite feature is computed on a per Xn record basis). If this is not done then biases may, again, be introduced into the analysis via the inclusion of transient behaviors. V. MANET T RANSIENT T ESTING R ESULTS In the results presented below the STARS framework is used to provide insights into the start-up behaviors of three
737
the need to perform statistically rigorous analysis of network simulation features on a feature by feature basis prior to making conclusions about how a particular MANET may be operating. Of the three features presented only one had any runs that were stationary within the general sub-200 second thresholds typically applied in ad hoc transient removal processes. In most of the experiment run these typical ad hoc thresholds were significantly exceeded. Moreover, clear differences exist when the steady-state statistical behaviors are entered for different features. This highlights that the use of a fixed transient thresholds for all measured features is not a reliable approach. In addition, for each measured feature a number of experiment runs produced non-stationary data, within the case of normalized overhead this accounts for > 20% of the runs. This highlights the issue that MANET simulation results performed with small numbers of runs, (i.e., 10s of runs) have the potential, for some features, to produce quite misleading results particularly if averaging across the runs is used without testing for ergodicity.
packet delivery ratio ï 412 of 500 samples stationary 1 0.9 0.8
CDF ï F(x)
0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 100
200
300
400 500 600 700 800 start of stationary interval (seconds)
900
1000
1100
(a) CDF of Packet Delivery Ratio normalized overhead ï 394 of 500 samples stationary 1 0.9 0.8
CDF ï F(x)
0.7 0.6 0.5 0.4
VI. L IMITATIONS AND F UTURE W ORK
0.3
As discussed above, the current implementation of the framework only provides automated testing support for determining stationary and ergodic periods under researcher provided experiment configurations. As highlighted by the above figures, the statistical behaviors within experiments can be quite sensitive to experiment configurations. The framework does not currently support searching the space of possible configurations to determine, for example: a) the best configuration to use, b) the configuration settings that are least sensitive to changes in the random seeds, or c) whether a configuration exists that would enable the experiments’ expressed behaviors to be contained within some desired statistical envelope(s). Obviously, for the MANET example, routing protocols that have a higher degree of insensitivity to start-up configurations, (i.e., random seeds) are in general more desirable, all else being equal, to protocols with high sensitivities. Extending the testing framework to support such processing is relatively straightforward but it significantly increases the computing resources required to perform the resulting testing; hence, the process is under way to map the framework into the grid and cloud computing APIs that are now commonplace such that these much larger computing platforms can be leveraged for more extensive research programs.
0.2 0.1 0 100
200
300 400 500 600 700 start of stationary interval (seconds)
800
900
(b) CDF of Normalized Overhead data delivery duration ï 473 of 500 samples stationary 1 0.9 0.8
CDF ï F(x)
0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0
200
400
600 800 1000 1200 1400 start of stationary interval (seconds)
1600
1800
(c) CDF of Data Delivery Duration
Fig. 4: CDFs of the time of commencement of steady-state behaviors across 500 MANET experiment runs for the three network features considered here.
VII. C ONCLUSIONS
for data packets to arrive at their destination. The time period begins when the packet is sent from the network-layer to the link-layer after a route has been discovered by the DYMO protocol. Figure 4c shows the CDF of the observed stationary periods for the 473 (94.6%) of experiment runs which exhibited stationary behavior, (i.e., 27 (or 5.4%) of the runs were non-stationary). For this feature 37 of the 500 runs become stationary prior to 200 seconds but the mean value at which stationarity is achieved is just below 740 seconds, with 118 runs taking more than 1200 seconds to become stationary.
This work has provided an overview of a testing framework designed to support statistically rigorous testing within simulation-based network research. In particular, the work highlighted, through the discussed MANET simulation results, the need for statistical rigour within network simulations. It showed through a set of DYMO MANET simulations that the standard ad hoc approach of seeking to eliminate start-up transient via applying fixed thresholds is an insufficient approach to achieve the desired end-goal. Instead, the results presented demonstrate the need to perform rigorous stationarity and
This simple set of MANET experiments clearly highlights
738
ergodicity testing on a feature by feature basis if the transient effects are to be accurately removed from the subsequent downstream analysis. Although the current STARS framework has been tailored to use of an existing small scale research cluster, work is underway to extend the STARS framework such that it can seamlessly make use of available cloud and grid infrastructures.
[20] “Reliable Server Pooling.” [Online]. Available: http://en.wikipedia.org/ wiki/Reliable server pooling [21] “GNU-R.” [Online]. Available: http://sonic.net/∼ rknop/php/linux/R.php [22] “Kolmogorov-smirnov test.” [Online]. Available: http://www.physics. csbsju.edu/stats/KS-test.html [23] J. S. Bendat and A. G. Piersol, Random Data: Analysis and Measurement Procedures. Wiley-Interscience; 3rd edition, 2000. [24] M. McGuire, “Stationary distributions of random walk mobility models for wireless ad hoc networks,” Proceedings of the 6th ACM international symposium on Mobile ad hoc networking and computing, MobiHoc ’05, pp. 90–98, 2005.
ACKNOWLEDGMENT This research has been support by grant from the Natural Sciences and Engineering Research Council of Canada (NSERC), the Canadian Innovation Foundation (CFI), the British Columbia Knowledge Development Fund (BC KDF), and generous equipment support by IBM Canada Inc. R EFERENCES [1] “The network simulator-ns2.” [Online]. Available: http://www.isi.edu/ nsnam/ns/ [2] “OPNET.” [Online]. Available: http://www.opnet.com/ [3] “OMNeT++.” [Online]. Available: http://www.omnetpp.org/ [4] “Emulab: Network emulation testbed home.” [Online]. Available: http://www.emulab.net/ [5] “PLANETLAB: An open platform for developing, deploying and accessing planetary scale services.” [Online]. Available: http://www. emulab.net/ [6] G. Koltsidas, S. Karapantazis, G. Theodoridis, and F. Pavlidou, “A detailed study of dynamic manet on-demand multipath routing for mobile ad hoc networks,” International Conference on Wireless and Optical Communications Networks, IFIP, pp. 1–5, Jul. 2007. [7] G. Koltsidas, F.-N. Pavlidou, K. Kuladinithi, A. Timm-Giel, and C. Gorg, “Investigating the performance of a multipath DYMO protocol for adhoc networks,” Proceedings of IEEE 18th International Symposium on Personal, Indoor and Mobile Radio Communications, pp. 1–5, Sep. 2007. [8] Peebles Peyton Z. Jr., Probability, Random Variables, and Random Signal Principles. McGraw Hill, 2001. [9] V. Paxon and S. Floyd, “Wide-area traffic: The failure of poisson modeling,” Proceedings of IEEE/ACM Transactions on Networking, vol. 3, no. 3, pp. 226–244, June 1995. [10] M. Crovella and A. Bestavros, “Self-Similarity in World Wide Web Traffic: Evidence and Possible Causes,” Proceedings of IEEE/ACM Transactions on Networking, vol. 5, no. 6, pp. 835–846, Dec. 1997. [11] “Cloud versus cloud: A guided tour of Amazon, Google, AppNexus, and GoGrid,” [Online]. Available: http://www.infoworld.com/d/cloud-computing/ cloud-versus-cloud-guided-tour-amazon-google-appnexus-and-gogrid-122. [12] A. J. H. Fran Berman, Geoffrey Fox, Grid Computing: Making the Global Infrastructure a reality, 1st ed. Wiley Interscience, 2003. [13] “Cluster computing grid.” [Online]. Available: http://www.ccgrid.org/ [14] M. Thoppian, S. Venkatesan, H. Vu, R. Prakash, and N. Mittal, “Improving performance of parallel simulation kernel for wireless network simulations,” in IEEE Military Communications Conference, MILCOM 2006, Oct 2006, pp. 1–6. [15] T. Dreibholz, “SimProcTC: A powerful tool-chain for setup, distributed processing and analysis of OMNeT++ simulations,” March 2009. [Online]. Available: http://www.iem.uni-due.de/∼ dreibh/omnetpp/ [16] S. Sroka and H. Karl, “Using Akaroa2 with OMNeT++,” Proceedings of the Second international OMNeT++ workshop, pp. 43–50, Jan. 2002. [17] R. Seggelmann, I. Rungeler, M. Tuxen, and E. P. Rathgeb, “Parallelizing OMNeT++ simulations using xgrid,” Proceedings of the 2nd International Conference on Simulation Tools and Techniques, pp. 1–8, 2009. [18] S. Lin, X. Cheng, and J. Lv, “A visualized parallel network simulator for modeling large-scale distributed applications,” in Eighth International Conference on Parallel and Distributed Computing, Applications and Technologies, PDCAT ’07, Dec. 2007, pp. 339–346. [19] A. Varga and A. Sekercioglu, “The OMNeT++ discrete event simulation system,” Proceedings of the European Simulation Multiconference, 2001.
739
