Features within MANETs. Deepali Arora, Eamon Millman and Stephen W. Neville. Department of Electrical and Computer Engineering. University of Victoria.
2011 IEEE 22nd International Symposium on Personal, Indoor and Mobile Radio Communications
On the Statistical Behaviors of Network-level Features within MANETs Deepali Arora, Eamon Millman and Stephen W. Neville Department of Electrical and Computer Engineering University of Victoria P.O. Box 3055 STN CSC, Victoria, B.C., CANADA, V8W 3P6 {darora, emillman, sneville}@ece.uvic.ca
Abstract—Event-based simulation has become a primary means of pursuing mobile ad hoc network (MANET) research. The stochastic nature of MANETs has been well studied with respect to mobility models, but less work has looked at the statistical behaviors of network layer features, (e.g., PDR, delay, hops and routing overhead). Fundamentally, issues such as “When do start up transients end?” and “Do all MonteCarlo runs indeed arrive at the same steady-state distributions?” have not been well explored. This work explores these issues through using the DYMO routing protocol and the OMNeT++ simulation framework as exemplars. By applying distribution free Kolmogorov-Smirnov goodness-of-fit tests it is shown that, for network-layer features: a) MANET start-up transients can persist far longer than previously reported, b) transient durations can vary significantly from feature to feature and with varying node velocities, and c) Monte-Carlo runs of a given MANET scenario can produce distinct behavioral modes. It is then discussed whether these issues are likely inherent to MANETs and their routing protocols or an artifact of OMNeT++.
I. I NTRODUCTION The unfolding wide-scale growth of the smart-phone market is in the process of enabling pragmatic real-world deployment environments for MANETs and MANET based applications and services. The quality of these applications and services will innately rest on the behavior of the underlying MANET network; hence, these behaviors need to be well understood. Currently, much of the existing MANET literature has been simulation based, (i.e., through tools such as ns-2 [1], OpNet [2], and OMNeT++ [3]). Obviously, a number of arguments exist as to how accurately such simulations represent the realworld. In most cases an approach that fails in simulation will, with high likelihood, also fail in the real-world. Hence, simulation becomes a useful tool in assessing whether or not the expense of real-world testing is warranted. Of course, that a technique works in simulation is an insufficient basis to claim that it would work in the real world. In light of this perspective, this work seeks to better understand the scope of statistical behaviors expressible within network-level MANET features where this understanding is then informative for both simulation-based and real-world deployment based MANET research. Significant research has explored the statistical nature of MANET mobility models [4]. But, far less research has sought to explore the statistical behaviors of MANETs’ network-layer features. Instead, the general approach has been to address
978-1-4577-1348-4/11/$26.00 ©2011 IEEE
the known issue of MANET start-up transients by merely excising the first X seconds of simulated run-time and then reporting MANET performance results as averages across a set of Monte-Carlo runs, (i.e., generally in the 10s to 100s of runs). This leads to the general statistical questions of: a) What guarantee exists that the start-up transients have indeed been removed?, and b) What guarantee exist that the averaging is being performed over data drawn from the same underlying distributions? This work seeks to directly test these questions. This is done by using DYMO as the exemplar MANET routing protocol and OMNeT++ as the exemplar event-based simulation platform. Early work showed a discrepancy between NS-2 and OMNeT++ simulations [5] but nearly a decade of maturation in both simulators has occurred since, leading to more recent work showing consistency between these simulation platforms [6]1 . Overall, this work can be seen as taking a step back from general MANET research directions and instead seeking to understand more fully the statistical behaviors expressed by network-level MANET features. Obviously, statistical characteristics in Monte-Carlo MANET simulations can arise due to either: a) MANET-level effects, (i.e., routing protocol, node mobility, , etc.), or b) artifacts of the simulator. To mitigate the likelihood of the latter this work focuses on: i) the well studied DYMO routing protocol, ii) the random walk mobility model [7], which specifically addresses the random waypoint models known non-uniform statistical artifacts [4], iii) the very simple free space wireless propagation model, and iv) OMNeT++ as a well used event-based network simulation research platform. Hence, simulator or protocol implementation deficiencies that would have given rise to the observed statistical characteristics would, most likely, have already been identified and removed by the active research community that uses OMNeT++, as these would tend not to exist as subtle defects. As such, the issues highlighted in this work are unlikely to represent easily fixable simulation (or simulator) defects, as was the prior case for the random waypoint mobility model. Instead, they suggest that core characteristics exist within MANET routing protocols that make their networklevel features prone to the observed statistical complexities. 1 It should also be noted that [5] compared single runs of the NS-2 and OMNeT simulators for a particular scenario. Hence, it does not seek to present a general statistical comparison of NS-2 and OMNeT++’s overall consistency (or lack thereof)
1145
In turn, this implies that routing protocols designed to reduce the observed statistical complexities may be researchable and implementable. To be clear, though, the objectives of this work are solely to highlight the statistical issues and not to directly assess or resolve their causes, the latter being precluded by space limitations. To our knowledge, prior works have not sought to assess the statistical behaviors of network-level features within MANETs, irrespective of the simulator used. Moreover, these issues are harder to expose in real-world testbeds due the innate difficulties in achieving the levels of control and repeatability required for statistically rigor across MonteCarlo runs. Within this work the distribution-free two sample Kolmogorov-Smirnov goodness-of-fit test is used to: a) directly test, within a simulation run, when (and if) networklevel MANET features, (e.g., PDR, delay, hop count, routing overhead) enter their steady-state distributions, and b) across ensembles of multiple simulation runs, to test these discovered steady-state distributions for their statistical similarity to each other, thereby identifying modes of behavior that occur across the ensemble. This work highlights that: a) start-up transient durations can persist for far longer than is generally assumed, b) these transient durations can vary from network feature to network feature and with mobility model velocities, and c) generally, although steady-state distributions do indeed tend to occur, a minority of experiments never enter a steady-state. These results imply that substantial statistical care must be taken when engaging in MANET research to formally assess: a) statistical stationarity2 . and b) statistical ergodicity3. Additionally, this formal statistical testing must be done on a feature by feature basis. Fortunately, the advent of low-cost easily accessible high performance computing (HPC) and cloud computing place the computation resources required for such testing within easy access of the average researcher. The outline for the work is as follows. Section 2 places this work within the context of the related works in the area. Section 3 presents the details of the performed MANET simulations. Section 4 introduces the statistical testing process that was applied. Section 5 presents the results obtained through applying this process to assess the network-layer data produced from multiple DYMO experiment runs. Section 6 concludes the work. II. R ELATED W ORK This work focuses directly on assessing the statistical behaviors of DYMO MANETs, although the issues brought forward by this work are by no means unique to DYMO. Hence, of primary interest and relevance are the significant number of bake-off performance studies which have focused on assessing DYMO against other routing protocols. These 2 A statistical process is stationary if statistical properties are independent of time [8]., (e.g., a process is called wide sense stationarity if its mean and standard deviation are time independent) 3 A statistical process is termed ergodic if its time averaged statistics equal its ensemble averaged statistics; hence, ergodicity by its nature implies stationarity [8]. Pragmatically, a lack of ergodicity means that the expressed statistical behaviors across Monte-Carlo runs vary from run-to-run.
prior studies have tended to be carried out for relatively short simulated run times(≤ 500 seconds) and have either not stated how start-up transients have been addressed [9]–[11] or applied ad hoc approaches to address these transients [12]. As with [13], a general approach has been to apply averaging across Monte-Carlo runs to deduce general trends but without first confirming that the averaged data is indeed statistically ergodic. The main conclusion drawn by these studies, from a performance perspective, is that DYMO can provide higher network throughputs but at the cost of increased network delays. While these studies have provided insight into DYMO’s behavior, questions remain as to whether the results are indeed: a) free of start-up transient effects, as claimed, and b) that all averaging has been meaningfully applied4 . This work applies KS goodness-of-fit testing to directly test issues (a) and (b). Significant research has also sought to assess the degree of realism that MANET simulations may or may not provide. All simulations, in general, apply simplifying abstractions to achieve tractability. Success in simulation provides no guarantee of real-world success. But, failure in simulation is nearly always predictive of real-world failure. Moreover, true Monte-Carlo runs are inevitably hard to enact in real-world settings, given the innate difficult in applying the level of control and repeatability required by the scientific method, (i.e., outside of highly constrained and, therefore, highly atypical environments). Assessing the underlying statistical characteristics of network-level MANET features is, therefore, best pursued via simulation. If statistical issues are shown to exist even in quite simple simulation scenarios, then it is difficult to argue that such issues will cease to exist once the added complexities of real-world deployments are introduced. From this perspective, the issue of the realism of MANET simulators (and simulations) is outside of the context of this work. III. S IMULATION D ETAILS The conducted DYMO [14] simulations used a mixture of off-the-shelf resources and custom tools. The simulation model defining network area, node density, mobility, input traffic parameters, communication protocols and channel characteristics was implemented via the OMNeT++ event-based simulation engine. The OMNeT++ DYMO simulator was then wrapped within a custom MPI-based job control framework, designed to automate the support for the required statistical testing [15]. This job control framework allowed the significant number of conducted simulations to be run within a cluster of 28 dual processor CentOS 3.0 GHz IBM blade machines. Each MANET simulation itself consists of a standard MANET model of 50 nodes moving within a 300m × 600m area, where each node has a transmission range of 98 meters, as per [16], [17]. All MANET nodes have their movements controlled via the random walk mobility model [7]. The radio propagation 4 As a trivial example, if samples are drawn from two normal distributions, distributed as N (0, 1) and N (10, 1), are averaged then the sample mean will be 5, whereas this value occurs rarely in either distribution. This highlights the innate problem with applying averaging across differing underlying distributions, (i.e., across non-ergodic sets).
1146
IV. S TATISTICAL ANALYSIS The core questions of when start-up transients end and when averaging can be meaningfully applied can be directly answered via applying statistical goodness-of-fit tests to the generated MANET simulation data, (e.g., the Kolmogorov Smirnov (KS) test, the Pearson’s Chi-square test, or the Anderson Darling test [18]). Within this work, the underlying analytical form of the empirical distributions are not known, (i.e., a distribution-free test is required), thereby, precluding the use of the Chi-squared test. For computational efficiency the KS-test was chosen, although it is a weaker test than the Anderson Darling test, (i.e., passing the KStest does not imply that the Anderson Darling test would also be passed). Moreover, the KS-test assesses the maximum difference that can exist between two empirical cumulative density functions (CDFs) if the distributions are to be seen as statistically indistinguishable from each other under a given confidence level, selected as 95% for all tests. Whereas, the Anderson Darling test would focus on assessing the cumulative difference between empirical CDFs. To be clear, passing the KS-test for two empirical CDF’s merely denotes that under the given confidence level no statistical evidence exists that the two CDFs are different from each other. Passing the test expressly cannot be interpreted as evidence that the two CDFs are the same, as this requires knowing the analytical distributions. With only empirical CDFs one can never answer whether the two distributions are indeed the same. One can only answer whether evidence exists that they are different, (e.g., with low probability any given empirical CDF could be drawn from both a Gaussian and uniform CDF and, hence, the KS-test could be passed). Hence, when the KS-test is used in this work to assess statistical stationarity and statistical ergodicity the precise statement associated with passing the given KS-test is that no statistical evidence exists to claim that the given data sets are, respectively, non-stationarity or do not exhibit ergodicity. As is well known, it is innately impossible to ever formally prove that data (or signals) are either stationarity or ergodic [19]. Hence, the KS-testing process can only be used to prove the negative hypotheses. Therefore, passing the KS-test, within this work, implies a lack of evidence for the negative hypothesis and expressly not that sufficient evidence exists to support the positive hypothesis. A. Testing for stationarity The issue of identifying when the start-up transients end within a given experiment can be directly viewed as seeking to identify when the underlying distribution for the measured random process enters into its steady-state distribution. More particularly, assumed that X is the measurement feature of interest, then the collected measurement samples for X during an experiment run n can be denoted as Xn = {hxk , tk i|k = 1, . . . , Kn }, where xk denotes the measurement sample and
Experiment Ensemble
model is a free-space path loss model with path loss parameter α = 2 and, for simplicity, only constant bit rate (CBR) traffic is considered.
{
x1(t) x2(t)
xN(t) t
0 sec
Texp
(a) The ensemble of records produced by N experiment runs for measurement feature X. Direction of KS-testing
}W }W }W }W }W }W
5
5
4
4
p 3
3 2
2
1
1
0
0 0
q
time
(b) The stationarity tests Wqp window tiling. KS-test CDF
CDF
KS-test CDF
CDF
KS-test Xn(t)
CDF
CDF
CDF
CDF
tnstat
(c) The KS-test stationarity testing process as applied right-to-left along an Xn record for a given set of W p windows.
Fig. 1: Illustration of the ensemble of records produced via rerunning a given experiment N times, showing Xn ’s random sampling intervals and the window tilings used for stationarity testing.
tk denotes the time, in simulated time, when the given measurement was made. For simplicity it can be assumed that all experiment runs begin at t = 0 and end at t = Texp . Hence, ∀n and ∀tk ∈ Xn tk ∈ [0, Texp ]. Due to the event-based nature of the simulations, the per-experiment sample time τn given by τn = tk+1 − tk will itself also be a random variable of, generally, an unknown distribution. Xn is therefore a two dimensional random process as its xk values can be viewed as having been drawn from an underlying cumulative distribution Pn (x). But, as start-up transients are known to exists for MANET simulations, this cumulative distribution is itself time dependent and more accurately denoted as Pt,n (x), where t ∈ [0, Texp ]. Within MANET research it is generally assumed that Xn ’s underlying distribution
1147
evolves toward a steady-state distribution. Hence, there should exist a tstat ∈ [0, Texp ] past which Pt,n (x) = Pn (x), (i.e., n past which Pt,n (x) becomes time independent5). Therefore, stat the time periods [0, tstat n ] and [tn , Texp ] will exist where Xn ’s underlying distribution, respectively, is statistically nonstationary and, formally, where no evidence exists that it is not stationary. Following general conventions, the latter will be termed Xn ’s statistically stationary period. Assessing the end of the MANET’s per-experiment start-up transients can be seen as equivalent to testing Xn for stationarity, (i.e., empirically deducing tstat n ). The Pt,n (x) distributions cannot be tested directly as only the measurements contained in Xn exist. Hence, the Xn record must be divided into non-overlapping windows of time, denoted as Wn = {Wq,n |q = 1, . . . , Qn }, where for q = 2, . . . , Qn the windows are of equal width and the W1,n widows holds any remaining data. Empirical CDFs can then be generated from the data in each Wq,n window, where these CDFs are denoted by Pˆq,n (x). The window width is chosen based on the observed numbers of xk events in Xn to ensure that every window, excluding the first window, contains sufficient data such that reasonable Pˆq,n (x) estimates are produced, (i.e., there are not so few hxk , tk i samples in Wq,n that the uncertainty in Pˆq,n (x) would be overly high6 ). The statistical similarity of adjacent empirical CDFs can then be assessed via the two sample KS-test, which assesses the maximum difference between the observed Pˆq,n (x) and Pˆq−1,n (x) empirical CDFs7 . More particularly, this KS-testing process is conducted from right-to-left, (i.e., from q = Qn to q = 1) given the presumption within MANET’s that Xn will evolve over time towards its steady-state distribution. Additionally, to improve the statistical quality of the analysis, if Pˆq,n (x) and Pˆq−1,n (x) pass the KS-test then they are averaged together to give a new averaged empirical CDF Pˆq¯,n (x) which is then used in a one side of the next KStest. Each time the KS-test is passed the averaged empirical CDF Pˆq¯,n (x) is updated, (i.e., Pˆq¯,n (x) denotes a running average). The effect of this averaging-in of the individual Pˆq¯,n (x) estimates is that the created Pˆq¯,n (x) has a lower variance, (i.e., is a better estimator) than would be the case if Pˆq¯,n (x) was simply formed from the composite of all of the data to the right of the current Pˆq,n (x) to be tested. The basic shape of empirical CDFs under the two methods will be the same. But, the first method’s Pˆq¯,n (x) estimate will have a variance of σ/K where K is the number of CDFs averaged-in wheres the second approach would have a variance of σ [19]. This full process is then iteratively repeated from rightto-left until the first KS-test fails for some window Wq,n at which point tstat is set to the minimum time present in n the Xn records used to create Pˆq¯,n (x). In this manner the overall testing process is conservative in that stationarity, or 5 It should be noted that if P t,n (x) = Pn (x) then all statistics that can be calculated on Pt,n (x) will also be time independent. 6 Assessing the true uncertainty in P ˆq,n (x) requires that Pt,n (x) be known, which is not. Hence, pragmatically the Wn are set such that each Wq,n for q > 1 has at least 150 observed data samples. 7 Other stricter distribution free goodness-of-fit tests, such as the AndersonDarling test, could be used, but goodness-of-fits test such as Pearson’s χ2 -test, requiring analytical knowledge of the Pt,n (x) cannot be used.
more precisely no evidence of non-stationarity, exists for all for which Pt,n (x) would but there may exist t < tstat t ≥ tstat n n also pass the stationarity test. Of course testing for such points would require knowledge of Pt,n (x) which is not available. Moreover, to date, no generally accepted theoretic arguments exists within the MANET literature as to the true analytical nature of Pt,n (x). It is fully possible that no KS-tests will pass for any of the constructed empirical CDFs. In this case the window widths are increased by a power of 2 and the full KS-testing process is repeated under the new larger windows. More generally, a set of W P = {Wnp |p = 1, . . . , P } are produced where the widow widths increase by powers of 2 for each p. Care is taken to ensure that P is set such that |WnP | ≥ 5 to preserve the meaningfulness of the KS-testing process. The first p for which there exist empirical CDFs passing the KStesting is used to define tstat n . Hence, the testing process seeks to identify the finest resolution tstat possible while preserving n the meaningfulness of the statistical tests. Innately, the overall process can be seen as seeking to balance the issues that: i) as window size decreases uncertainty in the Pˆq,n (x) estimates increases, due to the decreasing number of data points available to construct each Pˆq,n (x) estimate, and ii) as window sizes increase the variance in the Pˆq¯,n (x) estimate increases as less averaging occurs and, moreover, the available resolution for the tstat estimate decreases. Hence, n preference in deducing tstat is given to the smallest window n width, (i.e., p level) for which passed KS-tests exists. It remains fully possible that for a given Xn record no KStests are passed for any of the P windows widths. Such a result denotes an experiment run where Xn never entered into a steady-state distribution during the experiment’s run time, (i.e., a case where the start-up transient(s) persisted and never settled out). The occurrence of such Xn records and their frequency denotes the degree to which highly undesirable MANET configurations may exist. In general, for an arbitrary dynamic random process there is no guarantee that it must evolve towards a steady-state distribution. Instead, significant care would need to be taken to ensure that such a desirable evolution would be guaranteed. Hence, as MANET routing protocols have not been designed with this aspect in mind, it is not surprising that, when sufficient numbers of Monte-Carlo simulations are run and statistically rigorous testing is applied, there may exist runs for which no steady-state distributions are observed. The more interesting result, as shown below, is that for the large majority of MANET runs steady-state distributions are indeed observed. The full stationarity testing process is illustrated in Figure 1. It should also be noted that, as the KS-testing is only ever performed on the current window’s data against the data held across the composite of all succeeding windows, by definition, the stationarity testing process expressly does not constitute or involve multiple hypothesis testing. B. Testing for ergodic sets across Monte-Carlo runs No guarantee exists for any ensemble of Monte-Carlo runs X = {Xn |n = 1, . . . , N } that all of the observed steadystate distributions Pˆq¯,n (x) will themselves be the same. More
1148
1
TABLE I: Number (percentage) of experiments exhibiting stationary periods out of an ensemble of 500 Monte-Carlo runs as a function of node velocity and MANET feature. 0m/s 5m/s
Stationarity √ × √ ×
PDR 425 75 421 79
Hops travelled 500 0 483 17
Delay 334 166 415 85
0.8 0.7
CDF−F(x)
Velocity
Vel = 0 m/s Vel = 5 m/s
0.9
NRO 500 0 474 26
0.6 0.5 0.4 0.3 0.2 0.1 0 0
200
400
600
800
1000
1200
1400
Start of stationary intervals for PDR (seconds)
(a) CDF of
tstat n
for PDR.
1
V. P ERFORMANCE
MEASURES AND SIMULATION RESULTS
Within this work four standard network layer MANET features are considered, namely packet delivery ratio (PDR), delay, hops travelled, and normalized routing overhead (NRO) [15]. Table 1 shows the number (percentage) of the 500 runs that exhibited stationary CDFs for these four features. It is clear that for certain features a significant number of experiments never enter steady-state CDFs, (i.e., the “startup” transients persist throughout the 1800 seconds of simulated MANET run time). Figure 2 further illustrates how the CDFs of the tstat are themselves feature and velocity n dependent. Moreover, transient behavior can clearly persist well past the traditional ad hoc transient cut-off times (denoted in gray). This provides strong evidence that the current ad hoc approaches to excise transients behaviors are insufficient. Instead, the results highlight the need to apply significant statistical care to ensure transient effects are indeed removed. Moreover, the fact that these complex statistical behaviors are
Vel = 0 m/s Vel = 5 m/s
0.9 0.8
CDF−F(x)
0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
200
400
600
800
1000
1200
1400
1600
1800
2000
Start of stationary intervals for hops travelled (seconds)
(b) CDF of tstat for hops traveled. n 1 Vel = 0 m/s Vel = 5 m/s
0.9 0.8
CDF−F(x)
0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
200
400
600
800
1000
1200
1400
1600
1800
2000
Start of stationary intervals for delay (seconds)
(c) CDF of
tstat n
for delay.
1 Vel = 0 m/s Vel = 5 m/s
0.9 0.8 0.7
CDF−F(x)
particularly, it could be the case that within the Monte-Carlo ensemble different modes of MANET behaviors are expressed. This would denote a MANET that is sensitive to its initial conditions, (i.e., the steady-state distribution to which the MANET evolves would depend on the MANET’s initial state). Via the above statistical stationarity testing process, the steady-state estimates, Pˆq¯, n(x), of stationary empirical CDFs were produced for each of the K experiments where tstat n values were found. Hence, these Pˆq¯,n (x) can be compared on a pair-by-pair basis via KS-testing. In this manner their KS-test p-values can be determined for all pairs that pass the KS-test, again at a 95% confidence level. These p-values can then be used to define the edges on a completely connected graph where the nodes denote the Monte-Carlo runs for which Pˆq¯,n (x) values exist. Modes exhibited within the MANET’s stationary behavior can then be detected as cliques within this p-value graph. By definition, these cliques will form sets of Pˆq¯,n (x)’s for which, on a pair-by-pair basis, no evidence exists that any of the Pˆq¯,n (x)’s are statistically dissimilar, (i.e., the cliques can be viewed as denoting ergodic sets from the ensemble of Monte-Carlo runs). As this process innately denotes a clustering process, it also expressly does not involve or require multiple hypothesis testing. Whereas, if a final test on the deduced cliques was applied , (i.e., to assess whether or not the cliques themselves were ergodic) then multiple hypothesis testing would need to be applied. In this work, though, we never assess any properties of the cliques, stopping instead at clique formation. Hence, multiple hypothesis testing is not required.
0.6 0.5 0.4 0.3 0.2 0.1 0 0
200
400
600
800
Start of stationary intervals for normalized routing overhead (seconds)
(d) CDF of
tstat n
for NRO.
tstat n
Fig. 2: CDFs of for the network features considered which indicates the commencement of steady state behaviors for a 50 node network for node velocities equal to 0 and 5 m/s.
exhibited in the very simple MANET scenarios studied suggest that real-world MANETs may be quite sensitive to their startup conditions, which in turn would make the engineering of MANETs significantly more complex. Table 2 highlights when ergodic sets across 15 Monte-Carlo runs for the 50 node MANET for PDR and NRO8 . This table shows that ergodicity never occurs for data produced in the first 600 seconds of simulated runtime. PDR exhibits ergodicity in the Monte-Carlo ensemble at 900 seconds of runtime but only until node velocities exceed 5.0 m/s at which point ergodicity is lost until 1200 seconds of runtime is reached. For NRO, no ergodic runs are produced until 900 seconds of runtime and then only for velocities > 5.0 m/s. 8 A smaller set of Monte-Carlo runs was required for the ergodicity analysis due to hardware limitations.
1149
TABLE II: Ergodicity across an ensemble of 15 experiments runs across different [tstat n√ , 1800] intervals for a 50 node MANET. (Check marks ( ) denoted the existence of ergodic records whereas cross marks (×) denote the lack of ergodic records.) Sim. Run-time 300 450 600 900 1200 1500
Feature PDR NRO PDR NRO PDR NRO PDR NRO PDR NRO PDR NRO
Avg. Node Velocity (m/s) 0.5 5.0 10.0 15.0 × × × × × × × × × × × × × × × × × × × × × × × × √ √ × × √ √ × × × √ √ √ √ √ √ √ √ × × √ √ √ √ √ √ √ √ √ √
0.0 × × × × × × √
negligible subsets of Monte-Carlo runs may be non-stationary throughout, and d) that direct testing for statistical ergodicity in Monte-Carlo runs is required prior to applying ensemble averaging. These results imply that the typical approach of ad hoc start-up transient removal followed by Monte-Carlo run averaging is statistically insufficient and could engender misleading results. As a very simple MANET scenario was studied the observed results are likely inherent to MANETs themselves and not addressable via simple fixes to the simulator or simulation process. This implies that real-world non-trivial MANETs may turn out to be significantly more challenging to engineer that previously suspected. R EFERENCES [1] [2] [3] [4]
At 1200 seconds of runtime NRO records are ergodic for velocities > 0.5 m/s and at 1500 seconds of runtime they are always ergodic. Hops travelled is not shown in the table as they were always ergodic for all tested cases. These results clearly show that ergodicity across Monte-Carlo runs is feature dependent, runtime dependent, and node velocity dependent. Hence, merely averaging across Monte-Carlo runs in MANET research could easily produce mis-leading results. Hence, formally statistically testing for ergodicity is a necessity prior to applying ensemble averaging. These results introduce many follow up questions, such as: Exactly why do the non-stationary runs exist?, (i.e., Why is convergence to a steady-state distribution not guaranteed?), Are the issues simulator caused or inherent to MANET operations?, How do these issues vary with different MANET protocols? , etc.Space limitations preclude a full discussion of these issues nor is it the intent of this work to assess the underlying rationale for the observed issues, as these are avenues of future work. It should be noted though that this work assess statistical behaviors in the context of a very simple MANET scenario. This reduces the possibility that a core simulator defect is the underlying cause of these behaviors. This is further supported in that when the pernode communications distances are increased to more typically 150m-200m distances, thereby reducing the MANET to a 1 to 2 hop network, many of the statistical complexities fade. This is consistent with the fading of these issues for the low velocity tests presented in this work, (i.e., larger communication distances effectively render the MANET as nearly static in a mobility sense when viewed from the network protocol level, as opposed to the mobility model view).
[5]
[6] [7]
[8] [9]
[10] [11] [12]
[13]
[14] [15]
[16]
[17]
VI. C ONCLUSIONS This work has applied a formal KS-testing process to assess the statistical behaviors of standard network-level MANET features within a multi-hop network. This testing showed that: a) start-up transients can persist for far longer then previously reported, b) transient duration statistics can change considerably from feature to feature and with node velocities, c) non-
[18]
[19]
1150
The network simulator-ns2. [Online]. Available: www.isi.edu/nsnam/ns/ Opnet. [Online]. Available: http://www.opnet.com/ Omnet++. [Online]. Available: www.omnetpp.org/ J. Yoon, M. Liu, and B. Noble, “Random waypoint considered harmful,” Twenty-Second Annual Joint Conference of the IEEE Computer and Communications, INFOCOM, vol. 2, pp. 1312–1321, March 2003. D. Cavin, Y. Sasson, and A. Schiper, “On the accuracy of manet simulators,” in Proceedings of the second ACM international workshop on Principles of mobile computing, ser. POMC ’02. New York, NY, USA: ACM, 2002, pp. 38–43. [Online]. Available: http://doi.acm.org/10.1145/584490.584499 T. Andel and A. Yasinsac, “On the credibility of MANET simulations,” Computer Communications, vol. 39, no. 7, pp. 48–54, July 2006. M. McGuire, “Stationary distributions of random walk mobility models for wireless ad hoc networks,” Proceedings of the 6th ACM international symposium on Mobile ad hoc networking and computing, MobiHoc, pp. 90–98, 2005. P. P. Z. Jr., Probability, Random Variables, And Random Signal Principles. McGraw Hill, 2001. M. Rahman and A. Al Muktadir, “The impact of data send rate, node velocity and transmission range on QoS parameters of OLSR and DYMO MANET routing protocols.” 10th international Conference on Computer and Information Technology, ICCIT, pp. 1–6, Dec. 2007. M. Krasnovsky and V. Wieser, “A performance of wireless ad-hoc network routing protocol,” Radioelektronika, 2007. 17th International Conference, pp. 1–3, April 2007. J. Benkovic, I. Kotuliak, and P. Truchly, “Comparison of DSR and DYMO routing protocols in building environments,” ELMAR ’09. International Symposium, pp. 109–112, Sept. 2009. G. Koltsidas, S. Karapantazis, G. Theodoridis, and F. Pavlidou, “A detailed study of dynamic MANET on-demand multipath routing for mobile ad hoc networks,” International Conference on Wireless and Optical Communications Networks, IFIP, pp. 1–5, Jul. 2007. M. Rahman, M. Azad, F. Anwar, and A. Abdalla, “A simulation based performance analysis of reactive routing protocols in wireless mesh networks,” International Conference on Future Networks, pp. 268–272, March 2009. I. D. Chakeres and C. E. Perkins, “Dynamic MANET on-demand routing protocol,” ETF Internet Draft, draft-ietf-manet-dymo-12.txt, Feb. 2008. E. Millman, D. Arora, and S. Neville, “STARS: A framework for statistically rigorous simulation-based network research,” in IEEE Workshops of International Conference on Advanced Information Networking and Applications (WAINA), March 2011, pp. 733 –739. A. Al-Maashri and M. Ould-Khaoua, “Performance analysis of MANET routing protocols in the presence of self-similar traffic,” Proceedings of 31st IEEE Conference on Local Computer Networks, pp. 801 –807, Nov. 2006. W. M. H. AhleHagh and D. Finkel, “Statistical characteristics of wireless network traffic and its impact on ad hoc network performance,” Applied Telecommunication Symposium, Sept. 2003. B. Lemeshko, S. Lemeshko, and S. Postovalov, “Comparative analysis of the power of goodness-of-fit tests for near competing hypotheses. i. the verification of simple hypotheses,” Journal of Applied and Industrial Mathematics, vol. 3, pp. 462–475, 2009, 10.1134/S199047890904005X. [Online]. Available: http://dx.doi.org/10.1134/S199047890904005X J. S. Bendat and A. G. Piersol, Random Data: Analysis and Measurement Procedures. Wiley-Interscience; 3rd edition, 2000.
