Common MANET Routing Protocols. Deepali Arora, Eamon Millman and Stephen W. Neville. Department of Electrical and Computer Engineering. University of ...
Non-equilibrium Statistical Behaviors within Common MANET Routing Protocols Deepali Arora, Eamon Millman and Stephen W. Neville Department of Electrical and Computer Engineering University of Victoria P.O. Box 3055 STN CSC, Victoria, B.C., CANADA, V8W 3P6
Abstract—Mobile adhoc networks (MANETs) have been widely proposed as solutions for delivering peered wireless services (i.e., for disaster recovery scenarios, etc). The quality of service (QoS) and quality of experience (QoE) delivered by such MANETs depends largely on the performance of their underlying routing protocols. Early research has begun to show that networklevel MANET behaviors tend not to follow standard equilibrium system assumptions (i.e., QoS measures not exhibit the statistical stationarity and ergodicity required for Birkhoff’s Ergodic theorem to apply). Hence, the standard approach of reporting MANET results in terms of averages computed across ensembles of Monte Carlo simulation runs is insufficient since ergodicity does not hold. This work extends such prior works by showing that non-equilibrium behaviors continues to hold even when much larger sets of Monte Carlo runs are analyzed (i.e., sets of 100 runs as opposed to the prior 15 runs) providing further credence to the claim that these MANETs behave more generally as non-equilibrium systems that has previously been supposed within the prior literature.
I. I NTRODUCTION The proliferation of smartphones and other Wi-Fi enabled mobile devices has begun to provide pragmatic real-world deployment environments for mobile adhoc networks (MANETs) and MANET-based services, particularly within urban cores (e.g., for peer-to-peer services, for disaster recovery scenarios, etc). The quality of service (QoS) and quality of experience (QoE) produced by such MANETs will innately depend on the behavioral characteristic of their underlying routing protocol(s). Literature exists that compares the performance of various MANET routing protocols [1], [2], [3], [4]. However, this literature provides contradictory guidance which suggests that the statistical behaviors of MANETs may be more complex that commonly supposed. The alternative claim, that the contradictory results arise as a result of the use of different simulation platforms (e.g., ns-2 [5], OmNet++ [6], or OpNet [7]), is not supported by existing literature that has shown that each platform produces results consistent with the others [8, 9]. Prior studies though have tended to be based on analyzing QoS measures in terms of their statistical averages computed across the conducted sets of Monte Carlo simulation runs. Such averaging, of course, is only valid if the QoS mea-
sures exists as a stationary and ergodic stochastic process1 . The basic untested hypothesis within much of the existing MANET resaerch is that MANETs must necessarily give rise to stationary and ergodic QoS and QoE measures (i.e., more formally, the presumption is that Birkhoff’s Ergodic Theorem[10] necessarily applies to MANETs). Recent work has begun to call into question this basic presumption by directly applying statistical hypothesis testing to analyze the stationarity and ergodicity of common MANET network-level QoS measures (e.g., normalized routing overhead (NRO), delay, packet delivery ratio (PDR))[11], [12]. Such works though have only sought to evaluate a small number of Monte Carlo runs (i.e., 15 runs). Hence, a reasonable concern is that the reported results may have simply arisen as statistical anomalies due to the relatively small ensembles analyzed. This work extends these prior works and addresses this basics concern by showing that nonequilibrium behaviors continue to be exhibited even when 100 Monte Carlo simulation runs are analyzed across each of the standard well-studied MANET routing protocols of: (i) adhoc on-demand distance vector (AODV) [13], (ii) dynamic MANET on-demand (DYMO) [14], and (iii) optimized link state (OLSR) [15] routing. The more in-depth analysis conducted in this work has shown that non-equilibrium MANET behaviors become even more prevalent with the larger Monte Carlo ensembles that are analyzed than they were in the 15 run ensembles. More formally this work applies formal statistical hypothesis testing to address the specific questions of: • Do standard MANET QoS measures necessarily always enter into a singular statistically steady-state behavior? (i.e., under ADOV, DYMO, and OLSR does the MANET behave as an ergodic system?) • Do the prior results showing that MANET can exhibit non-equilibrium behaviors continue to hold even when larger Monte Carlo ensembles are analyzed? The work explores these questions through empirical data analysis. It does not seeks to provide the underlying formal theoretical explaination for the observed non-equilibrium be1 As is well known, for averaging to be meaningful all elements being averaged must arise from the same underlying probability distribution.
ª*&&&
haviors. In part, this is a result of the standard, generally untested, equilibrium system presumption being the more restrictive assumption. More formally, if one is to make use of Birkhoff’s Ergodic theorem it is necessary to show that it holds across all the cases of interest. Claiming it may not always hold, as is done in this work, requires that counter examples are found. As a reasonably large ensembles of only very standard and quite simple MANET scenarios are studied in this work it becomes relatively unlikely that all of the counter examples found simply arise as statistical anomalies. The remainder of this paper is organized as follows. Section 2 discusses the results from some of existing literature and Section 3 provides an overview of the MANET simulation scenario used in this work. Section 4 provides an overview of stationary and ergodic stochastic process theory and discusses how the formal statistical hypothesis testing is performed to empirically assess the stationarity and ergodicity of the conducted Monte Carlo MANET simulations. Section 5 introduces the performance measures of interest in this work. Section 5 then provides the simulation results and their analysis. Section 6 then concludes the work. II. R ELATED W ORK The performance of the MANET routing protocols has been compared in number of different studies. For example, Clausen et al., [1] compared the adhoc on demand distance vector routing (AODV), dynamic source routing (DSR) and optimized link state (OLSR) routing protocols with respect to their packet delivery ratio, delay and routing overheads using Monte Carlo simulations in ns-2 for 50 node network within a 1000×1000 m area. The results were obtained by averaging over 30 simulation runs, each lasting for 250 seconds and showed that for higher node velocities, the pro-active routing protocols yielded lower packet delivery ratio (PDR), lower end-to-end delays but higher routing overheads when compared to the reactive routing protocols. [2] also assessed the performance of these three protocols for 100 node network with network area, traffic model and mobility model similar to [1] using OpNet with each Monte Carlo simulation lasting for 1000 seconds. There results contradicted [1] and suggested that pro-active routing protocols are better than the reactive routing protocols in terms of PDR even for high mobility scenario. [3] compared the performance of AODV, OLSR and a source routing-based multicast protocol (SRMP) within 70 node network within an area of 500m× 500m using ns2 for different mobility models. Their results showed that the OLSR outperformed the AODV with higher PDR and lower delays. [16] carried out simulations for 50 node network within 300×600 m area for different traffic models. The results were averaged over 10 simulation runs each lasting for 1000 seconds of simulation runtime. Their results contradicted [3] and suggested that AODV performed better than the OLSR in terms of PDR and delay, especially in the presence of node mobility. The above discussions highlights the contradictory results that exist in the literature, even with respect to the compar-
isons of very standard protocols such as AODV, DYMO, and OLSR. Moreover, the standard protocol literature has generally reported its results in terms of ensemble averages computed across few Monte Carlo simulations runs [1], [16], [17] or by discarding the first 40-50 sec. of simulated run time, as in [18], [19]. Once these transients are removed, it is then presumed that MANETs, in general, behave as ergodic processes across all potential measurement features of interest. The quality of these transient removal processes and the subsequent ergodic process assumption have not been directly tested within the existing literature. Here we show that such assumptions do not hold even under very basic scenarios and non-ergodic and nonstationary behaviors exist even when the relevant statistical analyses are carried out over large numbers of Monte Carlo simulation runs (i.e., ensembles of 100 runs each). III. MANET S IMULATION S CENARIOS The analyses presented in this work pertain to standard Monte Carlo simulations conducted under traditional and simple MANET scenarios but where the MANETs have been pre-tuned to begin in good operational states (i.e., pathological scenarios are actively avoided). The simulation model with its network area, node density, mobility, input traffic parameters, communication protocols and channel characteristics (shown in Table 2) were implemented via the OmNet++ event-based simulation engine using the standard OmNet++ configuration files. The OmNet++ simulator was then wrapped in the STARs custom MPI-based job control framework [20], which adds the statistical hypothesis testing processes described below to a standard MPI-based job control framework. All simulations were conducted on a cluster of 42 IBM blade dual 3.0 GHz processor computers each with 8 Gbytes of memory, with the simulations analyzed in this paper requiring several months of run-time on this cluster. The simulation were conducted using a standard 50 mobile nodes within a 300m by 600m area and following the random walk mobility model [21] in order to avoid the wellknow statistical bias inherent to random waypoint mobility models [22]. Per-node communication distances are tuned in accordance with [9] to ensure the expected 5 to 7 one-hop neighbors per node as required to produce a fully connected network. A Poisson On/Off traffic model is used for all simulations. Hence, the observed statistical complexities do not arise through the occurrence, for example, of self-similar traffic behaviors. Additionally, the channel model is the simplest free-space open-field model involving no obstructions or more complex channel fading issues. IV. S TATIONARY AND E RGODIC P ROCESSES Within probability and statistics stationarity and ergodicity refer respectively to whether the underlying probability distribution characterizing a stochastic process is: (i) time independent, and (ii) ensemble (or space) independent. For stationary processes there exists a guarantee that all transients die out in finite time. Ergodicity then guarantees that all instances of a given stationary process also always converge towards the
Theorem IV.1 (Birkhoff’s Ergodic Theorem2). If μ(.) is a Lebesgue measure on [0, 1] and T : [0, 1] → [0, 1] is a measure preserving mapping then T is ergodic if and only if,
TABLE I: MANET simulation parameters and values. Simulation Parameter Network Size Number of nodes Mobility Model Traffic model Channel model Per-node Transmission range Node Velocity Simulated run time Simulation runs per ensemble Routing protocols
Value 300m × 600m 50 Random Walk Poisson Free space (α = 2) 98 m 0 m/sec, 5 m/sec 1800 sec 15 and 100 AODV, DYMO, OLSR
K 1 lim 1A (T l (x)) = μ(A) K→∞ K k=1
for each measurable set A and for almost every x ∈ [0, 1] (i.e., for all x ∈ [0, 1] except for at most a set of measure 0), and where 1A (x) denotes a standard index function where 1A (x) = 1 if x ∈ A and 0 if x ∈ / A and T k (x) = T ◦ T ◦ · · ·◦ T (x) denotes applying the T mapping k times in succession.
same unique statistical steady-state. Together processes that are both stationary and ergodic, for the purposes of this work, are denoted as equilibrium systems (i.e., this denotes a system that always converges in time to the same (a unique) statistical steady-state behavior). Within measure theory both stationarity and ergodicity rest, in part, on an underlying requirement that the stochastic process of interest be measure preserving or, equivalently, in dynamical system’s parlance measure invariant. Assume that X = {x(t)|x(t) ∈ , t ∈ } denotes a MANET QoS measure of interest (i.e., a stochastic process), which for simplicity is assumed to be a continuous real-valued quantity. Denoted by A ∈ Ω an event associated with X, with Ω denoting the space of all such possible events. Typically, the events of interest are of the form A : x < X where X ∈ (−∞, ∞), as these characterize X ’s cumulative distribution function (CDF). A Lebesgue measure μ(A) can then be defined on all events A ∈ Ω, where μ : Ω → + . If μ(A) also meets the axioms of probability then P (A) = μ(A), were P (A) denotes A’s probability of occurrence. A transform T : Ω → Ω can then be defined as measure preserving (or measure invariant) with respect to μ(.) if for all measurable set A ∈ Ω it is the case that, μ(T −1 (A)) = μ(A)
(1)
where T −1 (A) denotes the pre-image of A. If T (.) denotes a time shift transform and if T (.) is bijective and measure preserving for all T then X is a strictly stationary process (i.e., all statistics associated with X ’s are time independent). Hence, by definition, a stationary process will have a time independent CDF (i.e., P t (x < X) = P (x < X) for all t). Stationarity does not guarantee that all instance of X will always converge to the same singular steady state. Assume that a collection of N instances of X exist and denote this ensemble by XN = {Xn |n = 1, . . . , N }. A transform T : Ω → Ω can then be defined such that Pt,n (x) = T [Pt,n (x)]. The question can now be asked as to whether this T is measure preserving for all possible n and n (i.e., is every ensemble record statistically indistinguishable from any another ensemble record). If T is measure preserving for all n, n ∈ N then X is an ergodic process. More formally, Birkhoff’s Ergodic theorem provide the standard engineering definition of ergodicity as,
As can be easily seen Birkhoff’s Ergodic theorem must hold if MANET QoS (or QoE) results are to be reported in terms of simple averages computed across the full Monte Carlo ensembles. The focus of this work is to directly via statistical test, K 1 l hypothesis testing, whether limK→∞ K 1 k=1 A (T (x)) = μ(A) indeed holds for a common QoS measures in AODV, DYMO, and OLSR. A. Statistical Hypothesis Testing Figure 1a illustrates the time series of records produced for a given QoS measure X = {Xn (t)|n = 1, . . . , N } from an ensemble of N Monte Carlo simulation runs, which differ only in their initial random seeds. Packet-level network simulation is an event-based simulation approach. Hence, each Xn (t) ∈ X is a stochastic process sampled at its own discrete stochastic sampling rate. In general, if any two Xn (t) and Xn (t) records are to be compared they must be re-sampled onto a common time frame (i.e., via interpolation). But, as this work focuses on assessing how the X CDFs evolve both over time and across the ensemble, this otherwise required re-sampling can be avoided. More particularly, statistical hypothesis testing can be used to directly test whether the empirical CDF constructed via the X data are stationary and ergodic or more precisely whether evidence exists in the data to show that stationarity and ergodicity do not hold (i.e., as is well known, one cannot prove that stationarity or ergodicty hold via emperical data analysis approaches but one can apply data analysis to show that they do not hold). As per Figure 1b, each individual Xn (t) record can be divided into sets of non-overlapping windows Wpq defined over p = 1, . . . , P progressively coarser dyadic p scale levels. At each scale level p an ordered set Wq |q = 1, . . . , Qp of windowed Xn (t) data is then produced. Empirical estimates Pˆq,p (x) of Xn (t)’s CDF for each of these constructed windows can then be produced, beginning from the finest scaled, rightmost WQ1 1 window. As per Figure 2 and limiting consideration to just the p = 1 scale level, the PˆQ,1 (x) CDF can then compared with the estimated CDF PˆQ1 −1,1 (x) produced from the WQ1 1 −1 window located to its immediate left. This comparison is done formally via hypothesis testing using the distribution-free and relatively weak Kolmogorov-Smirnov test (KS-test). If these 2 This
formalization of Birkhoff’s Ergodic Theorem is taken from [23].
Experiment Ensemble
{
x1(t) x2(t)
xN(t) 0 sec
Texp
t
(a) The ensemble of records produced by N Monte Carlo experiment runs for measurement feature X. Direction of KS-testing
5 4 p 3 2 1 0 0
q
time
}W }W }W }W }W }W
(b) The stationarity tests Wqp window tiling. KS-test CDF
CDF
KS-test CDF
Xn(t)
CDF
5 4 3 2 1 0
the P scale levels are tested and all windows fail the test, then the Xn (t) record is marked as being non-stationary. The minimum and maximum window sizes are controlled, based on the Xn (t) data, to ensure that every window contains at least 100 data points and the P -level contains no less than 5 windows. To improve the statistical accuracy of the testing process, each CDF passing the KS-test is averaged with the CDF against which it was tested with this new CDF now forming one side of the next KS-test. This ensures that the testing process becomes increasingly constrained as one moves from right to left thereby ensuring that the computed tstat n values are conservative. MANETs are known to have start-up transients. Hence, tstat conservatively marks the end of these n transients for each Monte Carlo run. To test for ergodicity, all of the stationary Pˆn (x) empirical CDFs within a set of Monte Carlo runs are then compared, again via KS-testing, on a pair-by-pair basis. A fully connected graph is constructed with each stationary Pˆn (x) representing the nodes and the pair-by-pair KS-test p-values representing the graph edges. All edges below a 95% confidence level are removed and the remaining graph cliques are identified. These cliques are then labeled as X ’s ergodic modes, as no evidence exists within each clique that the empirical CDFs it contains have been drawn from different underlying distributions. The full details of this testing process, its rationale, and its use of the KS-test, can be found in the previously published work of [12]. It should be noted that as the testing process only performs pairwise comparisons of CDFs it does not constitute (or require) multiple hypothesis testing [12].
KS-test CDF
CDF
CDF
V. MANET S IMULATION R ESULTS AND A NALYSIS
CDF
For this works analysis three standard performance measures are considered: a) Packet delivery ratio (PDR): The ratio of the delivered application packets to sent packets, b) Average delay: The total time (in seconds) that a packet takes to travel from its source to the end-point destination, and c) Routing overhead: The percentage of total packets transmitted in the network that are for route establishment.
tnstat
(c) The KS-test stationarity testing process as applied right-to-left along an Xn record for a given set of W p windows.
Fig. 1: Illustration of the ensemble of records produced via rerunning a given experiment N times, showing Xn ’s random sampling intervals and the window tilings used for stationarity testing.
two empirical CDFs pass the KS-test at a 95% confidence level then, formally, no evidence exists that PˆQ1 −1,1 (x) and PˆQ1 ,1 (x) have been drawn from different distributions. Therefore, based on the available evidence Xn (t) can be considered to be stationary with respect to the WQ1 1 and WQ1 windows. The testing process then continues leftward until the first Pˆq,1 (x) failing the KS-test is found. The left most edge of the last Wq,1 window to pass the KS-test then, conservatively, denotes the earliest time that Xn (t) can be considered to have entered its stationary behavior. This time is denoted as tstat n . If no Wq,1 windows are found to fail the KS-test then p is incremented by one and the process is repeated. If all of
Figure 2 shows the empirical CDFs of the time from the start of simulation when each of PDR, NRO, and delay begin to enter their steady-state statistical behaviors (i.e., the emperical CDFs of tstat for each QoS measure) for n AODV, DYMO and OLSR. Only the results for the 5m/s node velocity case are show, as largely by definition, at 0 m/s the MANET will behave as an equilibrium system due the lack of node movement. Samples which were not observed to exhibit stationary behaviors for a given feature anywhere in t ∈ [0, 1800] were excluded from the results (i.e., experiment runs that were denoted as being non-stationary over the entire 1800 seconds of simulated MANET run time are excluded). Figure 2 therefore shows the probability distribution for when stationarity is achieved for each QoS measure over the full set of conducted experiments.
1.1
1
0.9
CDF−F(X)
0.8
0.7
0.6
DYMO (vel = 5 m/s, 100 sims) DYMO (vel = 5 m/s, 15 sims) AODV (vel = 5 m/s, 100 sims) AODV (vel = 5 m/s , 15 sims) OLSR (vel = 5 m/s, 100 sims) OLSR (vel = 5 m/s, 15 sims)
0.5
0.4
0.3
0.2
0.1
0
200
400
600
800
stat n
t
1000
1200
1400
1600
1800
(seconds)
(a) CDF of tstat for PDR. n 1
0.9
0.8
CDF−F(X)
0.7
0.6
0.5
0.4
DYMO (vel = 5 m/s, 100 sims) DYMO (vel = 5 m/s, 15 sims) AODV (vel = 5 m/s, 100 sims) AODV (vel = 5 m/s , 15 sims) OLSR (vel = 5 m/s, 100 sims) OLSR (vel = 5 m/s, 15 sims)
0.3
0.2
0.1
0
0
200
400
600
800
stat
tn
1000
1200
1400
1600
1800
1400
1600
1800
(seconds)
(b) CDF of tstat for NRO. n 1
DYMO (vel = 5 m/s, 100 sims) DYMO (vel = 5 m/s, 15 sims) AODV (vel = 5 m/s, 100 sims) AODV (vel = 5 m/s , 15 sims) OLSR (vel = 5 m/s, 100 sims) OLSR (vel = 5 m/s, 15 sims)
0.9
0.8
CDF−F(X)
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
200
400
600
800
stat
tn
1000
1200
(seconds)
(c) CDF of tstat for delay. n
Fig. 2: CDFs of tstat for PDR, NRO, and delay for each n of DYMO, AODV, and OLSR for average node velocities of 5 m/s and for both small 15 and large 100 Monte Carlo simulation runs (sims) ensembles.
From Figure 2 it is clear that PDR and NRO under OLSR never entered a statistical steady state (i.e., the start up transients persisted throughout the 1800 seconds of simulated MANET run time). This results holds for both the small 15 run ensembles and the larger 100 run ensembles. AODV and DYMO, by comparison perform better in that PDR, NRO, and delay all exhibited stationarity for both the small and large ensembles. These results highlight that the stationarity of the MANET QoS measures depends on both the QoS measure itself and the routing protocol being used. Generally speaking, except for NRO, DYMO tends to yield the narrowest tstat distributions or earliest tstat times implying that DYMO n n outperforms the other protocols in terms of its ability to limit the persistence of its start-up transients. These results also clearly highlight that the generally applied ad hoc techniques for transient removal within MANET research are insufficient as the transient behaviors can clearly extend well past the first few minutes of the MANET’s operation. More particularly, cases clearly exists where the MANET never settles down into a steady state even after 30 minutes of simulated run time. Obviously, this is problematic for real-world MANETs as nodes would be commonly expected to come and go over any given 30 minute interval. Table II and Table III provides more details of the observed non-stationary, stationary, and ergodic behaviors exhibited by each routing protocol and, again, only for the more interesting 5 m/s average velocity case. Table II in particular, suggests that the tested QoS measures are likely to exhibit stationarity in about one-half to one-third of the runs, excluding the nonstationary behaviors of PDR and NRO under OLSR. Moreover, even when stationarity is achieved a multiplicity of ergodic modes exist, ranging from 2 to 4 modes depending on the routing protocol and the QoS measure. But with only 15 runs per Monte Carlo ensemble the data of Table II, previously reported in [12], is more suggestive than it is truly informative. To address this limitation the same sets of experiments were re-run and re-analyzed for larger Monte Carlo ensembles of 100 runs per experiment, where again these runs only differ in their initial random seeds. The results from the analysis of these larger Monte Carlo runs is shown in Table III. From this table it becomes clear that for OLSR NRO and PDR remain non-stationary for all of the runs, echoing the results of the smaller ensemble and confirming that prior results did not arise as a simple statistical anomaly. It also becomes clearer in the analysis of the larger ensembles that although OLSR delay remains better behaved it is now only stationary in 24% of the runs compared with the 60% of runs suggested by Table II. DYMO and AODV also can be clearly seen to fail to converge to steady-state statistical behaviors in nearly threequarters of the runs, although both protocols now produce runs in which NRO and PDR are stationary which OLSR could not achieve. It can be observed though that when stationary runs occur they produce a large multiplicity of ergodic modes. The differences between DYMO and AODV though are less clear in the larger ensemble analysis. It should be noted that each ergodic modes must contain at least two runs and a run may
TABLE II: Number of non-stationary, stationary, and ergodic experiment runs for the AODV, DYMO, and OLSR MANETs with a 5 m/s average node velocity for the smaller 15 run Monte Carlo ensembles. QoS Measure
Delay PDR NRO
OLSR Nonstationary Runs 6 of 15 15 of 15 15 of 15
Stationary Runs 9 of 15 0 of 15 0 of 15
DYMO Number of Ergodic Modes 4 -
Nonstationary Runs 8 of 15 7 of 15 2 of 15
Stationary Runs 7 of 15 8 of 15 8 of 15
AODV Number of Ergodic Modes 2 2 4
Nonstationary Runs 9 of 15 7 of 15 10 of 15
Stationary Runs 6 of 15 8 of 15 5 of 15
Number of Ergodic Modes 3 2 3
TABLE III: Number of non-stationary, stationary, and ergodic experiment runs for the AODV, DYMO, and OLSR MANETs with a 5 m/s average node velocity for the larger 100 run Monte Carlo ensembles. QoS Measure
Delay PDR NRO
OLSR Nonstationary Runs 76 of 100 100 of 100 100 of 100
Stationary Runs 24 of 100 0 of 100 0 of 100
DYMO Number of Ergodic Modes 7 -
Nonstationary Runs 81 of 100 77 of 100 67 of 100
belong to more than one ergodic mode (i.e., runs 1 and 2’s stationary distributions both pass the KS-test and run 1 and 3’s stationary distributions pass the KS-test, but runs 1 and 3 fail the test producing two modes - one containing runs 1 and 2 and one containing runs 2 and 3). The results of Table II directly indicated that packet-level Monte Carlo simulations of MANETs need not give rise to equilibrium systems for their QoS measures. Instead, nonequilibrium behaviors are in fact by far the more common case. These results highlight that insufficiency of the standard MANET analysis approach of reporting results in terms of simple full ensemble averages, as in many cases, would lead to the averaging of non-ergodic and/or non-stationary data sets which is clear violation of the requirements of Birkhoff’s Ergodic theorem. VI. C ONCLUSIONS Much of the current MANET literature has been based on the presumption that Birkhoff’s Ergodic theorem must apply and, therefore, that MANET in terms of their QoS and QoE measure of interest must behave as equilibrium systems (i.e., produce QoS and QoS measures that exist as stationary and ergodic stochastic processes). This work extends a set of prior work conducted on a small and limited set of 15 Monte Carlo runs per ensemble [24], [25] on a wider and larger study of 100 Monte Carlo runs per ensemble. By this larger analysis it has been shown that the prior non-equilibrium system observations did not arise as simple statistical anomalies. Instead this work highlights that non-equilibrium MANET behaviors are instead the common case whereas the standard equilibrium system presumption rarely holds. Even when QoS measures do exhibit stationarity they tend to produce a multiplicity of ergodic modes, which although similar are not statistically close enough to pass even the relatively weak KS hypothesis test for having been drawn form the same underlying distribution.
Stationary Runs 19 of 100 23 of 100 33 of 100
AODV Number of Ergodic Modes 11 8 10
Nonstationary Runs 77 of 100 74 of 100 77 of 100
Stationary Runs 23 of 100 26 of 100 23 of 100
Number of Ergodic Modes 13 17 16
Evident from these results is that a fuller analysis of the richness of the statistical behaviors of MANET QoS and QoE measure would require the analysis of Monte Carlo ensembles involving thousands to tens of thousands of runs which in turn would require months of run-time even on quite large high performance computing clusters. Additionally, it is clear that deeper theoretical analysis of MANETs and their operations is required if formal theoretical explanations are to be found as to why they strongly tend to behave as non-equilibrium statistical systems. This is a current focus of our on-going research into MANET behaviors and performance. R EFERENCES [1] T. H. Clausen, P. Jacquet, and L. Viennot, “Comparative study of routing protocols for mobile ad-hoc networks,” 2002. [2] C. Mbarushimana and A. Shahrabi, “Comparative study of reactive and proactive routing protocols performance in mobile ad hoc networks,” in Advanced Information Networking and Applications Workshops, 2007, AINAW ’07. 21st International Conference on, vol. 2, may 2007, pp. 79–84. [3] F. Yongsheng, W. Xinyu, and L. Shanping, “Performance comparison and analysis of routing strategies in mobile ad hoc networks,” in Computer Science and Software Engineering, 2008 International Conference on, vol. 3, dec. 2008, pp. 505–510. [4] D.-W. Kum, J.-S. Park, Y.-Z. Cho, and B.-Y. Cheon, “Performance evaluation of aodv and dymo routing protocols in manet,” in IEEE CCCN 2010, 2010. [5] K. Fall and K. Varadhan, “The network simulator-ns2,” http://www.isi.edu/nsnam/ns/doc/index.html. [6] O. Community, “ OMNeT++,” http://www.omnetpp.org/. [7] “OPNET,” http://www.opnet.com/.
[8] S. Kurkowski, T. Camp, and M. Colagrosso, “MANET simulation studies: the incredibles,” SIGMOBILE Mob. Comput. Commun. Rev., vol. 9, no. 4, pp. 50–61, 2005. [9] T. Andel and A. Yasinsac, “On the credibility of MANET simulations,” Computer Communications, vol. 39, no. 7, pp. 48–54, July 2006. [10] C. Walkden. Birkhoff’s Ergodic Theorem. [Online]. Available: http://www.maths.manchester.ac.uk/ ∼cwalkden/ergodic-theory/lecture21.pd%f [11] E. Millman, “Analyzing MANET jamming strategies,” 2011. [Online]. Available: http://dspace.library.uvic.ca: 8080/handle/1828/3745 [12] D. Arora, E. Millman, and S. W. Neville, “On the statistical behaviors of network-level features within manets,” in PIMRC, 2011, pp. 1145–1150. [13] C. Perkins, E. Belding-Royer, and S. Das, “Adhoc ondemand distance vector (AODV) routing protocol,” Internet Draft, www.ietf.org/rfc/rfc3561.txt, July 2003. [14] I. D. Chakeres and C. E. Perkins, “Dynamic MANET on-demand routing protocol,” ETF Internet Draft, draftietf-manet-dymo-12.txt, Feb. 2008. [15] T. Clausen and P. Jacquet, “Optimized link state routing protocol (OLSR),” Internet Draft, www.ietf.org/rfc/rfc3626.txt, Oct. 2003. [16] A. Al-Maashri and M. Ould-Khaoua, “Performance analysis of MANET routing protocols in the presence of self-similar traffic,” in IEEE 31st Conference on Local Computer Networks Proceedings, Nov. 2006, pp. 801– 807. [17] J. Yoon, M. Liu, and B. Noble, “A general framework to construct stationary mobility models for the simulation of mobile networks,” IEEE Transactions on Mobile Computing, vol. 5, no. 7, pp. 860–871, 2006. [18] G. Koltsidasand, S. Karapantazis, G. Theodoridis, and F. Pavlidou, “A detailed study of dynamic MANET ondemand multipath routing for mobile ad hoc networks,” in International Conference on Wireless and Optical Communications Networks, IFIP Proceedings, Jul. 2007, pp. 1–5. [19] G. Koltsidas, F. Pavlidou, K. Kuladinithi, A. TimmGiel, and C. Gorg, “Investigating the performance of a multipath DYMO protocol for ad-hoc networks,” in IEEE 18th International Symposium on Personal, Indoor and Mobile Radio Communications,PIMRC Proceedings, Sep. 2007, pp. 1–5. [20] E. Millman, D. Arora, and S. Neville, “Stars: A framework for statistically rigorous simulation-based network research,” in Advanced Information Networking and Applications (WAINA), 2011 IEEE Workshops of International Conference on, march 2011, pp. 733 –739. [21] M. McGuire, “Stationary distributions of random walk mobility models for wireless ad hoc networks,” in ACM international symposium on Mobile ad hoc networking and computing, MobiHoc Proceedings, 2005, pp. 90–98. [22] M. L. J. Yoon and B. Noble, “Random waypoint considered harmful,” Twenty-Second Annual Joint Conference
of the IEEE Computer and Communications, INFOCOM, vol. 2, pp. 1312–1321, March 2003. [23] A. Patrascioiu, “The ergodic hypothesis,” Proceedings of Los Almos Science Special Issue, pp. 263–279, 1987. [24] D. Arora, E. Millman, and S. Neville, “Assessing the performance of aodv, dymo, and olsr routing protocols in the context of larger-scale denser manets,” in Communications, Computers and Signal Processing (PacRim), 2011 IEEE Pacific Rim Conference on, aug. 2011, pp. 675–679. [25] ——, “Enabling statistically rich manet simulations through cluster computing,” Springer journal on cluster computing, 2013.
