Modeling and Test Generation for Worst-case ... - CiteSeerX

Modeling and Test Generation for Worst-case Performance Evaluation of MAC Protocols for Wireless AdHoc Networks Shamim Begum

Ahmed Helmy

Sandeep Gupta

Department of Electrical Engineering University of Southern California Email: [email protected]

Department of CISE University of Florida, Gainesville Email: [email protected]

Department of Electrical Engineering University of Southern California Email: [email protected]

Abstract—We propose a novel framework to critically analyze a given MAC protocol for wireless adhoc networks with respect to its correctness criteria and performance metrics. The framework is composed of wanted state generation and test scenario generation algorithms. The wanted state generation algorithm generates a set of conditions that meet our study objective. The test generation algorithm then generates complete test scenarios that satisfy our objective, e.g., minimize the value of a particular performance metric. The core of our search engine utilizes novel algorithms that use combinations of goal-oriented backward and forward search and implications as well as heuristics that enable the generation of worst case scenarios in manageable complexity for our practical purposes. We demonstrate the effectiveness of our approach by using our framework to analyze the worst case performance, in terms of throughput and fairness of IEEE 802.11 for adhoc networks. For all topologies, the worst case scenarios generated by our framework show the worst performance among all scenarios that we generate. The scenarios generated by our framework include the scenarios typically used for performance evaluation of IEEE 802.11 protocol. The case study of IEEE 802.11 shows that the complexity of our novel algorithms are quite practical.

I. I NTRODUCTION The medium access control (MAC) is the main determinant of the efficiency and fairness of sharing the limited bandwidth of a wireless channel. Published in 2004, a survey [16] presents 34 MAC protocols for wireless adhoc networks ranging from industry standards (e.g., IEEE 802.11 [17]) to research proposals (e.g., MACAW [19]). There are number of studies that analyze these protocols using analytical models (e.g., [20], [21], [22], [23], [24] for IEEE 802.11) and experience or intuition (e.g., [25], [26], [27] for IEEE 802.11). Simulation and testbed implementations are the main evaluation tools today for which the scenarios are either manually generated (e.g., [20]) for average case analysis or are based on intuition (e.g. [25], [21], [22], [23], [24], [26], [27]). These evaluation approaches do not use any scenario generation tool. Often times, actual deployment of a protocol reveals many problems leading to very poor performance, which have not been identified before. Thus providing a framework for generating MAC layer scenarios to uncover *This research was supported by NSF and NASA.

weakness or breaking points of a class of MAC protocols for wireless adhoc networks motivates this work. The goal of this work is to augment other evaluation approaches with a library of meaningful scenarios. These scenarios are semiautomatically generated using generic algorithms, yet these are protocol specific for better evaluation of the protocols beyond average case and towards the worst case. Test generation is mainly based on search techniques that search for valid sequences of protocol events that expose weaknesses in the design of a protocol. Traditional test generation approaches target verification and are based on forward search methods where the entire search space is exhaustively searched for test scenarios [1], [2]. Formal verification approaches use high-level system description languages to model and to analyze network protocols [6], [7], and involve determining the set of all reachable states of the models. We propose a test generation framework that, instead of adopting the validation approach, uses a “falsification approach” and directly targets the protocol conditions that adversely effect the protocol performance. Our main formalism is that of a finite state machine (FSM) which we use to represent MAC layer events and states of network nodes. Given a protocol performance metric (e.g., throughput), we first identify our study objective (e.g., minimize throughput) and a set of target events that meet our objective. We use wanted state generation algorithm to generate a set of protocol states that lead to the target events. From the set of wanted states, we use a mix of forward and backward search and implication techniques to generate a set of MAC layer scenarios leading towards the worst performance, including the scenario that leads to the worst case performance at MAC layer. Finally, we perform simulations of our scenarios to evaluate performance of the given protocol for real applications. Our framework is applicable to all CSMA/CA based wireless MAC protocols. We apply our method to IEEE 802.11 because of its popularity in industry and academia as well as its richness in semantics as a representative protocol in its class. Using our framework we have generated a library of MAC layer scenarios that minimize the maximum network throughput or maximize the difference of throughput achieved

by different nodes in the network, thus leading towards the worst case network throughput or fairness in IEEE 802.11 networks. We simulate these MAC layer scenarios using ns-2 simulation framework to validate if the worst case scenarios show the worst case throughput and fairness at MAC layer in simulation. Simulation results show that for all topologies, the worst case scenarios generated by our framework show the worst performance among all scenarios that we generate. The scenarios generated by our framework include the scenarios typically used for performance evaluation of IEEE 802.11 protocol. The case studies of IEEE 802.11, MACA [18] and MACAW [19] show that the complexity of our algorithms are quite practical. The main contributions of this paper are as follows. (1) FSM models: besides the widespread use in theoretical computer science [8], machine learning [9], synthesis of sequential circuits [10] and finite Markov chain process [11], more recently FSM models are used in verification of communication protocols [1], [2], [4] and model checking and accurate computing of numerical properties of network protocols [6], [7]. To the best of our knowledge, FSM models have never been used for performance analysis in the area of wireless networks. The contributory components of our FSM models are: a) models of time in a distributed partially synchronous system using a global clock, and b) models of events that could not have happened in the past using prohibited entity in our system. Timed automata models, for example, MMT automata[12], model of Lynch and Vaandrager [13] and Alur and Dil [3], and extended time automata [4] use multiple clock variables. However, classical timed automata has been proven to be incorrect for a system with more than two clocks [14]. Using a single (global) clock allows us to generate test scenarios for a network with more than two wireless nodes. (2) Search algorithms: we use a goal-oriented backward and forward search and implications as well as heuristics that allow us to generate worst case scenarios in realistic time frame. As we show in Section V that our proposed complexity reduction techniques are of two categories: 1) generic approaches that can be applied to any context, and 2) protocol specific approaches that can be applied only to the same class of protocols. The rest of the paper is organized as follows. Section II presents related work. Section III describes our proposed framework. Our case study on IEEE 802.11 is presented in Section IV. In Section V we present complexity of our algorithms. Section VI concludes with a brief outline of our future plans. II. R ELATED W ORK There is a large body of work on verification and conformance testing. Conformance testing verifies that a specific implementation conforms to a specification, whereas our testing technique evaluates a design (as opposed to an implementation). We refer only the most related work on the formal verification of network protocols. The main formalism of these works are that of a probabilistic timed automata. PRISM [5] exhaustively analyzes the set of all reachable states

Protocol Performance Metric Protocol Transition Table

Wanted_state generation

Wanted_states

Relation between wanted transitions

Test Scenario Generation

Topology

Test Scenarios

Simulation Worst Protocol Performance

Fig. 1.

An overview of our performance evaluation framework.

and has been used for model checking and accurate computing of numerical properties of network protocols [6], [7]. Our work is best positioned with [6] which formally verifies certain properties of IEEE 802.11, whereas we provide scenarios that expose the worst performance. Both these approaches are search based, and therefore are exhaustive. However, we use implications to reduce the search space and heuristics to guide the search to generate worst performance scenarios at practical complexities. We model time assuming a global clock to abstract various types of MAC timers for performance evaluation while techniques (e.g., [4]) based on timed automata [3] model time using multiple clocks for conformance testing. Search algorithms have been used in wide range of areas of computer science and computer engineering starting from theoretical computer science to VLSI chip testing. Among these, our approach is based on the search algorithms that are used in automated test generation techniques. VLSI chip testing [15] uses test vector generation to detect target faults. Test vectors may be generated based on circuit and fault models, using the fault-oriented technique, that utilizes implication techniques. We extend fault-oriented techniques as follows. First, we model protocol performance metric and use search based approach to generate states of a given topology in which the given performance is penalized. Second, given a target protocol state, we generate network scenarios that lead to the target state using a mix of backward and forward search algorithms. In this paper we present our basic framework to evaluate the worst case performance of basic CSMA/CA protocols for wireless adhoc network. We extend our basic framework to incorporate the study of new protocols, namely MAC protocols for quality of service and power control. The extended framework is presented in [29]. III. P ROPOSED F RAMEWORK Given a wireless MAC protocol, our goal is to evaluate the worst case performance of the protocol with respect to a given set of protocol performance metrics. Figure 1 presents our proposed performance evaluation framework. Inputs to the framework are: (1) the protocol model, (2) a network topology, and (3) a protocol performance metric. The framework uses two main algorithms: (1) wanted states generation, and (2) test scenario generation, to generate a set of MAC layer scenarios towards the worst case performance of the given metric for the topology. It then regenerates the MAC scenarios and simulates the scenarios using a simulation framework, and outputs the worst case performance for the given metric. Given a protocol performance metric, it first transforms the performance metric in a form that reflects our objective.

For example, given throughput, it transforms it into a set of wanted states that it minimizes or maximizes in later steps in order to generate scenarios that lead to minimum throughput in a given topology. The wanted states generation algorithm transforms the protocol performance objective into wanted states. Second, given a set of wanted states and a topology, the test scenario generation algorithm generates scenarios that maximize or minimize the study objective, which in turn, minimizes (or maximizes) the given protocol performance metric. The test scenarios such generated are MAC level scenarios towards the worst MAC performance in the given topology. Using time relations embedded in the test scenarios, we regenerate these test scenarios and simulate them in a simulation environment to evaluate the actual performance provided by the test scenarios. Using examples, Sections III-B and III-C respectively present a brief overview of models and algorithms of the proposed framework. A. Assumptions Our model is based on the following assumptions. (1) We assume a wireless node do not fail or run out of battery power. Therefore, a transmitter never stops in the middle of its transmission. (2) We do not model message loss explicitly, however, loss implied by collision and drop is implicitly modeled. (3) We assume a static network. We could have relaxed this assumption by allowing the neighborhood to change over time as the semantics of time allows the change of neighborhood at the granularity of message transmission (or reception) level. However, for typical automobile speeds for a mobile node, the movement during the period of a message transmission is not significant. (4) When a receiver is receiving transmissions from two nodes and is unable to cleanly receive signal from either nodes, the phenomenon is known as collision. When the receiver is able to cleanly receive the signal from the closer transmitter, the phenomenon is called capture. Our finite state machines are deterministic (DFSM) modeling only collision, not capture. (5) We only model virtual carrier sense mechanisms. (6) The carrier sensing range is twice the transmission range. (7) We use a global clock assuming all wireless nodes are synchronized. B. Basic models For illustration purposes, let us consider a simple 3-way handshake based CSMA/CA protocol P and a 4-node chain topology in which our objective is to generate scenarios that lead to worst case network throughput and fairness. Throughput is defined as the fraction of time the channel is used to successfully transmit payload bits [20]. Let
be the amount of time a receiver spends in successfully receiving payload (data) and  be the total time the transmitter spends since the first transmission opportunity to transmit the data, then we define throughput as: MAC throughput =






(1)

In our model,
denotes the time it takes for an intended receiver to successfully receive MAC data packet.  denotes the entire time a transmitter spends from the start of first RTS

0

1

2

3

Transmission range: G = { G0 = {1}, G1 = {0, 2}, G2 = {1, 3}, G3 = {2} } Carrier sense range: g = { g0 = {1, 2}, G1 = {0, 2, 3}, G2 = {0, 1, 3}, G3 = {1, 2} }

Fig. 2.

Model of network topology (Topology I).

No

Start state

Input event

End state

Output event

1

Idle

Packet

Transmit

RTS Transmit (start, end), RTS Receive (start,end)

2

Transmit

End of RTS

WCTS

Start WCTS timer

3

Idle

RTS Recv start

Receive

4

Receive

RTS Recv end

Transmit

5

WCTS

CTS Recv start

WCTS & Receive

6

WCTS & Receive

CTS Recv end

Transmit

7

Idle

RTS overhear start Receive

8

Receive

RTS overhear end

9

WCTS

WCTS timer expire Backoff

Start Backoff timer

10

Receive

RTS Recv start

Backoff

Start Backoff time

11

Backoff

Backoff timer expire

Transmit

RTS Transmit (start, end),RTS Receive (start, end)

Fig. 3.

Defer

Successful RTS (row 1)

CTS Transmit (start, end), CTS Receive (start, end),

Successful CTS (row 4)

Data Transmit (start, end), Data Receive (start, end)

Successful Data(row 6)

Start Defer timer

Transition table of a CSMA/CA protocol P.

transmission to the end of successful reception of acknowledgment. Note that if an acknowledgment is lost requiring a retransmission, the time it takes for successful reception of retransmitted data and acknowledgment is considered in  . Our objective is to generate the MAC scenario that maximizes  . Thus generating worst case throughput scenario involves finding the MAC scenario that leads to the maximum  among the scenarios that we generate. A simple definition of fairness of a protocol is its ability to allocate the channel bandwidth equitably [26]. We define a scenario to exhibit fairness if all nodes initiating handshakes get equal opportunity to transmit data. We define a scenario to exhibit unfairness if the handshake is unsuccessful for at least one of the nodes that initiate handshake in the scenario and one or more of the nodes are successful. Index of unfairness is defined as: Index of unfairness =

Number of nodes unsuccessful in handshake Total number of nodes that initiate handshake

(2)

Topology is modeled in terms of transmission range of each node in the network. Transmission range of node i is a set  containing the nodes who receive its transmission and can decode it. While its carrier sensing range is a set   containing nodes who receive its transmission, but can not decode it. Figure 2 represents a wireless network of 4 nodes where   1 ,   0, 2 ,   1, 3 , and  , and    1, 2 ,    0, 2, 3 ,    0, 1, 3 , and   2   1, 2 . Figure 3 presents a finite state machine model or the transition table of a protocol P representing a lossObjective

Reduce throughput

Increase throughput

Fig. 4.

Wanted transition

Wanted_states Transmitter Receiver

Unsuccessful RTS

Backoff (BO)

Backoff (BO)

Unsuccessful CTS

BO

BO

Unsuccessful Data

BO

BO

Silent drop of RTS

BO

Defer

Successful RTS

Transmit (Tx) Receive (Rx)

Successful CTS

Tx

Rx

Successful Data

Tx

Rx

List of wanted states generated automatically by algorithm.

Protocol performance objective function

Inputs to test generation

Target event identification

Wanted_state generation

Target events

Protocol transition table

Node 0 in BO

Wanted_state generation Wanted_states

Network topology

Initial partial scenario E1

Test scenario generation

Implication Algorithms

State, event

State, event, time relation

Consistency report

Search Algorithms State, event, time relation

Search Tree

Initial partial scenario

Test Scenarios

Simulation

Fig. 5.

Worst performance

Detail components of our framework.

less channel. Each row of the transition table represents a state transition of a node running the protocol. Each output event of a transition is associated with a variable representing the delay of the output event from the state transition. The protocol works as follows: upon receiving a packet from higher layer for node j, node i transmits an RTS (Request-To-Send) destined for node j, schedules a WCTS (Wait-for-CTS) timer to wait for the CTS from j. Upon receiving the RTS destined for it, node j transmits a CTS (Clear-To-Send) to node i. On overhearing the RTS, a node k defers access to the channel by scheduling a timer. If node i receives the CTS from j before its WCTS timer expires, it goes to Transmit state to transmit the data, otherwise, it goes to Backoff (BO) state after which it retransmits the RTS. Row 1 of the table presented in Figure 3 states the following transition. When a node in Idle state receives a Packet from higher layer, it changes to Transmit state and triggers RTS Transmit and RTS Receive events. C. Algorithms to generate MAC layer scenarios Figure 5 presents detail components of our framework. Given a metric, such as throughput, our objective is to generate scenarios that lead to the worst case value for the metric. We achieve this objective using the following steps. (1) Given throughput, we first transform it into a protocol condition that we maximize in later steps in order to generate scenarios that lead to minimum throughput in a given topology. We refer the conditions that meet our study objective as wanted states. Wanted states generation procedure transforms the protocol performance objective into wanted states. (2) Given a set of wanted states and a topology, the test scenario generation algorithm generates scenarios that maximize our study objective, which in turn, minimize throughput in the given topology. Wanted states generation: The wanted states generation algorithm first identifies Data Receive as the target event based on
and  in Equation 1. Note that the target event Data Receive is output in row 6 of the transition table presented in Figure 3. The algorithm then uses a backward search technique to generate the transitions necessary (rows 6, 4, and 1 in the figure) to trigger the target event. Note that the transitions necessary to trigger the Data Receive event nullifies our study objective (instead of meeting the objective). Starting from these transitions, the algorithm then applies a forward search technique to generate wanted transitions. Any of these wanted

Node 0 in Defer Node 1 in BO (E2) (E3)

State descriptions: Node 0: starts in BO at time t1, ends at time t2 Event descriptions: Relations between time variables: t1 < t2

Fig. 6.

Node 3 in Defer (E8)

Node 3 Node 2 in BO in Defer (E ) 7 (E6) Node 1 Node 2 In Defer in BO (E4) (E5)

Initial partial scenarios for throughput.

transitions is sufficient to generate a condition that meet our objective. Figure 4 presents wanted states, e.g., BO and Defer that reduce throughput by increasing  . Test (MAC scenario) generation: Now with the knowledge of wanted states that penalize throughput, our goal is to generate the details of a MAC layer scenario that leads to worst case network throughput in Topology I using the test generation algorithm (Figure 5). The inputs to the test generation algorithm are (1) topology, (2) transition table and (3) wanted states. We first construct an initial partial scenario that we input as the root node of the search tree. A scenario is defined in terms of network node states, protocol events and time relations between the events and state transitions and represents a global picture of the network. The test generation algorithm first constructs all possible initial partial scenarios using the wanted states and the topology. Figure 6 presents the set of initial partial scenarios. For example, there are eight alternative choices of nodes to be in one of the two wanted states when the objective is to minimize network throughput of the topology. Note that states of three other nodes of the topology are not specified in initial scenarios as they are to be specified by later steps of the algorithm. Among these choices, the algorithm proceeds with the first choice (partial scenario   ) which is the case when node 0 is in BO state. We use search and implications to generate all valid scenarios leading from all initial scenarios (  ,   ,...,  ). When a scenario is determined to be valid, we apply forward search to evaluate its consequences. The forward search is controlled by protocol rules, e.g., retransmission limit. Finally, using time relations included in output scenarios, we compare all output scenarios and select the scenario that leads to the worst network throughput (i.e., the scenario that maximizes  considering all nodes in the topology). To reduce search complexity we use heuristics to prioritize search alternatives. The algorithm also outputs a library of MAC scenarios for which the network throughput is very close to the worst case throughput. To generate worst case fairness scenarios, the algorithm first generates the set of initial partial scenarios as presented in Figure 7. In these scenarios, one of the four nodes is in a state that reduces throughput and another node is in a state that increases throughput. Sections III-C1 and III-C2 present the basic concept of our search and implications. Search and implication algorithms operate on a tree structure. A tree node   represents a network scenario describing states of network nodes, protocol events and time relations between the state transitions and the

Inputs to test generation Node 0 in BO, Node 1 in Rx

Initial partial scenario E1

State descriptions: Node 0: starts in BO at time t1, ends at time t2 Node 1: starts in Rx at time t3, ends at time t4 Event descriptions: Relations between time variables: t1≤ t3≤ t2 || t3≤ t1≤ t4

Fig. 7.

….

Node 0 in BO, Node 2 in Rx (E2)

Node 1 in BO, Node 2 in Rx (E5)

Node 0 in BO, Node 1 in BO, Node 3 in Rx Node 0 in Rx (E3) (E ) 4

Initial partial scenarios for fairness. Test scenario generation algorithm 2. Implication procedures

1. Search procedures

specify_scenario

create_tree enumerate

bk_implication succ_rx supp_timer

create_pred

fw_implication non_supp_timer

check_exist test_exist

Fig. 8.

Hierarchy of procedures used in our algorithms.

events. A state or an event is commonly denoted by an entity in this section. Figure 8 presents a hierarchy of main procedures used in our algorithms. Note that procedure test-exist, succ-rx, supp-timer and non-sup-timer are core of all procedures. As discussed in following sections, procedure test-exist is used to determine the existence of an entity in a scenarios while the other core procedures are used to determine if a scenarios is invalid. We present Lemmas 1-8 that provide an outline for completeness and correctness of our algorithms. For details, see our technical report [30] (Section 5). 1) Search Algorithms and Concepts: If entity y is sufficient to create an entity x, entity y is said to be a predecessor of x. Note that an entity may have one or more predecessors except the initial state and event. An entity x in tree node   is justified if at least one of its predecessors exists in the scenario   . The entity is unjustified if none of its predecessors exists in the scenario. If all entities of a scenario are justified, it is called a fully specified scenario. If one or more entities of the scenario are unjustified, it is called a partial scenario. Two events are said to be incompatible if their event IDs are different, or sources are known and different, or destinations are known and different. Given two compatible entities, x at time t! and y at t " , where y exists in scenario # , the existence of x in $ (with respect to y) can be determined using time stamp test test-exist presented in Table I. TABLE I test-exist TO TEST EXISTENCE OF x W. R . T. y. t =t Test t t Test t t Test Output Unsolvable don’t care don’t care x New Solvable Unsolvable Unsolvable x Old Solvable Unsolvable Solvable x May-be-old Solvable Solvable Unsolvable x May-be-old Solvable Solvable Solvable x May-be-old

% &

%(' &

%() &

Lemma 1: Test test-exist accurately determines existence of entity x at t ! in a given scenario # . Proof: Assume x exists in # and the only compatible entity is y. Therefore, t ! must be the same as t " leading to a solution to the t ! =t" test, no solution to t !+* t " and t !-, t " tests. If x does not exist in   , there will be no compatible entity y such that t ! =t " test has a solution. In all other cases, it is undecidable. i. Enumeration procedure: Procedure check-existence checks the existence of an entity in a scenario. Given x, we first use procedure create-pred to create ./! , predecessor set of x and then check existence of each of these predecessors using procedure check-exist. If at least one of the predecessors in . ! exists in the scenario, x is deemed justified. If none of the predecessors of . ! exists in the scenario, x is deemed unjustified. Given a scenario   , we consider the unjustified entities to create its child nodes. Procedure enumerate takes cross product between predecessor sets of all unjustified entities to create child nodes of # . Lemma 2: Given a tree node # , procedure enumerate enumerates all child nodes rooted at # . Proof: Given an entity x in   , procedure create-pred creates all its predecessors from the transition table. It directly follows from Lemma 1 that we can determine whether x is justified in   accurately. Procedure enumerate only considers unjustified entities. Cross product of the predecessor sets of all unjustified entities enumerates all possible combinations of predecessors, and therefore, all child nodes rooted at   . ii. Test scenario generation procedure: Given an initial partial scenario, we generate test scenarios leading to the initial scenario using procedure create-tree. It applies a depthfirst search (DFS) technique and uses procedure enumerate to enumerate child nodes. The search continues until (1) we reach at a leaf node at which all entities are justified, or (2) the scenario is pruned using implication rules presented in Section III-C2. According to Lemma 2 the search algorithm generates all test scenarios leading to the initial partial scenario. We then compare the generated scenario with a previously generated scenario, using our objective, to identify the scenario that meets the objective. Lemma 3: All entities are justified in a leaf node, i.e. , a leaf node is indeed a fully specified scenario. Proof: Let x be unjustified in leaf node # . From the definition of justification, no predecessor of x exists in 0 . Procedure enumerate must create its predecessors to enumerate child nodes of # that contradicts with the initial assumption that $ is a leaf node. Lemma 4: Given an initial partial scenario described in  , procedure create-tree is guaranteed to generate a test scenario if it leads to  . It is guaranteed to prune a scenario if it does not lead to  . Proof: See Section V of this paper. 2) Implication Rules: Implication is the process of determining state and event history that can be uniquely identified as a consequence of a given partial scenario. If an entity

x in scenario $ can uniquely be created from entity y, then backward implication of x results y. Entity y is added to the scenario as a result of backward implication. If x uniquely creates entity z, i.e., z can uniquely be created from x, then forward implication of x adds z to   . We use implications to further specify a partial scenario. While specifying a scenario, our implication rules generate prohibited entities that are precluded from the scenario. The prohibited entities are precluded from the partial scenario as long as they satisfy the time relation indicated by the rule. In the process of implication, if an entity generated to be added to the scenario is determined to exist in the prohibited list, or vice versa, we prune the scenario. We also prune using a state consistency rule. Thus the function of implication is to specify a partially specified scenario as well as to prune inconsistent branches of the search tree. Successful reception rule: A reception of a message in a scenario $ is confirmed as successful when the corresponding receive-end (RE) event is generated in # . In our model, forward implication of a receive-start (RS) event does not generate the RE event. Therefore, an RE event is generated either by backward implication or by create-pred procedure, and always is implied as successful. Lemma 5: Given a successful reception of a message m defined by m-RE 21 3 at time t 4 at node k and corresponding m-RS21 3 at time t 5 in a tree node   , there should not be any event n-RS6 1 7 at time t 8 or n-RE6 197 at time t! in   such that n is a message of the protocol, k : (  and ;6 ), and t5=< t8 < t4 and/or t5< t! < t4 . Proof: Let there exist an event n-RS6 1 7 at time t8 (or t! ) in the scenario   during the successful reception interval [t 5 , t4 ]. From the definition of collision, at time t 8 the receiving node k must change its state to >@? leading to a garbled reception that contradicts our initial assumption of the successful reception. Lemma 6: Given a suppressive timer mT with the timer set event mT-TS A1  at time t 5 and the timer end event mTTE A1  at time t 4 at node i in a tree node   , there should not be any event e3 1  at time t 8 in $ such that n is the event of the protocol for which the timer mT is waiting, i is :B3 , and t5 < t8 * t4 . Proof: Let there exist an event e3C19D at time t8 in   during interval [t 5 , t4 ] for which the timer was running. From the definition of suppressive timer, the timer must time out at time t8 ( * t4 ) that contradicts the scheduled timer expiration event at time t 4 . Lemma 7: Given a non-suppressive timer mT defined by timer expiration event mT-TE  1  at time t 4 and set event mTTS  1  at time t 5 in a tree node $ , there should not be any transmission event n-TS  1 3 or n-TE 1 3 at time t 8 in $ such that n is a message of the protocol, j is an element of the set  , and t5 < t8 < t4 . Lemma 8: If a node i is in state s  from time t  to t  and in state s from time t  to tE such that s G F s , then it leads to inconsistency if these time intervals overlap.

i. Implication procedures: We use procedure specifyscenario to further specify a given scenario using two procedures bk-implication and fw-implication to perform backward and forward implications, respectively. We present the basic idea of implication using an example of protocol . . Let us perform backward implication of event RTS  1  transmitstart (RTS-TSC1H ) at t in a scenario. Event RTS-TS is output in row 1 and row 11 of the transition table (Figure 3). Hence the event has two predecessors, p  and p  . In order for the predecessor p  to be the one that triggers this event, node 0 must (1) be in state Idle for a period [t I , t J ] such that t I * t and t JGK t and (2) not be in state BO for a period [t 5 , t 4 ] such that t 5 * t and t 4 K t . Condition 1 confirms the necessary state in which the input event Packet can trigger the state transition to generate the RTS-TS event while condition 2 rules out the predecessor p  . If none or more than one of the predecessors satisfy the condition, we continue without any implication. While performing implication for an entity, we first carry out the process of elimination described above to eliminate predecessors. If number of predecessors left is more than one, we check if there exists an entity common to all predecessors that we can derive uniquely. Given a scenario presented in a tree node   , procedure specify-scenario applies both backward and forward implications on all entities of the scenario as long as the implication procedures generate new information. Note that fwimplication is similar to bk-implication with the exceptions that it starts with the rows of the transition table where the entity is an input. IV. P ERFORMANCE

EVALUATION OF

IEEE 802.11 DCF

IEEE 802.11 [17] is based on single channel multiple access schemes. The channel access is based on both physical and virtual carrier sense. We consider the protocol mechanisms in DCF which is designed for adhoc networks and is based on CSMA/CA. Each node maintains a Network Allocation Vector (NAV) timer to monitor the channel status. When node 0 of Topology I wants to transmit data to node 1, it senses the channel status. If the channel is idle for a predefined period (DIFS), it sends RTS to 1 indicating the period T which it expects to reserve the channel if RTS/CTS is successful. Node 0 schedules a WCTS timer to limit the period of time to wait for CTS. When node 1 receives RTS, if its NAV timer is not running, it replies with a CTS packet in the next frame slot. Otherwise, it discards the RTS packet. While sending CTS, node 1 schedules a WData timer to limit the period of time to wait for data. All other nodes in the range of transmitter and receiver update their NAV with the value as given in the RTS/CTS packets. During the period when a node has NAV running, it defers access to the channel. When node 0 receives CTS before its WCTS timer expires, it transmits data while scheduling WACK timer to wait for ACK. If ACK from node 1 does not arrive within this period, node 0 backs off and contends for the channel again. Upon reception of data from node 0, node 1 sends an ACK back to node 0.

BO

0

Rx

1

2

RTS0,1

3 RTS2,3

(a). Worst case MAC scenario

0

1

2

Flow0,1

3

State Descriptions: Node 1: starts in BO at time t1 and end at time t2 Node 1: starts in Rx at time t3 and end at time t1 Event Description: RTS2,3 receive start at t1 RTS0,1 receive start at t3 RTS0,1 transmit start at t4 RTS2,3 transmit start at t5 Time relation: t4 = t3 – time to transmit RTS t5 = t1 – time to transmit RTS t3 ≤ t1 ≤ (t3 + time to receive RTS)

Flow2,3

(b). Respective flow level scenario

Fig. 9.

Fig. 10.

Flow Description: Flow0,1 start at x1 Flow2,3 start at x2 Flow time schedule: x1 ≤ x2

Worst case throughput and fairness scenario in Topology I.

Scenario 1 (Sys 1):

0

1

2

3

4

5

6

Scenario 2 (Sys 2):

0

1

2

3

4

5

6

Scenario 3 (Sys 3):

0

1

2

3

4

5

6

Scenario 4 (Sys 4):

0

1

2

3

4

5

6

Scenario 5 (Sys 5):

0

1

2

3

4

5

6

Scenario 6 (Sys 6):

0

1

2

3

4

5

6

Flow level scenarios in a half loaded seven node chain..

A. Worst-Case Scenarios Generated by Our Framework Figure 9.(a) presents the MAC scenario leading to the worst case network throughput as well as the worst case fairness in Topology I as generated by our framework. In this scenario a collision of RTSML  and RTSNL  at node 1 leads to zero throughput at node 1 and maximum throughput at node 3. The forward search from the collision shows that the simultaneous transmission attempts by nodes 0 and 2 cause a failure for node 0 and a success for node 2 leading to backoff only for node 0. The failure of node 0 to access the channel is followed by another failed transmission attempt as it has a higher value of contention window compared to node 2. Such unfairness leads to a sequence of failed attempts for one of the nodes and a sequence of successful attempts for the other node leading to a throughput reduction that is greater than the other scenarios under comparison. The framework outputs all scenarios leading to the input condition including the one that leads to the worst performance. To evaluate the impact of these MAC scenarios on the end performance, we re-generate these MAC scenarios to facilitate simulation. Our MAC scenarios provide time relations between MAC external events (e.g., Packet). We use these relations to schedule the respective flows. For example, if a MAC scenario requires overlapping of RTS (triggered by Packet) messages, we schedule the respective flows to be concurrent in order to guarantee the overlap. Figure 9.(b) presents the corresponding flow level scenario that we simulate. Figure 10 presents a set of flow level scenarios that we re-generate from the library of automatically generated MAC scenarios for a seven node chain topology. Note that systematic Scenario 1 (represented as Sys 1) represents the worst case fairness and throughput scenario. Figure 11 graphically presents the worst case throughput scenarios generated for grid and cross topologies. Note that these scenarios are output of the framework among all scenarios the framework generates for the respective topologies.

2

3

0

1

RTS3,1

BO

RTS0,1

(a). 2x2 grid topology.

4 BO

0

RTS4,1

1

2

RTS2,1

RTS0,1

3

RTS3,1

(b). 3x3 cross topology.

Fig. 11.

Worst case throughput scenarios in grid and cross.

B. Simulation Results We simulate our flow level scenarios in order to evaluate the end performance achieved by our scenarios in simulation environment. We use ns-2 [31] for our simulation. We use a node distance of 250 meters. Transmission and carrier sensing ranges are set as 250 meters and 550 meters, respectively. To minimize the effect of routing we use static routing in all our simulations. We use the protocol in its 2 Mbps data rate mode. We use CBR flows with packet size of 1500 bytes. We use CBR data rates of 0.2, 0.42, 1, 2 and 2.4 Mbps to represent network condition below, at and beyond saturation point. Note that the network is at saturation point when CBR data rate is 2 Mbps. We run simulations for throughput and fairness experiments for 50 seconds and 300 seconds, respectively. We use ns monitors to obtain throughput achieved by all destinations of all flows of a scenario that we simulate. We use Equation 3 to compute index of unfairness achieved by the scenario. Index of unfairness =

Min IPOQOSRUTMOWVX8#Y (Throughput of Flow ZL 3 ) (3) Max IPOHOSR$TNO[V28#Y (Throughput of Flow ZL 3 )

We compute the index of unfairness for all scenarios we simulate and report the scenario that shows the minimum index as the worst case fairness scenario. Using our framework, we generate scenarios for chain (upto 20 nodes), grid (2x2 and 3x3) and cross (3+3 and 5+5) topologies. For each of these topologies, we simulate all scenarios generated by our framework to determine if the worst case scenario reported by our framework shows the worst case performance in simulation. For chain topologies we compare scenarios in which network load is similar. For limitation of space, we only report throughput and fairness results for chain topologies in which half of all nodes are transmitting. For all other cases of chain (e.g., fully loaded chains) and other topologies, the worst case scenarios reported by our framework show the worst performance in simulation. We compare scenarios presented in Figure 10 to identify which of these scenario leads to worst case network throughput and fairness using simulation. Among these scenarios Sys 1 shows the worst case performance in both of the performance metric. Figure 12 presents network throughput of these scenarios at five CBR rates. X-axis of these graphs represent the number of nodes in the chain topology and Y-axis represents total network throughput in Kbps. Note that for all network conditions, Sys 1 shows the worst throughput as we expected. Among these scenarios in half loaded chains, Sys 2 has been used in [28] to evaluate the performance of IEEE 802.11 DCF. The reduction of throughput achieved by Sys 1, our worst case scenario, compared to Sys 2 is about 21% and 19% at CBR rate of 0.2 and 0.4 Mbps, respectively.

(a) CBR rates = 0.2 Mbps

(d) CBR rates = 2 Mbps

2500

7000

Sys 1

Network throughput (Kbps)

Network throughput (Kbps)

8000 2000 Sys 2

1500

Sys 3 Sys 4

1000

Sys 5 Sys 6

500

0 0

5

10

15

20

25

Number of nodes

6000

Sys 1 Sys 2

5000

Sys 3

4000

Sys 4 Sys 5

3000

Sys 6

2000 1000

(b) CBR rates = 0.42 Mbps

0 0

4500

5

Network throughput (Kbps)

10

15

20

25

Number of nodes

4000 3500 Sys 1

3000

Sys 2

2500

Sys 3

2000

Sys 4 Sys 5

1500

Sys 6

1000

(e) CBR rates = 2.4 Mbps

500 0 0

5

10

15

20

8000

25

Number of nodes

Network throughput (Kbps)

6000

Sys 1

5000

Sys 2 Sys 3

4000

Sys 4

3000

Sys 5 Sys 6

2000

Network throughput (Kbps)

7000

(c) CBR rates = 1 Mbps 8000 7000

6000

Sys 1 Sys 2

5000

Sys 3

4000

Sys 4 Sys 5

3000

Sys 6

2000 1000 0

1000

0

5

0 0

5

10

15

20

10

15

20

25

Number of nodes

25

Number of nodes

Fig. 12.

Network throughput of scenarios in half loaded chain topologies.

TABLE II I NDEX OF UNFAIRNESS IN FOUR NODE HALF LOADED CHAIN SCENARIOS . Data rate 0.2 Mbps 0.42 Mbps 1 Mbps 2 Mbps 2.4 Mbps

Sys 1 0.61 0.61 0.23 0.07 0.06

Sys 2 1 1 0.92. 0.93. 0.96.

Sys 3 1 1 0.95 0.9 0.97

Sys 5 1 1 1 0.97 0.99

Sys 6 1 1 0.95 0.99 0.96

Table II presents index of unfairness calculated using Equation 3 achieved by scenarios Sys 1 thru Sys 6 at various data rates in four node chain topologies. An index of 1 represents a perfectly fair scenario. The lower the index the more unfair the scenario is towards the flow receiving the minimum throughput. Note that in all data rates, Sys 1 shows the most unfairness. Also note that the unfairness of scenario Sys 1 increases with data rate. Sys 1 has been used for analysis of unfairness of the protocol in [23]. Our results augment the previous work by showing that the scenario exhibits the worst case fairness in chain topologies among scenarios with similar load. We simulate scenario Sys 1 for longer chains (n = 6,..., 20) to evaluate the extent to which unfairness of the protocol exists. Simulation results show that unfairness towards a single flow tends to degrade as the chain gets longer, however, the unfairness on average tends to improve. V. A NALYSIS

OF THE

F RAMEWORK

Correctness and completeness: Given a partial scenario at a tree node $ , Lemma 2 guarantees that we enumerate all child nodes rooted at $ . We first construct the initial partial scenario and copy it to the root of search tree. We then apply a DFS search technique. The search and implication continue until we (1) reach a leaf node, or (2) we prune the scenario based on our implication rules. As we have shown that our algorithm detects a leaf node correctly, and prunes a tree node correctly, we can guarantee the completeness of our algorithm in the sense that it will always generate all valid scenarios that lead to the initial partial scenario. According to Lemma 3 the algorithm stops enumeration at a leaf node and hence, ensures the generation of valid scenarios. According to Lemmas 5, 6, and 7 the algorithm detects the existence of prohibited event entries in the scenario, while Lemma 8 shows that any

inconsistency in the state history is identified, and hence the algorithm correctly identifies all invalid scenarios to prune. Therefore, the completeness is guaranteed. A. Complexity Our search-based framework covers the whole search space and hence is guaranteed to generate the worst case scenario provided sufficient run-time is allowed. To reduce the time to generate these scenarios, we have developed three main techniques: 1) implications, 2) prohibited list (PL), and 3) heuristics. In this section, we present a brief analysis of our test results to evaluate the effectiveness of these techniques in reducing time complexity of our algorithms using the case study of IEEE 802.11. Case studies of MACA and MACAW also show similar reduction of complexity. However, for the limitation of space, we only report test results of IEEE 802.11 in this paper. Total number of states, events, and transitions of the protocol are 15, 31, and 132, respectively. As we discuss next, concepts of using implications and PL are generic and hence can be used to reduce complexity of any protocol under study. However, the heuristics we develop are specific to the class of CSMA/CA protocol, and hence may not be useful in reducing complexity of a different class of protocols. Complexity Reduction Techniques: Given a partial scenario, implication procedures are used to perform both spatial and temporal implications. Given an entity, spatial implication is used to derive states or events at different nodes in the network, while temporal implication is used to derive events occurred at different times. Implication procedures are also used to determine if the given scenario is invalid. Thus implication reduces complexity 1) by generating information earlier and without search, and 2) detecting and pruning an invalid scenario earlier in search tree. Prohibited list (PL) is a component of our scenarios that describes a list of events (with time relations) that should not occur. When an event is generated, it is checked against the PL. If the event is proven to match any event in the PL, we prune the scenario. Thus the sole purpose of using PL is pruning a partial scenario. To increase the likelihood of generating the worst case scenario in practical time we have implemented following two heuristics. The heuristics are based on the protocol performance objective and the class of protocols under study. According to heuristics single state (HSS), among alternative states, we choose a single state over a combined state. Thus it allows us to explore the simpler choices earlier. Note that this heuristic only chooses among alternatives. If a branch from the simpler choice is found to be inconsistent, we still explore all branches from the more complex choices. The second heuristic is based on the concept of dependency index of messages. If the precondition of message \ ! is another message \ " , then message \ ! is defined as dependent on message \ " and hence, the dependency index of message \ ! is one higher than that of message \ " . Dependency index of the external event is defined as zero. Among alternative messages, we choose a message with the least dependency index. If a subtree rooted at a less dependent message is pruned, it is likely that the subtrees rooted at the more dependent messages will also be pruned.

TABLE IV T EST RESULTS WITH I MPLICATION O N (T OPOLOGY I).

5 0

1

2

3

Test case

4

(a) Topology I 0

1

2

3

(b) Topology II

Fig. 13.

Topologies used for complexity analysis. TABLE III R EFERENCE TEST CASES W. R . T. REDUCTION TECHNIQUES . Test case no. 1 2 3 4 5 6 7 8 9-16

Implication On On On On On On On On Off

HSS Off Off Off Off On On On On Off/On

HDI Off Off On On Off Off On On Off/On

PL Off On Off On Off On Off On Off/On

Thus this heuristic not only chooses among alternatives, it also prunes subtrees rooted at more dependent messages if a subtree rooted at less dependent message is pruned. Thus the risk of using this heuristic is that we may prune a branch which might contain the worst case scenario. Test results: Figure 13 presents two topologies that we use to generate MAC scenarios using our framework. We input these topologies, a transition table of IEEE 802.11, and an objective function of throughput. All tests are run on a 2.1GHz Pentium IV computer. Wanted state generator first automatically generates following three wanted states, namely, BO, Defer1 and Defer2. The test generator then generates initial partial scenarios with four nodes and three wanted states. We turn each of the four techniques On or Off in order to evaluate the complexity of the test generator with and without the respective technique. For example, when we turn implication Off, no implication procedures are called. Table III presents test cases that we refer in this section to illustrate the usefulness of reduction techniques that we develop. For example, test case 1 refers to the case when implication is On while all other techniques are turned Off. Complexity reduction using implications: We first run tests to generate one valid scenario in Topology I. With implications Off (test cases 9 thru 16), no valid scenario is generated in 48 hour runs. In some cases, our tool runs for seven days at which point we terminate it manually. We perform the same set of experiments but with implication turned On (test cases 1 thru 8). Table IV presents execution times and pruned statistics of these tests. The columns entitled s and L represent total number of scenarios (valid or pruned) and the average path length of the scenarios, respectively. The column entitled ET represents the execution time (CPU time) for the run in seconds. Note that, we generate a valid scenario in the order of minutes, in the worst case and in the order of seconds in general with implication On. Therefore, implication reduces the complexity of search significantly as no scenarios are generated in couple of days when implication is Off and scenarios are generated in couple of minutes when it is On. Complexity reduction using prohibited list: In all cases

1 2 3 4 5 6 7 8

Valid scen. s L 1 8 1 8 1 8 1 8 1 4 1 4 1 4 1 4

by PL s L 0 -1 11 8 0 -1 1 7 0 -1 0 -1 0 -1 0 -1

Pruned scenarios by Impl. by HDI s L s L 46 9 0 -1 27 8 0 -1 2 6 9 5 1 5 9 5 0 -1 0 -1 0 -1 0 -1 0 -1 0 -1 0 -1 0 -1

Total s L 46 9 38 8 11 5 11 5 0 -1 0 -1 0 -1 0 -1

ET sec. 1683 1406 36 49 2 2 2 1

but one in Table IV, using PL does not help in reducing search time as we do not prune many scenarios using PL. However, when all heuristics are turned off (cases 1 and 2), we prune 11 scenarios earlier when using PL (case 2) which would otherwise be pruned by state consistency (i.e., implication) in later stage. In these cases, using PL reduces execution time as it prunes many scenarios earlier in search tree. Note that using PL is costly as we need to create prohibited entities and access the PL each time an event is generated which eventually increases execution time of the respective test. Thus using PL reduces the complexity of search when several branches are pruned using it. Prohibited list is used to prune scenarios both in the context of temporal and spatial implication. For example, successful reception at a node creates prohibited entities which must not have occurred with respect to the time of the successful reception. It also prohibits all events that could occur in the neighborhood of the receiver. In order to explore the use of PL in reducing complexity in the context of spatial implication, we run the same set of experiments for Topology II (Figure 13.(b)). Table V presents execution times of these experiments. Note that no valid scenario is generated in more than one hour for each cases when PL is Off. When PL is On, we are able to generate a valid scenario in the order of seconds. Branching factor of Topology II is much higher compared to branching factor of Topology I as node 1 has one more neighbor with a node degree of 2 (in Topology I, node 1 has only one neighbor of node degree 2). This leads to increased search complexity for which no valid scenario is generated in an hour. When PL is used, it specifies a scenario using prohibitions. Let us compare odd numbered cases (PL Off) with the even numbered cases (PL On). Using PL specifies scenarios for which a fully specified valid scenario is generated in couple of seconds. When PL is Off, these inconsistencies are detected by implications, however, no scenario is generated in an hour. Using PL reduces complexity of search in topologies with higher node degree, as PL may generate scenarios earlier by specifying prohibitions. Complexity reduction using heuristic HSS: Using HSS, we order choices during enumeration. Let us compare cases 1-4 (HSS Off) respectively with cases 5-8 (HSS On) of Table IV. In this example, when transitions of the FSM are not already ordered, using HSS reduces execution times from hundreds of seconds to couple of seconds. Note that execution times do

not increase when HSS is used as it does not cost to order choices. Using HSS in ordering choices may lead to valid scenarios faster. The reduction may be from the order of days to the order of seconds in the best case. Complexity reduction using HDI: Let us compare cases 1

automatically generated the worst case multihop scenarios for IEEE 802.11 networks that show as much as 60% reduction of throughput compared to the scenarios typically used for multihop performance evaluation of the protocol [25], [27].

TABLE V

[1] G. Holzmann, ”Design and validation of computer protocols”, Prentice Hall, ISBN 0L35399254, 1991. [2] Lin et al., ”Protocol verification using reachability analysis”, Computer Communication Rev., vol. 17, no. 5, pp. 126-135, Oct ’87. [3] R. Alur and D. L. Dil, “A theor of timed automata”, Theoretical Computer Science, 126(2), pp. 183-235, April ’94. [4] Berrada et al., “Testing communication systems: a model, a methodology and a tool”, Testing of communication systems ’05, pp. 111-128. [5] M. Kwiatkowska et al, “PRISM 2.0: a tool for probabilistic model checking”, QEST ’04, pp. 322-323, Sep ’04. [6] M. Kwiatkowska et al, “Probabilistic model checking of the IEEE 802.11 wireless local area network protocol”, PAPM/PROBMIV 2002. [7] M. Duflot, M. Kwiatkowska, G. Norman and D. Parker, “A formal analysis of Bluetooth device discovery”, ISOLA 2004. [8] P. Linz, ”Formal Languages and Automata”, 4th edition, Jones and Bartlett publisher, ISBN-13: 978-0-7637-3798-6, 2006. [9] T. Mitchell, ”Machine Learning”, 1st edition, McGraw-Hill Corp, ’97. [10] F. Hill and G. Peterson”, ”Introduction to the Theory of Switching Theory and Logical Design”, 3rd edition, John Wiley and Sons, 1981. [11] T. Booth, ”Sequential Machines and Automata Theory”, 1st edition, John Wiley and Sons, Inc., 1967. [12] M. Merritt, F. Modugno and M. R. Tuttle, ”Time constrained automata”, 2nd International Conference on Concurrency Theory, August 1991. [13] N. Lynch and F. Vaandrager, ”Forward and backward simulations for timing-based systems”, Real-Time: Theory in practice, June, 1991. [14] P. Bouyer, ”Untameable timed automata!”, STACS 2003. pp. 620-631. [15] N. Jha and S. Gupta, ”Testing of Digital Systems”, Cambridge University Press, 2003. [16] R. Jurdak et al, ”A Survey, Classification and Comparative Analysis of Medium Access Control Protocols for Ad Hoc Networks”, IEEE Communication Surveys and Tutorials, 1st quarter 2004. [17] IEEE Std 802.11-1997 Information Technology - telecommunications And Information exchange Between Systems-Local And Metropolitan Area Networks-specific Requirements-part 11: Wireless Lan Medium Access Control (MAC) And Physical Layer (PHY) Specifications. [18] P. Karn, ”MACA - A New Channel Access Method for Packet Radio”, 9th ARRL/CRRL Computer Networking Conference, pp. 134-140, 1990. [19] Bharghavan et al., ”MACAW: a media access protocol for wireless LANs”, SIGCOMM ’94, pp. 212-225, Sep ’94. [20] G. Bianchi, “Performance analysis of the IEEE 802.11 distributed coordination function”, IEEE JSAC, pp. 535-547, Mar 2000. [21] N. Gupta and P. R. Kumar, ”A Performance Analysis of the 802.11 Wireless Lan Medium Access Control”, Communications in Information and Systems, Vol. 3, No. 4, pp. 279-304, Sep ’04. [22] A. Abdrabou and W. Zhuang, ”Service time Approximation in IEEE 802.11 Single-hop Ad Hoc Networks”, IEEE ToWC, Jan ’08. [23] B. Bensaou, Y. Wang and C. C. Kuo, ”Fair Medium Access in 802.11 based Wireless Ad-Hoc Networks”, Mobihoc ’00, pp. 99-106,. [24] T. Nandagopal, T. E. Kim, X. Gao and V. Bharghavan, ”Achieving Mac layer fairness in Wireless Packet Networks”, Mobicom ’00, pp. 87-98. [25] J. Li, C. Blake, D. S. De Couto, H. Lee and R. Morris, ”Capacity of Ad Hoc Wireless Network”, Mobicom ’01, pp. 61-69, July 2001. [26] Barrett et al., ”Analyzing the short-term fairness of IEEE 802.11 in wireless multi-hop radio networks”, MASCOTS ’02, pp. 137-144. [27] J. K. Pathamasuntharam, A. Das, P. Mohapatra, A. Das, ”A Flow Control Framework for Improving Throughput and Energy Efficiency in CSMA/CA based Wireless Multihop Networks”, WowMom ’06. [28] K. Xu, M. Gerla and S. Bae, ”How Effective is the IEEE 802.11 RTS/CTS Handshake in Ad Hoc Networks?”, Globecom ’02. [29] S. Begum, A. Helmy and S. Gupta, ”Performance Analysis of Wireless MAC Protocols using a Search Based Framework”, MASCOTS ’07. [30] S. Begum, ”Test generation framework for evaluation of wireless MAC protocols”, USC EE Dept - CENG Technical report CENG-2007-5, http://ceng.usc.edu/assets/001/55933.pdf. [31] L. Breslau et al., ”Advances in Network Simulation”, IEEE Computer”, vol. 33, no. 5, pp. 59-67, May 2000.

T EST RESULTS WITH I MPLICATION O N (T OPOLOGY II). Test case 1 2 3 4 5 6 7 8

Valid scen. s L 0 -1 1 2 0 -1 1 2 0 -1 1 2 0 -1 1 2

by s 0 0 0 0 0 0 0 0

PL L -1 -1 -1 -1 -1 -1 -1 -1

Pruned scenarios by Impl. by HDI s L s L 124 12 0 -1 0 -1 0 -1 122 12 0 -1 0 -1 0 -1 126 12 0 -1 0 -1 0 -1 123 12 0 -1 0 -1 0 -1

ET sec. 3605 6 3604 6 3620 6 3670 6

and 2 (HDI Off) respectively with cases 3 and 4 (HDI On) of Table IV. Using HDI, in both cases, reduces execution times from the order of minutes to the order of seconds. In case 1, all scenarios are pruned by state consistency check at an L of 9, while in case 3, 9 scenarios are pruned at L of 5 using HDI. Similarly, among 27 scenarios that are pruned at an L of 8 in case 2, 9 scenarios are pruned at an L of 5 when using HDI in case 4. In other cases when HSS is On, using HDI does not reduce execution times. Now let us discuss the cost of using HDI. First, it does not increase execution time. However, HDI prunes branches based on likelihood that the branch might not lead to a valid scenario (unlike other pruning strategies that prune scenarios based on the fact that the branch will not lead to a valid scenario). This characteristics of HDI might lead to pruning of a branch which in fact contains a valid scenario. So, there is always a risk of using this heuristic. Using HDI, in the best case, can reduce complexity from the order of minutes to the order of seconds. However, in the worst case, it can prune a branch actually leading to a valid scenario. VI. S UMMARY

AND

F UTURE W ORK

We propose a framework for worst case performance evaluation of wireless adhoc MAC protocols. Given a protocol performance metric, we use search and implication based test generation algorithms to generate the worst performance scenario. We have used our framework to analyze worst throughput and fairness of IEEE 802.11 DCF. For all topologies, the worst case scenarios generated by our framework show the worst performance among all scenarios that we generate. The scenarios generated by our framework include the scenarios typically used for performance evaluation of IEEE 802.11. The case studies of 802.11, MACA and MACAW show that the complexity of our algorithms are quite practical. We use our framework to generate worst case throughput scenarios for fully loaded chains in which all nodes are transmitting and receiving. In these cases, we generate multihop scenarios using another level of abstraction on top of our MAC layer model, and hence, is not within the scope of this paper. Using this higher level of abstraction, we have

R EFERENCES