Markov Chain Existence and Hidden Markov Models in Spectrum

1

Markov Chain Existence and Hidden Markov Models in Spectrum Sensing Chittabrata Ghosh, Student Member, IEEE, Carlos Cordeiro, Member, IEEE, Dharma P. Agrawal, Fellow, IEEE, and M. Bhaskara Rao (Invited Paper)

Abstract—The primary function of a cognitive radio is to detect idle frequencies or sub-bands, not used by the primary users (PUs), and allocate these frequencies to secondary users. The state of the sub-band at any time point is either free (unoccupied by a PU) or busy (occupied by a PU). The states of a sub-band are monitored over L consecutive time periods, where each period is of a given time interval. Existing research assume the presence of a Markov chain for sub-band utilization by PUs over time, but this assumption has not been validated. Therefore, in this paper we validate existence of a Markov chain for subband utilization using real-time measurements collected in the paging band (928-948 MHz). Furthermore, since the detection of idle sub-bands by a cognitive radio is prone to errors, we probabilistically model the errors and then formulate a spectrum sensing paradigm as a Hidden Markov model that predicts the true states of a sub-band. The accuracy of our proposed method in predicting the true states of the sub-band is substantiated using extensive simulations. Index Terms—Cognitive Radio, Hidden Markov Models, Markov chain, Spectrum sensing, Viterbi Algorithm

I. I

E

VER increasing bandwidth demand for wireless devices has compelled regulatory bodies and the research community to explore innovative strategies and techniques that can offer improved radio spectrum utilization without adversely affecting admissible interference to existing incumbent users, also referred to as the primary users (PUs). Due to spatial and C. Ghosh is with the Department of Computer Science, University of Cincinnati, Cincinnati, OH-45221 USA. e-mail: [email protected] C. Cordeiro is with Intel Corporation, Oregon, USA. e-mail: [email protected] D. P. Agrawal is with the Department of Computer Science, University of Cincinnati, Cincinnati, OH-45221, USA. e-mail: [email protected] M. B. Rao is with the Center for Genome Information, College of Medicine, University of Cincinnati, Cincinnati, OH-45267, USA. e-mail: [email protected]

temporal variation of spectrum utilization by the PUs, static allocation of wireless bandwidth is observed to be spectrally inefficient. Therefore, dynamic use of wireless spectrum seems to be the panacea for higher spectral efficiency. With its opportunistic spectrum sharing capability, cognitive radio (CR) [1] has emerged as a technology by which spectrum white spaces - that is, spectrum not in use by PUs - can be identified and reused by what are known as the secondary users (SUs). A primary role of a CR is to detect white spaces and opportunistically allocate them among requesting SUs based on availability over time and space. This, in turn, is conducive to increasing the spectral efficiency and the channel capacity. The sub-band occupancy at any time instant can be considered as a state, which can be either free (unoccupied by a PU) or busy (occupied by a PU). The states of a sub-band are monitored over L consecutive time periods, where each time period is of a given time interval. Existing research [2]- [6] assume existence of a Markov chain, representing utilization of each sub-band by a PU over L time periods. However, to the authors’ best knowledge, this fundamental assumption has never been validated while it is critical that it be done for each frequency band of interest. Further, the constituents of a Markov chain, namely, initial probability and transition matrix need to be estimated and then utilized in analytical modeling. In this paper, we validate existence of a Markov chain by collecting real-time measurements [7] in the paging spectrum (928-948 MHz). While in this paper we focus on the paging spectrum, the same methodology can be applied to any other spectrum band. Since the true states (occupancy by PUs in reality) of a sub-band are never known (i.e., hidden) to the CR, in [2]- [5] the authors have extended their idea of improvising Hidden Markov model (HMM) in spectrum sensing. One of the critical parameters of HMM is the set of emission probabilities [8], i.e., emission

2

of an alphabet out of a set of alphabets by a hidden state. The authors have used well-known algorithms to predict the set of emission probabilities. In this paper, we exploit a novel idea in defining the emission probabilities. Additionally, with the parameters of the HMM and error probabilities known, obtaining the likelihood solution is faced with computational complexities. In this paper, we use the Viterbi algorithm to reduce complexity. We assess the effectiveness of our method in predicting the true states of the sub-band by performing extensive simulations. The code for the Viterbi algorithm is developed and its usefulness is checked using simulations. The rest of the paper is organized as follows. Section II covers various issues in spectrum sensing and detection. Section III breifly presents our system model using a Markov chain. Section IV validates the assumption of a Markov chain in the spectrum utilization by the PUs. Section V introduces the concept of HMM and how it fits well in designing an efficient spectrum sensing. Section VI evaluates our HMM approach for spectrum sensing in a Cognitive Radio Network (CRN) through simulations. Finally, Section VII draws the conclusion. II. S M  P F The configuration of the proposed spectrum sensing model for a specific sub-band is shown in Figure 1. The same model can be applied for several sub-bands within the operating spectrum. Power measurements are collected for a sub-band at regular time intervals (in seconds) spanning over an observation period (typically, hundreds of seconds). These measurements are translated into binary occupancy data Y that serves as the historic data for offline reference by the CR as shown in Figure 1. Based on the retrieved measurements, the CR is trained to perform a validation check to ensure Markovian property of the spectrum occupancy by PUs in the sub-band under consideration. If the sub-band occupancy follows a Markov chain, the Markovian parameters are estimated. The CR also simultaneously senses the spectrum for PU occupancy and passes this information X to the HMM block and is used in generating the predicted results 0 by the Viterbi algorithm. This predicted output X , as well as the output X generated by the CR, can now be compared with the actual PU occupancy Y to scrutinize the accuracy of the proposed prediction mechanism. The details of each block is described in the following sections. We define an observation period τ = {1, 2, · · · , T }, where each i in τ represents the ith sensing duration. We also define a sequence Y = {y1 , · · · , yT }, which

Figure 1. sensing.

The system model implemented for enhanced spectrum

represents the true states in the corresponding time periods. The entity yi =1 if the sub-band is free at the ith time instant and yi =0 otherwise. The CR output generated by a sensing mechanism is represented by a sequence X = {x1 , · · · , xT } of sensed states in the corresponding time periods. The entity xi = 1 if the state of the sub-band is sensed to be free at the ith sensing slot and xi = 0 otherwise. The sequence X represents the prediction of the true state sequence Y = {y1 , · · · , yT }. In reality, the true state sequence Y is unobservable. Collecting real-time measurements is time-consuming and expensive when considering an extensively wide spectrum over a period of weeks and months. Hence, a CR senses the spectrum and these sensing data gives rise to the sensed sequence X as described earlier. The readings provided by the sensing mechanisms are prone to errors in the form of mis-detections and false-alarms, and these can be in the order of 10% [6]. Therefore, the error probabilities are expressed as Pr(xi = 1|yi = 0) for the probability of mis-detection and Pr(xi = 0|yi = 1) for the probability of false-alarm. Our research work is an endeavor of enhancing the sensing accuracy of any sensed state sequence X obtained under the assumption that the true state vector Y and predictor X are governed by an HMM. III. M  M  T S   V A. Markov Chain modeling The sequence Y is modeled as a Markov chain, which is characterized by an initial distribution π = (p0 , p1 ) and one-step transition matrix P = (pi j )(2×2) , i, j ∈ S . This means that Y = y1 , y2 , · · · , yT is a Markov chain with state space S = {0, 1}, the distribution of y1 is π and Pr(yn = j|y1 = i1 , · · · , yn−2 = in−2 , yn−1 = i = Pr(yn = j|yn−1 = i), = pi j , for every i1 , i2 , · · · , in−2 , i, j ∈ S and 2 ≥ n ≤ T .

(1)

3

Table I S   E

These stipulations give the joint distribution of y1 , y2 , · · · , yT and is expressed as: Pr(y1 = i1 , y2 = i2 , · · · , yT = iT ) = pi1 pi1 i2 pi2 i3 · · · piT −1 iT , (2) for all i1 , i2 , · · · , iT ∈ S . B. Markov chain assumption validation As discussed before, previous research works have considered that the spectrum occupancy by the PUs follows a Markov chain model. Here we substantiate this assumption with results from real-time measurements conducted on the paging band (928 MHz to 948 MHz) [7]. To prove that the Markov chain model fits well with the PU occupancy, we follow a cross-validation technique. First, we define a threshold of −68.8 dBm for power values obtained from the experiment on five different sub-bands namely, 929.04, 929.06, 929.08, 929.10, and 929.56. The threshold is set to µ+3σ, where µ is the mean and σ is the standard deviation of the received signal power (in dBm) computed over 500 time sweeps, i.e., the observation period of our experiment. Each time sweep is of 1.68 seconds duration. Now for each time sweep, ranging from 1 to 500, we decide whether the power value is higher (state is 0) or lower (state is 1) than the defined threshold. Once we have the states over the observation period of 500 durations, we extract the probabilities required to obtain the transition matrix of the Markov chain using the first 400 readings. Next, we estimate the remaining 100 states using the transition matrix parameter. Since we conducted the experiment for one day, the initial distribution of the paging bands are not available. Hence, we assumed that out of the 100 readings, the first reading, i.e., the 401th power measurement is known to us. With this information, and the extracted transition matrix from the previous 400 readings, we estimated the remaining 99 states for each paging band. The statistical parameters of our estimation are provided in Table I, which proves that almost 92% of the time the predicted transition matrix matches the actual one computed based on the occupancy of the considered sub-bands. IV. HMM P E The key idea here is the introduction of an HMM in modeling the evolution of occupancy/non-occupancy of a sub-band by its PU over time using the measurements obtained by the CR. In this section, we outline the basic

Freq. (MHz) 929.04 929.06 929.08 929.10 929.56

Estimation Statistics (%) Min Max I Quartile III Quartile Mean 89 100 95 97 95.7230 83 98 90 93 91.2740 84 98 90 93 91.2090 83 97 90 93 91.2660 91 99 95 97 95.8170

a01

a00

Hidden States

yi = 0

yi = 1

a11

a10 e0(0)

e1(1) e0(1)

e1(0) xi = 0

Observed States

xi = 1

State Space S: {0, 1} Hidden States: yi ; Observed States: xi Transition matrix P = aij, i = 0, 1 and j = 0, 1 Initial distribution: p0 = Pr(yi = 0) and p1 = Pr(yi = 1) i

Emission probabilities, ey (xi)

Figure 2. Hidden Markov model representation in spectrum sensing

operational system of an HMM and then consider the model under the purview of CR. The true states Y of sub-band occupancy are never observable and are needed to be sensed using different sensing techniques. Hence, the Markov chain, constituting the true sequence Y, is hidden and the name for this type of model is hidden Markov model (HMM) [8], [9]. A HMM is a stochastic process created by two interrelated probabilistic functions. One of these functions is the above mentioned Markov chain with a finite number of states. The other is a set of random functions, referred to as the alphabet, wherein each function generates a symbol related to a state in the Markov chain. The general concept of an HMM is illustrated in Figure 2. A system over discrete time 1, 2, 3, · · · is changing stochastically from one state to another, within a defined state space S . Let Yn be the state in which the system is in at time n. The process is assumed to be Markovian. The evolution of the sequence Y1 , Y2 , · · · is hidden. However, the hidden sequence can be represented by a sequence of symbols from the alphabet Ω = {0, 1, 2, · · · , N}. A state

4

Table II E   Observed states/True states

0

or alternatively by using a-posteriori distributions over states [8]. 1

The output of spectrum sensing is can now be formulated on the basis of the generic HMM defined 1 δ (1 − ) above. The hidden sequence is the sub-band occupancy sequence Y = Y1 , · · · , YT , the observed sequence k can produce a symbol b from a distribution over all X1 , · · · , XT is the sequence of decisions generated by possible symbols b = 0, 1, · · · , N and its probability can the sensing technique used by a SU. In the spectrum sensing HMM, the challenge lies in framing emission be represented as: probabilities which has precise correlation with output of ek (b) = Pr(Xn = b|Yn = k). (3) any spectrum sensing technique. Once the emission probabilities are computed, maximum likelihood approach [8] These probabilities are known as emission probabilities can be adopted to estimate the hidden sequence Y. as shown in Figure 2. The system in state i can emit any Therefore, here we follow the following steps: one of the symbols from the alphabet with the following • Develop emission probabilities; distribution: • Correlate these probabilities with a spectrum sensing technique; and S tate o f the system : i, i = 0, 1, 2, · · · , M • Estimate the hidden sequence Y using maximum Alphabet : 0 1 2 ··· N likelihood approach. Emission probability : ei (0) ei (1) ei (2) · · · ei (N). Once the sensed sequence is obtained from a sensing technique, the maximum likelihood approach calculates Let Xn be the emitted symbol by the system at time n. The process X1 , X2 , · · · is independent with each Xn the probabilities of all possible sub-band occupancy setaking values 0, 1, 2, · · · , N with the following distribu- quences, i.e., the joint occurrence of the sensed sequence and the occupancy sequence. This joint occurrence of tion: both the sequences is interpreted as the joint distribution Pr(Xn = b|Yn = i) = ei (b), of the two sequences. The maximum likelihood approach consists of the following procedure: b = 0, 1, 2, · · · , N, and • Step 1: Compute the joint distribution of the sensed i = 0, 1, 2, · · · , M. (4) sequence x = X1 = x1 , X2 = x2 , · · · , XT = xT and The process X1 , X2 , · · · is observable. In the context of a possible sub-band occupancy sequence y = Y1 = prediction of the sub-band occupancy, symbol b and the y1 , Y2 = y2 , · · · , YT = yT ; ith state ranges between 0 and 1. As mentioned earlier, • Step 2: Compute joint distributions for all possible the predicted states obtained from a CR can be prone to sub-band occupancy sequences; and errors. The probability of predicting a state to be free • Step 3: Find the distribution which gives the maxwhen it is actually occupied is known as probability imum probability and the corresponding sub-band of mis-detection (PMD) [6] denoted by δ. Similarly, occupancy sequence is the estimate of the true subprobability of predicting a state to be occupied when band occupancy. it is free is known as probability of false-alarm (PFA) We define P(x; y) as the the joint distribution of the denoted by . Using Eq. (4), PMD and PFA can be sequence x generated by the spectrum sensing technique mathematically expressed as: and the occupancy sequence y. The joint distribution can be written as: Pr(Xn = 0|Yn = 0) = e0 (0) or (1 − δ), Pr(x; y) Pr(Xn = 1|Yn = 0) = e0 (1) or δ, = Pr(x1 , x2 , · · · , xT ; y1 , y2 , · · · , yT ) Pr(Xn = 0|Yn = 1) = e1 (0) or , = Pr(getting the data x under the path y) Pr(Xn = 1|Yn = 1) = e1 (1) or (1 − ). (5) = [Pr(Y1 = y1 )Pr(X1 = x1 |Y1 = y1 )] × Therefore, the emission probability matrix is specified in [Pr(Y2 = y2 |Y1 = y1 )Pr(X2 = x2 |Y2 = y2 )] × Table II. [Pr(Y3 = y3 |Y2 = y2 )Pr(X3 = x3 |Y3 = y3 )] × · · · 0

(1 − δ)



Based on the observed process, the hidden sequence can be estimated, either by finding the most likely one,

[Pr(YT = yT |YT −1 = yT −1 )Pr(XT = xT |YT = yT )]. (6)

5

Using notations for transition probability and emission probability defined in Eqs. (1) and (3), Eq. (6) can be further written as: Pr(x; y)

3000

4000 Pfa = 0.2, Mean = 72.30, Std = 5.11

2500 2000 1500 Frequency 1000

Pfa = 0.15, Mean = 75.94, Std = 5.04

3000 2000 Frequency 1000

500

= [py1 ey1 (x1 )] × [ay1 y2 ey2 (x2 )] × · · · × [ayT −1 yT eyT (xT )]. = p y1 ×

ΠTi=1 ayi yi+1 eyi (xi ).

(7)

Eq. (7) shows that the probability P(x; y) can be computed if the initial distribution, transition matrix, and emission probabilities are known. The occupancy sequence Y1 = y∗1 , Y2 = y∗2 , · · · , YT = y∗T , which maximizes P(x; y) over all paths y, is the path y∗ we seek. The predicted path y∗ = (y∗1 , y∗2 , · · · , y∗T ) is called the maximum likelihood sequence. For a given data x, the likelihood prediction of the underlying path of the Hidden Markov chain requires computation of the joint probability P(x; y) for every possible path y of length T . Even for moderate values of T , the set of all possible paths is astronomically large. For example, if T = 100 and M = 1, the number of paths is 2100 , which is incomprehensible to handle. The computational complexity of the likelihood approach involves 2T × (2T ) multiplications. Computation time required to find the maximum likelihood sequence on a Intel 3.2 GHz processor with 1GB RAM is 21.303348 seconds for T = 10, 86.221069 seconds for T = 12, and 373.666768 seconds for T = 14 for 10, 000 iterations. V. V  S  We have considered a typical case to validate our proposed model and determine accuracy of the Viterbi algorithm. The transition matrix is defined below. ! 0.3 0.7 T ransition matrix P = . 0.2 0.8 The initial distribution for each case is determined by the following steady-state equation: (p0 , p1 ) × P = (p0 , p1 ) p0 + p1 = 1.

(8)

We have simulated the Viterbi algorithm under two different scenarios: Scenario 1 : δ = 0.05 and (1 − ) = 0.95, 0.9, 0.85 and 0.8. Scenario 2 :  = 0.05 and (1 − δ) = 0.95, 0.9, 0.85 and 0.8. In the simulation work for each case, the initial distribution and transition matrix are fixed. The process consists of four steps. • Step 1: Using the initial distribution and transition matrix, simulate the Markov chain of length L = 100,

0 60

70

80

90

0 50

100

60

3000 2500 2000

80

90

100

3000 Pfa = 0.1, Mean = 79.70, Std = 5.03

Pfa = 0.05, Mean = 84.22, Std = 4.30

2500 2000

1500 Frequency 1000

1500 Frequency 1000

500

500

0 50

70

Percentage of accuracy

Percentage of accuracy

60 70 80 Percentage of accuracy

0 60

90

70 80 90 Percentage of accuracy

100

Figure 3. Frequency distribution of prediction accuracy percentage of the Viterbi algorithm with mis-detection probability (Pmd) δ = 0.05 and false alarm probability (Pfa)  specified in the inset of each histogram (Case I, Scenario 1)

2500 2000 1500 Frequency 1000

4000 Pmd = 0.2, Mean = 75.60, Std = 5.1

3000 2000 Frequency 1000

500 0 50

60

70

80

90

100

0 50

Percentage of accuracy

70

80

90

100

90

100

4000 Pmd = 0.1, Mean = 76.44, Std = 4.95

3000

2000

2000

Frequency 1000

Frequency 1000

0 50

60

Percentage of accuracy

4000 3000

Pmd = 0.15, Mean = 76.07, Std = 5.00

60

70

80

Percentage of accuracy

90

100

0 50

Pmd = 0.05, Mean = 76.74, Std = 4.96

60

70

80

Percentage of accuracy

Figure 4. Frequency distribution of prediction accuracy percentage of the Viterbi algorithm with  = 0.05 and δ specified in the inset of each histogram (Case I, Scenario 2)

leading to a path y1 , y2 , · · · , y100 . • Step 2: Under each scenario for each choice of  and δ, generate data x1 , x2 , · · · , x100 using the simulated path y1 , y2 , · · · , y100 . • Step 3: Apply the Viterbi algorithm detailed in Section IV to the data x1 , x2 , · · · , x100 to predict the underlying path as y∗1 , y∗2 , · · · , y∗100 . • Step 4: Calculate prediction accuracy (PA) by PA =

#{1 ≤ i ≤ 100 : y∗i = yi } × 100. 100

(9)

Repeat Step 1 to 4 for 10,000 times. The histogram of these PA percentages are as shown in Figure 3 (Scenario 1) and Figure 4 (Scenario 2). Under scenario 1 shown in Figure 3 (δ = 0.05

6

and  = 0.05, 0.1, 0.15, 0.20), the prediction accuracy decreases as  is increased. The standard deviation (Std) of accuracy is more or less stable around 5.0. For the chosen initial distribution and transition matrix, there will be high propensity of true state being 1 in the path generated by the Markov chain and transitions 0 to 1 and 1 to 1 are much more common. If the false alarm (reading 1 as 0) probability increases, the accuracy also decreases. Under scenario 2 shown in Figure 4 ( = 0.05 and δ = 0.05, 0.1, 0.15, 0.2), the prediction accuracy is more or less stable around 76% and the standard deviation of accuracy is also stable around 4.98. It is less frequent to have the State 0 in the path generated by the Markov chain and the percentage mis-readings (reading 0 as 1) do remain stable. VI. C  F  Existing research assumes the existence of Markov chain in the spectrum occupancy by PUs. In this paper, we have validated this assumption using real-time measurements collected in the paging band (928-948 MHz). In practice, the true path of the states is hidden to the SU and the only data available to the SU are the sensed data. Hence, the spectrum sensing is prone to errors in the form of mis-detection and false alarm. Therefore, here we exploit these probabilities to frame the spectrum sensing problem into a Hidden Markov model paradigm. We have also used the likelihood method for predicting true states. The computational complexity that arises due to likelihood approach is then reduced and validated by using the Viterbi algorithm. Finally, we have performed simulations to illustrate the prediction accuracy of the Viterbi algorithm. As future research, we plan to conduct extensive simulations to determine empirical relationship between prediction accuracy and parameter values. In addition, we plan to explore the use of the Baum-Welch algorithm for estimating parameters. R [1] B. Fette, Cognitive Radio Technology, Communication Engineering Series, Elsevier, Burlington, MA01803, USA. [2] T. W. Rondeau, C. J. Rieser, T. M. Gallagher, and C. W. Bostian, “Online modeling of wireless channels with hidden markov models and channel impulse responses for cognitive radios,” 2004 IEEE MTT-S Digest, pp. 739-742. [3] K. Kim, I. A. Akbar, K. K. Bae, J-S. Um, C. M. Spooner, and J. H. Reed, “Cyclostationary approaches to signal detection and classification in cognitive radio,” IEEE 2007, pp. 212-215. [4] C-H. Park, S-W. Kim, S-M. Lim, and M-S. Song, “HMM based channel status predictor for cognitive radio,” IEEE Proc. of Asia-Pacific Microwave Conference, 2007.

[5] I. A. Akbar and W. H. Tranter, “Dynamic spectrum allocation in cognitive radio using hidden markov models: Poisson distributed case,” IEEE Southeast Conference, 2007, pp. 196-201. [6] Q. Zhao and A. Swami, “A decision-theoretic framework for opportunistic spectrum access,” IEEE Wireless Comm. Mag., vol. 14, no. 4, pp. 14-20, August 2007. [7] C. Ghosh, D. P. Agrawal, S. Pagadarai, and A. M. Wyglinski, “Statistical Spectrum Occupancy Modeling Employing Radio Frequency Measurements,” submitted to IEEE Trans. on Wireless Communications, December, 2008. [8] R. Durbin, S. Eddy, A. Krogh, and G. Mitchison, Biological Sequence Analysis, Cambridge University Press, UK, 2004. [9] T. Koski, Hidden Markov Models for Bioinformatics, Kluwer Academic Publisher, 2001.