EUROPEAN TRANSACTIONS ON TELECOMMUNICATIONS Euro. Trans. Telecomms. (in press) Published online in Wiley InterScience (www.interscience.wiley.com) DOI: 10.1002/ett.1139
Letter Information Theory Choosing the parameters of multiple-repetition strategiesy Thijs Veugen* TNO Information and Communication Technology, Delft, The Netherlands
SUMMARY The class of multiple-repetition strategies is known to achieve capacity for specific discrete memoryless channels with noiseless feedback. We show, given an arbitrary discrete memoryless channel with noiseless feedback, that the repetition parameters should be chosen close to the self information of a transmission error to maximise the transmission rate. Furthermore, it is indicated how close the channel capacity can be approached in the general case. Finally, we compare some results on a related problem: the capacity of a discrete noiseless channel with constraints on the input symbols. Copyright # 2006 AEIT.
1. INTRODUCTION In Reference [1], Schalkwijk presented repetition strategies for the binary symmetric channel with noiseless feedback. These block-coding schemes achieve capacity for several values of the channel error probability and are easy to implement. Later, Schalkwijk and Post [2] showed that similar recursive coding schemes exist with fixed coding delay as well as variable coding delay. The computed bit error exponent for their variable coding delay is comparable to Horstein’s [3], which is the largest known error exponent for the binary symmetric channel with noiseless feedback, although Horstein’s scheme is rather difficult to implement. Becker [4] generalised Schalkwijk’s repetition strategies to multiple-repetition strategies for arbitrary discrete memoryless channels with noiseless feedback. In [5], Veugen showed that for each multiple-repetition strategy, a discrete memoryless channel with noiseless feedback
exists for which this multiple-repetition strategy achieves capacity. Later results focus on recursive strategies rather than block-coding schemes. Ooi [6] developed a class of strategies that are also applicable to the discrete memoryless channel with noiseless feedback. These iterative strategies achieve high transmission rates and large error exponents. More recently, Tchamkerten [7, 8] presented variable length coding schemes for discrete memoryless channels with feedback but with unknown error probabilities. Shulman [9] discusses a sequential decoding scheme for channels with unknown error probabilities, with or without feedback. And Simsek [10] focuses on the anytime channel coding problem for discrete memoryless channels with feedback, where ‘anytime’ denotes that the estimate of any given bit is permitted to change at any time and the estimate monotonically improves in time. The work of Kramer [11] and Kim [12] is mentioned as an example of recent related work on Gaussian channels with feedback.
* Correspondence to: Dr. Thijs Veugen, TNO Information and Communication Technology, PO Box 5050, 2600 GB Delft, The Netherlands. E-mail:
[email protected] y
The research for this letter was done at Eindhoven, University of Technology [13].
Copyright # 2006 AEIT
Received 4 January 2006 Revised 25 April 2006 Accepted 29 May 2006
THIJS VEUGEN
In this paper we focus on multiple-repetition strategies and show, given an arbitrary discrete memoryless channel with noiseless feedback, that the repetition parameters should be chosen close to the self-information of a transmission error to maximise the transmission rate, and how close this transmission rate approaches channel capacity. The simplest case, namely the binary symmetric channel, is considered first. Then, the general case is considered. In the next section is described how the remaining parameters of multiple-repetition strategies, namely the symbol precoding distribution, can be computed. Results of a related problem are compared, namely the problem of computing the capacity of a discrete noiseless channel with constraints on the input symbols. We complete this paper with an example and conclusion.
2. THE BINARY SYMMETRIC CASE Each binary symmetric repetition strategy is characterised by its repetition parameter k. The strategy works as follows: when a 0 is erroneously received as a 1, then this transmission error is ‘corrected’ by sending k additional 0 s. Similarly, when a 1 is erroneously received as a 0, this transmission error is ‘corrected’ by sending k additional 1 s. Note that the transmitter knows when a bit is erroneously received due to the noiseless feedback link. The receiver will scan the received sequence for occurrences of 10k and 01k , and replace them by 0 and 1 respectively, thereby ‘correcting’ the transmission error. Due to the error correcting mechanism all input sequences have to be precoded such that the subsequences 10k and 01k do not occur. It follows that for a binary symmetric repetition strategy to make sense, the repetition parameter k has to satisfy k 3. The type of coding scheme that can be used in the binary symmetric case can be either block coding [1], or recursive coding with fixed or variable delay [2]. Suppose a binary symmetric repetition strategy with repetition parameter k is used for a binary symmetric channel with error probability p. Then the transmission rate R for this strategy is easily computed [1] as: R ¼ ð1 kpÞlog2 ck ;
ð1Þ
where ck is the solution c > 1 of ck 2ck1 þ 1 ¼ 0. This strategy is known [5] to achieve capacity when p ¼ pk ; pk being the solution 0 < p < ½ of 2pð2 2pÞk1 ¼ 1. It is easily shown that pk ¼ 1 ½ ck . Copyright # 2006 AEIT
Since the graphical presentation of R as a function of p is a straight line that hits the capacity curve C ¼ 1 hðpÞ at p ¼ pk , the distance between R and C satisfies: C R ¼ Dð pkpk Þ;
ð2Þ
where D denotes the informational divergence. Equation (2) is easily generalised [13] to all symmetric channels as defined by Gallager [14, p. 94]. It would be interesting to find an expression such that k can be derived from pk . Such an expression could be used to determine the optimal repetition parameter for arbitrary error probability p. Lemma 1: Let k 3 and let pk be the solution 0 < p < ½ of 2pð2 2pÞk1 ¼ 1, then k ¼ dlog2 pk e. Proof: Define the function f ðpÞ for 0 < p < ½ as 2pð2 2pÞk1 1. The number pk is the unique solution of f ðpÞ ¼ 0. Since f ð2k Þ ¼ ð1 2k Þk1 1 < 0 and f ð21k Þ ¼ 2ð1 21k Þk1 1 > 0, it follows that 2k < pk < 21k . Consequently k 1 < log2 pk < k, and thus k ¼ dlog2 pk e. & The result of Lemma 1 can be explained as follows. The number log2 pk equals the self information of an erroneous transmission that occurs with probability pk. So, in order to inform the receiver that an error occurred, the transmitter would need at least log2 pk bits to transmit this information. Indeed, for a binary symmetric repetition strategy, this message consists of k repetitions of the bit that was received erroneously. The following Theorem shows that for arbitrary error probability p, the repetition parameter k ¼ dlog2 pe is a good choice for maximising the transmission rate. Theorem 1: Let k 3 and p > 0 such that k ¼ dlog2 pe. Let C be the channel capacity 1 hðpÞ and R the transmission rate, then C R ¼ Dðpkpk Þ < p. Proof: From Lemma 1 follows that k ¼ dlog2 pk e, so both p and pk are in the interval ð2k ; 21k Þ, so pk =2 p 2pk . Since C R ¼ Dðpkpk Þ (see Equation (2)), the worst cases are p ¼ pk =2 and p ¼ 2pk . First, suppose p ¼ 2pk , then by definition of informa1p tional divergence Dðpkpk Þ ¼ p þ ð1 pÞ log2 1p=2 < p. Second, suppose p ¼ pk =2, then Dðpkpk Þ ¼ p þ _ 2pÞÞ ¼ p ðð1 pÞ = ln 2Þ ð1 pÞ log2 ðð1 pÞ=ð1 lnð1 p=ð1 pÞÞ. Since ln ð1 xÞ x þ x2 for 0 x ½, it follows that Dðpkpk Þ p ð1=ln 2 1Þ þ p2 = ð1 pÞ ln 2. Since p ¼ pk =2 < 21k =2 1=8, we conclude that Dðpkpk Þ < pð1=ln 2 1þ1 =ð7 ln 2ÞÞ < p. & Euro. Trans. Telecomms. (in press) DOI:10.1002/ett
MULTIPLE-REPETITION STRATEGIES
Since . . . < 2k pk < 21k pk1 < . . . , two consecutive pk differ approximately a factor 2. Furthermore, because for each fixed factor g > 0, lim p!0 Dðpkg pÞ= p ¼ ðg 1Þ=ln 2 log2 g, (i.e. a positive constant) we know that the difference between channel capacity and transmission rate will increase by a multiple of p when a less optimal repetition parameter is chosen. In other words, the order of approximation in Theorem 1 is tight for decreasing error probability p. In the next section we will generalise the result of Theorem 1 to arbitrary discrete memoryless channels.
3. THE GENERAL CASE In the general case we have a discrete memoryless channel with input alphabet @, output alphabet }, and channel probabilities pxy for x 2 @; y 2 }. W.l.o.g. we assume @ }. For each channel error probability pxy we have a repetition parameter kxy ðx 2 @; y 2 }; x 6¼ yÞ. Note that kxy ¼ 1 when y 2 = @. As in the binary symmetric case, all input sequences have to be precoded such that the subsequences yxkxy do no longer occur. Especially for asymmetric channels, the precoding step is also needed to fix the symbol distribution of the input sequence in order to optimise the transmission rate [13, 5]. Let MðnÞ denote the set of @-ary sequences without the forbidden subsequences such that each symbol x in @ occurs exactly nx times. Then the precoding rate Rp as a function of the symbol precoding distribution q is defined as: logj@j MðqnÞ Rp ðqÞ ¼ lim n!1 n where the elements of the vector qn are rounded to the nearest integer. Let x denote the expected number of transmissions needed to send an x-symbol of the precoded sequence and correct all occurring transmission errors. As shown in Reference [5], the number x satisfies ! X 1 kxy pxy x ¼ 1 ð3Þ y2}; y6¼x
and the transmission rate R is consequently computed by Rp ðqÞ R ¼ sup X q qx x x2@
Copyright # 2006 AEIT
ð4Þ
We would like to generalise the results of the previous section to the general case. So, given an arbitrary set of channel error probabilities, we would like to find the repetition parameters that maximise the transmission rate (each set of repetition parameters has a different optimising (see Equation (4)) symbol precoding distribution). Inspired by the choice of the repetition parameter k in the previous section, a suitable choice for the repetition parameters in the general case would be 2 3 6 pxy 7 7 kxy ¼ 6 6logj@j X p 0 7 6 xy 7 6 7 y0 2@
ð5Þ
for each x; y 2 @; x 6¼ y. Note that when a symbol x is erroneously received as a symbol y such that y 2 = @, both transmitter and receiver know that this symbol is received erroneously and has to be retransmitted. Therefore the probability that a symbol x is erroneously received as a symbol y such that y 2 @ is actually 0 1n 1 X X pxy @ pxy pxy0 A ¼ X pxy0 n¼0 y02 =@ y0 2@
The following Theorem shows that these repetition parameters indeed lead to a transmission rate that approximates channel capacity. Theorem 2: Let @ and } be arbitrary sets of channel inputs and outputs respectively. Define kxy ¼ 1 for each x 2 @ and y 2 }=@. Suppose that for arbitrary channel error probabilities pxy , C denotes the channel capacity, the repetition parameters kxy satisfy Equation (5), and R satisfies Equation (4). Then there are constants "; c > 0 such that X X CRc pxy ; when pxy < " x2@;y2}; x6¼y
x2@;y2}; x6¼y
The result of Theorem 2 is comparable to the result of Theorem 1. It shows that even in the general case the repetition parameters should be chosen close to the self information of the transmission error in order to maximise the transmission rate. This choice becomes more critical for decreasing error probabilities. The example at the end of this article shows the values of all involved parameters for a specific (asymmetric) binary channel. Intuitively the result of Theorem 2 seems correct, but in order to rigorously proof Theorem 2, we need two additional lemmas. The first Lemma provides an interesting estimate of the precoding rate in terms of the repetition Euro. Trans. Telecomms. (in press) DOI:10.1002/ett
THIJS VEUGEN
parameters. The notation ¼_ is used to denote equality in order of magnitude (i.e. first order approximation), and H@ denotes @-ary entropy. Lemma 2: Let q be a symbol precoding distribution. Then the corresponding precoding rate satisfies ! X : kxy qy qx Rp ðqÞ¼ H@ ðqÞ þ logj@j 1 ð6Þ x;y2@; x6¼y
Proof: Let HðnÞ denote the set of @-ary sequences such that each symbol x in @ occurs exactly nx times. We derive: Prðm 2 MðqnÞm 2 HðqnÞÞ Y Pr ‘‘no forbidden subsequence at i’’m 2 H qn 1in
¼
Y 1in
1
X
1
Pr mi . . . miþkxy ¼ yx
x;y2@; x6¼y
X
! m 2 H qn
kxy
!n
qy qkxxy x;y2@; x6¼y
Consequently, M qn H qn
1
X
!n
qy qkxxy x;y2@; x6¼y
Taking logarithms to the basej@j at both sides and dividing by n leads to Equation (6). & According to Equation (4), the transmission rate is maximised by choosing a proper symbol precoding distribution. In the proof of Theorem 2 we choose the symbol precoding distribution q such that a capacity achieving channel input distribution Q is accomplished. Note that this is not sufficient for actually achieving channel capacity. The relation between a symbol precoding distribution q and its induced channel input distribution Q and vice versa, is determined by the numbers x in (3): qx x Qx =x Qx ¼ P or qx ¼ P ð7Þ qy y Qy =y y2@
By using the results of Lemma 2 and 3 we are able to proof Theorem 2. Proof of Theorem 2: As before P all logarithms are taken to the base j@j. Assume pP :¼ x2@;y2}; x6¼y pxy P < ". Since for : each Px 2 @, log y2@ pxy ¼ logð1 y2}n@ pxy Þ ¼ ln1j@j y2}n@ pxy for small enough ", we derive using Equation (5) that for each x; y 2 @, x 6¼ y: log pxy p < kxy < 1 log pxy . From this estimate of kxy an estimate of x can P be obtained. Using Equation (3) we obtain 1=x ¼ 1 þ y6¼x pxy log pxy þ OðpÞ, where OðpÞ is used to denote that the absolute difference is bounded by a constant times pPfor small p. Furthermore, since pxx log pxx ¼ P : pxx logð1 y6¼x pxy Þ¼ pj@jxx y6¼x pxy ¼ OðpÞ, we derive the nice equation: ð9Þ 1=x ¼ 1 H@ ðpx Þ þ OðpÞ where px equals the @-ary vector with components pxy, y 2 @. Let Q be an arbitrary channel input distribution. Let q be the symbol precoding distribution satisfying Equation (7). Then the transmission rate R(q) in Equation (4) that corresponds with P symbol precoding distribution q satisfies RðqÞ ¼ Rp ðqÞ x2@ Qx =x . From Equation (6) follows that ! X X kxy Qx =x þ O qx RðqÞ ¼ H@ ðqÞ ð10Þ
1 PLemma 3: Let Qx ¼ j@j þ "x for each x 2 @, such that x2@ "x ¼ 0. Then for small "x
x;y2@; x6¼y
x2@
P From P Equation (7) follows that H@ ðqÞ x2@ Qx = x ¼ x2@ ðQx =x Þlog using Pqx , which we split P Equation (7) P into x2@ ðQx =x Þlog Qx and x2@ ðQx =x Þlog ðx y2@ Qy =y Þ. Using Equation (9) the first part equals X H@ ðQÞ þ H@ ðpx ÞQx logQx þ OðpÞ
y2@
When the channel error probabilities are small, as is the case in Theorem 2, the capacity achieving channel input distribution will be close to uniform. The second lemma that we need to prove Theorem 2, gives an estimate of the entropy of a channel input distribution that is close to uniform.
Copyright # 2006 AEIT
j@j X 2 H@ Q ¼_ 1 " ð8Þ lnj@j x2@ x P P Proof: We derive H@ ðQÞ ¼ P x Qx logQx ¼ x Qx : 1 ðlog j@j þ logð1 þ j@j"x ÞÞ ¼1 ln1j@j x Qx j@j"x , from which Equation (8) follows. Note that all logarithms are to the base : j@j, and lnð1 þ "Þ ¼ " for small ". &
x2@
We will show that the second part is O(p). We frequently use that H@ ðpx ÞH@ ðpy Þ ¼ OðpÞ for arbitrary x; y 2 @, which follows from the fact that lim p!0 p log2 p ¼ 0. From Equation (9) follows that x ¼ 1 þ H@ ðpx Þ þ OðpÞ. When combining this estimate of x with the estimate of 1=y in Equation (9) we obtain X X Qy =y ¼ 1 þ H@ ðpx Þ Qy H@ ðpy Þ þ OðpÞ x y2@
y2@
Euro. Trans. Telecomms. (in press) DOI:10.1002/ett
MULTIPLE-REPETITION STRATEGIES
and consequently ln x
X
! Qy =y
¼ H@ ðpx Þ
y2@
X
Qy H@ ðpy Þ þ OðpÞ
y2@
Finally, we derive X Qx x2@
x
log x
X
!
x2@
Qy =y
y2@
! X 1 X Qx H@ ðpx Þ Qy H@ ðpy Þ þ OðpÞ ¼ ln j@j x2@ y2@ which indeed is O(p) because the first term equals 0. Summarizing, we transformed Equation (10) into RðqÞ ¼ H@ ðQÞ þ
X
H@ ðpx ÞQx logQx þ OðpÞ
x2@
þO
X
!
ð11Þ
qkxxy
x;y2@; x6¼y
On the other hand, define the channel capacity C(Q) for a given channel input distribution Q by: X X Qx pxy log pxy ð12Þ CðQÞ :¼ H} ðpÞ þ x2@
for some x0 with px0 y pxy . Combining the results of the first and the second observation shows that the double sum indeed is O(p). Summarizing, we obtain X Qx H@ ðpx Þ þ OðpÞ ð13Þ CðQÞ ¼ H@ ðQÞ
y2}
where p is the corresponding channel output distribution P which satisfies py ¼ x2@ Qx pxy for each y 2 }. Note that the channel capacity C is the maximum of C(Q) over all Q. Since py ¼ Qy þ OðpÞ for y 2 @, we obtain X Qx H@ ðpx Þ CðQÞ ¼ H@ ðQÞ
Remind that our goal is to prove C R ¼ OðpÞ by comparing Equations (11) and (13). By substituting a uniform channel input distribution in Equation (13) we obtain P ~ be the chanC 1 ð1=j@jÞ x2@ H@ ðpx Þ þ OðpÞ. Let Q nel input distribution that achieves channel Pcapacity: ~ x ¼ ð1=j@jÞ þ "x for each x 2 @ with Q x2@ "x ¼ 0: From the result of Lemma 3 and Equation (13) follows that: 1 X C ¼1 H@ ðpx Þ j@j x2@ X j@j X 2 "x H@ ðpx Þ " þ OðpÞ ln j@j x2@ x x2@ Combining Pthis with the previous Plower bound on C we derive that x2@ "x H@ ðpx Þ þ lnj@jj@j x2@ "2x is either negative or OðpÞ. We will derive that "2x ¼ OðpÞ for each x 2 @. First define x such that "x ¼ x H@ ðpx Þ, then X j@j X 2 X 2 j@j 2 "x H@ ðpx Þ þ " ¼ H@ ðpx Þ x þ ln j@j x2@ x ln j@j x x2@ x2@ ð14Þ
Each term in Equation (14) is negative if and only if lnj@jj@j x 0, so each negative term is OðH@2 ðpx ÞÞ and consequently OðpÞ. When the total sum in Equation (14) x2@ ! is negative, the sum of the negative terms exceeds the X X þ Qx pxy log pxy log Qx0 px0 y þ OðpÞ sum of the positive terms in absolute value, so each positive term must also be OðpÞ. Since the total sum is either x0 2@ x2@; y2}n@ negative or OðpÞ, we derive that the total sum is always We will show thatP the last (double) sum is O(p). The first OðpÞ. observation is that x0 2@ QP x0 px0 y Qx pxy , so the double The second step is to show that for each x, "x H@ ðpx Þ ¼ sum is upper bounded by x2@; y2}n@ pxy Qx logQx which H 2 ðp Þ ¼ OðpÞ. Three cases are distinguished: x @ x is indeed O(p). P * When ð2 ln j@jÞ=j@j x 0, then clearly The second observation is that x0 2@ Qx0 px0 y maxx0 2@ px0 y , so x H 2@ ðpx Þ ¼ OðpÞ ! X pxy 2 Qx0 px0 y pxy log pxy log pxy log * When x 0, then x H @ ðpx Þ is smaller than the sum p x0 y x0 2@ of all positive terms in Equation (14) which must be pxy pxy 1 OðpÞ since the total sum as well as the sum of all ¼ p x0 y log px0 y e ln j@j p x0 y p x0 y negative terms is OðpÞ. Copyright # 2006 AEIT
Euro. Trans. Telecomms. (in press) DOI:10.1002/ett
THIJS VEUGEN
*
When x ð2 ln j@jÞ=j@j, then jx j x þ ðj@j= ln j@jÞ 2x , so again jx jH 2@ ðpx Þ is smaller than the sum of all positive terms in Equation (14) and must therefore be OðpÞ.
We conclude that for each x, "x H@ ðpx Þ ¼ OðpÞ, and therefore, since the total sum in Equation (14) is OðpÞ, that P 2 " is OðpÞ. So "2x ¼ OðpÞ for each x 2 @. P x2@ x We continue by showing that the sum Px2@ H@ ðpx Þ Qx logQx of Equation (11) equals the sum x2@ Qx H@ ðpx Þ of Equation (13) up to O(p) when Q equals the capa~ This follows city achieving channel input distribution Q. : 1 from the fact that logj@j Qx ¼ ln1j@j lnðj@j ð1 þ j@j"x ÞÞ ¼ j@j 1þ ln j@j "x and "x H@ ðpx Þ ¼ OðpÞ. P k It only remains to show that the term Oð x;y2@;x6¼y qxxy Þ of Equation (11) is O(p) when Q equals the capacity ~ We know that achieving channel input distribution Q. ~ x ¼ 1=j@j þ "x and "2 ¼ OðpÞ for each x 2 @. By using Q x 1 Equation (9) we derive that Qx =x ¼ j@j ð1 H@ ðpx ÞÞþ "x þ OðpÞ. From Equation (7) follows that for q ¼ ~ q, the ~ symbol precoding distribution corresponding with Q: ! X 1 1 : 1 þ "x H@ ðpx Þ 1þ H@ ðpy Þ "y þOðpÞ; qx ¼ j@j j@j j@j y2@ so qx ¼
1 1 þ "x H@ ðpx Þ j@j j@j 1 X 1 H@ ðpy Þ "y þ OðpÞ: þ j@j y2@ j@j
ð15Þ
We know that log pxy p < kxy < 1 log pxy , so : log qkxy log pxy log pxy jlogðj@jqx Þjþjlog qx j¼ x j@j 1 qx þ Oð1Þ: log pxy ln j@j j@j Since for each x0 2 @ both "x0 log pxy and H@ ðpx0 Þlog pxy go to 0 when p approaches 0, we conclude using Equation k (15) that jlog qxxy log pxy j is bounded by a constant when k p approaches 0, and consequently qxxy ¼ OðpÞ: Finally, it follows that ~ Rð~ C R CðQÞ qÞ ¼ OðpÞ:
&
Note that in the proof of Theorem 2 we only used log pxy p < kxy < 1 log pxy , so we might just as well Copyright # 2006 AEIT
have chosen kxy ¼ dlog pxy e in Equation (5). However, for larger error probabilities Equation (5) will usually lead to higher transmission rates.
4. COMPUTING THE SYMBOL PRECODING DISTRIBUTION In the previous section we showed how to choose the repetition parameters for arbitrary discrete memoryless channels with feedback. When the strategy is implemented, the symbol precoding distribution also has to be known. Since fixing the symbol precoding distribution is only used to fine-tune the transmission rate, and it introduces an extra computational effort, one might consider allowing all precoded sequences as long as the forbidden subsequences yxkxy do not appear. For those applications where it’s worth to gain a slight improvement in transmission rate at the cost of extra precoding, we show how to compute the symbol precoding distribution for the general case. The first way is to follow the line of the proof of Theorem 2. There we choose the channel input distribution equal to a channel capacity achieving channel input distribution. Consequently, by using Equation (7), the corresponding symbol precoding distribution can be computed. Besides that it might be difficult to compute a capacity achieving channel input distribution, the corresponding symbol precoding distribution does not necessarily optimise Equation (4). The second way is to directly solve the optimisation problem in Equation (4). This can be done using Theorem 8 of Shannon [15, p. 400]. This theorem is related to Shannon’s [15, p. 384] Theorem 1. In his Theorem 1 Shannon describes how the capacity of a discrete noiseless channel with constraints on the input symbols can be computed and in Theorem 8 he shows how the corresponding state probability distribution can be computed. In Reference [13] is shown how exactly this result of Shannon can be applied to our type of constrained sequences. It is interesting to note that Shannon’s [15, p. 384] Theorem 1 is very similar to Veugen’s [6, p. 2223] Theorem 1. The first difference is that Veugen’s Theorem 1 only applies to a specific type of constrained sequences and Shannon’s to all types of constrained sequences. The second difference is that Shannon’s Theorem 1 only holds for integer durations, while Veugen’s Theorem also holds for real durations. In Reference [16] Veugen showed that his Theorem 1 can be generalised to arbitrary constrained sequences, thereby extending Shannon’s result to real valued durations. Euro. Trans. Telecomms. (in press) DOI:10.1002/ett
MULTIPLE-REPETITION STRATEGIES
5. EXAMPLE We illustrate the results of this article for a specific asymmetric binary channel [13] with error probabilities p01 ¼ 0.001 and p10 ¼ 0.003. The capacity of this channel is 0.9795744419 bits which is achieved by a channel input distribution of (0.5021500421, 0.4978499579). Since we should choose the repetition parameters close to the self information of the transmission errors we choose k01 ¼ 10 and k10 ¼ 8 (note that we use binary logarithms). The expected number of transmissions needed to correctly send a symbol is computed by Equation (3): 0 ¼ 1.010101010 and 1 ¼ 1.024590164. From the capacity achieving channel input distribution we can compute a nearly optimal symbol precoding distribution using Equation (7). Given the repetition parameters, this leads to a symbol precoding distribution of (0.5057103962, 0.4942896038) and a transmission rate of 0.9794932057. This transmission rate can be slightly increased to Equation (4) by fine-tuning the symbol precoding distribution. Using the results of Shannon [15, pp. 421–423] this optimal symbol precoding distribution can be computed: (0.5070054890, 0.4929945110). It leads to a transmission rate of 0.9794986025. In this example the difference between the channel capacity and the best transmission rate is about 0.00008 which is significantly smaller than the total error probability 0.004 from Theorem 2. To see the consequence of choosing non-optimal repetition parameters we compute the rate of a symmetric strategy with repetition parameter 9 on our example channel. The precoding rate in this case equals Rp ¼ log2 c9 which is 0.997134849 and the transmission rate is computed by Equation (4): 0.979104173. The difference with the channel capacity is about 0.0005 in this case, slightly more than for the best choice of repetition parameters. If we would degrade further to a repetition parameter of eight we would obtain a transmission rate of 0.978219846, so the difference with the channel capacity becomes indeed in the order of the total error probabilities.
6. CONCLUSION We showed that the class of multiple-repetition strategies has some nice information theoretic properties. The repe-
Copyright # 2006 AEIT
tition parameters should be chosen close to the self information of the corresponding error probabilities, which is intuitively right. With these repetition parameters the transmission rate approximates the channel capacity such that the first order approximation of the difference will be tight for small error probabilities. We also indicated how to compute the remaining strategy parameters, namely the symbol precoding distribution, thereby generalising Shannon’s [15, p. 384] Theorem 1.
REFERENCES 1. Schalkwijk JPM. A class of simple and optimal strategies for block coding on the binary symmetric channel with noiseless feedback. IEEE Transactions on Information Theory 1971; 3(17):283– 287. 2. Schalkwijk JPM, Post KA. On the error probability for a class of binary recursive feedback strategies. IEEE Transactions on Information Theory 1973; 4(19):498–511. 3. Horstein M. Sequential transmission using noiseless feedback. IEEE Transactions on Information Theory 1963; 9:136–143. 4. Becker DW. Multiple-repetition feedback coding. Ph.D. dissertation. University of California, 1973. 5. Veugen T. A simple class of capacity-achieving strategies for discrete memoryless channels with feedback. IEEE Transactions on Information Theory 1996; 6(42):2221–2228. 6. Ooi. JM-H. A framework for low-complexity communication over channels with feedback. Ph.D. thesis, Massachusetts Institute of Technology, 1998. 7. Tchamkerten A. Feedback communication over unknown channels. Ph.D. dissertation, E´cole polytechnique fede´rale de Lausanne, 2005. 8. Tchamkerten A, Telatar E. A feedback strategy for binary symmetric channels. ISIT 2002, Lausanne. p. 362. 9. Shulman N. Communication over an unknown channel via common broadcasting. Ph.D. thesis, Tel Aviv University, 2003. 10. Simek HT. Anytime channel coding with feedback. Ph.D. thesis, University of California, 2004. 11. Kramer G. Feedback strategies for white Gaussian interference networks. IEEE Transactions on Information Theory 2002; 6(48):1423– 1438. 12. Kim Y-H. Feedback capacity of the first order moving average Gaussian channel. ISIT 2005, 416–420. 13. Veugen T. Multiple-repetition coding for channels with feedback. Ph.D. dissertation, Eindhoven University of Technology, 1997. 14. Gallager RG. Information theory and reliable communication, Wiley: New York, 1968. 15. Shannon CE. A mathematical theory of communication. The Bell System Technical Journal, 1948; 27(3):379–423. 16. Veugen T. Constrained sequences and optimization. In EIDMA winter meeting on coding theory, information theory and cryptology, van Tilborg HCA, Willems FMJ (eds). 1994, 22.
Euro. Trans. Telecomms. (in press) DOI:10.1002/ett
THIJS VEUGEN
AUTHOR BIOGRAPHY Dr. Thijs Veugen (1969) graduated in 1991 in mathematics and in computer science at Eindhoven University of Technology, the Netherlands, both with distinction. At the same university he obtained a Ph.D. in information theory. He conducted research on electronic cash at the National Research Institute for Mathematics and Computer Science in the Netherlands and has several scientific publications in the areas of information theory and cryptography. He has been scientific software engineer at Statistics Netherlands for 3 years. Since June 1999 he is engaged in applied research at TNO on several areas within information security.
Copyright # 2006 AEIT
Euro. Trans. Telecomms. (in press) DOI:10.1002/ett
