Exponential Strong Converse for Source Coding with Side ... - MDPI

entropy Article

Exponential Strong Converse for Source Coding with Side Information at the Decoder † Yasutada Oohama Department of Communication Engineering and Informatics, University of Electro-Communications, Tokyo 182-8585, Japan; [email protected]; Tel.: +81-42-443-5358 † This paper is an extended version of our paper published in 2016 International Symposium on Information Theory and Its Applications, Monterey, CA, USA, 6–9 November 2016; pp. 171–175. Received: 31 January 2018; Accepted: 20 April 2018; Published: 8 May 2018







Abstract: We consider the rate distortion problem with side information at the decoder posed and investigated by Wyner and Ziv. Using side information and encoded original data, the decoder must reconstruct the original data with an arbitrary prescribed distortion level. The rate distortion region indicating the trade-off between a data compression rate R and a prescribed distortion level ∆ was determined by Wyner and Ziv. In this paper, we study the error probability of decoding for pairs of ( R, ∆) outside the rate distortion region. We evaluate the probability of decoding such that the estimation of source outputs by the decoder has a distortion not exceeding a prescribed distortion level ∆. We prove that, when ( R, ∆) is outside the rate distortion region, this probability goes to zero exponentially and derive an explicit lower bound of this exponent function. On the Wyner–Ziv source coding problem the strong converse coding theorem has not been established yet. We prove this as a simple corollary of our result. Keywords: source coding with side information at the decoder; the rate distortion region; exponent function outside the rate distortion region; strong converse theorem

1. Introduction For single or multi terminal source coding systems, the converse coding theorems state that at any data compression rates below the fundamental theoretical limit of the system the error probability of decoding cannot go to zero when the block length n of the codes tends to infinity. On the other hand, the strong converse theorems state that, at any transmission rates exceeding the fundamental theoretical limit, the error probability of decoding must go to one when n tends to infinity. The former converse theorems are sometimes called the weak converse theorems to distinguish them with the strong converse theorems. In this paper, we study the strong converse theorem for the rate distortion problem with side information at the decoder posed and investigated by Wyner and Ziv [1]. We call the above source coding system the Wyner and Ziv source coding system (the WZ system). The WZ system is shown in Figure 1. In this figure, the WZ system corresponds to the case where the switch is close. In Figure 1, the sequence ( X n , Y n ) represents independent copies of a pair of dependent random variables ( X, Y ) which take values in the finite sets X and Y , respectively. We assume that ( X, Y ) has a probability distribution denoted by p XY . The encoder ϕ(n) outputs a binary sequence which appears at a rate R bits per input symbol. The decoder function ψ(n) observes ϕ(n) ( X n ) and Y n to output a sequence Z n . The t-th component Zt of Z n for t = 1, 2, · · · , n take values in the finite reproduction alphabet Z . Let d : X × Z → [0, ∞) be an arbitrary distortion measure on X × Z . The distortion between x n ∈ X n and zn ∈ Z n is defined by

Entropy 2018, 20, 352; doi:10.3390/e20050352

www.mdpi.com/journal/entropy

Entropy 2018, 20, 352

2 of 32

d( x n , zn ) :=

n

∑ d ( x t , z t ).

t =1

Xn ϕ(n) ( X n ) - ϕ(n) rate R

X

- ψ(n) 

- Zn

r

Yn

 r

Y

Figure 1. Source encoding with a fidelity criterion with or without side inforamion at the decoder.

In general, we have two criteria on d( X n , Z n ). One is the excess-distortion probability of decoding defined by   1 (n) (n) (n) n n Pe ( ϕ , ψ ; ∆) := Pr d( X , Z ) ≥ ∆ . (1) n The other is the average distortion defined by "  1 1 n n ∆ n : =E d ( X, Z ) : = ∑ n n ( x n ,zn )∈X n ×Z n 

=

1 n

n

∑

∑

n

∑ d( xk , zk )

# Pr{ X n = x n , Z n = zn }

k =1

d( xk , zk )Pr { Xk = xk , Zk = zk } .

k=1 ( xk ,zk )∈X ×Z

A pair ( R, ∆) is ε-achievable for p XY if there exist a sequence of pairs {( ϕ(n) , ψ(n) )}n≥1 such that for any δ > 0 and any n with n ≥ n0 = n0 (ε, δ) 1 log k ϕ(n) k ≤ R + δ, n

(n)

Pe ( ϕ(n) , ψ(n) ; ∆) ≤ ε,

where k ϕ(n) k stands for the range of cardinality of ϕ(n) . The rate distortion region RWZ (ε| p XY ) is defined by

RWZ (ε| p XY ) = { ( R, ∆) : ( R, ∆) is ε-achievable for p XY } . Furthermore, set

RWZ ( p XY ) :=

\

RWZ (ε| p XY ).

ε >0

On the other hand, we can define a rate distortion region based on the average distortion criterion, a formal definition of which is the following. A pair ( R, ∆) is achievable for p XY if there exist a sequence of pairs {( ϕ(n) , ψ(n) )}n≥1 such that for any δ > 0 and any n with n ≥ n0 = n0 (δ), 1 log k ϕ(n) k ≤ R + δ, n

∆(n) ≤ ∆ + δ.

˜ WZ ( p XY ) is defined by The rate distortion region R ˜ WZ ( p XY ) := { ( R, ∆) : ( R, ∆) is achievable for p XY } . R If the switch is open, then the side information is not available to the decoder. In this case the communication system corresponds to the source coding for the discrete memoryless source (DMS) ˜ DMS ( p X ) in a similar manner to the definition specified with p X . We define the rate distortion region R

Entropy 2018, 20, 352

3 of 32

˜ WZ ( p XY ). We further define the region RDMS (ε| p X ), ε ∈ (0, 1) and RDMS ( p X ), respectively in of R a similar manner to the definition of RWZ (ε| p XY ) and RWZ ( p XY ). ˜ DMS ( p X ), RDMS (ε| p X ), ε ∈ (0, 1), and RDMS ( p X ) Previous works on the characterizations of R ˜ are shown in Table 1. Shannon [2] determined RDMS ( p X ). Subsequently, Wolfowiz [3] proved that ˜ DMS ( p X ) = RDMS ( p X ). Furthermore, he proved the strong converse theorem. That is, if ( R, ∆) ∈ R / ( n ) ( n ) ∞ ˜ RDMS ( p X ), then for any sequence {( ϕ , ψ )}n=1 of encoder and decoder functions satisfying the condition 1 lim sup log || ϕ(n) || ≤ R, (2) n→∞ n we have (n)

lim Pe ( ϕ(n) , ψ(n) ; ∆) = lim Pr

n→∞

n→∞



1 d( X n , Zn ) ≥ ∆ nk



= 1.

(3)

The above strong converse theorem implies that, for any ε ∈ (0, 1), ˜ DMS ( p X ) = RDMS ( p X ) = RDMS (ε| p X ). R (n)

Csiszár and Körner proved that in Equation (3), the probability Pe ( ϕ(n) , ψ(n) ; ∆) converges to one exponentially and determined the optimal exponent as a function of ( R, ∆). The previous works on the coding theorems for the WZ system are summarized in Table 1. ˜ WZ ( p XY ) was determined by Wyner and Ziv [1]. Csiszár and Körner [4] The rate distortion region R ˜ proved that RWZ ( p XY ) = RWZ ( p XY ). On the other hand, we have had no result on the strong converse theorem for the WZ system. Main results of this paper are summarized in Table 1. For the WZ system, we prove that if ( R, ∆) is out side the rate distortion region RWZ ( p XY ), then we have that for any sequence {( ϕ(n) , ψ(n) )}∞ n=1 of (n)

encoder and decoder functions satisfying the conditionin Equation (2), the quantity Pe ( ϕ(n) , ψ(n) ; ∆) goes to zero exponentially and derive an explicit lower bound of this exponent function. This result corresponds to Theorem 3 in Table 1. As a corollary from this theorem, we obtain the strong converse result, which is stated in Corollary 2 in Table 1. This results states that we have an outer bound with √ O(1 n) gap from the rate distortion region RWZ ( p XY ). Table 1. Previous results on the converse coding theorems for DMS, WZ. Main results in the present paper on WZ are also included. Characterization of the Rate Distortion Region

Strong Converse

Strong Converse Exponent

DMS

Shannon [2] (1959) ˜ DMS ( p X )) (Explicit form of R Wolfowitz [3] (1966) ˜ DMS ( p X ) = RDMS ( p X )) (R

Wolfowitz [3] (1966) (RDMS (ε| p X ) = RDMS ( p X ) for any ε ∈ (0, 1))

Csiszár and Körner [4] (1981) (The optimal exponent)

WZ

Wyner and Ziv [1] (1976) ˜ WZ ( p XY )) (Explicit form of R Csiszár and Körner [4] (1981) ˜ WZ ( p XY ) = RWZ ( p XY )) (R

Corollary 2 (Outer with
√bound O 1/ n gap from the rate distortion region, RWZ (ε| p XY ) = RWZ ( p XY ) for any ε ∈ (0, 1))

Theorem 3 (Lower bound F of the opt. exp. G)

To derive our result, we use a new method called the recursive method. This method is a general powerful tool to prove strong converse theorems for several coding problems in information theory. In fact, the recursive method plays important roles in deriving exponential strong converse exponent for communication systems treated in [5–8].

Entropy 2018, 20, 352

4 of 32

2. Source Coding with Side Information at the Decoder In the following argument, the operations E p [·] and Var p [·], respectively, stand for the expectation and the variance with respect to a probability distribution p. When the value of p is obvious from the context, we omit the suffix p in those operations to simply write E[·] and Var[·]. Let X and Y be finite sets and {( Xt , Yt )}∞ t=1 be a stationary discrete memoryless source. For each t = 1, 2, · · · , the random pair ( Xt , Yt ) takes values in X × Y , and has a probability distribution p XY = { p XY ( x, y)}( x,y)∈X ×Y . ∞ We write n independent copies of { Xt }∞ t=1 and {Yt }t=1 , respectively, as

X n = X1 , X2 , · · · , Xn and Y n = Y1 , Y2 , · · · , Yn . We consider a communication system depicted in Figure 2. Data sequences X n is separately encoded to ϕ(n) ( X n ) and is sent to the information processing center. At the centerm the decoder function ψ(n) observes ϕ(n) ( X n ) and Y n to output the estimation Z n of X n . The encoder function ϕ(n) is defined by (4) ϕ(n) : X n → Mn = { 1, 2, · · · , Mn } . Let Z be a reproduction alphabet. The decoder function ψ(n) is defined by ψ(n) : Mn × Y n → Z n .

(5)

Let d : X × Z → [0, ∞) be an arbitrary distortion measure on X × Z . The distortion between x n ∈ X n and zn ∈ Z n is defined by d( x n , zn ) :=

n

∑ d ( x t , z t ).

t =1

The excess-distortion probability of decoding is (n) Pe ( ϕ(n) , ψ(n) ; ∆)



= Pr

 1 n n d( X , Z ) ≥ ∆ , n

(6)

where Z n = ψ(n) ( ϕ(n) ( X n ), Y n ). The average distortion ∆(n) between X n and Z n is defined by ∆(n) : =

1 1 E [d( X n , Z n )] := n n

n

∑ Ed(Xt , Zt ).

t =1

In the previous section, we gave the formal definitions of RWZ (ε| p XY ), ε ∈ (0, 1), RWZ ( p XY ), ˜ WZ ( p XY ). We can show that the above three rate distortion regions satisfy the following property. and R

X

Xn ϕ(n) ( X n ) - ϕ(n) rate R

- ψ(n) 

- Zn

Yn Y Figure 2. Wyner–Ziv source coding system.

Entropy 2018, 20, 352

5 of 32

Property 1. ˜ WZ ( p XY ) are closed convex sets of R2 , where (a) The regions RWZ (ε| p XY ), ε ∈ (0, 1), RWZ ( p XY ), and R +

R2+ := {( R, ∆) : R ≥ 0, ∆ ≥ 0}. (b) RWZ (ε| p XY ) has another form using (n, ε)-rate distortion region, the definition of which is as follows. We set

RWZ (n, ε| p XY ) = {( R, ∆) : There exists ( ϕ(n) , ψ(n) ) such that 1 (n) log || ϕ(n) || ≤ R, Pe ( ϕ(n) , ψ(n) ; ∆) ≤ ε}, n which is called the (n, ε)-rate distortion region. Using RWZ (n, ε| p XY ), RWZ (ε| p XY ) can be expressed as !

RWZ (ε| p XY ) = cl

[ \

RWZ (n, ε| p XY ) ,

m ≥1 n ≥ m

where cl(·) stands for the closure operation. Proof of this property is given in Appendix A. ˜ WZ ( p XY ) was determined by Wyner and Ziv [1]. To describe their result It is well known that R we introduce auxiliary random variables U and Z, respectively, taking values in finite sets U and Z . We assume that the joint distribution of (U, X, Y, Z ) is pUXYZ (u, x, y, z) = pU (u) p X |U ( x |u) pY | X (y| x ) p Z|UY (z|u, y). The above condition is equivalent to U ↔ X ↔ Y, X ↔ (U, Y ) ↔ Z. Define the set of probability distribution p = pUXYZ by

P ( p XY ) := { p = pUXYZ : |U | ≤ |X | + 1, U ↔ X ↔ Y, X ↔ (U, Y ) ↔ Z }, P ∗ ( p XY ) := { p = pUXYZ : |U | ≤ |X | + 1, U ↔ X ↔ Y, Z = φ(U, Y ) for some φ : U × Y → Z }. By definitions, it is obvious that P ∗ ( p XY ) ⊆ P ( p XY ). Set

R( p) := {( R, ∆) : R, ∆ ≥ 0 , R ≥ I p ( X; U |Y ), ∆ ≥ E p d( X, Z )}, R( p XY ) :=

[ p∈P ( p XY )

R( p), R∗ ( p XY ) :=

[

R( p).

p∈P ∗ ( p XY )

We can show that the above functions and sets satisfy the following property: Property 2. (a) The region R( p XY ) is a closed convex set of R2+ . (b) For any p XY , we have R( p XY ) = R∗ ( p XY ). Proof of Property 2 is given in Appendix C. In Property 2 Part (b), R( p XY ) is regarded as another expression of R∗ ( p XY ). This expression is useful for deriving our main result. The rate region RWZ ( p XY ) was determined by Wyner and Ziv [1]. Their result is the following:

Entropy 2018, 20, 352

6 of 32

Theorem 1 (Wyner and Ziv [1]). ˜ WZ ( p XY ) = R∗ ( p XY ) = R( p XY ). R On RWZ ( p XY ), Csiszár and Körner [4] obtained the following result. Theorem 2 (Csiszár and Körner [4]). ˜ WZ ( p XY ) = R∗ ( p XY ) = R( p XY ). RWZ ( p XY ) = R We are interested in an asymptotic behavior of the error probability of decoding to tend to one as n → ∞ for ( R, ∆) ∈ / RWZ ( p XY ). To examine the rate of convergence, we define the following quantity. Set (n)

(n)

Pc ( ϕ(n) , ψ(n) ; ∆) := 1 − Pe ( ϕ(n) , ψ(n) ; ∆),   1 (n) G (n) ( R, ∆| p XY ) := min − log Pc ( ϕ(n) , ψ(n) ; ∆). n ( ϕ(n) ,ψ(n) ): (1/n) log k ϕ(n) k≤ R

By time sharing, we have that G (n+m)



 nG (n) ( R, ∆| p XY ) + mG (m) ( R0 , ∆0 | p XY ) nR + mR0 n∆ + m∆0 p , . XY ≤ n+m n+m n+m

(7)

Choosing R = R0 and ∆ = ∆0 in Equation (7), we obtain the following subadditivity property on XY ) }n≥1 :

{ G (n) ( R, ∆| p

G (n+m) ( R, ∆| p XY ) ≤

nG (n) ( R, ∆| p XY ) + mG (m) ( R, ∆| p XY ) , n+m

which together with Fekete’s lemma yields that G (n) ( R, ∆| p XY ) exists and satisfies the following: lim G (n) ( R, ∆| p XY ) = inf G (n) ( R, ∆| p XY ).

n→∞

n ≥1

Set G ( R, ∆| p XY ) := lim G (n) ( R, ∆| p XY ), n→∞

G( p XY ) := {( R, ∆, G ) : G ≥ G ( R, ∆| p XY )}. The exponent function G ( R, ∆| p XY ) is a convex function of ( R, ∆). In fact, from Equation (7), we have that for any α ∈ [0, 1] G (αR + α¯ R0 , α∆ + α¯ ∆0 | p XY ) ≤ αG ( R, ∆| p XY ) + α¯ G ( R0 , ∆0 | p XY ), where α¯ = 1 − α. The region G( p XY ) is also a closed convex set. Our main aim is to find an explicit characterization of G( p XY ). In this paper, we derive an explicit outer bound of G ( p XY ) whose section by the plane G = 0 coincides with RWZ ( p XY ).

Entropy 2018, 20, 352

7 of 32

3. Main Results In this section, we state our main results. We first explain that the rate distortion region R( p XY ) can be expressed with two families of supporting hyperplanes. To describe this result, we define two sets of probability distributions on U ×X ×Y ×Z by

Psh ( p XY ) := { pUXYZ : |U | ≤ |X |, U ↔ X ↔ Y, X ↔ (U, Y ) ↔ Z }. Q := {q = qUXYZ : |U | ≤ |X |}. Let µ¯ = 1 − µ. We set R(µ) ( p XY ) :=



min

p∈Psh ( p XY )

¯ + µ∆ ≥ R(µ) ( p XY )}. {( R, ∆) : µR

\

Rsh ( p XY ) :=

¯ p ( X; U |Y ) + µE p d( X; Z ) , µI

µ∈[0,1]

Then, we have the following property: Property 3. For any p XY , we have

Rsh ( p XY ) = R( p XY ).

(8)

Proof of Property 3 is given in Appendix D. For µ ∈ [0, 1] and λ, α ≥ 0, define (µ,λ)

ωq|| p ( x, y, z|u) # " # " q X |YU ( x |y, u) q X ( x )qY | XU (y| x, u)q Z|UYX (z|u, y, x ) + λ µ¯ log + µd( x, z) , := log p X ( x ) pY | X (y| x )q Z|UY (z|u, y) p X |Y ( x | y ) oi h n (µ,λ) Ω(µ,λ,α) (q| p XY ) := − log Eq exp −αωq|| p ( X, Y, Z |U ) , Ω(µ,λ,α) ( p XY ) := min Ω(µ,λ,α) (q| p XY ), q∈Q

¯ + µ∆| p XY ) := F (µ,λ,α) (µR

¯ + µ∆) Ω(µ,λ,α) ( p XY ) − λα(µR . 1 + (4 + λµ¯ )α

Furthermore, set F ( R, ∆| p XY ) :=

¯ + µ∆| p XY ), F (µ,λ,α) (µR

sup µ∈[0,1],λ,α≥0

G( p XY ) := {( R, ∆, G ) : G ≥ F ( R, ∆| p XY )} . We next define a functions serving as a lower bound of F ( R, ∆| p XY ). For each p = pUXYZ ∈ Psh ( p XY ), define " (µ) ω˜ p ( x, y, z|u)

:= µ¯ log

p X |YU ( x |y, u)

#

+ µd( x, z) , p X |Y ( x | y ) h n oi ˜ (µ,λ) ( p) := − log E p exp −λω (pµ) ( X, Y, Z |U ) . Ω

Furthermore, set ˜ (µ,λ) ( p XY ) := Ω

min p∈Psh ( p XY )

˜ (µ,λ) ( p), Ω

˜ (µ,λ) ( p XY ) − λ(µR ¯ + µ∆) Ω ¯ | p XY ) := F˜ (µ,λ) (µR + µ∆ , 5 + λ(1 + µ¯ )

Entropy 2018, 20, 352

8 of 32

F˜ ( R, ∆| p XY ) :=

sup

¯ + µ∆| p XY ). F˜ (µ,λ) (µR

λ≥0,µ∈[0,1]

We can show that the above functions satisfies the following properties: Property 4. (a) The cardinality bound |U | ≤ |X | appearing in Q is sufficient to describe the quantity Ω(µ,λ,α) ( p XY ). Furthermore, the cardinality bound |U | ≤ |X | in Psh ( p XY ) is sufficient to describe the quantity ˜ (µ,λ) ( p XY ). Ω (b) For any R, ∆ ≥ 0, we have F ( R, ∆| p XY ) ≥ F˜ ( R, ∆| p XY ). ˜ (µ,λ) ( p) exists and is nonnegative. (c) Fix any p = pUXY ∈ Psh ( p XY ) and µ ∈ [0, 1]. For λ ∈ [0, 1], Ω (λ)

For p = pUXYZ ∈ Psh ( p XY ), define a probability distribution p(λ) = pUXYZ by n o (µ) p(u, x, y, z) exp −λω˜ p ( x, y, z|u) h n oi . p(λ) (u, x, y, z) := (µ) E p exp −λω˜ p ( X, Y, Z |U ) ˜ (µ,λ) ( p) is twice differentiable. Furthermore, for λ ∈ [0, 1/2], we have Then, for λ ∈ [0, 1/2], Ω h i d ˜ (µ,λ) (µ) Ω ( p) = E p(λ) ω p ( X, Y, Z |U ) , dλ i h d2 ˜ (µ,λ) (µ) ˜ ω ( X, Y, Z | U ) . Ω ( p ) = − Var ( λ ) p p dλ2 ˜ (µ,λ) ( p) is a concave function of λ ∈ [0, 1/2]. The second equality implies that Ω (d) For (µ, λ) ∈ [0, 1] × [0, 1/2], define ρ(µ,λ) ( p XY ) :=

max

(ν,p)∈[0,λ] ×Psh ( p XY ): ˜ (µ,λ) ( p) Ω ˜ (µ,λ) ( p XY ) =Ω

h i (µ) Var p(ν) ω˜ p ( X, Y, Z |U ) ,

and set ρ = ρ( p XY ) :=

max

(µ,λ)∈[0,1]×[0,1/2]

ρ(µ,λ) ( p XY ).

Then, we have ρ( p XY ) < ∞. Furthermore, for any (µ, λ) ∈ [0, 1] × [0, 1/2], 2 ˜ (µ,λ) ( p XY ) ≥ λR(µ) ( p XY ) − λ ρ( p XY ). Ω 2

(9)

(e) For every τ ∈ (0, (1/2)ρ( p XY )), the condition ( R + τ, ∆ + τ ) ∈ / R( p XY ) implies ρ( p XY ) 2 F˜ ( R, ∆| p XY ) > ·g 10



τ ρ( p XY )



> 0,

where g is the inverse function of ϑ ( a) := a + (1/5) a2 , a ≥ 0. Proof of Property 4 Part (a) is given in Appendix B. Proof of Property 4 Part (b) is given in Appendix E. Proofs of Property 4 Parts (c), (d), and (e) are given in Appendix F. Our main result is the following:

Entropy 2018, 20, 352

9 of 32

Theorem 3. For any R, ∆ ≥ 0, any p XY , and for any ( ϕ(n) , ψ(n) ) satisfying (1/n) log || ϕ(n) || ≤ R, we have (n)

Pc ( ϕ(n) , ψ(n) ; ∆) ≤ 5 exp {−nF ( R, ∆| p XY )} .

(10)

It follows from Theorem 3 and Property 4 Part (d) that if ( R, ∆) is outside the rate distortion region, then the error probability of decoding goes to one exponentially and its exponent is not below F ( R, ∆| p XY ). It immediately follows from Theorem 3 that we have the following corollary. Corollary 1. For any R, ∆ ≥ 0 and any p XY , we have G ( R, ∆| p XY ) ≥ F ( R, ∆| p XY ).

(11)

Furthermore, for any p XY , we have

G( p XY ) ⊆ G( p XY ) := {( R, ∆, G ) : G ≥ F ( R, ∆| p XY )} .

(12)

Proof of Theorem 3 will be given in the next section. The exponent function in the case of ∆ = 0 can be obtained as a corollary of the result of Oohama and Han [9] for the separate source coding problem of correlated sources [10]. The techniques used by them is a method of types [4], which is not useful for proving Theorem 3. In fact, when we use this method, it is very hard to extract a condition related to the Markov chain condition U ↔ X ↔ Y, which the auxiliary random variable U ∈ U must satisfy when ( R, ∆) is on the boundary of the set R( p XY ). Some novel techniques based on the information spectrum method introduced by Han [11] are necessary to prove this theorem. From Theorem 3 and Property 4 Part (e), we can obtain an explicit outer bound of RWZ (ε| p XY ) with an asymptotically vanishing deviation from RWZ ( p XY ) = R( p XY ). The strong converse theorem immediately follows from this corollary. To describe this outer bound, for κ > 0, we set

R( p XY ) − κ (1, 1) := {( R − κ, ∆ − κ ) : ( R, ∆) ∈ R( p XY )}, which serves as an outer bound of R( p XY ). For each fixed ε ∈ (0, 1), we define κn = κn (ε, ρ( p XY )) by s

 ! 10 5 κn := ρ( p XY )ϑ log nρ( p XY ) 1−ε s     (a) 10ρ( p XY ) 5 2 5 log + log . = n 1−ε n 1−ε

(13)

Step (a) follows from ϑ ( a) = a + (1/5) a2 . Since κn → 0 as n → ∞, we have the smallest positive integer n0 = n0 (ε, ρ( p XY )) such that κn ≤ (1/2)ρ( p XY ) for n ≥ n0 . From Theorem 3 and Property 4 Part (e), we have the following corollary. Corollary 2. For each fixed ε ∈ (0, 1), we choose the above positive integer n0 = n0 (ε, ρ( p XY )) Then, for any n ≥ n0 , we have

RWZ (n, ε| p XY ) ⊆ R( p XY ) − κn (1, 1). The above result together with !

RWZ (ε| p XY ) = cl

[ \ m ≥1 n ≥ m

RWZ (n, ε| p XY ) ,

Entropy 2018, 20, 352

10 of 32

yields that for each fixed ε ∈ (0, 1), we have

RWZ (ε| p XY ) = RWZ ( p XY ) = R( p XY ). Proof of this corollary will be given in the next section. The direct part of coding theorem, i.e., the inclusion of R( p XY ) ⊆ RWZ (ε| p XY ) was established by Csiszár and Körner [4]. They proved a weak converse theorem to obtain the inclusion RWZ ( p XY ) ⊆ R( p XY ). Until now, we have had no result on the strong converse theorem. The above corollary stating the strong converse theorem for the Wyner–Ziv source coding problem implies that a long standing open problem since Csiszár and Körner [4] has been resolved. 4. Proof of the Main Results In this section, we prove Theorem 3 and Corollary 2. We first present a lemma which upper bounds the correct probability of decoding by the information spectrum quantities. We set Sn := ϕ(n) ( X n ), Z n := ψn ( ϕ(n) ( X n ), Y n ). It is obvious that Sn ↔ X n ↔ Y n , X n ↔ ( Sn , Y n ) ↔ Z n . Then, we have the following: Lemma 1. For any η > 0 and for any ( ϕ(n) , ψ(n) ) satisfying (1/n) log || ϕ(n) || ≤ R, we have (n)

Pc ( ϕ(n) , ψ(n) ; ∆) ≤ pSn X n Y n Zn



(i)

Q n (Xn ) 1 log X , n pXn (X n )

η≥

(14)

(ii)

Q Y n | S X n (Y n | S n , X n ) 1 n η ≥ log , n p Y n | X n (Y n | X n )

(15)

(iii)

Q X n | S Y n Z n ( X n | Sn , Y n , Z n ) 1 n η ≥ log , n p X n | Sn Y n ( X n | Sn , Y n )

(16)

(iv)

Q X n | S Y n ( X n | Sn , Y n ) 1 n R + η ≥ log , n p X n |Y n ( X n | Y n )  1 n n ∆ ≥ log exp {d( X , Z )} + 4e−nη . n

(17) (18)

The probability distribution and stochastic matrices appearing in the right members of Equation (18) have (i)

a property that we can select them arbitrary. In Equation (14), we can choose any probability distribution QXn (ii)

on X n . In Equation (15), we can choose any stochastic matrix QYn |S

nX

(iii)

we can choose any stochastic matrix QXn |S

nY

(iv)

stochastic matrix QXn |S

nY

n

n Zn

n

: Mn × X n → Y n . In Equation (16),

: Mn × Y n × Z n → X n . In Equation (17), we can choose any

: Mn × Y n → X n .

Proof of this lemma is given in Appendix G. Lemma 2. Suppose that, for each t = 1, 2, · · · , n, the joint distribution pSn X t Y n of the random vector Sn X t Y n is a marginal distribution of pSn X n Y n . Then, for t = 1, 2, · · · , n, we have the following Markov chain:

Entropy 2018, 20, 352

11 of 32

Xt ↔ Sn X t−1 Ytn ↔ Y t−1

(19)

or equivalently that I ( Xt ; Y t−1 |Sn X t−1 Ytn ) = 0. Proof of this lemma is given in Appendix H. For t = 1, 2, · · · , n, set ut := (s, x t−1 , ynt+1 ). Let Ut := (Sn , X t−1 , Ytn+1 ) be a random vector taking values in Mn × X t−1 × Ytn+1 . From Lemmas 1 and 2, we have the following: Lemma 3. For any η > 0 and for any ( ϕ(n) , ψ(n) ) satisfying (1/n) log || ϕ(n) || ≤ R, we have the following: (n)

Pc ( ϕ(n) , ψ(n) ; ∆) ≤ pSn X n Y n Zn 1 η≥ n 1 η≥ n 1 η≥ n R+η ≥ ∆≥



(i)

n

Q Xt ( Xt )

t =1

p Xt ( Xt )

∑ log

,

(ii)

n

QY |U X (Yt |Ut , Xt )

t =1

pYt | Xt (Yt | Xt )

∑ log

t

t

t

,

(20)

(iii)

n

Q X |U Y Z ( Xt |Ut , Yt , Zt )

t =1

p Xt |Ut Yt ( Xt |Ut , Yt )

∑ log

t

t t t

,

(iv)

1 n

n

Q X |U Y ( Xt |Ut , Yt )

t =1

1 n

p Xt |Yt ( Xt |Yt ) )

∑ log ed(Xt ,Zt )

∑ log

t

n

t t

,

+ 4e−nη ,

t =1

where for each t = 1, 2, · · · , n, the following probability distribution and stochastic matrices: (iv)

(iii)

(ii)

(i)

Q Xt , QY |U X , Q X |U Y Z , and Q X |U Y t

t

t

t

t

t t t

t t

appearing in the first term in the right members of Equation (21) have a property that we can choose their values arbitrary. Proof. On the probability distributions appearing in the right members of Equation (18), we take the (i)

following choices. In Equation (14), we choose Q X n so that (i)

QXn (X n ) = (ii)

In Equation (15), we choose QY n |S (ii)

QY n | S

n

Xn

(Y n | S n , X n ) =

nX

n

n

(ii)

t =1

t n n X Yt+1

(iii)

In Equation (16), we choose Q X n |S (iii)

Q X n |S

n n nY Z

nY

(i)

(21)

t =1

so that

∏ QY | S t

n

∏ Q Xt ( Xt ).

n Zn

(Yt |Sn , X t , Ytn+1 ) =

n

∏ QYt |Xt Ut (Yt |Ut Xt ). (ii)

(22)

t =1

so that

( X n | Sn , Y n , Z n ) =

n

∏ QXt |Sn Xt−1 Ytn Zt (Xt |Sn X t−1 , Ytn , Zt )

t =1 n

=

(iii)

∏ QXt |Ut Yt Zt (Xt |Ut Yt Zt ).

t =1

(iii)

(23)

Entropy 2018, 20, 352

12 of 32

In Equation (16), we note that p X n | Sn Y n ( X n | Sn , Y n ) =

n

(a) n

∏ pXt |Sn Xt−1 Yn (Xt |Sn , X t−1 , Yn ) = ∏ pXt |Sn Xt−1 Ytn (Xt |Sn , X t−1 , Ytn )

t =1 n

=

t =1

∏ pXt |Ut Yt (Xt |Ut , Yt ).

(24)

t =1

(iv)

Step (a) follows from Lemma 2. In Equation (17), we choose Q X n |S (iv)

Q X n |S

n

Yn

( X n | Sn , Y n ) =

n

nY

n

n

so that

∏ QXt |Sn Xt−1 Ytn (Xt |Sn , X t−1 , Ytn ) = ∏ QXt |Ut Yt (Xt |Ut , Yt ). (iv)

t =1

(iv)

(25)

t =1

From Lemma 1 and Equations (21)–(25), we have the bound of Equation (21) in Lemma 3. To evaluate an upper btound of Equation (21) in Lemma 3, we use the following lemma, which is well known as the Cramér’s bound in the large deviation principle. Lemma 4. For any real valued random variable A and any θ ≥ 0, we have Pr{ A ≤ a} ≤ exp [(θa + log E[exp(−θ A)])] . (n)

Here, we define a quantity which serves as an exponential upper bound of Pc ( ϕ(n) , ψ(n) ). For each t = 1, 2, · · · , n, let Qt be a set of all (i)

(iv)

(iii)

(ii)

Q t = ( Q X t , QY |U X , Q X |U Y Z , Q X |U Y ) . t

t

t

t

t

t t t

t t

Set

Qn :=

n

∏ Qt , Qn :=

t =1

n

Qt

on t =1

∈ Qn .

Let P (n) ( p XY ) be a set of all probability distributions pSn X n Y n Zn on Mn ×X n ×Y n ×Z n having the form: ( ) pSn X n Y n Zn (s, x n , yn , zn ) = pSn | X n (s| x n )

n

∏ pXt Yt (xt , yt )

p Z n |Y n S n ( z n | y n , s ) .

t =1

For simplicity of notation, we use the notation p(n) for pSn X n Y n Zn ∈ P (n) ( p XY ). We assume that pUt Xt Yt Zt = pSn X t Y n Zt is a marginal distribution of p(n) . For t = 1, 2, · · · , n, we simply write pt = t

pUt Xt Yt Zt . For p(n) ∈ P (n) ( p XY ) and Qn ∈ Qn , we define θ  pYt | Xt (Yt | Xt )    (ii) QY | X U (Yt | Xt , Ut ) 

  p (X ) t Xt Ω(µ,λ,θ ) ( p(n) , Qn | p XY ) := − log E p(n)  ∏ (i)  Q (X ) t =1 

n

Xt

t

t

t

t



 θ    λθ  µ¯    n  n p Xt |Yt ( Xt |Yt ) p Xt |Ut Yt ( Xt |Ut , Yt )      e−µd(Xt ,Zt ) × ∏   ,  ∏ (iii) (iv)     Q X |U Y Z ( Xt |Ut , Yt , Zt ) Q X |Y U ( Xt |Ut , Yt ) t =1 t =1 t

t t t

t

t

t

where, for each t = 1, 2, · · · , n, the following probability distribution and stochastic matrices:

Entropy 2018, 20, 352

13 of 32

(i)

(iv)

(iii)

(ii)

Q X t , Q X |U Y , Q X |U Y Z , QY | X U t

t t

t

t

t t t

t

t

appearing in the definition of Ω(µ,λ,θ ) ( p(n) , Qn | p XY ) are chosen so that they are induced by the joint distribution qt = qUt Xt Yt Zt ∈ Qt . By Lemmas 3 and 4, we have the following proposition: Proposition 1. For any µ ∈ [0, 1], λ, θ ≥ 0, any qn ∈ Qn , and any ( ϕ(n) , ψ(n) ) satisfying (1/n) log || ϕ(n) || ≤ R, we have    1 (µ,λ,θ ) (n) n (n) ¯ + µ∆) Ω ( p , Q | p XY ) − λθ (µR Pc ( ϕ(n) , ψ(n) ; ∆) ≤ 5 exp −n [1 + (3 + λµ¯ )θ ]−1 . n ¯ + µ∆), the bound we wish to prove is obvious. In the Proof. When Ω(µ,λ,θ ) ( p(n) , Qn | p XY ) ≤ nλθ (µR ( µ,λ,θ ) ¯ + µ∆). We define five random following argument, we assume that Ω ( p(n) , Qn | p XY ) > nλθ (µR variables Ai ,i = 1, 2, · · · , 5 by

A1 =

1 n

1 A3 = n 1 A4 = n

(i)

n

∑ log

t =1

Q Xt ( Xt ) p Xt ( Xt )

, A2 =

1 n

(ii)

n

QY | X U (Yt | Xt , Ut )

t =1

pYt | Xt (Yt | Xt )

∑ log

t

t

t

,

(iii)

n

Q X |U Y Z ( Xt |Ut , Yt , Zt )

t =1

p Xt |Ut Yt ( Xt |Ut , Yt )

∑ log

t

t t t

,

(iv)

n

∑ log

Q X |U Y ( Xt |Ut , Yt )

t =1

t

t t

p Xt |Yt ( Xt |Yt )

, A5 =

1 n

n

∑ log ed(Xt ,Zt ) .

t =1

By Lemma 3, for any ( ϕ(n) , ψ(n) ) satisfying (1/n) log || ϕ(n) || ≤ R, we have (n)

Pc ( ϕ(n) , ψ(n) ; ∆) ≤ pSn X n Y n Zn { Ai ≤ η for i = 1, 2, 3, A4 ≤ R + η, A5 ≤ ∆} + 4e−nη ¯ 4 + µA5 ) ≤ λ(µR ¯ + µ∆) + (3 + λµ¯ )η } + 4e−nη ≤ pSn X n Yn Zn { A1 + A2 + A3 + λ(µA

= pSn X n Yn Zn { A ≤ a} + 4e−nη ,

(26)

where we set ¯ 4 + µA5 ), A := A1 + A2 + A3 + λ(µA ¯ + µ∆) + (3 + λµ¯ )η. a := λ(µR Applying Lemma 4 to the first term in the right member of Equation (26), we have h i (n) Pc ( ϕ(n) , ψ(n) ; ∆) ≤ exp θa + log E p(n) [exp(−θ A)] + 4e−nη    1 (µ,λ,θ ) (n) n ¯ + µ∆) + (3 + λµ¯ )θη − Ω = exp n λθ (µR ( p , Q | p XY ) + 4e−nη . n

(27)

We choose η so that 1 ¯ + µ∆) + θ (3 + λµ¯ )η − Ω(µ,λ,θ ) ( p(n) , Qn | p XY ). −η = λθ (µR n Solving Equation (28) with respect to η, we have η=

1 (µ,λ,θ ) (n) ¯ + µ∆) ( p , Qn | p XY ) − λθ (µR nΩ

1 + (3 + λµ¯ )θ

.

(28)

Entropy 2018, 20, 352

14 of 32

For this choice of η and Equation (27), we have (n)

Pc ( ϕ(n) , ψ(n) ; ∆) ≤ 5e−nη    −1 1 (µ,λ,θ ) (n) n ¯ + µ∆) = 5 exp −n [1 + (3 + λµ¯ )θ ] Ω ( p , Q | p XY ) − λθ (µR , n completing the proof. Set Ω(µ,λ,θ ) ( p XY ) := inf

min

max

n≥1 p(n) ∈P (n) ( p XY ) Qn ∈Qn

1 (µ,λ,θ ) (n) n Ω ( p , Q | p XY ). n

By Proposition 1, we have the following corollary. Corollary 3. For any µ ∈ [0, 1], λ ≥ 0, for any θ ≥ 0, and for any ( ϕ(n) , ψ(n) ) satisfying (1/n) log || ϕ(n) || ≤ R, we have ( " #) ¯ + µ∆) Ω(µ,λ,θ ) ( p XY ) − λθ (µR (n) (n) (n) Pc ( ϕ , ψ ; ∆) ≤ 5 exp −n . 1 + (3 + λµ¯ )θ We shall call Ω(µ,λ,θ ) ( p XY ) the communication potential. The above corollary implies that the analysis of Ω(µ,λ,θ ) ( p XY ) leads to an establishment of a strong converse theorem for Wyner–Ziv source coding problem. In the following argument, we drive an explicit lower bound of Ω(µ,λ,θ ) ( p XY ). We use a new technique we call the recursive method. The recursive method is a powerful tool to drive a single letterized exponent function for rates below the rate distortion function. This method is also applicable to prove the exponential strong converse theorems for other network information theory problems [5–7]. Set Ft := ( p Xt |Ut Yt , Qt ), F t := {Fi }it=1 . For each t = 1, 2, · · · , n, define a function of (ut , xt , yt , zt ) ∈ Ut ×X ×Y ×Z by (µ,λ,θ )

( xt , yt , zt |ut ) θ   λµ¯    p (x ) p p p ( y | x ) ( x | u , y ) ( x | y ) Yt | Xt t t Xt |Ut Yt t t t Xt |Yt t t Xt t −λµd( xt ,zt )   . e := (iii) (iv)   Q(i) ( x ) Q(ii)  ( y t | x t , u t ) Q X | U Y Z ( x t | u t , y t , z t ) Q X |Y U ( x t | u t , y t ) Xt t Y |X U

f Ft

t

t

t

t

t t t

t

t

t

By definition, we have n o exp −Ω(µ,λ,θ ) ( p(n) , Qn | p XY )

=

∑n

pSn Y n (s, yn )

s,y

∑ n n

x ,z

n

p X n Zn |Sn Y n ( x n , zn |s, yn ) ∏ f Ft

(µ,λ,θ )

( x t , y t , z t | u t ).

t =1

For each t = 1, 2, · · · , n, we define the conditional probability distribution (µ,λ,θ )

p X t Z t |S

n

Y n ;F t

:=

n

(µ,λ,θ )

p X t Z t |S

n

Y n ;F t

( x t , zt |s, yn )

o ( x t ,zt ,s,yn )∈X t ×Z t ×Mn ×Y n

by t

(µ,λ,θ )

p X t Z t |S

n

( x t , zt |s, yn ) := Ct−1 (s, yn ) p X t Zt |Sn Yn ( x t , zt |s, yn ) ∏ f F Y n ;F t i =1

(µ,λ,θ ) i

( xi , yi , zi | ui )

(29)

Entropy 2018, 20, 352

15 of 32

where Ct (s, yn ) :=

∑

x t ,zt

t

p X t Zt |Sn Y n ( x t , zt |s, yn ) ∏ f F

(µ,λ,θ )

i =1

i

( xi , yi , zi | ui )

(30)

are constants for normalization. For t = 1, 2, · · · , n, define (µ,λ,θ )

Φt,F t

(s, yn ) := Ct (s, yn )Ct−−11 (s, yn ),

(31)

where we define C0 (s, yn ) = 1 for (s, yn ) ∈ Mn ×Y n . Then, we have the following lemma: Lemma 5. For each t = 1, 2, · · · , n, and for any (s, yn x t , zt ) ∈ Mn ×Y n ×X t ×Z t , we have (µ,λ,θ )

p X t Z t |S

(µ,λ,θ )

nY

n ;F t

( x t , zt |s, yn ) = (Φt,F t

(µ,λ,θ )

(s, yn ))−1 p X t−1 Yt−1 |S (µ,λ,θ )

× p Xt Zt |Sn X t−1 Yn ( xt , zt |s, x t−1 , zt−1 , yn ) f Ft (µ,λ,θ )

Φt,F t

(s, yn )=

∑

x t ,zt

(µ,λ,θ )

p X t −1 Z t −1 | S

nY

n ; F t −1

nY

n ;F t −1

( x t−1 , zt−1 |s, yn )

( x t , y t , z t | u t ).

(32)

( x t−1 , zt−1 |s, yn ) (µ,λ,θ )

× p Xt Zt |Sn X t−1 Yn ( xt , zt |s, x t−1 , zt−1 , yn ) f Ft

( x t , y t , z t | u t ).

(33)

Furthermore, we have o n exp −Ω(µ,λ,θ ) ( p(n) , Qn | p XY ) =

∑n

s,y

n

pSn Y n (s, yn ) ∏ Φt,F t

(µ,λ,θ )

(s, yn ).

(34)

t =1

The equality in Equation (34) in Lemma 5 is obvious from Equations (29)–(31). Proofs of Equations (32) and (33) in this lemma are given in Appendix I. Next, we define a probability distribution of the random pair (Sn , Y n ) taking values in Mn ×Y n by (µ,λ,θ ) n n t ( s, y )= n Y ;F

pS

t

(µ,λ,θ ) C˜ t−1 pSn Y n (s, yn ) ∏ Φi,F i (s, yn ),

(35)

i =1

where C˜ t is a constant for normalization given by C˜ t =

t

∑n pSn Yn (s, yn ) ∏ Φi,F i

s,y

For t = 1, 2, · · · , n, define

(µ,λ,θ )

(s, yn ).

i =1

(µ,λ,θ )

Λt,F t

:= C˜t C˜ t−−11 ,

(36)

where we define C˜ 0 = 1. Set (µ,λ,θ ) t n t n t−1 ( s, x , y t , z t ) n X Yt Zt ;F

pS

:=

∑

yt−1 ,zt−1

(µ,λ,θ )

= p U X Y Z ; F t −1 ( u t , x t , y t , z t ) t

t t t

(µ,λ,θ ) (µ,λ,θ ) pS Y n ;F t−1 (s, yn ) p X t−1 Zt−1 |S Y n ;F t−1 ( x t−1 , zt−1 |s, yn ) n n

× p Xt Zt |X t−1 Zt−1 Sn Yn ( xt , zt | x t−1 , zt−1 , s, yn ). Then, we have the following:

(37)

Entropy 2018, 20, 352

16 of 32

Lemma 6. n o exp −Ω(µ,λ,θ ) ( p(n) , Qn | p XY ) =

n

∏ Λt,F t

(µ,λ,θ )

,

(38)

t =1

(µ,λ,θ )

Λt,F t

∑

=

(µ,λ,θ )

ut ,xt ,yt ,zt

(µ,λ,θ )

p U X Y Z ; F t −1 ( u t , x t , y t , z t ) f F t t

t t t

( x t , y t , z t | u t ).

(39)

Proof. By the equality Equation (34) in Lemma 5, we have o n exp −Ω(µ,λ,θ ) ( p(n) , Qn | p XY ) = C˜ n =

n

(a) n

∏ C˜ t C˜ t−−11

=

t =1

(µ,λ,ν,θ )

Step (a) follows from the definition in Equation (36) of Λt,F t (µ,λ,θ )

Lemma 6. Multiplying Λt,F t

∏ Λt,F t

(µ,λ,θ )

.

(40)

t =1

. We next prove Equation (39) in

= C˜ t /C˜ t−1 to both sides of Equation (35), we have (µ,λ,θ ) (µ,λ,θ ) pS Y n ;F t (s, yn ) n t (µ,λ,θ ) C˜ t−−11 pSn Y n (s, yn ) Φi,F i (s, yn ) i =1 (µ,λ,θ ) (µ,λ,θ ) pS Y n ;F t−1 (s, yn )Φt,F t (s, yn ). n

Λt,F t

(41)

∏

= =

(42)

Taking summations of Equations (41) and (42) with respect to (s, yn ), we have (µ,λ,θ )

∑n pSn Yn ;F t−1 (s, yn )Φt,F t (µ,λ,θ )

Λt,F t

=

(a)

(µ,λ,θ )

=

(µ,λ,θ )

(s, yn )

s,y

∑n pSn Yn ;F t−1 (s, yn ) ∑

(µ,λ,θ )

x t ,zt

s,y

p X t −1 Z t −1 | S

nY

n ; F t −1

( x t−1 , zt−1 |s, yn ) (µ,λ,θ )

× p Xt Zt |X t−1 Zt−1 Sn Yn ( xt , zt | x t−1 , zt−1 , s, yn ) f Ft =

∑

∑

s,x t ,ynt ,zt yt−1 ,zt−1

(µ,λ,θ )

× p Xt Zt |X t−1 Zt−1 Sn Yn ( xt , zt | x t−1 , zt−1 , s, yn ) f Ft (b)

=

∑

ut ,xt ,yt ,zt

(µ,λ,θ )

(µ,λ,θ )

p U X Y Z ;F t −1 ( u t , x t , y t , z t ) f F t t

( xt , yt , zt |ut )

(µ,λ,θ ) (µ,λ,θ ) pS Y n ;F t−1 (s, yn ) p X t−1 Zt−1 |S Y n ;F t−1 ( x t−1 , zt−1 |s, yn ) n n

t t t

( x t , y t , z t | u t ).

( x t , y t , z t | u t ).

Step (a) follows from Equation (33) in Lemma 5. Step (b) follows from the definition in Equation (37) of

(µ,λ,θ ) pU X Y Z ;F t−1 . t t t t

The following proposition is a mathematical core to prove our main result. Proposition 2. For θ ∈ [0, 1), we choose the parameter α such that α=

α θ ⇔θ= . 1−θ 1+α

(43)

Then, for any λ ≥ 0, µ ∈ [0, 1] and for any θ ∈ [0, 1), we have Ω(µ,λ,θ ) ( p XY ) ≥

Ω(µ,λ,α) ( p XY ) . 1+α

(44)

Entropy 2018, 20, 352

17 of 32

Proof. Set

Qˆ n := {q = qUXYZ : |U | ≤ |Mn ||X n−1 ||Y n−1 |}, ˆ n(µ,λ,α) ( p XY ) := min Ω(µ,λ,α) (q| p XY ). Ω q∈Qˆ n

Then, by Lemma 6, we have (µ,λ,θ )

Λt,F t

=

∑

ut ,xt ,yt ,zt

(µ,λ,θ )

(µ,λ,θ )

p U X Y Z ; F t −1 ( u t , x t , y t , z t ) f F t t

t t t

( x t , y t , z t | u t ). (µ,λ,θ )

For each t = 1, 2, · · · , n, we recursively choose qt = qUt Xt Yt Zt so that qUt Xt Yt Zt = pU X Y Z ;F t−1 and (i)

(ii)

(iii)

t

(iv)

t t t

choose Q Xt , QY | X U , Q X |U Y Z , and Q X |Y U appearing in t

t

t

t

t t t

t

t

t

(µ,λ,θ )

( xt , yt , zt |ut ) θ   λµ¯     p (x ) pYt | Xt (yt | xt ) p Xt |Ut Yt ( xt |ut , yt ) p Xt |Yt ( xt |yt ) Xt t   e−λµd( xt ,zt ) :=    Q(Xi) ( xt ) QY(ii|)X U (yt | xt , ut ) Q(Xiii|)U Y Z ( xt |ut , yt , zt ) Q(Xiv|)Y U ( xt |ut , yt )  t

f Ft

t

t

t

t

t t t

t

t

t

such that they are the distributions induced by qUt Xt Yt Zt . Then, for each t = 1, 2, · · · , n, we have the following chain of inequalities:  θ  λµ¯ −λµd( Xt ,Zt )   p (X ) p p ( X | Y ) e t t p Xt |Ut Yt ( Xt |Ut , Yt ) Yt | Xt (Yt | Xt ) Xt |Yt (µ,λ,θ ) t   Xt Λt,F t = Eqt   λµ¯   q Xt ( Xt ) qYt | Xt Ut (Yt | Xt , Ut ) q Xt |Ut Yt Zt ( Xt |Ut , Yt , Zt ) q X |Y U ( Xt |Ut , Yt ) t t t  θ λµ¯ −λµd( Xt ,Zt )   p (X ) p p ( X | Y ) e t t ( Y | X ) q ( X | U , Y ) t t t t t Yt | Xt Xt |Ut Yt Xt |Yt t  Xt = Eq t  λµ¯  q Xt ( Xt ) qYt | Xt Ut (Yt | Xt , Ut ) q Xt |Ut Yt Zt ( Xt |Ut , Yt , Zt ) q X |Y U ( Xt |Ut , Yt )  t t t ( )θ  p Xt |Ut Yt ( Xt |Ut , Yt )  × (45) q Xt |Ut Yt ( Xt |Ut , Yt )   θ   1− θ λµ¯ −λµd( Xt ,Zt )  1−θ  p ( Xt ) p p ( X | Y ) e t t ( Y | X ) q ( Z | U , Y ) t t t Yt | Xt t t Zt |Ut Yt Xt |Yt    Xt ≤  Eq t   λµ¯   q X ( Xt ) qYt | Xt Ut (Yt | Xt , Ut ) q Zt |Ut Xt Yt ( Zt |Ut , Xt , Yt ) q X |Y U ( Xt |Ut , Yt ) t 

(a)

t

(

× Eq t (b)

(

= exp

p Xt |Ut Yt ( Xt |Ut , Yt )

)!θ

t

t

n o θ = exp −(1 − θ )Ω(µ,λ, 1−θ ) (qt | p XY )

q Xt |Ut Yt ( Xt |Ut , Yt ) ) ( ) ( ) ˆ (nµ,λ,α) ( p XY ) (d) Ω(µ,λ,α) (qt | p XY ) (c) Ω Ω(µ,λ,α) ( p XY ) − ≤ exp − = exp − . 1+α 1+α 1+α

Step (a) follows from Hölder’s inequality and the following identity: q Xt |Ut Yt ( Xt |Ut , Yt ) q Xt |Ut Yt Zt ( Xt |Ut , Yt , Zt )

=

q Zt |Ut Yt ( Zt |Ut , Yt ) q Zt |Ut Xt Yt ( Zt |Ut , Xt , Yt )

for t = 1, 2, · · · , n. (µ,λ,α)

ˆn Step (b) follows from Equation (43). Step (c) follows from the definition of Ω

( p XY ). Step (d) ˆ (nµ,λ,α) ( p XY ). follows from that by Property 4 Part (a), the bound |U | ≤ |X |, is sufficient to describe Ω Hence, we have the following:

Entropy 2018, 20, 352

18 of 32

max

qn ∈Qn

≥

1 (µ,λ,θ ) (n) n Ω ( p , Q | p XY ) n

(a) 1 (µ,λ,θ ) (n) n 1 Ω ( p , Q | p XY ) = − n n

n

∑ log Λt,F t

(µ,λ,θ )

(b)

≥

t =1

Ω(µ,λ,α) ( p XY ) . 1+α

(46)

Step (a) follows from Equation (38) in Lemma 6. Step (b) follows from Equation (45). Since Equation (46) holds for any n ≥ 1 and any p(n) ∈ P (n) ( p XY ), we have Ω(µ,λ,θ ) ( p XY ) ≥

Ω(µ,λ,α) ( p XY ) . 1+α

Thus, we have Equation (44) in Proposition 2. Proof of Theorem 3: For θ ∈ [0, 1), set α=

θ α ⇔θ= . 1−θ 1+α

(47)

Then, we have the following: 1 log n (b)

≥

=

(

)

5

(a)

≥

(n)

Pc ( ϕ(n) , ψ(n) ; ∆)

¯ + µ∆) Ω(µ,λ,θ ) ( p XY ) − λθ (µR 1 + θ (3 + λµ¯ )

1 λα ¯ (µ,λ,α) ( p + µ∆) XY ) − 1+α ( µR 1+ α Ω α 1 + 1+α (3 + λµ¯ ) ¯ + µ∆| p XY ). F (µ,λ,α) (µR

=

¯ + µ∆) Ω(µ,λ,α) ( p XY ) − λα(µR 1 + α + α(3 + λµ¯ )

Step (a) follows from Corollary 3. Step (b) follows from Proposition 2 and Equation (47). Since the above bound holds for any positive λ ≥ 0, µ ∈ [0, 1], and α ≥ 0, we have 1 log n

(

5 (n)

Pc ( ϕ(n) , ψ(n) ; ∆)

)

≥ F ( R, ∆| p XY ).

Thus, Equation (10) in Theorem 3 is proved. Proof of Corollary 2: Since g is an inverse function of ϑ, the definition in Equation (13) of κn is equivalent to   s   κn 10 5 = log . (48) g ρ( p XY ) nρ( p XY ) 1−ε By the definition of n0 = n0 (ε, ρ( p XY )), we have that κn ≤ (1/2)ρ( p XY ) for n ≥ n0 . We assume that for n ≥ n0 , ( R, ∆) ∈ RWZ (n, ε| p XY ). Then, there exists a sequence {( ϕ(n) , ψ(n) ) }n≥n0 such that for n ≥ n0 , we have 1 (n) log || ϕ(n) || ≤ R, Pe ( ϕ(n) , ψ(n) ; ∆) ≤ ε. n Then, by Theorem 3, we have (n)

1 − ε ≤ Pc ( ϕ(n) , ψ(n) ; ∆) ≤ 5 exp {−nF ( R, ∆| p XY )}

(49)

for any n ≥ n0 . We claim that for n ≥ n0 , we have ( R + κn , ∆ + κn ) ∈ R( p XY ). To prove this claim, we suppose that ( R + κn∗ , ∆ + κn∗ ) does not belong to R( p XY ) for some n∗ ≥ n0 . Then, we have the following chain of inequalities:

Entropy 2018, 20, 352

19 of 32

   (a) κ n∗ n∗ ρ( p XY ) 2 5 exp [−n∗ F ( R, ∆| p XY )] < 5 exp − ·g 10 ρ( p XY )       ∗ (b) n ρ( p XY ) 1−ε 10 5 = 5 exp − = 5 exp log = 1 − ε. log 10 n∗ ρ( p XY ) 1−ε 5

(50)

Step (a) follows from κn∗ ≤ (1/2)ρ( p XY ) and Property 4 Part (e). Step (b) follows from Equation (48). The bound of Equation (50) contradicts Equation (49). Hence, we have ( R + κn , ∆+κn ) ∈R( p XY ) or equivalent to ( R, ∆) ∈ R( p XY ) − κn (1, 1) for n ≥ n0 , which implies that for n ≥ n0 ,

RWZ (n, ε| p XY ) ⊆ R( p XY ) − κn (1, 1), completing the proof. 5. Conclusions For the WZ system, we have derived an explicit lower bound of the optimal exponent function on the correct probability of decoding for for ( R, ∆) ∈ / RWZ ( p XY ). We have described this result in Theorem 3. The determination problem of the optimal exponent remains to be resolved. This problem is our future work. In this paper, we have treated the case where X and Y are finite sets. Extension of Theorem 3 to the case where X and Y are arbitrary sets is also our future work. Wyner [12] investigated the characterization of the rate distortion region in the case where X and Y are general sets and {( Xt , Yt )}∞ t=1 is a correlated stationary memoryless source. This work may provide a good tool to investigate the second future work. Acknowledgments: I am very grateful to Shun Watanabe for his helpful comments. Conflicts of Interest: The authors declare no conflicts of interest.

Appendix A. Properties of the Rate Distortion Regions In this Appendix, we prove Property 1. Property 1 Part (a) can easily be proved by the definitions of the rate distortion regions. We omit the proofs of this part. In the following argument, we prove Part (b). Proof of Property 1 Part (b): We set

RWZ (m, ε| p XY ) =

\

RWZ (n, ε| p XY ).

n≥m

By the definitions of RWZ (m, ε| p XY ) and RWZ (ε| p XY ), we have that RWZ (m, ε| p XY ) ⊆ RWZ (ε| p XY ) for m ≥ 1. Hence, we have that [

RWZ (m, ε| p XY ) ⊆ RWZ (ε| p XY ).

(A1)

m ≥1

We next assume that ( R, ∆) ∈ RWZ (ε| p XY ). Set (δ)

RWZ (ε| p XY ) := {( R + δ, ∆) : ( R, ∆) ∈ RWZ (ε| p XY )}. Then, by the definitions of RWZ (n, ε | p XY ) and RWZ ( ε| p XY ), we have that, for any δ > 0, there exists n0 (ε, δ) such that for any n ≥ n0 (ε, δ), ( R + δ, ∆) ∈ RWZ (n, ε| p XY ), which implies that

Entropy 2018, 20, 352

20 of 32

(δ)

\

RWZ (ε| p XY ) ⊆

RWZ (n, ε| p XY )

n≥n0 (ε,δ)

! [

= RWZ (n0 (ε, δ), ε| p XY ) ⊆ cl

RWZ (m, ε| p XY ) .

(A2)

m ≥1

Here, we assume that there exists a pair ( R, ∆) belonging to RWZ (ε| p XY ) such that !

( R, ∆) ∈ / cl

[

RWZ (m, ε| p XY ) .

(A3)

m ≥1

Since the set in the right hand side of Equation (A3) is a closed set, we have !

( R + δ, ∆) ∈ / cl

[

RWZ (m, ε| p XY )

(A4)

m ≥1

(δ)

for some small δ > 0. Note that ( R + δ, ∆) ∈ RWZ (ε| p XY ). Then, Equation (A4) contradicts Equation (A2). Thus, we have ! [

RWZ (m, ε| p XY ) ⊆ RWZ (ε| p XY ) ⊆ cl

m ≥1

[

RWZ (m, ε| p XY ) .

(A5)

m ≥1

Note here that RWZ (ε| p XY ) is a closed set. Then, from Equation (A5), we conclude that !

RWZ (ε| p XY ) = cl

[

RWZ (m, ε| p XY )

!

= cl

[ \

RWZ (n, ε| p XY ) ,

m ≥1 n ≥ m

m ≥1

completing the proof. Appendix B. Cardinality Bound on Auxiliary Random Variables Set R(µ) ( p XY ) := R

(µ)

( p XY ) :=



min

q∈Psh ( p XY )

min

q∈P ( p XY )



¯ q ( X; U |Y ) + µEq d( X, Z ) , µI

¯ q ( X; U |Y ) + µEq d( X, Z ) , µI

Since Psh ( p XY ) ⊆ P ( p XY ), it is obvious that R(µ) ( p XY ) ≥ R(µ) ( p XY ). We first prove the following lemma. Lemma A1. R(µ) ( p XY ) = R(µ) ( p XY ). To prove Lemma A1, we set

P1 ( p XY ) := { q1 = qUXY : |U | ≤ |X |, q XY = p XY , U ↔ X ↔ Y }, Q1 ( p XY ) := { q1 = qUXY : |U | ≤ |X | + 1, q XY = p XY , U ↔ X ↔ Y }, Q2 (qUXY ) := {q2 = q Z|UXY : qUXYZ = (qUXY , q2 ), X ↔ (U, Y ) ↔ Z }.

(A6)

Entropy 2018, 20, 352

21 of 32

By definition, it is obvious that

Psh ( p XY ) = {q = (q1 , q2 ) : q1 ∈ P1 ( p XY ), q2 ∈ Q2 (q1 )},

(A7)

P ( p XY ) = {q = (q1 , q2 ) : q1 ∈ Q1 ( p XY ), q2 ∈ Q2 (q1 )}.

(A8)

Proof of Lemma A1: Since we have Equation (A6), it suffices to show R(µ) ( p XY ) ≤ R(µ) ( p XY ) to prove Lemma A1. We bound the cardinality |U | of U to show that the bound |U | ≤ |X | is sufficient to describe R(µ) ( p XY ). We first observe that, by Equation (A8), we have R(µ) ( p XY ) =

min



min

q1 ∈Q1 ( p XY ) q2 ∈Q2 (q1 )

¯ q1 ( X; U |Y ) +µE(q1 ,q2 ) d( X, Z ) µI

o



= =

min

q1 ∈Q1 ( p XY )

 ¯ q1 ( X; U |Y ) +µ µI

n

min

min

q2 ∈Q2 (q1 )

E(q1 ,q2 ) d( X, Z )

o ¯ q1 ( X; U |Y ) + µE(q1 ,q∗ (q1 )) d( X, Z ) , µI 2

q1 ∈Q1 ( p XY )

where q2∗ = q2∗ (q1 ) = q∗Z|UY = {q∗Z|UY (z|u, y)}(u,y,z)∈U ×Y ×Z is a conditional probability distribution that attains the following optimization problem: min

q2 ∈Q2 (q1 )

E(q1 ,q2 ) d( X, Z ).

Observe that pX (x) =

∑

q U ( u ) q X |U ( x | u ) ,

(A9)

u∈U

¯ q1 ( X; U |Y ) + µE(q1 ,q∗ ) ( X, Z ) = µI 2

∑

pU (u)π (q X |U (·|u)),

(A10)

u∈U

where

∑

π (q X |U (·|u)) :=

q X |U ( x |u) pY | X (y| x )q∗Z|UY (z|u, y)

( x,y,z)∈X ×Y ×Z

× log

       qµ¯      

µ¯

( x |u)

X |U µ¯ pX (x)

      

pY (y)e−µd( x,z) "

∑

pY | X (y| x˜ )q X |U ( x˜ |u)

x˜ ∈X

#µ¯

.

     

For each u ∈ U , π (q X |U (·|u)) is a continuous function of q X |U (·|u). Then, by the support lemma,

|U | ≤ |X | − 1 + 1 = |X | is sufficient to express |X | − 1 values of Equation (A9) and one value of Equation (A10). Next, we give a proof of Property 4 Part (a). Proof of Property 4 Part (a): We first bound the cardinality |U | of U in Q to show that the bound |U | ≤ |X | is sufficient to describe Ω(µ,λ,α) ( p XY ). Observe that qX (x) =

∑

u∈U

q U ( u ) q X |U ( x | u ) ,

(A11)

Entropy 2018, 20, 352

22 of 32

n o exp −Ω(µ,λ,α) (q| p XY ) =

∑

qU (u)Π(µ,λ,α) (q X , q XYZ|U (·, ·, ·|u)),

(A12)

u∈U

where

∑ qXYZ|U (x, y, z|u) exp

Π(µ,λ,α) (q X , q XYZ|U (·, ·, ·|u)) :=

n

( x,y,z)

o (µ,λ) −αωq|| p ( x, y, z|u) .

∈X ×Y ×Z

The value of q X included in Π(µ,λ,α) (q X , q XYZ|U (·, ·, ·|u)) must be preserved under the reduction of U . For each u ∈ U , Π(µ,λ) (q X , q XYZ|U (·, ·, ·|u)) is a continuous function of q XYZ|U (·, ·, ·|u). Then, by the support lemma, |U | ≤ |X | − 1 + 1 = |X | is sufficient to express |X | − 1 values of Equation (A11) and one value of Equation (A12). We next bound the cardinality |U | of U in Psh ( p XY ) to show that the bound |U | ≤ |X | is sufficient to describe ˜ (µ,λ) ( p XY ). Observe that Ω pX (x) =

∑

p U ( u ) p X |U ( x | u ) ,

(A13)

u∈U

n o ˜ (µ,λ) ( p) = exp −Ω

∑

˜ (µ,λ) ( p X , p XYZ|U (·, ·, ·|u)), pU ( u ) Π

∑

n o (µ) p XYZ|U ( x, y, z|u) exp −λω p ( x, y, z|u) .

(A14)

u∈U

where ˜ (µ,λ) ( p X , p XYZ|U (·, ·, ·|u)) := Π

( x,y,z) ∈X ×Y ×Z

˜ (µ,λ) ( p X , p XYZ|U (·, ·, ·|u)) must be preserved under the reduction The value of p X included in Π of U . For each u ∈ U , Π(µ,λ) ( p X , p XYZ|U (·, ·, ·|u)) is a continuous function of p XYZ|U (·, ·, ·|u). Then, by the support lemma, |U | ≤ |X | − 1 + 1 = |X | is sufficient to express |X | − 1 values of Equation (A13) and one value of Equation (A14). Appendix C. Proof of Property 2 In this Appendix, we prove Property 2. Property 2 Part (a) is a well known property. Proof of this property is omitted here. We only prove Property 2 Part (b). Proof of Property 2 Part (b): Since P ∗ ( p XY ) ⊆ P ( p XY ), it is obvious that R∗ ( p XY ) ⊆ R( p XY ). Hence it suffices to prove that R( p XY ) ⊆ R∗ ( p XY ). We assume that ( R, ∆) ∈ R( p XY ). Then, there exists p ∈ P ( p XY ) such that R ≥ I p (U; X |Y ) and ∆ ≥ E p d( X, Z ). (A15) On the second inequality in Equation (A15), we have the following: ∆ ≥ E p d( X, Z ) =

"

∑

pUY (u, y)

≥

"

∑

pUY (u, y) min

∑

∑

z∈Z

(u,y)∈U ×Y

=

(u,y)∈U ×Y

pUY (u, y)

x ∈X

∑

p Z|UY (z|u, y)

z∈Z

(u,y)∈U ×Y

∑

∑

!# d( x, z) p X |UY ( x |u, y)

x ∈X

!# d( x, z) p X |UY ( x |u, y)

x ∈X

! ∗

d( x, z ) p X |UY ( x |u, y) ,

(A16)

Entropy 2018, 20, 352

23 of 32

where z∗ = z∗ (u, y) is one of the minimizers of the function

∑

d( x, z) p X |UY ( x |u, y).

x ∈X

Define φ : U × Y → Z by φ(u, y) = z∗ . We further define q = qUXYZ by qUXY = pUXY , q Z = qφ(U,Y ) . It is obvious that q ∈ P ∗ ( p XY ) and R ≥ I p ( X; U |Y ) = Iq ( X; U |Y ).

(A17)

Furthermore, from Equation (A16), we have ∆ ≥ E p d( X, Z ) ≥ Eq d( X, φ(U, Y )).

(A18)

From Equations (A17) and (A18), we have ( R, ∆) ∈ R∗ ( p XY ). Thus R( p XY ) ⊆ R∗ ( p XY ) is proved. Appendix D. Proof of Property 3 In this Appendix, we prove Property 3. From Property 2 Part (a), we have the following lemma. ˆ ) does not belong to R( p XY ). Then, there exist e > 0 and µ∗ ∈ [0, 1] such ˆ ∆ Lemma A2. Suppose that ( R, that for any ( R, ∆) ∈ R( p XY ) we have ˆ ) − e ≥ 0. µ¯∗ ( R − Rˆ ) + µ∗ (∆ − ∆ Proof of this lemma is omitted here. Lemma A2 is equivalent to the fact that if the region R( p XY ) ˆ ) outside the region R( p XY ), there exits a line which separates ˆ ∆ is a convex set, then for any point ( R, ˆ ) from the region R( p XY ). Lemma A2 will be used to prove Equation (8) in Property 3. ˆ ∆ the point ( R, Proof of Equation (8) in Property 3: We first recall the following definitions of P ( p XY ) and Psh ( p XY ):

P ( p XY ) := { pUXYZ : |U | ≤ |X | + 1, U ↔ X ↔ Y, X ↔ (U, Y ) ↔ Z }, Psh ( p XY ) := { pUXYZ : |U | ≤ |X |, U ↔ X ↔ Y, X ↔ (U, Y ) ↔ Z }. ˆ) ∈ ˆ ∆ We prove Rsh ( p XY ) ⊆ R( p XY ). We assume that ( R, / R( p XY ). Then, by Lemma A2, there ∗ exist e > 0 and µ ∈ [0, 1] such that for any ( R, ∆) ∈ R( p XY ), we have ˆ ≤ µ¯ ∗ R + µ∗ ∆ − e. µ¯ ∗ Rˆ + µ∗ ∆ Hence, we have ˆ ≤ µ¯ ∗ Rˆ + µ∗ ∆ (a)

=

≤

min



p∈P ( p XY )

min

p∈Psh ( p XY )

min

( R,∆)∈R( p XY )

{µ¯ ∗ R + µ∗ ∆} − e

µ¯ ∗ I p (U; X |Y ) + µ∗ E p d( X, Z ) − e 

∗ µ¯ ∗ I p (U; X |Y ) + µ∗ E p d( X, Z ) − e = R(µ ) ( p XY ) − e.

(A19)

Step (a) follows from the definition of R( pXY ). The inequality in Equation (A19) implies that ˆ ˆ (R, ∆) ∈ / Rsh ( pXY ). Thus, Rsh ( pXY ) ⊆ R( pXY ) is concluded. We next prove R( pXY ) ⊆ Rsh ( pXY ). We assume that (R, ∆) ∈ R( pXY ). Then, there exists q ∈ P ( pXY ) such that R ≥ Iq ( X; U |Y ), ∆ ≥ Eq d( X, Z ).

(A20)

Entropy 2018, 20, 352

24 of 32

Then, for each µ ∈ [0, 1] and for ( R, ∆) ∈ R( p XY ), we have the following chain of inequalities: (a)

¯ + µ∆ ≥ µI ¯ q ( X; U |Y ) + µEq d( X, Z ) µR   (b) ¯ q ( X; U |Y ) + µEq d( X, Z ) = R(µ) ( p XY ) = R(µ) ( p XY ). ≥ min µI q∈P ( p XY )

Step (a) follows from Equation (A20). Step (b) follows from Lemma A1. Hence, we have R( pXY ) ⊆ Rsh ( pXY ). Appendix E. Proof of Property 4 Part (b) In this Appendix, we prove Property 4 Part (b). We have the following lemma. 1 Lemma A3. For any µ ∈ [0, 1], λ ≥ 0, α ∈ [0, λ+ 1 ] and any q = qUXY ∈ Q( pY | X ), there exists p = pUXYZ ∈ Psh ( p XY ) such that ˜ (µ,λ) ( p). Ω(µ,γ,α) (q| p XY ) ≥ αΩ (A21) 1 This implies that for any µ ∈ [0, 1], λ ≥ 0, α ∈ [0, λ+ 1 ], we have

˜ (µ,λ) ( p XY ). Ω(µ,γ,α) ( p XY ) ≥ αΩ

(A22)

Proof. Since Equation (A22) is obvious from Equation (A21), we only prove Equation (A21). We consider the case where (µ, λ, α) satisfies (µ, λ) ∈ [0, 1], λ ≥ 0, and α ∈ [0, 1+1 λ ]. In this case, we have λ λα α λµ¯ ≤ ≤ 1+λ1 = 1. (A23) α¯ α¯ 1 − 1+ λ For each q = qUXYZ ∈ Q, we choose p = pUXYZ ∈ Psh ( p XY ) so that pU | X = qU | X and p Z|UY = q Z|UY . Then, for any (µ, λ, α) satisfying µ ∈ [0, 1], λ ≥ 0, and α ∈ [0, 1+1 λ ], we have the following chain of inequalities: n o exp −Ω(µ,λ,α) (q| p XY ) α   λµ¯ −λµd( X,Z )   p ( X ) p (Y | X ) ( X | Y ) e p q ( Z | U, Y ) Y|X Z |UY X |Y X  = Eq  λµ¯  q X ( X ) qY | XU (Y | X, U ) q Z|UYX ( Z |U, Y, X )  q X |UY ( X |U, Y ) α   )λµα λµ¯ ¯ −λµd( X,Z )  ( p p ( X | Y ) e p ( X | U, Y ) X |UY UXYZ ( X, Y, Z, U ) X |Y  = Eq  λµ¯  qUXYZ ( X, Y, Z, U )  q X |UY ( X |U, Y ) p ( X |U, Y ) X |UY

(a)





 

p ( X, Y, Z, U ) ≤ Eq  UXYZ qUXYZ ( X, Y, Z, U )  n

˜ = exp −αΩ

α λµ¯ p X |Y ( X |Y )e−λµd(X,Z)  λµ¯ p X |UY ( X |U, Y )

(µ,λ)

o

(

( p )  Eq 

p X |UY ( X |U, Y ) q X |UY ( X |U, Y )

Step (a) follows from Hölder’s inequality. Hölder’s inequality.

)λµ¯

(

 Eq 

p X |UY ( X |U, Y ) q X |UY ( X |U, Y )

)λµ¯ α α¯ α¯ 

¯ α α

α¯

n o (b) ˜ (µ,λ) ( p) .  ≤ exp −αΩ

(A24)

Step (b) follows from Equation (A23) and

Entropy 2018, 20, 352

25 of 32

Proof of Property 4 Part (b): We have the following chain of inequalities: F ( R, ∆| p XY ) =

≥

¯ + µ∆) Ω(µ,λ,α) ( p XY ) − λα(µR ¯ 1 + (4 + µλ)α µ∈[0,1],λ,α≥0 sup

sup

sup µ∈[0,1],λ≥0 α∈[0, 1 ] 1+ λ

¯ + µ∆) Ω(µ,λ,α) ( p XY ) − λα(µR ¯ )α 1 + (4 + µλ

˜ (µ,λ) ( p XY ) − λ(µR ¯ + µ∆)] α[Ω sup ¯ 1 + ( 4 + µλ ) α µ∈[0,1],λ≥0 α∈[0, 1 ] 1+ λ ˜ (µ,λ) ( p XY ) − λ(µR ¯ + µ∆) (b) Ω = sup = F˜ ( R, ∆| p XY ). ¯) 5 + λ ( 1 + µ µ∈[0,1],λ≥0 (a)

≥

sup

Step (a) follows from Lemma A3. Step (b) follows from α 1 = , sup ¯ )α 1 + (4 + µλ 5 + λ(1 + µ¯ ) α∈[0, 1+1 λ ] completing the proof. Appendix F. Proof of Property 4 Parts (c), (d), and (e) In this Appendix, we prove Property 4 Parts (c), (d), and (e). We first prove Parts (c) and (d). Proof of Property 4 Parts (c) and (d): We first prove Part (c). We first show that for each p ∈ Psh ( p XY ) ˜ (µ,λ) ( p)], for λ ∈ [0, 1], we ˜ (µ,λ) ( p) exists for λ ∈ [0, 1]. On a lower bound of exp[−Ω and µ ∈ [0, 1], Ω have the following: ˜ exp[−Ω

(µ,λ)

( p)] =

"

≥

#µλ ¯

∑

pUXYZ (u, x, y, z)

∑

pUXYZ (u, x, y, z) p X |Y ( x |y)e−d( x,z) .

(u,x,y,z) ∈U ×X ×Y ×Z (a)

p X |Y ( x | y ) p X |UY ( x |u, y)

e−µλd( x,z)

(A25)

(A26)

(u,x,y,z) ∈U ×X ×Y ×Y

Step (a) follows from that, for µ, λ ∈ [0, 1], "

p X |Y ( x | y ) p X |UY ( x |u, y)

#µλ ¯ e−µλd( x,z) ≥ p X |Y ( x |y)e−d( x,z) .

˜ (µ,λ) ( p)] takes some positive It is obvious that the lower bound of Equation (A26) of exp[−Ω ( µ,λ ) ˜ ˜ (µ,λ) ( p) ≥ 0 for λ ∈ [0, 1]. value. This implies that Ω ( p) exists for λ ∈ [0, 1]. We next show that Ω ( µ,λ ) ˜ On upper bounds of exp[−Ω ( p)] for λ ∈ [0, 1], we have the following chain of inequalities:

Entropy 2018, 20, 352

26 of 32

˜ exp[−Ω

(µ,λ)

∑

=

(u,x,y) ∈U ×X ×Y (b)

(a)

∑

( p)] ≤

pUXYZ (u, x, y, z) (u,x,y,z) ∈U ×X ×Y ×Z ¯ ¯ 1−µλ µλ pUY (u, y) p X |UY ( x |u, y) p X |Y ( x |y)

∑

pUY (u, y)

∑

pUY (u, y) = 1.

≤

∑

p X |Y ( x | y )

#µλ ¯

p X |UY ( x |u, y)

!1−µλ ¯ p X |UY ( x |u, y)

x ∈X

(u,y)∈U ×Y

=

"

∑

!µλ ¯ p X |Y ( x | y )

x ∈X

(A27)

(u,y)∈U ×Y

¯ ∈ [0, 1] and Step (a) follows from Equation (A25) and e−µλd( x,z) ≤ 1. Step (b) follows from µλ ˜ (µ,λ) ( p) is twice Hölder’s inequality. We next prove that that, for each p ∈ Psh ( p XY ) and µ ∈ [0, 1], Ω differentiable for λ ∈ [0, 1/2]. For simplicity of notations, set a := (u, x, y, z), A := (U, X, Y, Z ), A := U × X × Y × Z , (µ)

˜ (µ,λ) ( p) := ξ (λ). ω˜ p ( x, y, z|u) := ς( a), Ω Then, we have

" ˜ (µ,λ) ( p) = ξ (λ) = − log Ω

∑

# p A ( a)e−λς(a) .

(A28)

a∈A

(λ)

The quantity p(λ) ( a) = p A ( a), a ∈ A has the following form: p(λ) ( a) = eξ (λ) p( a)e−λς(a) .

(A29)

By simple computations, we have " 0

ξ (λ) = e

ξ (λ)

∑

# p ( a ) ς ( a )e

−λς( a)

a∈A

" ξ 00 (λ) = −e2ξ (λ)

=−

∑

p

(λ)

=

∑

p ( λ ) ( a ) ς ( a ),

a∈A

{ς( a) − ς(b)}2 −λ{ς(a)+ς(b)} e ∑ p( a) p(b) 2 a,b∈A

( a) p

(λ)

a,b∈A

#

{ς( a) − ς(b)}2 (b) = − ∑ p ( λ ) ( a ) ς2 ( a ) + 2 a∈A

"

∑

#2 p

(λ)

( a)ς( a)

≤ 0.

(A30)

a∈A

On upper bound of −ξ 00 (λ) ≥ 0 for λ ∈ [0, 1/2], we have the following chain of inequalities: (a)

∑

− ξ 00 (λ) ≤

(b)

p ( λ ) ( a ) ς2 ( a ) =

a∈A

=e

ξ (λ)

∑

p( a)

≤

p

e2ξ (λ)

s

p( a)ς2 ( a)e−λς(a)+ξ (λ)

a∈A

p

e−2λς(a)

q

a∈A

(d)

∑

ς4 ( a )

(c)

≤

p

e2ξ (λ)−ξ (2λ)

s

∑

p ( a ) ς4 ( a )

a∈A

∑

p ( a ) ς4 ( a ).

(A31)

a∈A

Step (a) follows from Equation (A30). Step (b) follows from Equation (A29). Step (c) follows from Cauchy–Schwarz inequality and Equation (A28). Step (d) follows from that ξ (2λ) ≥ 0 for 2λ ∈ [0, 1]. Note that ξ (λ) exists for λ ∈ [0, 1/2]. Furthermore, we have the following:

Entropy 2018, 20, 352

27 of 32

∑

p( a)ς4 ( a) < ∞.

a∈A

Hence, by Equation (A31), ξ 00 (λ) exists for λ ∈ [0, 1/2]. We next prove Part (d). We derive the ˜ (µ,λ) ( p XY ). Fix any (µ, λ) ∈ [0, 1] × [0, 1/2] and any p ∈ Psh ( p XY ). lower bound Equation (9) of Ω ˜ (µ,λ) ( p) with respect to λ around λ = 0, we have that, for any By the Taylor expansion of ξ (λ) = Ω p ∈ Psh ( p XY ) and for some ν ∈ [0, λ], ˜ (µ,λ) ( p) = ξ (0) + ξ 0 (0)λ + 1 ξ 00 (ν)λ2 Ω 2 h i λ2 h i (µ) (µ) = λE p ω˜ p ( X, Y, Z |U ) − Var p(ν) ω˜ p ( X, Y, Z |U ) 2 h i 2 (a) λ (µ) ≥ λR(µ) ( p XY ) − Var p(ν) ω˜ p ( X, Y, Z |U ) . 2

(A32)

Step (a) follows from p ∈ Psh ( p XY ), h i (µ) ¯ p ( X; U |Y ) + µE p d( X, Z ), E p ω˜ p ( X, Y, Z |U ) = µI and the definition of R(µ) ( p XY ). Let (νopt , popt ) ∈ [0, λ]× Psh ( p XY ) be a pair which attains ρ(µ,λ) ( p XY ). By this definition, we have that ˜ (µ,λ) ( popt ) = Ω ˜ (µ,λ) ( p XY ) Ω

(A33)

and that, for any ν ∈ [0, λ], Var

h (ν) popt

i (µ) ω popt ( X, Y, Z |U ) ≤ Var

h (νopt ) popt

i (µ) ω popt ( X, Y, Z |U ) = ρ(µ,λ) ( p XY ).

(A34)

On lower bounds of Ω(µ,λ) ( p XY ), we have the following chain of inequalities: h i 2 (b) a) ˜ (µ,λ) ( p XY ) (= ˜ (µ,λ) ( popt ) ≥ λR(µ) ( p XY ) − λ Var (ν) ω˜ (pµ) ( X, Y, Z |U ) Ω Ω opt popt 2 (c)

≥ λR(µ) ( p XY ) −

(d) λ2 (µ,λ) λ2 ρ ( p XY ) ≥ λR(µ) ( p XY ) − ρ( p XY ). 2 2

Step (a) follows from Equation (A33). Step (b) follows from Equation (A32). Step (c) follows from Equation (A34). Step (d) follows from the definition of ρ( p XY ). We next prove Part (e). For the proof we use the following lemma. Lemma A4. When τ ∈ (0, (1/2)ρ], the maximum of 1 5 + 2λ



1 − ρλ2 + τλ 2



for λ ∈ (0, 1/2] is attained by the positive λ0 satisfying ϑ ( λ0 ) : =

1 2 τ λ + λ0 = . 5 0 ρ

(A35)

Let g( a) be the inverse function of ϑ ( a) for a ≥ 0. Then, the condition of Equation (A35) is equivalent to λ0 = g( τρ ). The maximum is given by

Entropy 2018, 20, 352

28 of 32

1 5 + 2λ0



1 − ρλ20 + τλ0 2



=

ρ 2 ρ 2 λ0 = g 10 10

  τ . ρ

By an elementary computation we can prove this lemma. We omit the detail. Proof of Property 4 Part (e): By the hyperplane expression Rsh ( p XY ) of R( p XY ) stated Property 3 we have that when ( R + τ, ∆ + τ ) ∈ / R( p XY ), we have ∗

µ¯ ∗ ( R + τ ) + µ∗ (∆ + τ ) < R(µ ) ( p XY ) or equivalent to

∗

R(µ ) ( p XY ) − (µ¯ ∗ R + µ∗ ∆) > τ

(A36)

for some µ∗ ∈ [0, 1]. Then, for each positive τ, we have the following chain of inequalities: F˜ ( R, ∆| p XY ) ≥

∗ F˜ (µ ,λ) (µ¯ ∗ R + µ∗ ∆| p XY )

sup λ∈(0,1/2]

˜ (µ∗ ,λ) ( p XY ) − λ(µ¯ ∗ R + µ∗ ∆) Ω = sup 5 + λ(1 + µ¯ ∗ ) λ∈(0,1/2] )  (a) ∗ 1 1 2 ≥ sup − ρλ + λR(µ ) ( p XY ) − λ(µ¯ ∗ R + µ∗ ∆) 2 λ∈(0,1/2] 5 + 2λ     (b) (c) ρ 2 τ 1 1 2 − ρλ + τλ = . > sup g 2 10 ρ λ∈(0,1/2] 5 + 2λ Step (a) follows from Property 4 Part (d). Step (b) follows from Equation (A36). Step (c) follows from Lemma A4. Appendix G. Proof of Lemma 1 To prove Lemma 1, we prepare a lemma. Set (

A˜ n :=

p n (xn ) 1 ≥ −η x : log X(i) n QXn (xn )

)

n

,

An := A˜ n × Mn × Y n × Z n , Acn := A˜ cn × Mn × Y n × Z n ,   n |xn )   p ( y n n 1 Y |X B˜ n := (s, x n , yn ) : log (ii) ≥ −η ,   n Q (yn | x n , s) n

B˜ nc

Y n | X n Sn n

Bn := B˜ n × Z := ×Z ,   n n   p X n |S Y n ( x |s, y ) 1 Cn := (s, x n , yn , zn ) : log (iii) n ≥ −η ,   n Q X n |S Y n Zn ( x n |s, yn , zn ) , Bnc

n

n

n

D˜ n := {(s, x , y ) : s = ϕ

(n)

(x

n

(iv) ), Q X n |S Yn ( x n |s, yn ) n n

≤ Mn enη p X n |Yn ( x n |yn )},

Dn := D˜ n × Z n , Dnc := D˜ nc × Z . Then, we have the following lemma: Lemma A5.

pSn X n Y n Zn (Acn ) ≤ e−nη , pSn X n Y n Zn (Bnc ) ≤ e−nη , pSn X n Y n Zn (Cnc ) ≤ e−nη , pSn X n Y n Zn (Dnc ) ≤ e−nη .

Entropy 2018, 20, 352

29 of 32

Proof. We first prove the first inequality. We have the following chain of inequalities:

∑

pSn X n Y n Zn (Acn ) = p X n (A˜ cn ) =

p Xn ( x n )

x n ∈A˜ cn

(a)

e−nη Q X n ( x n ) ≤ e−nη ∑ Q X n ( x n ) = e−nη .

∑

≤

(i)

(i)

xn

x n ∈A˜ cn

Step (a) follows from the definition of An . We next prove the second inequality. We have the following chain of inequalities: (a) pSn X n Y n Zn (Bnc ) = pSn X n Y n (B˜ nc ) =

∑

pSn X n (s, x n ) pY n | X n (yn | x n )

(s,x n ,yn )∈B˜ nc (b)

∑

≤

(ii)

(s,x n ,yn )∈B˜ nc

≤ e−nη

e−nη pSn X n (s, x n ) QY n |S

nX

n

(yn |s, x n )

pSn X n (s, x n ) QY n |S X n (yn |s, x n ) = e−nη . ∑ n n n (ii)

s,x ,y

Step (a) follows from the Markov chain Sn ↔ X n ↔ Y n . Step (b) follows from the definition of Bn . On the third inequality, we have the following chain of inequalities: (a)

pSn X n Y n Zn (Cnc ) = (b)

∑

≤

p X n |Sn Y n ( x n |s, yn ) pSn Y n Zn (s, yn , zn )

(iii)

(s,x n ,yn ,zn )∈Cnc

∑ n n

≤ e−nη

∑

(s,x n ,yn ,zn )∈Cnc

e−nη Q X n |S

nY

(iii)

s,x ,y ,zn

Q X n |S

nY

n Zn

n Zn

( x n |s, yn , zn ) pSn Yn Zn (s, yn , zn )

( x n |s, yn , zn ) pSn Yn Zn (s, yn , zn ) = e−nη .

Step (a) follows from the Markov chain X n ↔ Sn Y n ↔ Z n . Step (b) follows from the definition of Cn . We finally prove the fourth inequality. We have the following chain of inequalities: pSn X n Y n Zn (Dnc ) = pSn X n Y n (D˜ nc )=

∑

s∈Mn

≤

e−nη Mn

∑

∑

s∈Mn ( x n ,yn ):ϕ(n) ( x n )=s, p X n |Y n ( x n | y n ) ≤(1/Mn )e−nη (iv)

× Q X n |S

≤

e−nη Mn

(iv)

Q X n |S

n Yn

nY

n

∑

p X n |Y n ( x n | y n ) p Y n ( y n )

( x n ,yn ):ϕ(n) ( x n )=s, p X n |Y n ( x n | y n ) ≤(1/Mn )e−nη (iv) × Q X n |S ,Y n ( x n |s,yn ) n

( x n |s, yn ) pYn (yn )

( x n |s,yn )

Q X n |S Y n ( x n |s, yn ) pY n (yn ) = e−nη . ∑ ∑ n n n (iv)

s∈Mn x ,y

Proof of Lemma 1: We set 

En :=

(s, x n , yn , zn ) :

 1 d( X n , Zn ) ≤ ∆ . n

Set R(n) := (1/n) log Mn . By definition, we have

Entropy 2018, 20, 352

30 of 32

pSn X n Y n Zn (An ∩ Bn ∩ Cn ∩ Dn ∩ En )  (i) Q n (Xn ) 1 , = pSn X n Yn Zn η ≥ log X n pXn (X n ) (ii)

Q Y n | X n S (Y n | X n S ) 1 η ≥ log , n p Y n | X n (Y n | X n ) (iii)

Q X n | S Y n Z n ( X n | Sn Y n Z n ) 1 n η ≥ log , n p X n | Sn Y n ( X n | Sn Y n ) (iv)

Q X n | S Y n ( X n | Sn Y n ) 1 n log , n p X n |Y n ( X n | Y n )  1 n n ∆ ≥ log exp {d( X , Z )} . n Then, for any ( ϕ(n) , ψ(n) ) satisfying R(n) + η ≥

R(n) =

(A37)

1 log Mn ≤ R, n

we have pSn X n Y n Zn (An ∩ Bn ∩ Cn ∩ Dn ∩ En )  (i) Q n (Xn ) 1 , ≤ pSn X n Yn Zn η ≥ log X n pXn (X n ) (ii)

Q Y n | X n S (Y n | X n S ) 1 η ≥ log , n p Y n | X n (Y n | X n ) (iii)

Q X n | S Y n Z n ( X n | Sn Y n Z n ) 1 n η ≥ log , n p X n | Sn Y n ( X n | Sn Y n ) (iv)

Q X n | S Y n ( X n | Sn Y n ) 1 n , log n p X n |Y n ( X n | Y n )  1 n n ∆ ≥ log exp {d( X , Z )} . n

R+η ≥

(A38)

Hence, it suffices to show (n)

(n)

(n)

Pc ( ϕ1 , ϕ2 , ψ(n) ; ∆) ≤ pSn X n Y n (An ∩ Bn ∩ Cn ∩ Dn ∩ En ) + 4e−nη to prove Lemma 1. By definition, we have (n)

Pc ( ϕ(n) , ψ(n) ; ∆) = pSn X n Y n Zn (En ) . Then, we have the following: (n)

Pc ( ϕ(n) , ψ(n) ; ∆) = pSn X n Y n Zn (En )

= pSn X n Yn Zn (An ∩ Bn ∩ Cn ∩ Dn ∩ En ) + pSn X n Yn Zn [An ∩ Bn ∩ Cn ∩ Dn ]c ∩ En ≤ pSn X n Yn Zn (An ∩ Bn ∩ Cn ∩ Dn ∩ En ) + pSn X n Yn Zn (Acn ) + pSn X n Yn Zn (Bnc ) + pSn X n Yn Zn (Cnc ) + pSn X n Yn Zn (Dnc ) (a)

≤ pSn X n Yn Zn (An ∩ Bn ∩ Cn ∩ Dn ∩ En ) + 4e−nη .

Step (a) follows from Lemma A5.




Entropy 2018, 20, 352

31 of 32

Appendix H. Proof of Lemma 2 In this Appendix, we prove Lemma 2. Proof of Lemma 2: We have the following chain of inequalities: I ( Xt ; Y t−1 |Sn Xt−1 Ytn ) = H (Y t−1 |Sn X t−1 Ytn ) − H (Y t−1 |Sn X t Ytn ) (a)

≤ H (Y t−1 | X t−1 ) − H (Y t−1 |Sn X n Ytn ) = H (Y t−1 | X t−1 ) − H (Y t−1 | X n Ytn ) (b)

= H (Y t−1 | X t−1 ) − H (Y t−1 | X t−1 ) = 0.

Step (a) follows from that Sn = ϕ(n) ( X n ) is a function of X n . Step (b) follows from the memoryless property of the information source {( Xt , Yt )}∞ t =1 . Appendix I. Proof of Lemma 5 In this Appendix, we prove Equations (32) and (33) in Lemma 5. (µ,λ,θ )

Proofs of Equations (32) and (33) in Lemma 5: By the definition of p X t Zt |S Y n ,qt ( x t , zt |s, yn ), for t = n 1, 2, · · · , n, we have t

(µ,λ,θ )

p X t Z t |S

n

Y n ;F t

( x t , zt |s, yn ) = Ct−1 (s, yn ) p X t Zt |Sn Yn ( x t , zt |s, yn ) ∏ f F

(µ,λ,θ )

i =1

i

( x i , y i , z i | u i ).

(A39)

Then, we have the following chain of equalities: t

(a)

(µ,λ,θ )

p X t Z t |S

n

Y n ;F t

( x t , zt |s, yn ) = Ct−1 (s, yn ) p X t Zt |Sn Yn ( x t , zt |s, yn ) ∏ f F

(µ,λ,θ )

i =1

t −1

i

( xi , yi , zi | ui )

= Ct−1 (s, yn ) p X t−1 Zt−1 |Sn Yn ( x t−1 , zt−1 |s, yn ) ∏ f F

(µ,λ,θ )

( xi , yi , zi | ui ) i i =1 (µ,λ,θ ) × p Xt |Zt |X t−1 Zt−1 SYn ( xt , zt | x t−1 , zt−1 , s, yn ) f Ft ( xt , yt |ut ) n (b) Ct−1 ( s, y ) (µ,λ,θ ) p ( x t−1 , zt−1 |s, yn ) = Ct (s, yn ) X t−1 Zt−1 |Sn Y n ;F t−1 (µ,λ,θ ) × p Xt |Zt |X t−1 Zt−1 Sn Yn ( xt , zt | x t−1 , zt−1 , s, yn ) f Ft ( xt , yt , zt |ut ) (µ,λ,θ ) (µ,λ,θ ) = (Φt (s, yn ))−1 p X t−1 Zt−1 |S Yn ;F t−1 ( xt , yt , zt |ut ) n (µ,λ,θ )

× p Xt |Zt |X t−1 Zt−1 Sn Yn ( xt , zt | x t−1 , zt−1 , s, yn ) f Ft

( x t , y t , z t | u t ).

(A40)

Steps (a) and (b) follow from Equation (A39). From Equation (A40), we have (µ,λ,θ )

Φt,qt

(µ,λ,θ )

(s, yn ) p X t Zt |S

(µ,λ,θ )

= p X t −1 Z t −1 | S

nY

n ; F t −1

nY

n

( x t , zt |s, yn )

(A41)

( x t−1 , zt−1 |s, yn ) (µ,λ,θ )

× p Xt Zt |X t−1 Zt−1 Sn Yn ( xt , zt | x t−1 , zt−1 , s, yn ) f Ft

( x t , y t , z t | u t ).

Taking summations of Equation (A41) and (A42) with respect to x t , zt , we obtain (µ,λ,θ )

Φt,F t

(s, yn )=

∑

x t ,zt

(µ,λ,θ )

p X t −1 Z t −1 | S

nY

n ; F t −1

( x t−1 , zt−1 |s, yn ) (µ,λ,θ )

× p Xt Zt |X t−1 Zt−1 Sn Yn ( xt , zt | x t−1 , zt−1 , s, yn ) f Ft completing the proof.

( x t , y t , z t | u t ),

(A42)

Entropy 2018, 20, 352

32 of 32

References 1. 2. 3.

4. 5.

6.

7.

8.

9. 10. 11. 12.

Wyner, A.D.; Ziv, J. The rate-distortion function for source coding with side information at the decoder. IEEE Trans. Inf. Theory 1976, 22, 1–10. [CrossRef] Shannon, C.E. Coding theorems for a discrete source with a fidelity criterion. IRE Nat. Conv. Rec. 1959, 7, 142–163. Wolfowiz, J. Approximation with a fidelity criterion. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability; University of California Press: Berkeley, CA, USA, 1966; Volume 1, pp. 566–573. Csiszár, I.; Körner, J. Information Theory: Coding Theorems for Discrete Memoryless Systems; Academic: London, UK, 1981. Oohama, Y. Exponent function for one helper source coding problem at rates outside the rate region. In Proceedings of the 2015 IEEE International Symposium on Information Theory, Hong Kong, China, 14–19 June 2015; pp. 1575–1579. Oohama, Y. Strong converse exponent for degraded broadcast channels at rates outside the capacity region. In Proceedings of the 2015 IEEE International Symposium on Information Theory, Hong Kong, China, 14–19 June 2015; pp. 939–943. Oohama, Y. Strong converse theorems for degraded broadcast channels with feedback. In Proceedings of the 2015 IEEE International Symposium on Information Theory, Hong Kong, China, 14–19 June 2015; pp. 2510–2514. Oohama, Y. Exponent function for asymmetric broadcast channels at rates outside the capacity region. In Proceedings of the 2016 IEEE InternationalSymposium on Information Theory and Its Applications, Monterey, CA, USA, 30 October–2 November 2016; pp. 568–572. Oohama, Y.; Han, T.S. Universal coding for the Slepian-wolf data compression system and the strong converse theorem. IEEE Trans. Inf. Theory 1994, 40, 1908–1919. [CrossRef] Slepian, D.; Wolf, J.K. Noiseless coding of correlated information sources. IEEE Trans. Inf. Theory 1973, 19, 471–480. [CrossRef] Han, T.S. Information-Spectrum Methods in Information Theory; Springer: Berlin, Germany, 2002; The Japanese edition was published by Baifukan-Publisher, Tokyo, 1998. Wyner, A.D. The rate-distortion function for source coding with side information at the decoder-II: General sources. Inf. Control 1978, 38, 60–80. [CrossRef] c 2018 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access

article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).