On “Soft” Bit Allocation - CiteSeerX

614

IEEETRANSACTIONS

ON ACOUSTICS, SPEECH, ANDSIGNALPROCESSING,

VOL. ASSP-35, NO. 5, MAY 1987

On “Soft” Bit Allocation A. M. BRUCKSTEIN

r s ’

to transmit information on successive realizations of the , X,) overaJixed capacity commuvector ( X , , X,, nication channel. We note that the optimal E, and the corresponding bit-count B , do not correspond to a point on the rate-distortion curve for aninformation source producing the vector of random variables. The assumption of independent quantization of the elements in the source vector necessarily makes this coding scheme a suboptimal one. It is, however, a very popular block quantization process, widely used in speech coding and image processing applications. The bit allocation process, as described above, does not allow any flexibility in either reducing the bit rate when the quantization error is smaller than required, or in increasing it when it becomes temporarily important to im, X,. With prove the precision of describing X , , X,, I.INTRODUCTION present technology it is not a problem to dynamically alUPPOSE we are given M independent random vari- locate channel capacity in order to accommodate a nonables, X , , X,, * , X , with variances of 0: 2 ui 2 stationary information source, and improved performance u32 2 * * 2 ui, and we separately quantize them using may certain!y be expected to result from adaptation of the si quantization levels for Xi. Then the total quantization quantization process to slowly varying statistical propererror, with optimal quantization,is approximately pro- ties of the sources or to changing demands on precision. In this paper we propose a “soft” bit allocation proportional to cedure. It will be assumed that two functions, CQ(.) and M C,( that measure the cost incurred by quantization error and by bit usage, respectively, are given, and we shall and the total number of bits needed to code the si determine an integer quantization level allocation { s i that possible discretized levels forthevector ( X l , X,, , minimizes their sum for a source vector characterized by the given set of variances { u:, a;, , a;}. X,) is The total cost to be minimized, over all feasible alloM M cations { s i } , is therefore the function

Abstract-If a random variable X with variance is quantized optimally, being mapped into s discrete levels, the quantization error is roughly proportional to 6 ’ / s 2 . In many applications in speech coding and image digitization, we are given sets of random variables and we have to quantize them in some way so as to represent their realizations with as few discretized levels (bits) as possible. This has to be done while also minimizing the total quantization error. The quantization level or bit allocation process should therefore be the result of a compromise between the total discretization error and the number of bits used to represent the realizations of random vectors. This paper presents a solution of this problem, extending the classical bit allocation methods in which a fixed, prescribed number of bits have to be allocated to minimize the quantization error. The general soft bit allocation process is useful in designing variable rate (adaptive) coders as devised for opposed to the classical bit allocation procedures that were information transmission over communication channels requiring constant rate encoding.

-

---

S

-

e),

IIE , -

--

/ M

We note that the quantization error €or uni€ormly distributed random variables is exactly proportional to the expression given by (1). The expression a 2 / s 2 is also a very good approximation of the error in quantizing a random variable of variance u 2 to s levels, and therefore, (1) is a good approximation for the total error incurred by independent quantization of M random variables. The classical bit allocation or integer quantization level allocation problem is to choose the natural numbers si so as to meet the requirement that B T = B (a certain predetermined integer), while minimizing EQ. The motivation for such a bit allocation process may be the requirement Manuscript received April 18, 1986; revised July 26, 1986. The author is with theDepartment of Electrical Engineering, Technion-Israel Institute of Technology, Haifa, 32000, Israel. IEEE Log Number 8613495.

cQ

where and GD are monotone increasing and smooth (differentiable) functions. The minimization of this function has to be carried out under the constraint that si are natural numbers greater than one. 11. THEOPTIMALLEVELALLOCATION PROCESS The optimal integer quantization level allocation problem defined in the previous section is an integer programming problem. We shall first assume that the si may take any real value in the interval [ 1, 03 ), and solve the problem of minimizing the cost function (3), under this constraint alone. Then the integers closest to the optimal solution may be chosen as the integer level allocation, or a search process around the optimal solution may be done.

0096-35 18/87/0500-0614$01.OO 0 1987 IEEE

BRUCKSTEIN:

615

BIT ALLOCATION

Thevector {si] that minimizes the cost functional G ( (si1) is characterized using a classical result in optimization under simple inequality constraints (see, e.g., [14, p. 761). Since we need to have si 1 1 for all i , the necessary conditions for optimality of an assignment { ST} are

ues of E$ and B;. These values should be substituted in (6), and then the optimality conditions (7) will show whether the guess for W was correct or not. This process must be started with the maximal value for W , W = M , and repeated for decreasing values of W until a consistent set of level assignments is obtained [satisfying the optimality conditions (6)]. Note that the existence and uniqueness of solutions for the set of nonlinear equations (8)-(9) dependsonthe ,( These choice of the cost functions G,( .) and C functions should be chosen to allow the use of a reasonable number of bits if the quantization error is very severely penalized, and conversely, should not prohibit an increase in quantization error if the number of Sits is limited to a small value by the corresponding penalty function. In the sequel we shall assume that W = M yields a consistent solution, i.e., that we have s? > ‘1 for i = 1, 2, 3, * * . , M . This assumption is very reasonable fora quantization level assignment problem, implying that we will in fact code each of the elements of the vector (Xl, x,, , XM),providing at least one bit of information about all of them. Under this assumption, the equations (8) and (9) for the optimal total bit usage and the corresponding quantization error are a).

This yields the following result:

1

for s:

>1

(51

for s t = 1 where Defining

denote the derivatives of the cost functions. \

M

B; =

C i=l

M

log,

S? =

log2 II ai i= 1

we can rewrite the optimality conditions ( 5 ) as follows:

= 1

;s:

when Pk{s?j

I1.

(7)

Clearly, for every choiceof ( si1, the sequence Pk will be decreasing with k (since we assumed the variances are decreasing with the index). Thus, in searching for an assignment that satisfies the necessary conditions for optimality , we may start with an assumption that siare greater than one for i IW IM . Using the necessary conditions for optimality , one can write the following equations for the quantization error E$ and the total number of bits used B;:

In all cases, the B; bits will be allocated proportionally to log, ai;however, the interesting game here is how many bits are chosen to be distributed in the soft allocation process. 111. SOMEINTERESTING PARTICULAR CASES We shall next analyze two particular cases of the allocation process outlined above. In the first case, the bit usage cost function is defined in a way that makes it very expensive to exceed a certain predetermined total number of bits B , whereas the second case is one for which the quantization error excess above a certain allowed level E is severely penalized.

M

B; =

C

i= 1

log2 si* = log,

II

ai

i=l

A. Linear Quantization Error Cost and Nonlinear BitUsage Penalty Let us consider that

Under the assumption that W is known, (8) and (9) constitute a nonlinear system of equations that yield the val-

616

IEEE TRANSACTIONS ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOL. ASSP-35, NO. 5 , MAY 1987

In this case, from (10) and (1 I), we obtain

This gives

from which it is easy to see that E: + E , if n -+ 00. Solving for E; in the equation above and substituting into (19), we also have that

and as n

-+

03,the optimal number of bits approaches

Bf(n=

m) =

1 -log2 2

” (p) i=l

1 2

IT

= -log2

from which we can see that, as n of allocated bits becomes

--t 00

, the total number

BF(n = 03) = B .

(15)

This is hardly an unexpected result, since the bit usage cost defined by (12) becomes, as n grows to infinity, a “barrier” function forcing (15). The quantization error induced by ( 14) will be

and, as n

-+

03,the quantization error tends to

The method therefore yields the classical bit allocation solution for theabove-considered class of cost functionals in the limit of high values for n , when bit-usage cost becomes a barrier-type penalty function.

B. Linear Bit-Usage Cost and Nonlinear Quantization Error Penalty Here we shall introduce the cost functionals

Now (10) and (1 1) yield

M i=l

o’

(q).

(22)

This is thecase in which we enforcea certain maximal quantization error E and ask for the number of bits necessary to achieve it. Indee4, B , = C K l log2 ( o ’ / ( E / M ) ) implies a choice of s: = o ; / ( E / M ) . This means that we set the quantization levels so as to achieve the same error, E / M = o ; / s ; , in each component Xi.

IV. CONCLUSIONS We have outlined an approach to quantization level allocation that is based on a functional controlling the tradeoff between quantization error and bit usage. This method generalizes the classical bit allocation procedure that requires B bits to be allocated to M variables so as to minimize the quantization error. As a particular case of this general approach, we also solve the dual problem of assigning a variable number of bits so as to enforce an upper bound on the quantization error. There is a rather large body of literature devoted to the bit, or more generally, to the channel capacity allocation problem. The problem is sometimes referred to as the block-quantization problem, or the problem of allocating a communication channel to a vector source of information in conjunction with optimal coding schemes achieving the rate-distortion bound. The optimal bit allocation for block quantization was first obtained by Huang and Shultheiss [ 11, and subsequently, extensions for channel allocation with general source coding (not necessarily via quantizations) were discussed by Goodman [2]? [3] and Segall [6]. For various applications in speech and image coding see [4], [ 5 ] , [7]-[12]. In particular? the work of Dubnowski and Crochiere [8] deals with the problem of variable rate coding of speech. It isproposed there to perform dynamic bit allocation within temporal blocks of waveform samples, each block being encoded with a fixed number of bits. This results in in a classical bit allocation solution within each block of data, the local rate fluctuations being due to the varying number of quantization levels allocated to successive samples. The process described in this paper would be more appropriate to a subband coding scenario, as opposed to this time-domain

617

BRUCKSTEIN: SOFT BIT ALLOCATION

approach; however, one could also adapt the block rate to accommodate slow, interblock temporal variations in either source statistics or precision demands. For completeness, we should also mention a recent paper of Shoham and Gersho [ 131, which deals with the bit allocation problem in the context of vector quantizers. The references mentioned above only deal with the problem of optimally allocating a predetermined channel capacity or total bit rate. The soft bit allocation method described in this paper should be appropriate in many of the scenarios described there and it is equally applicable to the general channel allocation problem, with any coding scheme having a known rate-distortion function. We simply have to assign cost functions to channel capacity usage and to the total distortion incurred in the coding process. In order to readily apply the soft bit allocation process in either speech or image coding examples we still have proto deal with the following problem. The solution vided by (7) is generally not a set of integers as required. Therefore, a certain search process on the integer coordinate lattice points around the optimal solution is required. The search for the optimal set of integers that minimizes the cost functional is an integer programming process. It is, however, expected that choosing the closest integers to the optimal assignments obtained in the process described above will yield a suboptimal solution close to the one that would be obtained via integer programming. We finally note that Medan, in work on subband speech coding [ 111, provides a practical integer quantization level allocation procedure based on a dynamic programming approach. As mentioned above, however, his work only deals with the allocation of a fixed total number of bits or quantization levels.

ST

ACKNOWLEDGMENT Thanks are due to Dr. Y. Medan, of the IBM Scientific Center in Haifa, for useful discussions and the very careful reading of a first draft of this paper, which resulted in several corrections and improvements of presentation. I also wish to thank the reviewers for their comments. REFERENCES [ l ] J. J. Y . Huang and P. M. Schultheiss, “Block quantization IEEE Trans.Commun. relatedGaussianrandomvariables,” vol. CS-11, pp. 289-296, Sept. 1963. [2] L. M. Goodman, “Optimum rate allocation for encoding sets alog messages,” Proc. IEEE, vol. 53, pp. 1776-1777, Nov.

of corSyst.,

of an1965.

[3] -, “Channel encoders,” Proc. IEEE, vol. 55, pp. 127-128, Jan. 1967. [4] P. A. Wintz and A. J. Kurtenbach, “Waveform error control in PCM telemetry,” IEEE Trans.Inform.Theory, vol. IT-14, pp. 650-661, Sept. 1968. [ 5 ] A. Habibi and P. A. Wintz, “Image coding by linear transformation and block quantization,” ZEEE Trans. Commun. Technol., vol. COM19, pp. 50-62, Feb. 1971. 161 A. Segall, “Bit allocation and encoding for vector sources,” IEEE Trans. Inform. Theory, vol. IT-22, pp. 162-169, Mar. 1976. [7] R . Zelinski and P. Noll, “Adaptive transform coding of speech signals,” IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP33, pp. 299-309, Aug. 1977. [8] J. J. Dubnowskiand R. E. Crochiere,“Variableratecoding of speech,” Bell Syst. Tech. J . , vol. 58, no. 3, pp. 577-600, Mar. 1979. 191 J. D. Gibson, “Bounds on the performance and dynamic bit allocation for sub-band coders,” in Proc. 1981 Con$ Acoust., Speech, Signal Processing, Mar. 1981, pp. 836-839. [IO] M. Honda and F. Itakura, “Bit allocation in time and frequency domains for predictive coding of speech,”ZEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-32, pp. 465-473, June 1984. [ 1I] Y.Medan, “An optimal integer allocation scheme for sub-band coding of speech,” IBM Haifa Sci. Center, Rep. 016, June 1985. [I21 J. D. Gibson, “Notes onbitallocationinthetimeandfrequency vol. domains,” IEEE Trans.Acoust.,Speech,SignalProcessing, ASSP-33, pp. 1609-1610, Dec. 1985. [13] Y. Shoham and A. Gersho, “Efficient codebook allocation for an arbitrary set of vector quantizers,” in Proc. ICASSP, Tampa, FL, Mar. 1986, pp. 1696-1699. [ 141 D. Bertsekas, ConstrainedOptimizationandLagrangeMultiplier Methods. New York:Academic. 1982.

Alfred M. Bruckstein was born in Sighet, Transylvania,onJanuary 24, 1954. Hereceivedthe B.Sc. andM.Sc.degreesinelectricalengineering, from the Technion-Israel Institute of Tech1977 and 1980, respectively, nology,Haifa,in and thePh.D. degree in electricalengineering from Stanford University, Stanford, CA, in 1984. Since October 1984 he has been with the Faculty of Electrical Engineering at the Technion. His research interests are in estimation theory, signal and image processing, algorithmic aspects of inverse scattering, point processes, and mathematical models in neurophysiology. Dr. Bruckstein is a member of SIAM and Sigma Xi.