Fast Nearest Neighbor Search of Entropy-Constrained ... - CiteSeerX

1

Fast Nearest Neighbor Search of Entropy-Constrained Vector Quantization

Mary Holland Johnson, Richard E. Ladner, Eve A. Riskin Abstract |

Entropy-constrained vector quantization (ECVQ) [3] offers substantially improved image quality over vector quantization (VQ) at the cost of additional encoding complexity. We extend results in the literature for fast nearest neighbor search of VQ to ECVQ. We use a new, easily computed distance that successfully eliminates most codewords from consideration. Keywords | Entropy-constrained vector quantization, ECVQ, fast full search of ECVQ, fast full search of vector quantization, Voronoi diagram I. Introduction

Full search vector quantization [8] encodes an input vector by choosing its nearest neighbor from a codebook according to a distortion measure such as the Euclidean distance. The nearest neighbor is found by computing the distortion from the input vector to each codeword, which is a computationally intensive operation. Algorithms to reduce search complexity [9], [5] narrow the eld of candidate codewords for which the distortion must be calculated. These techniques have their roots in pattern recognition techniques for nding nearest neighbors [12], [4]. Most fast full search measures in the literature have been limited to metrics such as the Euclidean distance. We described extensions of existing fast full search techniques to entropyconstrained vector quantization (ECVQ) in [7]. In this correspondence, we present another fast full search technique for ECVQ. An algorithm for fast nearest neighbor search, presented by Orchard [9] and extended by Huang et al. [5], precomputes and stores the distance between each pair of codewords. Given an input vector V, a current best codeword Cb , and a candidate codeword Cj , if then

jjV ? Cj jj  jjV ? Cb jj;

jjCj ? Cb jj  2jjV ? Cb jj:

Graphically, this constrains the search to a hypersphere centered on the current best codeword, with a radius of twice the smallest distortion calculated so far.

This work was supported by The Washington Technology Center, an NSF Young Investigator Award, a Sloan Research Fellowship, U. S. Army Research Oce Grant DAAH004-96-1-0255, NSF Grant CCR-9732828, and AT&T Bell Laboratories. This work appeared in part in the 1996 Proceedings of ICIP. Mary H. Johnson was with the Department of Electrical Engineering, Box 352500, University of Washington, Seattle, WA 98195-2500. She is now with the Naval Undersea Warfare Command, Newport, RI. Richard E. Ladner is with the Department of Computer Science and Engineering, Box 352500, University of Washington, Seattle, WA 98195-2350. Eve A. Riskin is with the Department of Electrical Engineering, Box 352500, University of Washington, Seattle, WA 98195-2500.

In our work, we generalize this fast full search technique to a class of vector quantizers that use a Lagrangian distortion measure. This measure sums the squared error and some constant assigned to each codeword. For ECVQ [3], a variable rate algorithm, the distortion measure trades o squared error with the codeword rate: J (V; Ci ) = jjV ? Ci jj2 + Ri :

(1)

Here, V is the input vector, Ci is the ith codeword in the codebook, and Ri is the cost in bits of sending the label for codeword Ci . We could use other distortion measures such as jjV ? Ci jj + Ri ; (2) but this would require taking a square root as part of the search process. This was examined in [6] and gives comparable results to using the measure in Equation 1. Other nearest neighbor techniques have been proposed based on the examination of the boundaries between codewords [2], [1], [10]. These techniques can also be generalized for use with a Lagrangian distortion measure. II. A Boundary-Based Approach

Putting a spherical constraint on the Lagrangian distortion measure J (V; Ci ) is not obvious. In [7], we formulated constraints directly analogous to the spherical constraint of [9]. Here we use a Voronoi diagram as a visualization tool to describe a fast algorithm that does not require taking square roots. The Voronoi diagram, as depicted in Figure 1, shows the boundaries of each VQ codeword cell. Any vector falling inside the Voronoi cell drawn around a codeword is closest to that codeword. In this small Voronoi diagram, all codewords have adjacent mean-squared error (MSE) boundaries. While this would not be the case for larger codebooks, because we consider just two codewords at a time, only the boundary between any two codewords is used. The following discussion uses this small Voronoi diagram to compare the decision boundary computed by the MSE with the boundary computed by the Lagrangian distortion measure. When we use the Euclidean distortion, jjV ? Ci jj, or equivalently, squared error, jjV ? Ci jj2 , to choose the closest codeword, the cells appear as shown by the thin lines in Figure 1. The line segments depicting the boundaries between codewords are segments of the perpendicular bisector of the line segment between two codewords. In N -space, the boundary is the plane perpendicular to and bisecting the line segment between the two codewords. Finding a

2

nearest neighbor using these codeword boundaries is equivalent to the spherical constraint described by Orchard [9]. We can calculate a similar set of boundaries for ECVQ codewords. The ECVQ boundary between codewords Ci and Cj is de ned to be the plane of equal distortion described by: J (V; Ci ) = J (V; Cj ) (3) or jjV ? Ci jj2 + Ri = jjV ? Cj jj2 + Rj : (4) We restate Equation 4 as the more familiar equation of a plane n
V = d [11]: 2 2 (Cj ? Ci )
V = jjCj jj ? jjCi jj2 + (Rj ? Ri ) ;

where and

(5)

n = C j ? Ci d=

jjCj jj2 ? jjCi jj2 + (Rj ? Ri ) :

2 The plane (Cj ? Ci )
V is perpendicular to the vector n from Ci to Cj and, for  = 0, will bisect the line segment connecting Ci and Cj , as expected. For Ci and Cj that share a boundary, because n is perpendicular to the boundary plane (see Figure 1), the shortest distance from Ci to the ECVQ boundary between Ci and Cj , B (Ci ; Cj ), will be found along this vector for all values of : jdj ? n
Ci : B (Ci ; Cj ) = (6) jjnjj

From Equation 5, we obtain:

distortion between them. As shown in Figure 2, any codeword Cj for which J (V; Cj ) < J (V; CI ), must satisfy the condition B (CI ; Cj )2 < jjV ? CI jj2 : (8) Therefore, to implement our fast full search, we need only compute the Lagrangian distortion for the codewords satisfying this condition. III. Implementation

We calculate our boundary condition, B (Ci ; Cj )2 , for each pair of codewords, Ci and Cj ahead of time. A codebook of size K requires a matrix of size K 2. Each row of the matrix is sorted in increasing order for ease of searching. The matrix computation is not insigni cant, but is easily precomputed and stored. To encode, we calculate the squared error from an input vector V to an initial codeword CI , and the Lagrangian distortion J (V; CI ). We can choose CI because it was the best codeword for a neighboring vector or by doing a binary search of the codeword norms. We then calculate the distortion J (V; Cj ) for each vector Cj that ful lls the boundary condition of Equation 8 in increasing order. If J (V; Cj ) < J (V; CI ), we have found a better codeword. We can then continue to compare codewords against the boundaries of our original guess, CI , or switch to compare against the better codeword Cj . In our implementation, we switch codewords and keep track of distortions already computed. We found that switching to a better codeword decreased the average number of codewords searched by a factor of four. IV. Experimental Results

We trained our size 512 ECVQ codebook on a set of (7) ve images from the USC database using 4  4 vectors. We tested our fast ECVQ search algorithms on the image Equation 7 expresses the shortest distance to the bound- \Lenna," also from the USC database. For a size 512 ECVQ codebook, with  = 1, our fast algoary in terms of the Euclidean distance between the two rithm searched an average of only 21.8 codewords, meaning codewords and the dierence in rate terms. Note that the that 95.7% of codewords were eliminated from consideradistance is not symmetric (B (Ci ; Cj ) 6= B (Cj ; Ci )). tion. On a Hewlett-Packard 755 workstation, the average The ECVQ boundaries as shown in Figure 1 will not be run time was 1.7 seconds compared to approximately 31.2 equidistant from the two codewords, but will be parallel to seconds for exhaustive search. Thus, our method provides the MSE boundary and oset by an amount dependent on speedup of almost 20:1 for this example. the dierence in rate between the two. The boundary will For larger values of , we get codebook reduction because move in the direction of the higher rate codeword because some codewords are never used. As an example, for  = of the second term in Equation 7. 2666, our eective codebook size is 290. In this case, we are When two codewords have very dierent rates, the difable to eliminate 99% of the codewords from consideration ferences in the rate terms may be large enough to make the distance to the boundary negative. That is, instead of (an average of 2.8 codewords are searched). The percent a boundary between the two codewords, we have a bound- eliminated is larger because the boundary value B (Ci ; Cj ) ary on the far side of one of the codewords. We can easily assigned to more probable codewords will be larger and the check for this condition when evaluating the boundaries condition in Equation 8 is satis ed by fewer codewords. As  varies from 1 to very large values, the average run time and eliminate the appropriate codeword. Now that we can evaluate the boundaries between ECVQ did not vary by more than 1.1 seconds. codewords, we can identify any codewords that may proV. Conclusions duce a lower distortion. We use B (Ci ; Cj )2 to avoid calculating square roots. Given an input vector V, and an We have presented a fast full search algorithm for ECVQ initial codeword CI , we compute J (V; CI ), the Lagrangian and other Lagrangian distortion measures. The boundary B (Ci ; Cj ) =

jjCj ? Ci jj + (Rj ? Ri ) : 2 2jjCj ? Ci jj

3

condition can be precomputed and compared directly with the squared error between the input vector and any codeword. Calculations are simple and require no square roots. Experimental results show that this technique eliminates over 95% of the codewords from consideration and produces a 20:1 speedup in calculation time. VI. Acknowledgement

The authors wish to thank Agnieszka Miguel for help with the nal draft of the paper. References [1] D. Cheng and A. Gersho. A fast codebook search algorithm for nearest neighbor pattern matching. In Proceedings of ICASSP, pages 265{268. IEEE Acoustics Speech and Signal Processing Society, 1986. [2] D. Cheng, A. Gersho, B. Ramamurthi, and Y. Shoham. Fast search algorithms for vector quantization and pattern matching. In Proceedings of ICASSP, pages 9.11.1{9.11.4. IEEE, 1984. [3] P. A. Chou, T. Lookabaugh, and R. M. Gray. Entropy-constrained vector quantization. IEEE Transactions on Acoustics, Speech and Signal Processing, 37(1):31{42, January 1989. [4] C. D. Feustel and L. G. Shapiro. The nearest neighbor problem in an abstract metric space. Pattern Recognition Letters, 1:125{128, December 1982. [5] C. M. Huang, Q. Bi, G. S. Stiles, and R. W. Harris. Fast full search equivalent encoding algorithms for image compression using vector quantization. IEEE Transactions on Image Processing, 1(3):413{416, July 1992. [6] M. H. Johnson. Data Compression for Low Bit Rate Transmission of Marine Imagery. PhD thesis, University of Washington, Seattle, WA, June 1999. [7] M. H. Johnson, E. A. Riskin, and R. L. Ladner. Fast nearest neighbor search for ECVQ and other modi ed distortion measures. In Proceedings of ICIP-96, volume III, pages 423{426. IEEE, September 1996. [8] Y. Linde, A. Buzo, and R. M. Gray. An algorithm for vector quantizer design. IEEE Transactions on Communications, 28:84{ 95, January 1980. [9] M. Orchard. A fast nearest-neighbor search algorithm. In Proceedings of ICASSP, pages 2297{2300. IEEE Acoustics Speech and Signal Processing Society, 1991. [10] M. R. Soleymani and S. D. Morgera. An ecient nearest neighbor search method. IEEE Transactions on Communications, COM-35(6):677{679, June 1987. [11] G. Strang. Introduction to Linear Algebra. Wellesley-Cambridge Press, 1998. [12] E. Vidal. An algorithm for nding nearest neighbors in (approximately) constant average time. Pattern Recognition Letters, 4:145{157, July 1986.

4

C0 n MSE Boundaries C1 B(C ,C ) 1

0

C

2

ECVQ Boundaries

Fig. 1. Voronoi diagram of a two-dimensional, size three codebook. In this example, 0 1 2 The thin lines are the mean squared error boundaries. The dark lines are the ECVQ boundaries. Note how the ECVQ boundaries have shifted towards the higher rate codewords. The normal vector n is drawn for the boundary between C0 and C1 . R

< R

< R :

C0 V

C1

1111111111111111 0000000000000000 00000000000 11111111111 - C 0|| || V 11111111111 0000000000000000 1111111111111111 00000000000 0000000000000000 1111111111111111 00000000000 11111111111 0000000000000000 1111111111111111 00000000000 11111111111 0 1 0000000000000000 1111111111111111 00000000000 11111111111 0 1 0000000000000000 1111111111111111 00000000000 11111111111 0 1 00000000000 11111111111 00000000000 11111111111 00000000000 11111111111 ) 11111111111 00000000000 00000000000 11111111111 ,C 1 00000000000 11111111111 00000000000 11111111111 (C 0 00000000000 11111111111

B

C

2

ECVQ Boundaries

Fig. 2. Input vector V is closest to C1 in a Lagrangian sense. Assume we initially have calculated jjV ? C0 jj. Because (C0 C1 )  jjV ? C0jj, we must check codeword C1, and will nd that (V C1 ) (V C0 ). B

J

;

< J

;

;