sMAE: An Improved Block Matching Criterion

sMAE: An Improved Block Matching Criterion V. Fotopoulos1 and A. N. Skodras1,2 1

2

Electronics Laboratory, University of Patras, GR-26110 Patras, Greece Fax: +30 61 997456 Email: [email protected]

Computer Technology Institute, PO box 1122, GR-26110 Patras, Greece Tel.: +30 61 997463 Fax: +30 61 997456 Email: [email protected]

Abstract In this paper a new variation of the well known MAE (Mean Absolute Error) matching criterion for video coding applications, is introduced. The MAE criterion is simpler than the MSE criterion because of the lack of multiplication operations, but it achieves a lower performance in terms of PSNR and MSE. In this work we impose an additional statistical constraint over the MAE thus filling the performance gap between the original MAE and MSE criteria. This is accomplished by multiplying the MAE by the deviation of the mean value of the differences. Simulation results show that the proposed criterion performs always better than the MAE, at the expense of an increased computational complexity.

maximised (or minimised, depending on the criterion). Some of these criteria are well known functions from the field of signal processing, such as the Mean Squared Error (MSE) and the Cross Correlation Function (CCF), while others are relatively new, fitted to the needs of image processing and video coding, such as the Pel Difference Classification (PDC) [10], the Minimum of Maximum Differences (MiniMax) [11] and the Signatures [12]. A brief overview of the matching criteria is given in section 2. The proposed criterion is introduced in section 3 and the simulation results are discussed in section 4.

1. Introduction

The most well known criteria for block matching motion estimation are the following [2,7]:

There is a high temporal correlation between consecutive frames in a video sequence. In order for this temporal redundancy to be reduced or eliminated numerous motion estimation / compensation techniques are used. The overall performance of a real time videocoding system depends highly on the speed and accuracy of the motion estimation scheme. Several motion estimation approaches have been proposed so far in the open literature such as pelrecursive [1], block matching algorithms (BMA) [2-5], frequency domain techniques [6] e.t.c. The BMA techniques have dominated the commercial applications because of their simplicity and regularity, two facts that facilitate their realisation in hardware [79]. A very important part of a BMA is the matching criterion which is a two dimensional function that matches a block from the current frame to another block of the reference frame by searching in a certain area of candidate blocks. We define the matching as the location of the reference frame where the function is

2. Overview of Matching Criteria

• Mean Absolute Error (MAE) or Mean Absolute Difference (MAD) defined as:

MAE ( i , j ) =

1 N2

N

N

∑ ∑ x t (k , l ) − x t −1 (k + i, l + j ) k =1 l =1

where (Ei , Ej) = (i , j) | min MAE(i , j) • Mean Squared Error (MSE) defined as:

MSE (i , j ) =

1 N N 2 ⋅ ∑ ∑ [xt (k , l ) − x t −1 (k + i , l + j )] N 2 k =1 l =1

where (Ei , Ej) = (i , j) | min MSE(i , j) • Cross Correlation Function (CCF) defined as: N N

CCF(i, j ) =

∑∑ xt (k, l )⋅ xt−1 (k + i, l + j ) k =1 l =1

1

 N N x 2 (k, l ) 2  N N x 2 (k i, l + + t t −1 ∑∑  ⋅ ∑∑ k =1 l =1 k =1 l =1 where (Ei , Ej) = (i , j) | max CCF(i , j)

1

2 j ) 

• Pel Difference Classification (PDC) or Maximum Pel Count (MPC) defined as: ( T k , l , i , j ) = 1 , if

1 19 19 1 Matrix1

x t (k , l ) − x t − 1 (k + i , l + j ) ≤ Threshold Value

(i , j ) = ∑ ∑ T (k , l , i , j ) N

PDC

N

k =1 l = 1

where (Ei , Ej) = (i , j) | max PDC(i , j) • Minimum of Maximum Differences (MiniMax)

MiniMax (i , j ) = x t (k , l ) − x t −1 (k + i , l + j ) max

where (Ei , Ej) = (i , j) | min MiniMax(i , j) • Signatures, which are the same as the MAE criterion, the difference being that only groups of pixels that match a specified pattern are used in the calculations. In all the above definitions Ei, Ej are the motion vector components and i,j∈[-p, p], where p is the dimension of the search window, usually being equal to 7 or 15 [2]. Each of those criteria has certain advantages and disadvantages. So MSE is optimum in terms of PSNR and MSE, closely followed by the CCF and MAE criteria. The others like the PDC, the MiniMax and the Signatures are not that good from the PSNR and MSE points of view but they are much faster calculated. This is better illustrated in Tables 1 and 2. Table 1 gives the comparative PSNR results for some known video sequences with frequencies of 10 and 12 Hz. Table 2 summarises the number of operations for each criterion thus providing a measure of speed. In all cases the motion estimation has been carried out only between original frames, not reconstructed, to ensure a fair comparison for all criteria (i.e. all having the same input).

3. The sMAE Criterion It is seen from Table 1 that the MSE criterion gives the best performance from the PSNR point of view, followed by the MAE criterion. It is also seen that there is space for improvement between these two criteria. What we would like to do is to introduce a matching criterion achieving a performance between these two, perhaps with more computations than MAE but still without the multiplication operations that MSE involves. What we thought of is to maintain the MAE criterion but also add a statistical constraint. In this way our criterion will never perform below the MAE. An example of the need for this constraint can be realised by examining the following matrices.

12 12 12 12 Matrix2

Let us suppose that these two matrices contain the absolute differences of the current and the reference blocks of size 2 by 2 at two candidate positions. The MAE for the first matrix is 10 and the MSE for the same matrix is 181. For the second matrix we have a MAE of 12 and a MSE of 144. So, if MAE were used as the matching criterion the first candidate would have been selected, even though it gives a really bad MSE, meaning also a worst PSNR. On the other hand, the second matrix would be a better choice but can we select it without using the computationally intensive MSE criterion? And this is not the only case. How do we choose between candidates of the same MAE values? Which one is the best? We define the mean deviation according to the following formula:

s (i , j ) = with

N −1 N −1

∑ ∑ (diff (k , l , i , j ) − MAE (i , j )) diff ( k , l , i, j ) = xt (k , l ) − x t −1 (k + i , l + j ) k =0 l =0

and k, l, i, j∈[-p, p]. Then we can define a new criterion, that we call 8MAE, as follows:

sMAE = s ⋅ MAE where s=s(i,j) This new version of the MAE criterion still calculates the MAE value but it also combines it with the 8 parameter which is a measure of smoothness. In this way, smoother candidates are selected in every case resulting in better measurements and higher quality of the output sequence. At the same time less significant losses at the DCT stage are introduced due to the lower frequencies of the resulting difference matrices. The block diagram for the calculation of the proposed sMAE matching criterion is given in Fig. 1. The steps for this calculation are summarised below: 1. Subtract the candidate block from the searching block and form the matrix of differences 2. Calculate the MAE criterion for the candidate block, based on the matrix of differences 3. Subtract the MAE from each element of the matrix of differences and calculate the 8 parameter 4. Output the product of the square root of 8 times the MAE.

4. Results and Conclusions We have tested the proposed matching criterion on various video sequences. The comparative results between this and the existing ones are given in Tables 3 and 4. Table 3 summarises the results concerning the quality performances of the criteria in terms of the PSNR achieved, while Table 4 depicts their compression efficiency. The input conditions were the same and Fram e t-1

Fram e t

C an didate Blocks

Search in g Block

sequence an improvement of 0.17dB has been achieved. For the rest of the video sequences the results weren’t so impressive because MAE and MSE performed very closely. The important thing to notice is that there always has been an improvement, because we actually have succeeded in refining the MAE criterion by the 8 factor and in removing the ambiguity for those candidates having the same MAE. However, the drawback of the algorithm is that it requires twice the number of operations that MAE does. More specifically, while MAE needs 2N2 additions / subtractions and N2 comparisons / absolute values for its calculation (N being the dimension of the rectangular blocks), sMAE requires 4N2 additions / subtractions and 2N2 comparisons / absolute values. Further research on this is being conducted aiming at reducing this computational complexity. REFERENCES

-

M atrix of D ifferen ces

MAE C alculation

___ √ s C alculation 8M A E

x

output

Fig. 1- Diagram of the steps for the sMAE calculation

the 8MAE criterion lies in between the MAE and MSE. The relative improvement (R.I.) defined as

R .I . =

PSNR PSNR

sMAE MSE

− PSNR − PSNR

MAE

⋅ 100

MAE

is also included in Table 3. It can be seen that a significant filling of the gap between the MAE and MSE performance has been accomplished. There is also a slight improvement in the size of the output streams, except for the case of the “Silence” sequence, which however achieves the highest improvement in quality (0.17 dB). The PSNR results for the “Silence” sequence are shown in Figure 2. It is observed that the 8MAE curve is always above that of MAE. It is clearly shown that there has been a significant improvement in all cases that the difference between MAE and MSE is considerable. Thus in “Silence”

[1]. G. Tziritas and C. Labit: “Motion Analysis for Image Sequence Coding”, Elsevier Science B.V., 1994 [2]. V. Bhaskaran and K. Konstantinides: “Image and Video Compression Standards: Algorithms and Architectures”, Second Edition, Kluwer Academic Publishers, 1997 [3]. R. Li, B. Zeng and M. L. Liou: “A New Three-Step Search Algorithm for Block Motion Estimation”, IEEE Trans. on Circuits and Systems for Video Technology, vol.4, no.4, August 1994, pp. 438-442 [4]. M. Ghanbari: “The Cross-Search Algorithm for Motion Estimation”, IEEE Trans. on Communications, vol.38, no.7, July 1990, pp. 950953 [5]. B. Liu and A. Zaccarin: “New Fast Algorithms for the Estimation of Block Motion Vectors”, IEEE Trans. on Circuits and Systems for Video Technology, vol.3, no.2, April 1993, pp. 148-157 [6]. R.W. Young and N.G. Kingsbury: “FrequencyDomain Motion Estimation Using a Complex Lapped Transform”, IEEE Trans. on Image Processing, vol.2, no.1, January 1993, pp. 2-17 [7]. A.M. Tekalp: “Digital Video Processing”, Prentice Hall PTR, 1995 [8]. ITU-T DRAFT H.263, Standardization Sector of ITU, July 1995 [9]. T.Sikora:“MPEG Digital Video-Coding Standards”, IEEE Signal Processing Magazine, vol. 14, no.5, September 1997, pp. 82-100 [10].H. Gharavi & M. Mills: “Block Matching Motion Estimation Algorithms – New Results”, IEEE Trans. on Circuits and Systems for Video Technology, vol.37, no.5, May 1990, pp. 649-651

[11].M.J. Chen, L.G. Chen, T.D. Chiueh and Y.P. Lee: “A new block matching criterion for motion estimation and its implementation”, IEEE Trans. on Circuits and Systems for Video Technology, vol.5, no.3, June 1995, pp. 231-236 [12].Y. Wong: “An efficient heuristic-based motion estimation algorithm”, Proc. of the 1995 IEEE Int. Conf. on Image Processing (ICIP-95), pp. 205-208. Sequence Container Foreman News Silence

ACKNOWLEDGEMENTS The joint support of the Greek Secretariat for Research and Technology (GSRT) and of the British Council is gratefully acknowledged.

Table 1 - Mean PSNR measurements MAE MSE CCF PDC MiniMax 33.383 33.423 33.422 33.129 32.922 24.241 24.377 23.690 22.839 23.348 28.965 29.127 28.890 27.614 27.595 28.988 29.251 28.486 27.614 27.995

Signature 32.929 23.501 28.546 28.463

Table 2 Number and types of operations per block of size N x N Additions Multiplication Comparisons Increments Criterion Subtractions Division Absolute Values Decrements MAE 2N2 N2 2 2 MSE 2N N 3N2 CCF 3N2 2 N2 PDC N ≤ N2 2 2 MiniMax N N 16 Signature N2+16 Table 3 - Mean PSNR measurements Sequence MAE 8MAE MSE R.I. (%) Container 33.383 33.398 33.423 37.5 Foreman 24.241 24.290 24.377 36.0 News 28.965 29.016 29.127 31.5 Silence 28.988 29.156 29.251 63.9

Table 4 - Output stream sizes in bits Sequence MAE 8MAE Container 663468 662866 Foreman 1104916 1101417 News 763841 763365 Silence 773451 774421

 0$( V0$(



    


  

Figure 2 - PSNR measurements for the “Silence” sequence