Rate Control for Low Bit Rate Video via Enhanced ...

Rate Control for Low Bit Rate Video via Enhanced Frame Skipping Jae-Young Pyun, Yoon Kim, and Sung-Jea Ko Department of Electronics Engineering, Korea University 5-1 Anam-Dong, Sungbuk-ku, Seoul 136-701, Korea {jypyun, yooni, sjko}@dali.korea.ac.kr

Abstract. A scene-adaptive rate control algorithm with an enhanced frame skipping is proposed for low bit rate video coding. Most existing rate control algorithms for low bit rate video focus on bit allocation at the macroblock level under a bit rate constraint. The proposed rate control method looks at bit allocation from the frame level perspective and provides an effective bit allocation to reduce the temporal visual fluctuation of image quality such as flickering and motion jerkiness. It is demonstrated that the new rate control algorithm allocates target bits according to picture coding complexities and alleviates the problem of the frame skipping occurred by abrupt scene changes. Experimental results indicate that the proposed control method provides an effective alternative to existing frame-layer rate control for non-interactive streaming applications such as videocasting, video-monitoring, and video-based entertainment services.

1

Introduction

The quality of service (QoS) becomes a challenging task when we have to cope with transmission along channels with low bit rate. In H.263 encoding, rate control is a big issue to improve the video quality because the bit allocation among pictures greatly affects the picture quality. The reference rate control method known as TMN8 performs bit allocation for each frame according to the coding result of the previous frame [1,2]. The TMN8 rate control scheme starts from the assumption that rate control method should encode a video frame with the frame target rate. But the frame target rate can not be sufficient to encode a frame when the sequence characteristics are changed significantly. When scene changes abruptly, since the statistical characteristics of the current picture are very different from the previous, statistical information of the previous picture cannot be used for motion estimation of the future frame. To prevent the encoded bit rates from exceeding the maximum buffer size, the encoder must cease generating bits by skipping frames thereby causing an interruption to the smoothness of the video. To reduce the annoying picture degradation, we need a scene-adaptive bit allocation method for the frame-layer rate control based on rate-distortion (RD) models. The previous bit allocation approaches used the jumping window V. Kumar et al. (Eds.): ICCSA 2003, LNCS 2667, pp. 302–311, 2003. c Springer-Verlag Berlin Heidelberg 2003

Rate Control for Low Bit Rate Video via Enhanced Frame Skipping

303

method for the pre-analysis of scene characteristics [3,4]. However, the jumping window scheme is very sensitive to the length of the window and the size of encoder buffer. In this paper, we propose a sliding window scheme that moves the window one frame at a time to enhance the scene adaptivity. This scheme can reduce the quality fluctuation without increasing the length of the pre-analysis window. However, when the frame skipping occurs, both jumping and sliding windowbased rate control methods are not effective to handle the scene-changed frame due to the suddenly increased coding bit rate regardless of the target bit rate. To handle the abrupt scene change, we also introduce an enhanced frame skipping (EFS) that can manage the frames to be skipped by the prediction of the frame skipping. This paper is organized as follows. Frame-layer rate control method using the sliding window and EFS methods is described in Sect. 2. Sect. 3 presents experimental results. Conclusions are given in Sect. 4.

Conventional scene-adaptive rate control Modified function for the pre-analysis performance Added procedure for EFS Detected Raw video stream

Pre-analysis buffer (sliding window)

Scene analysis

Analysis results

Detection of abrupt scene change Not detected

Rate-distoriton model Encoding results

Encoded video stream

Encoding video stream

Estimation of generated bit rates

Scene-adaptive bit allocation (Lagrange Mulitplier)

Estimation of # of frames to be skipped

Rearrange frames to be skipped and unskipped

Fig. 1. System block diagram of the proposed frame-layer rate control

2

Scene-Adaptive Frame-Layer Rate Control

A block diagram of the overall video rate control system is shown in Fig. 1. The target bit rate is initially determined based on the scene analysis using the sliding window and optimized using Lagrangian method. For the R-D model estimation and Lagrangian optimization method, we adopt the conventional scene-adaptive rate control proposed in [4]. To handle the motion jerkiness by abrupt scene change, the prediction technique for the frame skipping is utilized.

304

2.1

J.-Y. Pyun, Y. Kim, and S.-J. Ko

Sliding Window for Pre-analysis

The performance of the jumping window method depends on the length of window. As the window length becomes longer, the more scene information is collected. However, increasing the window length causes the additional delay. We propose a sliding window method that can analyze scene characteristics without increasing the window size. Now, we define Si as the amount of bits to be shared among all frames within the sliding window, starting from i to i + NW − 1 . Si is initially determined by Si = k·(R·NW )/F , where R is the bandwidth of the transmission channel, NW is the number of frames in the period of the sliding window, F is the frame rate, and k is a control parameter for rate quantity (k = 0.5 for our experiment). At the encoding time of each frame, the target bit rate of the ith frame RiT is calculated as follows: RiT = (1 − k)

R M AD(Pi−1 , Pi ) + Si · , F M ADsum

(1)

where M AD(·, ·) is the mean of absolute difference between two frames, Pi is the i th frame at the current time instance, and M ADsum is the sum of MAD values of NW frames. However, since the bit allocation method depends on the MAD value, the buffer overflow and underflow might be occurred by various scene characteristics. To avoid this problem, the previous buffer occupancy Wprev is examined and the target bit rate is recalculated as ˆ T = max(min(RT , α·M + R/F − Wprev ), β·M + R/F − Wprev ), R i i

(2)

where M is the default maximum buffer size and α and β are parameters used for the safety margin to avoid the buffer overflow and underflow. In our experiment, α and β are set to 0.9 and 0.05, respectively. The next shared memory Si+1 can be rewritten as ˆ iT + R . Si+1 = Si − R F

(3)

Another consideration is the frame skipping after the abrupt scene change. If the skips of future frames are expected, the control of the bit allocation to the current frame is not helpful to reduce the quality degradation. Thus, the ˆ T = R/F only when the abrupt scene change is allocated bits are fixed to R i detected. 2.2

Lagrangian Optimization Process

For the frame-layer rate control, we employ an empirical data-based frame-layer R-D model using the quadratic rate model and the affine distortion model [5] with respect to the average quantization parameter (QP) in a frame, which is given by qi ) = (a¯ R(¯ qi−1 + b¯ qi−2 ) · M AD(Pî−1 , Pi ), qi ) = a q¯i + b , D(¯

(4) (5)


305

where a, b, a , and b are the model coefficients, Pî−1 is the reconstructed reference frame of ith frame, and q¯i is the average QP of all macroblocks in the ith frame. We consider a new formulation of frame-layer rate control based on the R-D model as follows: Determine q¯i , i = 1, 2, ..., NW to minimize

NW

qi ) D(¯

subject to

i=1

NW

Ri ≤ RNW ,

(6)

i=1

where Ri is the bit rate used for the ith frame and RNW is the sum of initial target bit rates of NW frames (RNW = NW · R/F ). This formulation minimizes the average distortion over the sliding window. The above optimization problem with bit budget constraint can be solved by the Lagrangian method, in which the cost function and constraint are combined by a Lagrange multiplier. Now, we define a cost function for the ith frame by combining qi ) + λi · B ˆ res , qi ) = D(¯ Ji (¯ i îres = B

i−1

˜ qi ) − Rj + R(¯

j=1

(7) i

ˆ jT , R

(8)

j=1

where Ji (¯ qi ) and λi is the cost function and the Lagrange multiplier for the ith frame, respectively. Note that we do not have to encode the current frame to determine the target bit rates of the current frame. Based on the rate and distortion models, we can determine the optimal QP to minimize the above cost function. It was shown qi ) is generally a convex function. Thus, we can obtain the optimal in [9] that Ji (¯ solution by using the gradient method in (7). However, note that what we finally i (¯ qi∗ ) that is the target bit budget for the ith frame. need is not q¯i∗ , but R q¯i∗ = arg min Ji (¯ qi ).

(9)

q¯i

Since the Lagrange multiplier should be adaptively adjusted in real-time rate control, only a sub-optimal solution with a low computational complexity is considered. Thus, we employ the adaptive adjustment rule [4] given by i

λi+1 = λi + ∆λ,

∆λ =

j=1

Rj

i ˆT R j

− 1.

(10)

j=1

Once the bit rate is allocated to the frame using the aforementioned framelayer rate control, existing TMN8 rate control algorithm can be effectively used i (¯ to allocate the bit budget to each macroblock with the solution R qi∗ ).

306

2.3

J.-Y. Pyun, Y. Kim, and S.-J. Ko

Enhanced Frame Skipping Method

To cope with the quality fluctuation and motion jerkiness occurred by scene changes in the frame-layer rate control, we propose an enhanced frame skipping method using the prediction method for the frame skipping. In the proposed method, when scene change is detected, the number of frames to be skipped is predicted and frames are nonconsecutively skipped by controlling the frame encoding time. First, we predict the encoded bit rates of FSC , where FSC is the first frame of a new scene when an abrupt scene change is detected. In (4), we do not have the value of q¯i of the unencoded future ith frame when ith frame is FSC . However, since the allocated bits are not sufficient to encode a frame when the sequence characteristics significantly change, macroblock-layer rate control sets all QP values to 31. Thus, we expect that q¯i is equal to about 31. Also, Pî−1 is simply replaced by Pi−1 due to minor differences to the prediction model when scene change occurs. Therefore, the predicted rates of the future ith frame and the encoder buffer occupancy are respectively given by i = max(R(¯ qi ), R/F ), R

(11)

= Wprev + R i − R/F. W

(12)

and

i and Wprev , where the In (12), the buffer occupancy is predicted by both R unknown value Wprev is set to M/2 for the simplicity of computation. S ) using the preSecond, we define the number of frames to be skipped (N dicted buffer occupancy W and maximum buffer size M after the abrupt scene change as S = W /M . N (13) S frames in the sliding window are skipped in advance. When S ≥ 2, half of N If N prev ) are skipped, the unskipped frames can be encoded at the these frames (N S idle encoding time left by the skipped frames. Then, the number of remaining ¯ next ) is determined after encoding FSC . After encoding frames to be skipped (N S the scene-changed frame FSC , we know the exact number of frames to be skipped prev and N ¯ next are defined as (NS ). Thus, N S S prev = N S /2, N S

(14)

¯ next = max(NS − N prev , 0). N S S

(15)

and

Now, we determine which frames are to be skipped. When ith frame Pi is equal to FSC , the skipped frames are given by Pi−2m and Pi+2n−1 , where m=1,


307

prev and n=1, 2, · · · , N ¯ next . After choosing the skipped frames, we 2, · · · , N S S modify the encoding time of the remaining unskipped frames within the analysis window. The modified encoding time of the unskipped frame is summarized as follows: Ti−2k+1 =

Ti−2k+1 −

1 F

prev , for k=N S

prev –1 Ti−2k−1 + F1 , for k=1,2,· · · ,N S 1 Ti = Ti−1 + , F ¯ next –1 Ti+2k + F1 , for k=N S Ti+2k = ¯ next –2. Ti+2k+2 − F1 , for k=1,2,· · · ,N S

(16)

prev =1 and Fig. 2(b) shows the frames to be skipped and unskipped when N S next ¯ N =2. In Fig. 2(c), the modified encoding time of the unskipped frames is S described. As the 2nd frame P2 is skipped, the 3rd frame P3 is encoded at the idle encoding time of P2 . Also, other unskipped frames P3 , P4 , and P6 are encoded at the idle encoding time. Consequently, the bit rates generated from the unskipped frames are the same as those of the conventional frame skipping as shown in Fig. 2(d). Fig. 2(e) shows that the skipped frames can be easily recovered by the frame interpolation technique at the decoder, since the skipped frames are not in succession. Moreover, since the number of skipped frames after the abrupt scene change is fewer than that of the conventional frame skipping, the proposed method can provide the fast quality restoration.

3

Experimental Results

Our rate control scheme has been implemented in a H.263+ standard codec. The different sequences are used for the experiment. The experiment sequence “Test 1” is a combination of 4 scenes (Miss America, Mother & daughter, News, and Akiyo) and another sequence “Test 2” also consists of different scenes (Bream, Container, Foreman, and Trevor). The input sequences are in QCIF format (176×144), and the frame rate F is 30 fps. First, we present the comparison between the jumping window method and the proposed sliding window method discussed in Sect. 2.1. For our experiments, we choose the target bit rate R equal to 64 Kbps and the window size for pre-analysis set to 12 frames. Table 1 shows that the proposed sliding window method has the similar average PSNR with other methods, but has the smallest standard deviation of PSNR among three schemes. It illustrates that the rate control using sliding window method reduces the PSNR fluctuation by allocating bit rates effectively based on the characteristics of the sequence. Moreover, as the encoder buffer size is larger than M , the standard deviation values at the sliding window method are reduced substantially at the expense of the larger buffer size.

308

J.-Y. Pyun, Y. Kim, and S.-J. Ko Scene change occurred

: Encoded frame : Unencoded frame : Unskipped frame I

P

P

P2

P1

P3

P4

P5

P6

P7

P8

Frame skipping

Frame skipping

Conventional frame skipping

(a) Frames to be skipped

I

P

P

P2

P1

P3

P4

P5

P6

P7

P8

Decision of skipped frames

(b) I

P

P

P1

P3

P6

P4

P8

Modified encoding time

(c) Target bit rates

P1

P3

P6

P4

P8

No change in the transmitted bit rates

(d) I

P

P

P1

P2

P3

P4

P5

P6

P7

P8

Display order (Decoder) Interpolation

(e)

Fig. 2. The enhanced frame skipping method. (a) Frame skipping by the abrupt scene change, (b) New arrangement of frames to be skipped by EFS, (c) Modified encoding times, (d) Transmitted bit rates after encoding frames, (e) Interpolation method to restore the skipped frames at the decoder

Second, we show the PSNR comparison between the conventional frame skipping and the proposed EFS method. For our experiments, we choose the encoder buffer equal to M and the window size set to 12 frames. Fig. 3 shows that the qualities of two schemes are deteriorated by the frame skipping, when scene


309

Table 1. PSNR results of pre-analysis methods Encoder Sequence buffer size Test 1 M 3M Test 2 M 3M

Avg. of PSNR No pre- Jumping Sliding analysis window window 35.575 35.559 35.585 35.622 35.629 35.632 30.763 30.773 30.790 30.830 30.854 30.861

STD. of PSNR No pre- Jumping Sliding analysis window window 3.910 3.887 3.884 3.805 3.765 3.724 3.217 3.215 3.209 2.950 2.877 2.867

Table 2. PSNR results improved by EFS method using MCI Target Sequence rates [Kbps] Test 1 32 64 Test 2 32 64

Avg. of PSNR SW SW+EFS SW+EFS +MCI 29.422 30.502 30.552 33.088 33.502 33.517 26.934 27.537 27.601 30.116 30.324 30.332

STD. of PSNR SW SW+EFS SW+EFS +MCI 4.229 3.464 3.401 4.393 3.867 3.861 3.527 2.470 2.349 2.927 2.708 2.706

changes abruptly. However, EFS can restore the image quality promptly due to the reduced number of skipped frames after the scene change. The statistical results of the experiment are collected from 80 frames surrounding each scenechanged frame. In Table 2, we can observe that the average PSNR is improved significantly while the standard deviation of PSNR is reduced by about 8-32%. Moreover, when EFS is implemented, the frame interpolation technique is helpful to guarantee smooth display at the decoder. Several interpolation methods have been proposed such as frame repetition, motion-compensated interpolation (MCI), and motion adaptive filtering [7]. Finally, we illustrate the effect of EFS by showing the PSNR result of the skipped frames restored using frame interpolation technique in Fig. 4. The results show that the proposed algorithm skips the frames to be recovered easily by using the frame interpolation technique.

4

Conclusions

In this paper, we presented the scene-adaptive frame-layer rate control based on the sliding window and enhanced frame skipping. The two main features of the proposed frame-layer rate control are to reduce the quality fluctuation and enhance the video quality at the expense of tolerable time-delay. Experimental results showed that EFS is 1-2dB better in the period of scene change than

310

J.-Y. Pyun, Y. Kim, and S.-J. Ko 38

Sliding window (SW) SW+EFS

36 34

PSNR[dB]

32 30 28 26 24

Miss America

News

Mother & daughter

22 140

160

180

200

220

240

260

280

Frame

Fig. 3. PSNR comparison between the conventional frame skipping and EFS method in test sequence “Test 1” (R=32 Kbps)

the conventional frame skipping method. The low bit rate control offering QoS guaranty to video transmission over the unreliable transmission channel will be our further research topic.

References 1. ITU-T/SG-15 Video Coding Experts Group, Video codec test model, TMN8, 1997. 2. J. Ribas-Corbera and S. Lei, “Rate control in DCT video coding for low-delay communications,” IEEE Trans. Circuits Syst. Video Technol., vol. 9, no. 1, pp. 172–185, Feb. 1999. 3. Y. Yokoyama and S. Nogaki, “A rate control method with pre-analysis for real-time MPEG-2 video coding,” in Proc. Int. Conf. Image Processing, pp. 514–517, Oct. 2001. 4. H. J. Song and K. M. Lee, “Adaptive rate control algorithms for low-bit-rate video under the networks supporting bandwidth renegotiation,” Signal Process.:Image Commun.(Elsevier Science), vol. 17, no. 10, pp. 759–779, Nov. 2002. 5. T. Chiang and Y. Zhang, “A new rate control scheme using quadratic rate distortion model,” IEEE Trans. Circuits Syst. Video Technol., vol. 7, no. 1, pp. 246–250, Feb. 1997. 6. D. Wu, Y. T. Hou, W. Zhu, H. J. Lee, T. Chiang, Y. Q. Zhang, and H. J. Chao, “On end-to-end architecture for transporting MPEG-4 video over the Internet,” IEEE Trans. Circuits Syst. Video Technol., vol. 10, no. 6, pp. 923–941, Sep. 2000. 7. B. T. Choi, S. H. Lee, and S. J. Ko, “New frame rate up-conversion using bidirectional motion estimation,” IEEE Trans. on Consumer Electron., vol. 46, no. 3, pp. 603–609, Aug. 2000.


311

34

PSNR[dB]

32

30

28

SW SW+EFS SW+EFS+MCI

26

24

News

Mother & daughter 244

246

248

250

252

254

256

Frame

Fig. 4. Quality restoration of the skipped frames recovered by MCI method in test sequence “Test 1”

8. H. J. Song and C. C. J. Kuo, “Rate control for low-bit-rate video via variableencoding frame rates,” IEEE Trans. Circuits Syst. Video Technol., vol. 11, no. 4, pp. 512–521, Apr. 2001. 9. L. J. Lin, A. Ortega, and C. C. J. Kuo, “Rate control using spline interpolated RD characteristics,” in Proc. of SPIE Visual Communication Image Processing, pp. 111–122, Mar. 1996.