spatio-temporal video error concealment using priority ... - Microsoft

Spatio-tem poralV ideo ErrorConcealm entusing Priority-ranked Region-m atching Y an Chen*1,X iaoyan Sun2,Feng W u2,ZhengkaiLiu1 and Shipeng Li2 1

U niversity ofScience and Technology ofChina,H efei230027 2 M icrosoftResearch A sia,Beijing 100080

Abstract— W hen transm itted over error-prone netw orks, com pressed video sequences m ay be received w ith errors.In this paper w e propose a priority-ranked region-m atching algorithm to recover the “lost” area ofthe decoded fram es,in w hich both tem poral and spatial correlations of the video sequence are exploited. In the proposed schem e, w e first calculate the priorities ofalledge pixels ofthe “lost” area and generate a priority-ranked region group. Then according to their priorities,the regions in the group w illsearch their best m atching regions tem porally and spatially.Finally,the “lost” area is recovered progressively by the corresponding pixels in the m atching regions. Experim ental results show that the proposed schem e achieves higher PSN R as w ell as better video quality in com parison w ith the m ethod adopted in H .264. Keywords-error concealm ent; region m atching; video concealm ent;block m atching;spatio-tem poralconcealm ent

I.

IN TRO D U CTIO N

W ith the explosive growth and great success of the Internet as well as the w ireless network, video services overnarrow band netw orks such as internetand m obile are becom ing m ore and m ore popular. H owever, w hen transm itted over error-prone netw orks, video sequences m ay be received with “lost” m acroblocks (M Bs)caused by cellm issing and channelerrors due to w hich the quality of the obtained video declines.Especially,com pressed video stream s are extrem ely vulnerable to transm ission errors because ofthe use ofpredictive coding and variable length coding (V LC) entropy coding schem es. In case of utilization of spatio-tem poral prediction, one error not only corrupts the current decoded fram e but also m ay propagate to succeeding fram es. M any error concealm entm ethods have been proposed to recover the artifacts caused by transm ission errors at decoder by m aking use of inherent correlations am ong spatially and/or tem porally adjacent blocks [1][2][4]. M axim ally sm ooth recovery is a typical spatial approach that exploits the neighboring inform ation to recover the “lost” M Bs by utilizing the sm oothness property of im age [1][3]. M oreover, content adaptable m ethod is also proposed for error concealm ent [5]. O ne the other hand, tem poral schem es are preferred for video signals to replace the dam aged M Bs by using tem poral correlations between sequential fram es. M any m ethods have been

presented to replace the “lost” M Bs by the m otion com pensated M Bs in the previous fram e w ith zero m otion vector (M V ), average or m edian M V of the spatially adjacent blocks [6]. Block m atching and neighbor m atching principles are often utilized in tem poral error concealm ent[7][8].A well-known algorithm is boundarym atching-based m otion vector recovery,w hich is adopted in H .264 and described in detailin [9].Later,Zheng etal proposed to recover the lost m otion vectors by Lagrange interpolation form ula [10]. M ore recently, hybrid algorithm s have been proposed to take advantages of both spatial and tem poral correlations to better recover the “lost” M Bs. In [11], a concealm ent schem e is presented using a tem poral replacem ent of the lost block through block m atching m ethod follow ed by a m esh-based transform ation to fill the block content with the correctly received surrounding area. Later, a spatio-tem poral interpolation for error concealm entis proposed in w hich the edges interrupted by the “lost” area are m apped to the reference fram e by contour m atching together with the snake-based edge m apping [12]. In this paper, a novel error concealm ent m ethod is proposed in which both spatial and tem poral correlations are exploited based on the priority-ranked region-m atching algorithm .The region-filling algorithm proposed in [13]is utilized in the proposed schem e to select a series of key points from the edges of the “lost” area and correspondingly results in a series of key regions centered atthese points.Then,the bestm atching regions ofthe key regions are searched both in current fram e and previous fram e. The pixels to be concealed in the key regions are recovered with the corresponding pixels inside those m atching regions.Finally,we update the edge and repeat the aforem entioned processes until all pixels in the “lost” M Bs are concealed. Experim ental results show that the proposed schem e can recover m ore details and leave less block artifacts com pared with the error concealm ent m ethod used in H .264. The rest of this paper is organized as follow s. The proposed algorithm is described in detail in Section 2. Section 3 shows the experim ental results.Finally,section 4 concludes this paper.

*This w ork has been done while the authoris w ith M icrosoftResearch A sia.

0-7803-9134-9/05/$20.00 ©2005 IEEE

II-1050

II. C O N CEA LM EN T A LG O RITH M B ASED O N PRIO RITY R A N K ED R EGIO N -M ATCH IN G In this paper,w e assum e thatthe “lost” M Bs have been detected and the corresponding M B-based status m ap of each fram e is available for decoding. A ccording to the status m ap, all correctly received M Bs are decoded first and then the “lost” M Bs are concealed by the proposed error concealm ent algorithm . D ifferent from com m on blocking-m atching-based error concealm ent m ethods, the error concealm ent schem e present in this paper enables m ultiple regions m apping and partialM B recovery atone concealm ent step. Thus, by fully m aking use of the correlations both inside fram e and betw een fram es, each “lost” M B will be filled partially and priorityprogressively in the proposed schem e. It is introduced in detailin the following subsection. A. The Priorities ofBoundary Pixels In the proposed error concealm ent schem e, the “lost” M Bs are restored by first reconstructing the significant pixels and then the others.W hethera pixelis significantor not is determ ined by the priority algorithm presented in [13] for region-filling.G iven an error M B,the priority of each edge pixelis com puted.In detail,for one edge pixel p,a NxN region RP is firstselected centered atpixelP(x,y). Then the priority of P(x,y) is calculated considering tw o aspects,the percentage ofthe “know n” pixels in region RP and the gradient and norm al of the edge line. Since the pixelP(x,y)belongs to the edge ofthe “lost” M B,itis hard to calculate its gradient ∇P(x,y) directly. Thus, in our m ethod, w e use the m axim um one of its 8-adjacent gradients to approach the gradient∇P(x,y)of P(x,y),thatis ∇ P ( x,y) = m ax( ∇ P ( m ,n) ) ,

(1)

w here x-1 ≤ m ≤ x+ 1,y-1 ≤ n ≤ y+ 1.

side effectof the false recovered pixels m ay propagate to other pixels.Itis apparentthatthe filling order w illplay a very im portantrole in the concealm ent.

Fig.1 “lost” M Bs in one fram e

Fig.1 shows a fram e w ith som e “lost” M Bs denoted by grey blocks.The dark line tells the edge ofthe “lost” area. G iven the priorities of the boundary pixels, a straightforward m ethod is to reconstruct the “lost” area once a region according to priority order of the boundary pixels. H owever, this m ethod is quite sensitive to false recovered regions. In the proposed schem e, w e w ill rank the priorities ofthe pixels and selectm ultiple regions to be concealed sim ultaneously. The rule of the regions selection is described as follow s. First,the priorities ofalln pixels belonging to the edge are calculated and a series of priorities (P0,P1,… ,Pn-1) is generated. Then, the series of priorities is rearranged in descending orderofm agnitude based on which a subsetof region priorities (R0, R1,… ,Rk-1) is presented with the follow ing two term s: 1) Each tw o pixels, Ri and Rj, can not be in a sam e NxN region,where 0R1>… >Rm -1,where Rm -1≥ h1.Secondly,the average value of the new priority group,

II-1051

C.Ranked-Priority Region M atching For each NxN region in the ranked-priority region group,the region m atching is perform ed both tem porally and spatially. In other w ords, the m ost sim ilar region is searched from currentfram e and the previous fram e.H ere the m ostsim ilarregion is defined as follows: )

ψ (q ) =

arg m in

D (ψ (q),ψ (p )) .

(4)

ψ (q )∈ϕ

H ere, ψ(p) indicates the region to be m atched, ψ(q) ) represents the candidate m atching region, ψ (q) is the resulted m atching region of ψ(q) and φ is the searching range.In form ula (4), D (ψ (q),ψ (p)) is defined as the sum of square differences of the “know n” pixels in the tw o regions.N otice thatallpixels should be “know n” pixels in case ofthe spatialcandidate regions. D .Region recovery H aving found the m ostsim ilar region,the “lost” pixels ofeach m atched region ψ(p)are recovered w ith the pixels ) insideψ (q) atcorresponding positions.A ll“known” pixels (including the previously filled ones)are unchanged.

A s shown in Fig.2,the PSN R of each recovered fram e of the proposed schem e and the H .264 m ethod are presented. O ur schem e can always provide higher PSN R com pared with the H .264 m ethod. The visual quality of the reconstructed fram es is also evaluated,as show n in Fig.3 and Fig.4,w here (a)and (b) are the undam aged and dam aged video fram es, respectively. Fig. 3-(c) and Fig. 4- (c) show the im ages reconstructed using w eighted pixel value averaging for intra pictures [2] and boundary-m atching-based m otion vector recovery for inter pictures [8] respectively, w hich are adopted in H .264.Fig.3- (d) and Fig.4- (d) show the results generated by the proposed algorithm . It can be seen that the concealed fram es provided by the proposed schem e are m ore sm ooth and with m ore details and less artifacts. The visual quality of the proposed schem e is m uch better than that of the schem e adopted in H .264.

Forem an C IF 38 36 34 32 PSNR

h2,is utilized to generate the ranked-priority region group G (R)with priorities R0 >R1>… >Rs-1,w here Rs-1 ≥ h2.

30 28

E.U pdate and Repeat A fter allthe priority-ranked regions are recovered,the “lost” area and its edge are updated. Then all the four steps m entioned above are repeated until the entire “lost” area is concealed.

26

P roposed H .264 E rror Free

24

Fram e N um .

22 1

3

5

7

9

11

13

15

17

19

21

23

25

27

29

31

33

35

37

Fig.2 “Forem an” sequence PSN R perform ance com parison vs. fram e num berat20% loss rate

III. EX PERIM EN TA L R ESU LTS The proposed error concealm ent schem e is evaluated based on the H .264 codec. The perform ance of the proposed m ethod is com pared with the error concealm ent m ethod adopted in H .264 [2][8][9].The JM 8.2 software is used in this experim ent.The sequence Forem an in CIF form atis encoded at30H z. In this experim ent,I fram e is coded every ten fram es and no B fram e is used. The quantization param eteris setto be 28. In the proposed schem e, the search range of region m apping in currentfram e is 64,w hile the search range of tem poralregion m apping is 32. From the third fram e,a num berofM Bs in every fram e of the video sequence are dropped according to the error pattern of 20% loss rate,which are assum ed to be caused by transm ission errors.Then, the corrupted fram e, which is the reference of succeeding fram e, is concealed by the proposed algorithm as w ell as the H .264 m ethod respectively.The experim entalresults are show n in Fig.2, Fig.3 and Fig.4.

IV . C O N CLU SIO N S In this paper, a spatio-tem poral error concealm ent algorithm based on priority-ranked region-m atching is proposed. Rather than recovering the M Bs in “lost” area one by one, the proposed schem e is able to conceal the “lost” M Bs from different directions according to the priorities ofthe edge pixels.Itm eans thatm ultiple regions can be recovered once in the proposed m ethod.Thus,the block artifacts are effectively reduced and m ore boundary inform ation can be restored. M oreover, the region m apping is perform ed both inside current fram e and in previous fram e.Therefore,m ore details and less blurs are presented. Com pared w ith the error concealm ent m ethod adopted in H .264,the sim ulation results dem onstrate that the algorithm we propose can im prove both visualquality and PSN R perform ance ofthe recovered video sequences. Itshould be noted that the proposed algorithm can be used not only for H .264, but also for other video com pression m ethods based on m otion com pensation.

II-1052

In this paper, only current fram e and previous fram e are used in the proposed region-m atching. For better reconstructed results, m ore reference fram es can be utilized to achieve even betterperform ance. R EFEREN CES [1] Y . W ang, Q . Zhu, “Error Control and Concealm ent for V ideo Com m unication:A Review ”,Proceedings ofthe IEEE,vol.86,no. 5,pp.974-997,M ay 1998 [2] P.Salam a, N .B.Shroff, and E.J.D elp, “Error concealm ent in encoded video,” Im age Recovery Techniques for Im age Com pression A pplications.N orw ell,M A:K luwer,1998. [3] Y . W ang, Q . –F. Zhu, and L. Shaw, “M axim ally sm ooth im age recovery in transform coding,” IEEE Trans.Com m un.,vol.41,pp. 1544-1551,O ct.1993. [4] S.C.H sia,“A n edge-oriented spatialinterpolation for consecutive block errorconcealm ent,” IEEE SignalProcessing Letters,vol.11, pp.577-580,June 2004. [5] R.F.Zhang,Y .H .Zhou,X .D .H uang,“Content-A daptive spatial error concealm ent for video com m unication,” IEEE Trans. O n Consum erElectronics,vol.50,pp.335-341,Feb.2004.

(a)

[6] P. H askell and D . M esserschm itt, “Resynchronization of m otion com pensated video affected by A TM cellloss,” Proc.ICA SSP’92, San Francisco,CA ,M ar.1992,pp.III545-548. [7] S.Tsekeridou and I.pitas,“M PEG-2 error concealm ent based on block-m atching principles” IEEE Trans. Circuits and System for V ideo Technology,V ol.10,pp.646-658,June 2000. [8] W . M . Lam , A . R. Reibm an, and B. Liu, “Recovery of lost or erroneously received m otion vectors,” Proc. ICASSP, vol.5, M ar. 1993,pp.417-420. [9] Y .–K .W ang,M .M .H annuksela,V .V arsa,A .H ourunranta,A nd M . G abbouj, “The error concealm ent feature in the H .26L test m odel,” Proc.ICIP,vol.2,pp.729-732,Sept.2002. [10] J.H .Zheng,L.P.Chau,“A tem poralerrorconcealm entalgorithm for H . 264 using Lagrange interpolation,” ISCAS’04, vol. 2, pp. 23-26,M ay 2004. [11] L.A tzori,G.B.D e N atale Francesco,C.Perra,“A spatio-tem poral concealm ent technique using boundary m atching algorithm and m esh-based w arping (BM A-M BW ),” IEEE Trans.O n M ultim edia, vol.3,pp.326-338,Sep.2001. [12] F. D e N atale and A . M atera, “V ideo Error Concealm ent using Spatio-Tem poral Interpolation w ith Snakes.” International Sym posium on Control, Com m unications and Signal Processing, pp.83-86,M arch,2004 [13] A . Crim inisi, P. P´erez, and K . Toyam a, “O bject Rem oval by Exem plar-Based Inpainting,” IEEE Proc. CV PR, 2:721-728,2003

(b)

(c)

(d)

Fig.3 The 30th fram e ofForem an at20% loss rate (a)error-free fram e;(b)error-dam aged fram e;(c)concealed w ith the m ethod adopted in H .264;(d)concealed w ith the proposed algorithm .

(a)

(b)

(c)

(d)

Fig.4 The 3th fram e ofForem an at20% lossrate (a)error-free fram e;(b)error-dam aged fram e;(c)concealed w ith the m ethod adopted in H .264;(d)concealed w ith the proposed algorithm

II-1053