AN INTRA-FRAME ERROR CONCEALMENT: NON-LINEAR PATTERN ALIGNMENT AND DIRECTIONAL INTERPOLATION W. Kumwilaisak † and F. Hartung‡ †Telecommunication Department, King Mongkut University of Technology, Thonburi, Bangkok, Thailand ‡ Ericsson Research, Mobility Applications Laboratory, Herzogenrath, Germany E-mail:
[email protected],
[email protected] ABSTRACT This paper presents a new intra-frame error concealment algorithm designed for both images and videos with nonlinear pattern alignment and directional interpolation. By this intra-frame error concealment algorithm, first the nonlinear pattern alignment based on the variant version of dynamic programming algorithm is applied to image pixels around the lost region. The non-linear pattern alignment effectively provides the best corresponding pixel pairs from the boundary pixels of the lost regions to dynamically capture a local image texture in terms of optimization metrics. Then, with derived corresponding pixel pairs, the directional interpolation algorithm is used to recover pixels in the lost region. From simulation in both images and videos, the proposed intra-frame error concealment provides the significant enhancement of the capability in capturing and recovering the lost image texture comparing to the bilinear interpolation approach. 1. INTRODUCTION Recovering corrupted or lost regions of images and videos to maintain acceptable perceptual quality is one of the most challenging tasks in multimedia transmission. When other error resilient techniques such as re-sync markers [1] or reversible variable length code [2] fail to recover the lost data under error-prone environment, an error concealment is often considered as the last approach to maintain a good quality of image and video at the decoder. The error concealment can use image data from both temporal and spatial domain to recover the lost region. The spatial-domain data is often used when there is low correlation among image data in the temporal domain. This may happen when there is no temporal image data available (i.e., only one single image is considered) or the temporal data does not correlate well with the considering image (i.e., scene cut in a video sequence). To effectively conceal the lost region with the spatial-domain data, the texture of the
lost region should be considered. In this paper, we propose a new way to perform error concealment in a spatial domain. The proposed approach is based on non-linear pattern alignment and directional interpolation. First the non-linear pattern alignment based on the variant version of dynamic programming algorithm is applied to image pixels around the lost region. The non-linear pattern alignment effectively provides the best corresponding pixel pairs from the boundary pixels of the lost regions to dynamically capture a local image texture in terms of optimization metrics. Then, with derived corresponding pixel pairs, the directional interpolation algorithm is used to recover pixels in the lost region. The rest of the paper is organized as follows. Section 2 describes the algorithm of non-linear pattern alignment, which searches for optimal matches of the pixels around the lost region. In Section 3, the directional interpolation is presented. Simulation results are given in Section 4. Finally, concluding remarks are given in Section 5. 2. NON-LINEAR PATTERN ALIGNMENT ALGORITHM The non-linear pattern alignment algorithm consists of three steps: initialization, matrix filling, and back tracing. To demonstrate the non-linear pattern alignment algorithm, let consider two pixel sequences representing by the sequence letter as G A A T T C A G T T A and G G A T C G A, respectively. Note that these two pixel sequences can be obtained from the uncorrupted pixels around the lost or corrupted regions and they are not necessary to have the same length. This provides a flexibility in selecting uncorrupted image pixels for error concealment. To optimally match this two letter sequence, the algorithm is stated as follows. 2.1. Initialization First, the scoring matrix with size (NS1 + 1) × (NS2 + 1) is set up (Fig.1A), where NS1 and NS2 are the length of
letter sequence 1 and sequence 2, respectively. From the sequences used for demonstrating the non-linear pattern alignment algorithm, NS1 and NS2 are equal to 11 and 7, respectively. Define Si,j as the score assigned to position (i, j) for declaring match or mismatch of the pixel at position j of sequence 1 and at position i of sequence 2 and w is the gap penalty. This corresponds to the position (i + 1, j + 1) in the scoring matrix (Fig.1A). However, when we describe the algorithm, we will use the position index based on the position of the pixels in the pixel sequence. Then we design the scoring scheme used in the non-linear pattern alignment as follows. Si,j = 1; if the pixel at position i in sequence 1 is matched with the pixel at position j in sequence 2. Si,j = 0; if the pixel at the position i in sequence 1 is mismatched with the pixel at the position j in sequence 2. w = 0; if a gap between pixel is applied in either sequence 1 or sequence 2. The initialization of scoring matrix can be illustrated (Fig.1A). Because the gap penalty is set to be 0, the first row and column of the scoring matrix are filled with 0.
is 1. The rest positions of column 1 and row 1 can be filled by the same techniques. Next, position (2,2) of the scoring matrix is considered (Fig.1C). The score at position (2,2) is the maximum value of 1 (mismatch), 1 (gap in sequence 2), or 1 (gap in sequence 1). Thus, the score at position (2,2) is equal to 1. At position (3,2), the letter at position 2 in sequence 1 and that at position 3 in sequence 2 are the same. Therefore the score is the maximum value of 2 (mismatch), 1 (gap in sequence 1), 1 (gap in sequence 2). Hence, the score at position (3,2) is 2. At position (2,4), the score is the maximum value of 1 (mismatch), 1 (gap in sequence 1), 2 (gap in sequence 2). Thus, the score filled in position (4,2) is 2. For the rest of positions in column 2, except position (7,2), consider similarly as position (4,2) since the letters are not matched. At position (7,2), the letter at position 7 of sequence 2 matches with the letter of sequence 1 at position 2. The score at position (7,2) is the maximum value of 2 (match), 1 (gap in sequence 1), or 1 (gap in sequence 2). Similar approaches described above are used to completely fill the scoring matrix. The completed filled matrix can be shown (Fig.1D).
2.2. Matrix Filling Next, beginning from the upper left hand corner of the matrix, the score filled at position (i, j), Mi,j , where i is a row index and j is a column index, can be computed from the scores at position (i − 1, j − 1), (i, j − 1), and (i − 1, j), and can be expressed as Mi,j = max[Mi−1,j−1 + Si,j , Mi,j−1 +w, Mi−1,j + w].
(1)
From Eq.1 derived above, the score at position (1,1) can be obtained. Given the first letter in sequence 1 and sequence 2 being the same, the score filled at position (1,1) of scoring matrix, M1,1 , can be expressed as M1,1 = max[M0,0 + 1, M1,0 + 0, M0,1 + 0] = max[1, 0, 0] = 1,
(2)
Because the gap penalty (w) is 0, the rest positions of row 1 and column 1 can be filled with score 1 (Fig.1B). For example, at position (1,2), the score is the maximum value of 0 (mismatch) from the diagonal direction, 0 (gap in the sequence 2) from the vertical direction, and 1 (gap in sequence 1) from the horizontal direction. As in the computation of score at position(1,2), the rest of row 1 can be similarly filled until it reaches position (1,8). Now, at position (1,8), the letters at position 1 from sequence 2 and at position 8 from sequence 1 are the same. The score at position (1,8) can be derived from the maximum value of 1 (match) from the diagonal direction, 1 (gap in sequence 2) from the horizontal direction, and 0 (gap in sequence 1) from the vertical direction. Therefore, the score filled at position (1,8)
2.3. Back Tracing From the completely filled matrix, the maximum alignment score located at position (7,11) in the scoring matrix of the two underlying sequences is equal to 6. The back tracing algorithm is applied to determine the optimal alignment solution of the letters in the underlying sequences. Begin at position (7,11) (Fig.2A). The neighboring positions consist of the position to the left (gap in sequence 2), the position along the diagonal (match/mismatch), and the position to the right (gap in sequence 1). The back tracing algorithm determines the path leading to the predecessor position of the current position from the matrix filling step. The predecessor position of position (7,11) is the position from the diagonal direction (i.e., position (6,10)). If there are more than one possible predecessor, any direction can be chosen. From position (6,10)(Fig.2B), the predecessor position of position (6,10) is the position corresponding to the neighboring position to the left of position (6,10), which implies the existence of a gap in sequence 2. Next, at position (6,9) (Fig.2C), the same consideration is applied at position (6,10) and the alignment can be expressed (Fig.2D). The corresponded optimal alignment of letter sequences described in this example can be shown as in Fig. 3. When dealing with the real value of pixels, in which its value can range from 0-255, the threshold is used for matching declaration. In other words, two considering pixels will declare as a match in the algorithm, if the condition in Eq. 3 is fulfilled. |P1 (i) − P2 (j)| ≤ Tth , (3) where P1 (i) is the pixel value at position i from image pixel
(A)
(B)
(C)
(D)
Fig. 1. Matrix initialization and Matrix filling.
(A)
(B)
(C)
(D)
Fig. 2. Back Tracing.
(A)
(B)
Fig. 3. (A) Optimal letter alignment. (B) The corresponding pairs of pixel obtained from the pattern alignment algorithm applied to Lena image.
of aligned pixel sequence obtained from applying the nonlinear pattern alignment to lost region of Lena image can be shown (Fig.3B). With the set of connection line Υ, the recovery of the lost pixels can be considered into two cases depending on the location of the lost pixels. These two cases can be described as follows. First, the position of the lost pixel is on the connection line li . Hence the recovered pixel could be simply obtained by L1 =
sequence 1, P2 (j) is the pixel value at position j from image pixel sequence 2, and Tth is the threshold. 3. DIRECTIONAL INTERPOLATION Suppose that the set of corresponding pixel pairs obtained from the non-linear pattern alignment can be defined as Ω = {(x(1), y(1)), . . . , (x(k), y(k))}, where k is the number of corresponding pixel pairs, x(i) is the pixel value obtained from pixel sequence 1, S1 , of corresponding pixel pair i, and y(i) is the pixel value obtained from pixel sequence 2, S2 , of corresponding pixel pair i. Then define the set of a connection line between the corresponding pixel pairs as Υ = {l1 , . . . , lk }, where li is the connection line link between pixel value x(i) and pixel value y(i). The example
ω1 x(i) + ω2 y(i) , ω1 + ω2
(4)
where L1 is the recovered pixel. ω1 and ω2 are the weighting coefficient for interpolation and are the distant between the lost pixel and corresponding pixel pairs (e.g., consider Fig.4, when x(i) = P1 and y(i) = P2 , then ω1 = d6 and ω2 = d5 ). Second, the position of the lost pixel is in the location between the 2 closest connection lines, which ∗ can be defined as lL = arg min∀li ∈Υ d(L2 , li ), j = 1, 2, 2 ,j where d(L2 , li ) is the distance between the location of the ∗ lost pixel L2 to the connection line li , and lL , j = 1, 2 are 2 ,j the 2 closest connection lines in Υ to the pixel L2 . With the pixel pairs corresponding to the 2 closest connection lines, the recovered pixel value could be computed as L2 = ω1 xL2 ,l (1) + ω2 xL2 ,l (2) + ω3 yL2 ,l (1) + ω4 yL2 ,l (2),
(5)
(A) PSNR = 41.9170 dB
Fig. 4. The directional interpolation for the lost pixels from the surrounding corresponding pair of pixel values. where ωj , j = 1, . . . , 4 are the weighting coefficient for interpolation, xL2 ,l (1) and yL2 ,l (1) are corresponding pixel ∗ values of the closest line lL of pixel sequence 1 and 2 2 ,1 respectively, and xL2 ,l (2) and yL2 ,l (2) are corresponding ∗ of pixel sequence 1 and pixel values of the closest line lL 2 ,2 2 respectively (e.g., consider Fig.4, when xL2 ,l (1) = P1 , yL2 ,l (1) = P2 , xL2 ,l (2) = P3 , and yL2 ,l (2) = P4 , then ω1 = dω1 ,, ω2 = dω2 , ω3 = dω3 , ω4 = dω4 , where ω = P4 1 j=1 dj ).
(B)PSNR = 43.0986 dB
Fig. 5. Subjective and objective quality of the recovered lost region of Lean image with A) Bilinear interpolation error concealment B) Proposed method.
(A)PSNR = 34.6 dB
(B)PSNR = 35.4 dB
Fig. 6. Subjective and objective quality of the recovered lost region of Mother and Daughter image with A) Bilinear interpolation error concealment B) Proposed method.
4. EXPERIMENTAL RESULTS We conducted experiments to study the performance of the proposed error concealment algorithm in Lena and Mother and Daughter image. The results of error concealment is obtained from applying the algorithms to Y component of image. We compare the results between the proposed algorithm and the bilinear interpolation approach [4]. For Lena image, we selected to eliminate the blocks of images with the size of 16x48 pixels (Fig. 5). The 16th GOB was removed from the first frame of Mother and Daughter video sequence. We set up the threshold of Eq. 3 equal to 40. The subjective and objective performance comparison is illustrated in Fig. 5 and 6. We can see that the proposed error concealment can provide much better recovered lost region comparing to bilinear interpolation. It can improve around two and one dB in Lena and Mother and Daughter image, respectively. In subjective quality, the proposed method can provide much better visible effect than that of bilinear interpolation. 5. CONCLUSION The new intra-frame error concealment algorithm based on the non-linear pattern alignment and directional interpolation was presented. The non-linear pattern alignment approach was used to dynamically detect the possibly existed texture in the lost region. Then, the directional interpolation
recovers the lost pixel based on the detected texture. From preliminary computer experiments, the proposed error concealment provided much better subjective and objective performance comparing to simple bilinear interpolation. 6. REFERENCES [1] S. Wenger, G.D. Knorr, J. Ott, and F. Kossentini,“Error resilience support in H.263+,” IEEE Trans. on Circuits and Systems for Video Technology, vol. 8, no. 7, pp.867877, November 1998. [2] J. Wen and J.D. Villasenor, “Reversible variable length codes for efficient and robust image and video coding,” in Proc. Data Compression Conference, DCC’98, Snowbird, Utah, pp. 471-480, March 1998. [3] S.B.Needleman and C.D. Wunsch, “A general method applicable to the search for similarities in the amino acid sequences of two proteins”, Journal of Molecular Biology. 48: 443-453, 1970. [4] M. Sun and A. Reibman, Compressed Video over Networks, Marcel Dekker Inc., 2001.
