output divided images by using conventional HDTV video signal interfaces, like HD-SDI's. In some camera systems, the out
SUPER HIGH RESOLUTION VIDEO CODEC SYSTEM WITH MULTIPLE MPEG-2 HDTV CODEC LSI’S Ken Nakamura, Takeshi Yoshitome, Yoshiyuki Yashima NTT Cyber Space Laboratories, NTT Corporation 1-1 Hikarinooka Yokosuka-city Kanagawa 239-0847 Japan 3840 pixel
1. INTRODUCTION Interest in Super High Resolution (SHR) video systems has grown in the last few years. SHR video has approximately double the height and width of conventional HDTV and has a 24 - 60 fps frame rate. Because of the high quality and the sense of reality they convey to the viewer, SHR systems are expected to be platforms for many new video communication applications, like digital cinema, virtual museums and public viewing of sports, concert and other events[1]. For SHR communication applications, it will be very important to reduce network bandwidth, because raw SHR video data is sent at 6 Gbit/s - 12 Gbit/s. Thus a CODEC system is indispensable for realization of these applications. We have already developed a coding and transmission system for SHR or over-HDTV video, which consists of several conventional MPEG-2 HDTV CODEC equipments and a new multi-channel frame synchronizer [2]. This system was economical, but it was also large and complex. The present report deals with a new SHR CODEC system that uses multiple MPEG-2 HDTV CODEC LSI’s. We have already developed MPEG-2 HDTV encoder system
;,(((
1 1 stream
2 ....
3
4
1
2
3
4
1
....
picture data
4
i) one stream output system 3840 pixel 2160 pixel
This super high resolution (SHR) video CODEC system is based on a spatial image division and multiple stream output approach and is implemented with multiple MPEG2 HDTV CODEC LSI’s. The encoder has two operation modes, a non-multiplexing mode, and a multiplexing mode. In the non-multiplexing mode, the encoder outputs four constant bit rate transport streams (CBR-TS’s) for which it can use two channel-synchronization schemes, system time clock (STC) sharing and sync-marker. In the multiplexing mode, the encoder outputs one CBR-TS comprised of four variable bit rate elementary streams (VBR-ES’s), where cooperative operation between the pre-encoder and main-encoder realize adaptive bit allocation between four channels. The encoder also has a video shift and padding function which reduces distortions caused by the image division. These functions enable high-efficiency, high-quality SHR video coding.
2160 pixel
ABSTRACT
1
3
2
4
4 streams ....
1
1
1
....
....
2
2
2
....
....
3
3
3
....
....
4
4
4
....
picture data
ii) multiple stream output system
Fig. 1. Spatial image division approach.
that uses multiple MPEG-2 SDTV encoder LSI’s [3][4], and have developed the world’s first MPEG-2 422P@HL CODEC LSI’s [5][6]. This time, we have constructed the SHR CODEC system with them. This system is small and energy efficient, yet enables high efficiency coding. In this paper, first, we describe the basic idea of our CODEC system, which is spatial image division and multiple stream output. Next, we explain the video shift and padding function, which solves the boundary distortion problem caused by spatial image division. Then we show how to synchronize the multiple streams in non-multiplexing mode and explain the strategy of the adaptive bit allocation in multiplexing mode. Finally, we discuss the system’s implementation. 2. ARCHITECTURE OF SHR CODEC SYSTEM This CODEC system adopts the spatial image division and multi-stream output approach. In spatial image division, the input image is divided into multiple sub-images and the encoder modules encode them in parallel. This approach is reasonable in terms of cost performance and extensibility. Spatial image division can use a one-stream output system, in which the sub-streams generated by the encoder modules are reconstructed into one SHR elementary stream (ES), or it can use a multiple-stream output system, where
,,,
,6&$6
Encoder
Decoder
1st/2nd Field
1
ENC1 1
2
ENC2
3
4
ENC3 ENC4
DEC1
line copy
2 DEC2
1
2
DEC3
3
4
3
DIST SRC1
4
4 CBR-TS
DEC4
active line
i) non-multiplexing mode
PRE
Encoder MAIN
ENC1
ENC1
Decoder
1
2
ENC2
ENC2
1 2 3
3
4
ENC3
ENC3
4
ENC4
ENC4
coding information.
SRC2
DEC1 DEC2
1
2
DEC3
3
4
line copy
Fig. 3. Video shift and padding function.
DEC4 1 CBR-TS (4 VBR-ES) ii)multiplexing mode
Fig. 2. Overview of the SHR CODEC system.
several HDTV streams generated by the encoder modules are output directly in parallel or multiplexed into one transport stream (TS), as shown in Fig.1. For SHR communication applications, we use the multiplestream output system, because the conventional HDTV decoders can decode its output stream, whereas the one-stream SHR output system needs dedicated SHR decoders. Fig.2 depicts the overview of our CODEC system. The CODEC has two output modes, multiplexing mode and nonmultiplexing mode. In non-multiplexing mode, encoder modules output constant bit rate (CBR) TS’s which are transmitted in parallel. In multiplexing mode, pre-encoder analyzes the input images and each encoder module in main-encoder outputs a variable bit rate (VBR) ES, and these are multiplexed into one CBR-TS. The multiplexing mode has the advantages of transmitter availability and coding efficiency. The non-multiplexing mode is needed when the bitrate of each stream is too high for the bandwidth of the network channel. 3. VIDEO SHIFT AND PADDING FUNCTION In spatial image division, there is a problem that the coding distortion caused by the mismatch of the image boundary and DCT block boundary might be visible at the boundary of the divided images under certain coding conditions. Generally, SHR or over-HDTV image camera systems output divided images by using conventional HDTV video signal interfaces, like HD-SDI’s. In some camera systems, the output image data does not fill the active line, e.g., 1920 1032 image data on 1920 1080, and the remaining lines are filled with black data or zero data. Because the MPEG-2 encoders transform the 8 8 image block by using a DCT and cut off the high-frequency components, if the edge of the image and the DCT block boundary do not match, the border will not be sharp and the image data will be eroded
by black data. Thus, the coding distortion will be visible at the boundary of the divided images. This problem also occurs at the border of ordinary coded images, but it is more visible in this kind of system because the boundaries of the divided images are positioned at the center of the image. In our encoder system, the video shift and padding function modules are placed before the encoder modules. This module can copy the top and bottom lines of image data onto the non-image data area, as shown in Fig. 3. The decoder system does nothing in regard to this operation because the copied data are aborted later. This module also can shift all the image data vertically so that the image border matches the DCT block boundary. This function can provide better quality than the line copy function can, although the image must be shifted back to its original position at the projector system. With this function, the distortion at the divided images’ boundary can be almost completely removed. 4. SYNCHRONIZATION STRATEGY IN NON-MULTIPLEXING MODE In non-multiplexing mode, the encoder outputs 4 CBR-TS’s. When the multiple streams are transmitted on different channels in parallel, the synchronization between channels is a very important issue. This CODEC system has two scheme for synchronization between channels. One method is that the encoder modules share a common STC (System Time Clock), as shown in Fig. 4. On the encoder side, a 27-MHz system clock and STC value are generated from the input video signal at the master encoder module and distributed to the other encoder modules because the input video signals are synchronized. Each encoder generates and outputs a PCR (Program Clock Reference) and PTS/DTS (Presenting Time Stamp / Decoding Time Stamp) from the 27-MHz system clock and STC value. On the decoder side, the master decoder module generates a 27-MHz system clock and STC from the received PCR’s and distributes the system clock and STC value to the other decoder modules. In order to absorb the disparity in the receiving timing between channels and to avoid under-
,,,
VCLK
Pre-Encoder
PLL STC_VALUE Video
VCXO
CLK27MHz
TS
Decoder Module
TS
Encoder Module
TS
Decoder Module
TS
Encoder Module
TS
Decoder Module
TS
Video
Video
Proce ssor
Proce ssor
CODEC LSI
CODEC LSI
CODEC LSI
bitrate, target_bit, Q
CLK27MHz
CLK27MHz Encoder Module
CODEC LSI
Video
Proce ssor
CLK27MHz
CLK27MHz
Video
target_bit, picture_biit, Q
Video
CLK27MHz
CLK27MHz
Video
CLK27MHz
STC_VALUE Encoder Module
Proce ssor
TS
Decoder Module
TS
Proce ssor
Proce ssor
Proce ssor
Proce ssor
CODEC LSI
CODEC LSI
CODEC LSI
CODEC LSI
Video
Fig. 4. Synchronization by STC sharing.
Main Encoder
1
2
Fig. 6. Adaptive bit allocation using pre-encoder. 3
1 bit3
4
0 bit2
1 bit1
main-encoder encodes the input image at a variable bitrate, which is given by the control processor. The bitrates of the encoder modules are changed simultaneously so that the bitrate of the multiplexed TS will be constant. These output streams are multiplexed into one CBR-TS with the cascade multiplexing function in the CODEC LSI’s.
0 bit0
Fig. 5. Sync marker. 6. IMPLEMENTATION flow or overflow of the buffer, the encoder system generates the PTS/DTS with a large enough timing margin. The other synchronization scheme is the sync marker scheme, which is useful mainly with multiple conventional HDTV decoders. Many conventional HDTV decoders do not have an STC sharing scheme and can’t deal with ancillary data like time code information. Our encoder put a sync marker on the top or bottom of the active line of each channel of the image, as shown in Fig. 5. On the receiving side, the decoded images are synchronized with the multichannel frame synchronizer that we have developed [2]. 5. BIT ALLOCATION STRATEGY IN MULTIPLEXING-MODE In multiplexing mode, four VBR-ES’s are multiplexed into one CBR-TS. For higher efficiency coding, an adaptive bit allocation between channels is needed. Our approach is to use a two-pass bit allocation system, as shown in Fig. 6. A pre-encoder encodes and analyzes the input images, and a control processor calculates the bitrate of each channel so as to make the image quality constant between channels. The
The SHR encoder system is comprised of the pre-encoder unit and the main encoder unit, and these units are realized with the same hardware; only the firmwares are different. Fig. 7 is a block diagram of the encoder unit hardware. Four synchronized video signals are input through the HD-SDI to the video buffers. These video buffers can delay the signals up to 20 frames. At the pre-encoder unit, the delay is 0 frames and at the main-encoder unit, it is about 15-20 frames for utilizing the pre-encoding information. This video buffer also has the video shift and padding function and can put the sync marker onto the input images. The video signals are encoded by the corresponding encoder LSI’s. In the multiplexing mode, the main-encoder module multiplexes the streams into one TS and outputs it. The encoder unit hardware is composed of four channel modules, each of which has a video buffer and CODEC LSI, the CODEC’s external memories, host processor, and other communication modules. The channel modules are connected to each other; serial cables are used for the high speed communication between the CODEC LSI’s , which is the cooperative coding control and the cascade bitstream multiplexing, and 100 Base-TX is used for the communica-
,,,
AC
Power Unit
LCD display
Control Processor
HD-SDI
Cable interface
Video Buffer
Table 1. Specification of SHR encoder system Communication Module
CODEC LSI
video format Cable interface
DVB-ASI
video encoder output stream
module#1 Control Processor
HD-SDI
Cable interface
Video Buffer
Communication Module
CODEC LSI
Cable interface
maximum bitrate
DVB-ASI
module#2 Control Processor
HD-SDI
Cable interface
Video Buffer
input interface output interface control interface power size(mm)
Communication Module
CODEC LSI
Cable interface
DVB-ASI
module#3 Control Processor
HD-SDI
Cable interface
Video Buffer
Communication Module
CODEC LSI
¢ ¢
1080i (60/59.94i) 4ch, 720p (60/59.94p) 4ch MPEG-2 422P@HL CODEC LSI 4 or 8( ) CBR-TS 4, or CBR-TS 1 (VBR-ES 4)( ) 80Mbps 4, or 160Mbps 1( ) HD-SDI 4ch, AES3id 4 DVB-ASI 1ch or 4ch( ) 100BASE-TX AC-100-240V 460(w) 440(D) 44(H) 1 or 2( )
¢ ¢ ¢ ¢ Ý ¢ ¢
¢
¢
Ý
¢ Ý ¢ Ý
¢
¢
Ý
Ýmultiplexing mode Cable interface
DVB-ASI
module#4 100 Base-TX
a video shift padding function and adaptive bit allocation using a pre-encoder for high-efficiency video coding. The hardware of this system comprises several channel modules that are connected to each other by high-speed serial cable and that cooperatively carry out the encoding process. In the future, we will increase the resolution of the video that the CODEC can handle.
Serial Cable
Fig. 7. Block diagram of encoder unit.
8. REFERENCES [1] T.Fujii, ”Super High Definition Digital Movie System”, SPIE VCIP’99, Vol.3653, pp.1412-1319, 1999. [2] T.Yoshitome, ”A scalable architecture for use in an over-HDTV real-time codec system for multiresolution video”, Visual Communications and Image Processing (VCIP) 2003, pp.1752-1759 Fig. 8. Encoder unit.
tions between the encoder units. The SHR decoder is comprised of four channel HDTV decoders, and it embodies the STC sharing and sync marker synchronization schemes described in Section 4. The decoder hardware unit has a similar configuration to the encoder hardware. These hardwares have extensibility with respect to the number of channels for encoding and decoding. Thus, video CODEC’s with various resolutions are configurable simply by adding channel modules and changing firmware. 7. CONCLUSION We developed a new SHR video CODEC system using multiple MPEG-2 HDTV CODEC LSI’s. The system has a non-multiplexing mode and a multiplexing mode, and it has
[3] T.Yoshitome, et al, ”Development of an HDTV MPEG2 encoder based on multiple enhanced SDTV encoding LSIs”, International Conference on Consumer Electronics(ICCE) 2001, pp. 160-161 [4] M.Ikeda, ”An MPEG-2 Video Encoder LSI with Scalability for HDTV based on Three-layer Cooperative Architecture”, Design, Automation and Test in Europe Conference (DATE) 1999, pp.44-50 [5] J.Naganuma, et al, ”Vasa: Single-chip MPEG-2 422P@HL CODEC LSI with Multi-chip Configuration for Large Scale Processing beyond HDTC Level”, HOT Chips 14, August 2002 [6] H.Iwasaki, et al, ”Single-chip MPEG-2 422P@HL CODEC LSI with Multi-chip Configuration for Large Scale Processing beyond HDTV Level”, Design, Automation and Test in Europe Conference (DATE) 2003, Designers’ Forum, March 2003, pp.2-7
,,,
