IEICE TRANS. FUNDAMENTALS, VOL.E88–A, NO.6 JUNE 2005
1443
PAPER
Special Section on Papers Selected from ITC-CSCC 2004
A Design of Real-Time JPEG Encoder for 1.4 Mega Pixel CMOS Image Sensor SoC Kyeong-Yuk MIN†a) , Student Member and Jong-Wha CHONG† , Member
SUMMARY In this paper, we propose a hardware architecture of realtime JPEG encoder for 1.4 mega pixels CMOS image sensor SoC which can be applied to mobile communication devices. The proposed architecture has an efficient interface scheme with CMOS image sensor and other peripherals for real-time encoding. The JPEG encoder supports the baseline JPEG mode, and processes motion images of which resolution is up to 1280 × 960 (CCIR601 YCrCb 4:2:2,15 fps) by real-time processing. The JPEG encoder supports 8 types of resolution, and can serve the 4 levels of image quality through quantization matrix. The proposed JPEG encoder can transfer encoded motion pictures and raw image data from CMOS image sensor to external device through USB 2.0 and a compressed still image is stored at external pseudo SRAM through SRAM interface. And proposed core can communicate parameters of encoding type with other host by I2C. The proposed architecture was implemented with VHDL and verified for the functions with Synopsys and Modelsim. The encoder proposed in this paper was fabricated in process of 0.18 µ of Hynix semiconductor Inc. key words: CMOS image sensor, encoder, JPEG, one-chip camera, SoC
1.
Fig. 1
Block diagram of JPEG encoder.
Introduction
Currently, high-quality image applications are performed on hand-held terminal. Such kind of hand-held device for video application requires a small-size and low-power video camera device with image compressor. A general digital camera system is composed of a image sensor, signal processor, image compressor/decompressor, video encoder, decoder, memory controller, interface to peripherals, and other devices [1]. There are two types of image sensor including CCD (Charge Coupled Device) and CMOS Image sensor. Among two image sensor, especially in image quality, recent progress in CMOS image sensors is creating new opportunities to develop a low-cost, low-power, one-chip digital video camera which has digitizing, signal processing, and image compression functions [2]–[5]. So we can realize low-cost and low-power one-chip camera by integrating image sensor, A/D signal processor and image compressor into one die by CMOS technology. This means that image sensor of CMOS camera system has a benefit in low-cost compared to that of CCD camera system. Thus, image sensors should be based on CMOS technology from the view points of low-power, low-cost, and feasibility of co-integration of the sensor and mixed signal processing. In this paper, we proposed archiManuscript received October 3, 2004. Manuscript revised January 5, 2005. Final manuscript received February 28, 2005. † The authors are with the Department of Electronics Engineering, Hanyang University, Seoul, Korea. a) E-mail:
[email protected] DOI: 10.1093/ietfec/e88–a.6.1443
tecture of real-time processing JPEG encoder for 1.4 mega pixel CMOS image sensor with attention in recent studies. Also, we propose the interface scheme with CMOS image sensor and other peripherals for real-time encoding. 2.
Implementation of High Speed Base-Line JPEG Encoder
The proposed architecture can compress and generate realtime image which is fully compliant with baseline JPEG standard [1], 1280×960 resolution, 4:2:2 image formats, and 15 fps. The proposed JPEG encoder is composed of DCT, quantizer, run-length coder, Huffman encoder and packer. The DCT in the proposed JPEG core has pipe-lined processing method which adapts distributed arithmetic operation method, and it is possible to process with small size hardware. Shown as Fig. 2, The DCT is composed of 3 level pipelined processes of which first stage receives the pixel data, second stage processes distributed arithmetic operation, and third stage generates coefficient element. Gate count of the DCT is 19500 except for a transposition matrix from synthesis. The proposed JPEG encoder can select image quality of 4 levels defined as “best,” “better,” “good,” and “normal.” The quality of image can be selected by using a Q-matrix value. In Fig. 3, Architecture of Huffman encoder for this proposed JPEG encoder is depicted. Huffman table is implemented by ROM instead of internal RAM. These are brief descriptions of the pipeline architecture of the baseline JPEG encoder.
c 2005 The Institute of Electronics, Information and Communication Engineers Copyright
IEICE TRANS. FUNDAMENTALS, VOL.E88–A, NO.6 JUNE 2005
1444 Table 1
Fig. 2
Resolution list of real-time encoding.
Block diagram of DCT.
(a)
Fig. 3
(b)
Block diagram of Huffman encoder.
Fig. 4 Sensor timing. (a) Timing map for CMOS image sensor with frame mode. (b) Detailed timing map for CMOS image sensor.
1) Implementation of Distributed arithmetic architecture of DCT core with pipelined three-stage. 2) Implementation of pipeline structure for Quantizer and RLC. 3) Implementation of pipeline structure for Huffman encoder 3.1) Huffman encoder can compress with 2 or 3 clocks per one pair of run and level value from RLC, because quantizer has many zero output [6]. 3.2) If there are many non-zero values from quantizer, then previous Huffman encoder cannot compress a pair of run and level value with every clock cycle by pipeline manner. 3.3) So, we propose a architecture of Huffman encoder which can compress a pair of run and level value per one clock. 4) Implementation of pipelined Packer. 3.
Interfacing with CMOS Image Sensor
Because of the difference between input orders from CMOS image sensor and processing orders for JPEG encoder, we need the input buffer rearranging order of interlaced inputs
to 8×8 block order. The proposed JPEG encoder can receive 8 types of input image resolution with YCrCb, 4:2:2 and 4:2:0 types listed in Table 1. The order of image data from CMOS image sensor is depicted in Fig. 4 [7]. External Pseudo SRAM size of 32 k × 8 is used as a buffer memory. Figure 4 describes the relation of data and sync signal from sensor module. Because of difference in start time between CMOS image sensor and JPEG encoder, JPEG encoder should make another refresh signal instead of master reset. This refresh signal is related with latency of input buffer. If the input data from image sensor has the 4:2:2 image format, 8 × 8 block output is available after 8th line input, and in the case of 4:2:0 image format, it is possible to make 8 × 8 block output after 16th line input. So we can get the equation of latency for variable block size and image format. T Latency = w ∗ c w: c:
(1)
width of input image constant 24 if 4:2:0 16 if 4:2:2
In Eq. (1), there are two values in constant c. In the
MIN and CHONG: A DESIGN OF REAL-TIME JPEG ENCODER
1445
case of 4:2:0, the number of luminance blocks divided by the number of chrominance blocks is 1.5. And the 8 × 8 block for chrominance component can be built after 16th line input from CMOS image sensor. Thus, we can get the number 24 by only multiplying 1.5 by 16. In the same manner, we can calculate another constant 16 in Eq. (1). 4.
Interfacing with Other Module
4.1 Command and Motion Mode We propose a JPEG encoder that can manage various resolutions and input image formats, and that can compress still images and motion pictures in real-time processing. In the case of the motion pictures, we support frame skipping mode which takes external host and memory size in account. Many control lines are needed to support these various modes. The proposed JPEG encoder adopts I2 C to communicate with other systems about current status of encoder and define encoder’s various modes. I2 C module which is applied to proposed architecture supports only slave mode because each commands are received from external hosts. Output data from USB2.0 are compressed data with motion mode and raw image data generated from CMOS image sensor. Assume that input images are 4:2:2 in image format, 1280 × 960 in resolution, and 15 in frame rate, then, data rate is up to 40 Mbps in real-time processing. Thus, we should adopt USB2.0 specification to this JPEG encoder for transmitting encoded data and raw image data to external host.
We propose the simple interface scheme for still mode using only three signals. To communicate control signal with host and JPEG encoder, Still Mode Enable, IRQ and READ ON signals are used. After the Still Mode Enable is set to high, JPEG encoder is operated by the Vsync that operates for the first time. The JPEG encoder operates only one time in this mode. When the JPEG encoder begins operating, IRQ signal is set to low by the Still Mode Enable signal. After the start of compressing, the still mode enable can set to high. When IRQ signal is set to high, this means that JPEG encoder completes compressing or keeps idle state. And READ ON signal must hold on high, when the external host read data from external SRAM for encoded bitstream. Next, the READ ON is set to low for do next operation. 5.
Timing Map for Real-Time Processing
There is a time difference between start of input image and processing of the JPEG encoding. Because of the different start time between CMOS image sensor and JPEG encoder, especially real time processing, the JPEG encoder should make refresh signal instead of master reset. Shown as Fig. 7, difference between reset for input buffer and frame refresh signal is calculated as Eq. (2).
4.2 Interface of Still Mode An interface scheme about still mode is depicted in Fig. 5.
(a)
(a)
(b)
(b)
Fig. 5 Data interface for still mode. (a) Control sequence of still mode interface. (b) Block diagram of still mode interface.
Fig. 6 Interface feature of JPEG encoder. (a) Interface of CIS and external host. (b) Detailed interface signal.
IEICE TRANS. FUNDAMENTALS, VOL.E88–A, NO.6 JUNE 2005
1446
Fig. 7 Table 2
Table 3
JPEG encoder control.
Calculation of latency.
Synthesized result of proposed JPEG encoder.
Fig. 8
Results of motion mode (640 × 480, 4:2:2, 30 fps, with 27 MHz).
Fig. 9
T interval = (w + n) ∗ 7 + w − 142 w: width of input image n: hsync interval The constant 142 is calculated as Table 2.
Result of still mode (640 × 480, 4:2:2).
(2) We can get a refresh time by subtracting 142 clocks from latency of input buffer. Characteristics of the proposed JPEG encoder
MIN and CHONG: A DESIGN OF REAL-TIME JPEG ENCODER
1447
Library: 0.18 µ hynix process Gate count: 65000 Operation Freq.: up to 120 MHz SRAM: Synchronous SRAM 32 k × 8, 64 × 11, 64 × 16 Critical path delay: 8.2 ns (DCT module) 6.
Conclusion
In this paper, the hardware architecture of real-time JPEG encoder for 1.4 mega pixel CMOS image sensor SoC is presented. Also, the interface scheme with CMOS image sensor and other peripherals for real-time encoding is proposed too. This architecture can be used in one-chip camera featured by 1.4 mega pixel resolution with CMOS image sensor. And the proposed JPEG core can be used in a wide range of video system applications, particularly in consumer product such as a cellular phone. Acknowledgments Author would like to thank to Oh-bong Kwon and Sung-kyu Chang in Hynix Semiconductor Inc. References [1] ISO 10918 (ISO/IEC), Digital Compression and Coding of Continuous Tone Still Images (JPEG). [2] E.R. Fossum, “CMOS image sensors: Electronic camera-on-a-chip,” IEEE Trans. Electron Devices, vol.44, no.10, pp.1689–1698, Oct. 1997. [3] S.K. Asada, H. Ohtsubo, T. Fujihira, and T. Imaide, “Development of low-power MPEG1/JPEG encode/decode IC,” IEEE Trans. Consum. Electron., vol.43, no.3, pp.639–644, Aug. 1997. [4] S. Okada, Y. Matsuda, T. Watanabe, and K. Kondo, “A single chip motion JPEG codes LSI,” IEEE Trans. Consum. Electron., vol.43, no.3, pp.418–422, Aug. 1997. [5] S. Kawahito, M. Yoshida, M. Sasaki, K. Umehara, D. Miyazaki, Y. Tadokoro, K. Murata, and A. Matsuzawa, “A CMOS image sensor with analog two-dimensional DCT-based compression circuit for onechip cameras,” IEEE J. Solid-State Circuits, vol.32, no.12, pp.2030– 2031, Dec. 1997. [6] L.V. Agostini, I.S. Silva, and S. Bampi, “Pipelined entropy coders for JPEG compression,” Proc. 15th Symposium on Integrated Circuits and Systems Design (SBCCI’02), pp.203–208, 2002. [7] “HV7151SP,” Data Sheet of CMOS Image Sensor with Image Signal Processing, Hynix Semiconductor Inc.
Kyeong-Yuk Min was born in Nonsan, Korea, on Sept. 26, 1966. He received the B.S degree in physics from Korea University, Seoul, Korea in 1992. He received the M.S. degree in 1996 and is now working toward the Ph.D. degree in electronics engineering from Hanyang University, Seoul, Korea. This author became a Member (M) of IEEE in 2002. His current research interests the digital Cinema, the hardware design of real-time H.264 encoder/decoder and JPEG2000 encoder/decoder.
Jong-Wha Chong was born in Nonsan, Korea, on March 10, 1950. He received the B.S. and the M.S. degree in electronics engineering from Hanyang University, Seoul, Korea, in 1975, and 1979 respectively. He received Ph.D. degree in electronics & communication engineering from Waseda University, Japan, in 1981. From 1979 to 1980, he was a researcher in C&C Research Center of Nippon Electronic Company (NEC). From 1983 to 1984, he was a researcher in the Korean Institute of Electronics & Technology (KIET). From 1986 to 1987, he was the visiting professor in the University of California at Berkeley, CA. From 1993 to 1994, he was the chairman of CAD & VLSI Society in the Institute of the Electronic Engineers of KOREA (IEEK). From 1995 to 1997, he was the visiting professor in the University of Newcastle uponTyne, UK. From 1997 to 1999, he was the dean of information and communication center, Hanyang University. Since 1981, he is a professor of department of electronics engineering, Hanyang University, Seoul Korea. Since 2002, he is the Vice chairman of the IEEK. His current research interests the design of ASIC emulation system, CAD for VLSI, H.264 encoder/decoder design, JPEG2000 encoder design, and communication circuit design, especially UWB modem design.
