Feb 11, 2003 - 12.8 A 128 x 128 Pixel 120dB Dynamic Range Vision. Sensor Chip for Image Contrast and Orientation. Extraction. Pierre-François Rüedi ...
ISSCC 2003 / SESSION 12 / CMOS IMAGERS, SENSORS AND DISPLAYS / PAPER 12.8
12.8
A 128 x 128 Pixel 120dB Dynamic Range Vision Sensor Chip for Image Contrast and Orientation Extraction
Pierre-François Rüedi, Pascal Heim, François Kaess, Eric Grenet, Friedrich Heitger, Pierre-Yves Burgi, Stève Gyger, Pascal Nussbaum Centre Suisse d'Electronique et de Microtechnique, Neuchâtel, Switzerland The conventional approach to process image information, is to acquire an image with a CCD or CMOS camera, convert the intensity distribution to a digital representation and process it by means of a digital processing device. This approach poses considerable problems in applications, (e.g. automotive or surveillance) where vision tasks have to be executed in real-time and at low system cost on scenes with high and uncontrolled variations of illumination. The vision sensor described in this paper is a step towards fulfilling these requirements by incorporating essential image processing operations on the sensor chip: (i) auto-exposure at the pixel level introduces a high dynamic range, (ii) real-time operation is achieved by pixel-parallel, analog computation of contrast magnitude and direction of image features and (iii) algorithmic post-processing is facilitated by a data communication scheme which allows conveyance of only the essential image and feature information, ordered according to signal magnitude. Contrast magnitude and direction of image features are obtained by taking spatial derivatives at each pixel. The direction of these derivatives is changed continuously over time by means of a global steering function [1]. The spatial derivative reaches a maximum when its direction matches the orientation of image features, thus defining the local gradient magnitude and direction. The contrast representation derived in the vision sensor is equivalent to normalizing the spatial gradient magnitude with the local intensity [∆I/IC]. Contrary to spatial gradient, the contrast representation does not depend on illumination strength, thus introducing considerable advantages for the interpretation of scenes. Figure 12.8.1 illustrates the pixel structure. At the beginning of a frame acquisition, capacitor Cint is reset and transistors M1 to M4 are turned on. Voltage VC, resulting from the integration of the photocurrent on Cint, is dispatched to the 4 neighbors (Left, Right, Top and Bottom). At the moment when VC reaches a fixed voltage level at each pixel, the differences in voltage of the neighboring pixels (VL, VR, VT, VB) do not depend any more on the illumination level but rather represent a measure of the local contrast. Therefore, when each individual VC reaches VREF, transistors M1 to M4 are turned off so that these 4 voltages are sampled on capacitors CL, CR, CT and CB. In addition, at the same time, a pulse encoding the address of the cell is emitted on the communication bus (PX, PY). The time when this pulse is emitted encodes the local luminance level. The spatial derivatives based on VL, VR, VT and VB are implemented by two 4-quadrant multipliers made of transistors operating in strong inversion. The sampled voltages are applied to their gate terminals while two steering functions (sine and cosine components) are applied to the respective source terminals of the multipliers using class-AB amplifiers. The output of the multipliers (Isteer) consists of a sine wave whose amplitude and phase represent the magnitude and direction of contrast. This signal is fed into a maximum detector reset at the beginning of each frame acquisition, and into a zero crossing detector. Figure 12.8.2 illustrates a frame acquisition and the
• 2003 IEEE International Solid-State Circuits Conference
whole sequence of operations for 2 pixels with different local illumination levels and contrasts. It is observed that the currents Isteer(t) increase as photocurrent integration proceeds. Pixel b, which receives the strongest illumination, is the first to reach VREF and thus to hold the voltages coming from the neighbors. After completion of this integration stage for all pixels, the steering functions are turned on. During the first steering period (peak period), the maximum detectors detect and memorize the maximum (Inorm) of currents Isteer(t). During the subsequent steering periods, signal ITH(t), which decreases monotonously with time, is compared to Inorm. Upon matching of Inorm and ITH(t), a pulse is emitted on PX and PY, thus the contrast magnitude is time encoded with the high contrasts preceding the lower ones. The contrast orientation is encoded by emitting a pulse at the first zero crossing with a positive slope of signal Isteer(t) following the emission of the contrast information. By this, the contrast direction is ordered in time according to the contrast magnitude, hence prioritizing the strong contrasts. This scheme allows to implement a threshold for contrast at the sensor level, thereby reducing the transmitted information and facilitating post-processing for vision tasks. Figure 12.8.3 illustrates a block diagram of the circuit. The core of the circuit is the pixel array. To facilitate the interfacing with an external micro-controller, the column and row encoders feed the pulses to a FIFO. The address is recorded in the FIFO together with an 8b temporal tag representing luminance during integration, and the contrast and orientation during the steering periods. A SPI interface is used to access the control registers in the digital interface. An analog bias block generates, among others, the contrast threshold current as well as the sine and cosine steering functions applied to each pixels. The top and middle rows of Fig. 12.8.4 illustrate the ordered dispatching of information based on signal amplitude. The top row depicts the accumulation of luminance during photocurrent integration, whereas the middle row shows the accumulation of contrast information during the steering periods. Computation of the contrast amplitude (first steering period) and transmission of contrast information (subsequent periods) typically take 2 ms. The bottom row illustrates the independence of the contrast representation on spatial inhomogeneities in the illumination level present in a scene. The top left image of Fig. 12.8.5 shows the ability of the sensor to represent very low contrasts, like shadings on a hand. The top right illustration represents a scene with a wide range of contrasts and illuminations. The filament of a lighted lamp bulb is visible, together with shadings on a face. The left and right bottom pictures show respectively the local contrast magnitude and orientation of a lighted lamp bulb. The orientations are encoded using gray levels. Figure 12.8.6 lists the characteristics of the sensor. The maximum integration time due to dark current is 150ms at 25OC. The maximum usable integration time is arbitrarily fixed to one tenth of this value. A contrast sensitivity of 2% is maintained up to a corner integration time of 3ms, then decreasing down to 10% for the maximum integration time of 15ms. This 2% sensitivity corresponds to 6b or a 30dB increase in the dynamic range of the sensor. References [1] M. Barbaro, P.-Y. Burgi, A. Mortara, P. Nussbaum and F. Heitger: "A 100 x 100 Pixel Silicon Retina for Gradient Extraction with Steering Filter Capabilities and Temporal Output Coding,” IEEE Journal of Solid-State Circuits, vo. 37-2, Feb 2002, pp 160.
0-7803-7707-9/03/$17.00
©2003 IEEE
ISSCC 2003 / February 11, 2003 / Salon 1-6 / 4:30 PM steering functions
cosZt
sinZt
[V]
VL(t) VR(t) VT(t) VB(t)
M1
CL
M2
CR
M3
CT
M4
Peak detector
t
integration time
Isteer(t)
steering currents Isteer [A] hold a
0-x detector
CB
VREF
max allowed
Inorm
cos(ωt) sin(ωt)
peak detection period
steering periods Ith decreasing theshold Inorm peak a peak b
pixel a
Cint VC(t)
PX
1
VBLACK PY
PD
VRST
Pulse emitter
pixel b
S
t
hold b
ITH
Sa
S Px Py
Figure 12.8.1: Pixel structure.
magnitude
Sb magnitude
angle
angle
t t
Figure 12.8.2: Frame acquisition sequence.
Row encoder
Accumulation of luminance with time during integration
128 x 128 Pixel array
Analog biases
Digital interface
.
FIFO
Column encoder
SPI
8-bits data bus
Contrast magnitude
0°
High luminance
Low luminance
Contrast
Control signals
Figure 12.8.3: Block diagram of the circuit. Contrast magnitude
12
Accumulation of output contrast with time
Figure 12.8.4: Luminance and contrast representations.
Contrast magnitude
Technology Pixel size / Fill factor Conversion factor Dark current Integration time Integration dynamic range Contrast dynamic range Total dynamic range Data compute and dispatch Frame rate Contrast sensitivity Orientation precision Power consumption
Contrast direction
Direction gray scale map (bottom right frame)
0.5 µm, 3-m, 2-p 69 x 69 µm2 / 9 % 46 µV/e0.3 pA at 25 °C 0.5 µs … 15 ms 90 dB 30 dB 120 dB 2 ms 1/(max integration time + 2 ms) 2% +/- 3° 300 mW at 3.3 V
360°
Figure 12.8.5: Contrast magnitude and direction.
• 2003 IEEE International Solid-State Circuits Conference
Figure 12.8.6: Circuit characteristics.
0-7803-7707-9/03/$17.00
©2003 IEEE
Figure 12.8.7: Chip micrograph.
12
• 2003 IEEE International Solid-State Circuits Conference
0-7803-7707-9/03/$17.00
©2003 IEEE
VL(t) VR(t) VT(t) VB(t)
M1
CL
M2
CR
M3
CT
M4
Peak detector Isteer(t) 0-x detector
CB
VREF
cos(ωt) sin(ωt) Cint
VC(t)
Inorm
1 PD
PX VBLACK PY
Pulse emitter
S ITH
VRST
Figure 12.8.1: Pixel structure.
• 2003 IEEE International Solid-State Circuits Conference
0-7803-7707-9/03/$17.00
©2003 IEEE
steering functions
cosZt
sinZt
[V]
max allowed
t
integration time steering currents Isteer [A] hold a
peak detection period
steering periods Ith decreasing theshold Inorm peak a peak b
pixel a pixel b
t
hold b S Px Py
Sa magnitude
Sb angle
magnitude
angle
t t
Figure 12.8.2: Frame acquisition sequence.
• 2003 IEEE International Solid-State Circuits Conference
0-7803-7707-9/03/$17.00
©2003 IEEE
Row encoder
128 x 128 Pixel array
Column encoder
Analog biases
.
Digital interface
FIFO
SPI
8-bits data bus Control signals
Figure 12.8.3: Block diagram of the circuit.
• 2003 IEEE International Solid-State Circuits Conference
0-7803-7707-9/03/$17.00
©2003 IEEE
Accumulation of luminance with time during integration
Accumulation of output contrast with time
High luminance
Low luminance
Contrast
Figure 12.8.4: Luminance and contrast representations.
• 2003 IEEE International Solid-State Circuits Conference
0-7803-7707-9/03/$17.00
©2003 IEEE
Contrast magnitude
Contrast magnitude
Contrast magnitude
Contrast direction
0°
Direction gray scale map (bottom right frame)
360°
Figure 12.8.5: Contrast magnitude and direction.
• 2003 IEEE International Solid-State Circuits Conference
0-7803-7707-9/03/$17.00
©2003 IEEE
Technology Pixel size / Fill factor Conversion factor Dark current Integration time Integration dynamic range Contrast dynamic range Total dynamic range Data compute and dispatch Frame rate Contrast sensitivity Orientation precision Power consumption
0.5 µm, 3-m, 2-p 69 x 69 µm2 / 9 % 46 µV/e0.3 pA at 25 °C 0.5 µs … 15 ms 90 dB 30 dB 120 dB 2 ms 1/(max integration time + 2 ms) 2% +/- 3° 300 mW at 3.3 V
Figure 12.8.6: Circuit characteristics.
• 2003 IEEE International Solid-State Circuits Conference
0-7803-7707-9/03/$17.00
©2003 IEEE
Figure 12.8.7: Chip micrograph.
• 2003 IEEE International Solid-State Circuits Conference
0-7803-7707-9/03/$17.00
©2003 IEEE
