IEICE TRANS. ELECTRON., VOL.E90–C, NO.10 OCTOBER 2007
1876
PAPER
Special Section on VLSI Technology toward Frontiers of New Market
An Image-Moment Sensor with Variable-Length Pipeline Structure Atsushi IWASHITA†a) , Takashi KOMURO† , Nonmembers, and Masatoshi ISHIKAWA† , Member
SUMMARY A 128 × 128 pixel functional image sensor was implemented. The sensor was able to capture images at 1,000 frame/s and extract the sizes and positions of 10 objects/frame when clocked at 8 MHz. The size of each pixel was 18 µm × 18 µm and the fill factor was 28%. The chip, 3.24 mm × 3.48 mm in size, was implemented with a 0.35 µm CMOS sensor process; the power consumption was 29.7 mW at 8 MHz. key words: smart image sensor, vision chip, CMOS image sensor, real-time vision, variable-length pipeline
1.
Introduction
Recently, visual information has been playing an important role in system automation. Until now, image sensors and powerful processors have been used to extract visual information from images. Although such image processing systems are relatively easy to construct, they are generally large and expensive [1]. On the other hand, there are sensors that can output only characteristics or features of an image, for example, position sensitive detectors (PSDs) [2]. They operate quickly and effectively, but they are restricted to outputting only the position of an object; they cannot separate the object from its background, for instance. Therefore, PSDs are only useful in a limited range of applications. Functional image sensors constructed from an image sensor and a processing module implemented in a single chip have been also developed. These systems, called vision chips, are small and low cost, but the type of information that can be extracted is restricted [3], [4], or the resolution and sensitivity are limited according to the size of the image processing circuit [5], [6]. Therefore, we developed a new functional image sensor specially designed for extracting moments. Moments are statistical values that contain information about areas, positions, orientations, and shapes of objects in an image. An image-moment M pq can be calculated from a binarized image I(x, y), coordinate weights x p ,yq and size of the array X,Y, as follows: M pq =
Y−1 X−1
x p yq I(x, y).
10 M01 is ( M M00 , M00 ), and the orientation is calculated from the 2nd moments. Other shape information, such as scaling, rotation and affine transformation invariants, are also derived from higher order moments [7]. This information can be used for system automation, such as measurement, identification, and control of objects. Our image-moment sensor could output various types of information about multiple objects in an image in realtime with a single chip. In addition, it had high sensitivity and high resolution because the processing circuit was much smaller than that of other vision chips. These advantages are especially suitable for real-time applications, such as robot systems using visual feedback, high-speed product inspection, and human-machine interfaces for portable devices. In this paper, we first explain the method of calculating the image-moments and propose a technique for accelerating the calculation. Next, we discuss the implementation of the circuit. Finally, we show the results of experiments and the performance of our chip design.
2.
Image-Moments
Figure 1 shows the architecture of the image-moment sensor. Each pixel of the sensor consists of a photo-detector (PD), a binarizer, a column selector, and a bit-serial adder. The PD includes a photodiode for photo-electric conversion
(1)
x=0 y=0
For example, the size of an object is M00 , the centroid Manuscript received February 23, 2007. Manuscript revised May 22, 2007. † The authors are with The University of Tokyo, Tokyo, 1138656 Japan. a) E-mail:
[email protected] DOI: 10.1093/ietele/e90–c.10.1876
Fig. 1
Architecture of the image-moment sensor.
c 2007 The Institute of Electronics, Information and Communication Engineers Copyright
IWASHITA et al.: AN IMAGE-MOMENT SENSOR WITH VARIABLE-LENGTH PIPELINE STRUCTURE
1877
Fig. 2
Moment of a rectangular region. (a)
and an amplifier for voltage signal output. The column selector outputs a logical AND of the binarized image I(x, y) and a column selection signal, col(x). Grayscale images can be captured by changing the threshold voltage of the binarization, Vref , by using the multiple-readout method. Each adder outputs a summation of the output of the column selector and the previous adder using bit-serial operation. The outputs of the last adders on each row, sum(y), give the summations in the horizontal direction by connecting the adders in series. The same operations are carried out on sum(y) and row selection signals, row(y), to calculate the summation in the vertical direction. Therefore, the sensor outputs the summation of the image, moment. In the following subsections, the method of calculating the image-moment with this architecture will be explained.
(b)
Fig. 3
(a) Bit-serial adder, and (b) summation circuit.
described in this paper can perform the calculation with order O(log N) by using bit-serial operations. First, a serial value v(k), constructed from a binary sequence, can be converted to a parallel value V, and vice versa, by the following equation: V=
K−1
2k v(k)
(3)
k=0
A bit-serial adder constructed from a register and a fulladder is shown in Fig. 3(a). The relationship between the inputs and outputs of the full-adder is given by the following equation: a(k) + b(k) + c(k) = s(k) + 2c(k + 1)
2.1 Characteristics of the Moments The output of the sensor, moment, is the summation of the whole image. However, the moment of a rectangular region can be calculated as shown in Fig. 2. Now, a region Rr is defined as follows: 1 (r0 ≤ r ≤ r1 ) (2) Rr = Rr0 r1 (r) = 0 (r < r0 , r1 < r)
If the value of the register becomes zero in the first state (k = 0) and the final state (k = K), the bit-serial adder outputs the summation of two serial values, a(k) and b(k), as a serial value s(k): c(0) = c(K) = 0 K−1 K−1 K−1 2k s(k) = 2k {a(k) + b(k)} + 2k c(k) k=0
pq , is derived The moment of the rectangular region, M from horizontal and vertical direction masks R x and Ry and coordinate weights x p and yq . pq = M
X1 Y1
=
Y−1 X−1
k=0
−
K−1 k=0
x y I(x, y) p q
R x x p Ry yq I(x, y)
x=0 y=0
The moments of rectangular regions can be obtained using such region selection. The moment of a free-form region can be calculated by adding the moments of individual rectangular regions in an external processor outside the sensor. The moments of a grayscale image can also be calculated by adding the moments of binary images captured with different thresholds. 2.2 Calculation of the Moments Moment calculation requires O(N 2 ) clocks when using a general-purpose processor, but the image-moment sensor
(5)
k=0
2k 2c(k + 1) =
−2K c(K) =
x=X0 y=Y0
(4)
K−1
2k {a(k) + b(k)} + c(0)
k=0 K−1
2k {a(k) + b(k)}
(6)
k=0
The summation circuit is also shown in Fig. 3(b). It is constructed from N bit-serial adders connected in series, and the first input b(0) is fixed at zero. 0 (n = 0) b(n) = (7) s(n − 1) (0 < n) The output of this circuit, s(N −1, k), becomes the summation of the N inputs, a(n, k): K−1 k=0
2 s(N − 1, k) = k
K−1 N−1
2k a(n, k).
(8)
n=0 k=0
If a serial value is constant, e(k) = e, the logical AND operation is equivalent to arithmetic multiplication: K−1 k=0
2k e(k) ∩ d(k) = e ×
K−1 k=0
2k d(k).
(9)
IEICE TRANS. ELECTRON., VOL.E90–C, NO.10 OCTOBER 2007
1878
Therefore, a parallel value M converted form the serial output of the sensor, moment(i, j), is given by the following equation: J−1 I−1
M =
2i+ j moment(i, j)
i=0 j=0 I−1
= =
2i
Y−1 J−1 {2 j row(y, j) ∩ sum(y, i)}
i=0
y=0 j=0
I−1
Y−1 J−1
2i
i=0
{
y=0 j=0
J−1 Y−1
=
2 j row(y, j)}sum(y, i)
y=0 j=0
Clock pair and master-slave operation.
Fig. 5
State of the array with a global clock.
Fig. 6
State of the array with delayed clocks.
i=0
Y−1 I−1 X−1
=
Fig. 4
I−1 2 j row(y, j) {2i col(x, i) ∩ I(x, y)}
2i col(x, i)
x=0 y=0 i=0
J−1
2 j row(y, j)I(x, y).
j=0
If the product of the region masks R x and Ry and coordinate weights x p and yq are input into row and column selection signal col(x) and row(y), the moment of a rectangular region can be obtained from the output of the sensor: I−1
2i col(x, i) = R x x p
i=0 J−1
2 row(y, j) = Ry y , j
q
(10)
j=0
M=
Y−1 X−1
pq . R x x p Ry yq I(x, y) = M
(11)
x=0 y=0
The number of input clocks, Nc , required for this calculation is given by the following equation: Nc = (p + 1) log2 X + (q + 1) log2 Y. 3.
(12)
Variable-Length Pipeline Structure
There are obvious problems with this architecture, however. For instance, the critical path is very long because a large number of full-adders are connected in series. This increases the amount of current because the output of the sensor changes many times. In addition, the very long path restricts the maximum frequency. These problems can be solved by constructing a pipeline structure using delayed clocks. The register in each bit-serial adder consists of three transparent latches in order to reduce the circuit area (Fig. 4). There are two clocks, Pm and P s , for master-slave operation, but only one signal can be High in this architecture. If there is a conflict between states M and S , the adder outputs the wrong result.
by a pair of global clocks. The maximum frequency of the pixel array farray is limited by the delay time of the pixel circuit t pixel , as follows: farray
