A Time-of-Flight Line Sensor – Development and Application Rolf Kaufmanna , Michael Lehmanna , Matthias Schweizera , Michael Richtera , Peter Metzlera , Graham Langa , Thierry Oggiera , Nicolas Blanca , Peter Seitza , Gabriel Gruenerb and Urs Zbindenb a CSEM b CSEM
SA, Photonics Division, Badenerstrasse 569, CH-8048 Zurich, Switzerland; SA, Robotics Division, Untere Gr¨ undlistrasse 1, CH-6055 Alpnach, Switzerland ABSTRACT
A new miniaturised 256 pixel silicon line sensor, which allows for the acquisition of depth-resolved images in real-time, is presented. It reliably and simultaneously delivers intensity data as well as distance information on the objects in the scene. The depth measurement is based on the time-of-flight (TOF) principle. The device allows the simultaneous measurement of the phase, offset and amplitude of a radio frequency modulated light field that is emitted by the system and reflected back by the camera surroundings, without requiring any mechanical scanning parts. The 3D line sensor will be used on a mobile robot platform to substitute the laser range scanners traditionally used for navigation in dynamic and/or unknown environments. Keywords: Distance-Measurement, 3D-Measurement, TOF, Demodulation, Lock-In, Range-Scanner
1. INTRODUCTION The ability to capture the environment in three dimensions brings tremendous new opportunities in several fields of application. Our approach is based on a time-of-flight (TOF) distance measuring principle. Competing approaches, such as triangulation methods (e.g. stereo vision), interferometric systems and radar systems, are well known from literature. Due to the rapid progress in microtechnologies, mainly in microelectronics, a system using the TOF-principle to capture entire environments has become increasingly precise and, nowadays, can fulfill the specifications of industrial applications. Systems based on the TOF-approach profit mainly from the tremendous progress in microtechnology. The main difficulty, and the reason that the TOF-system needs even further progress in microtechnology, is the fact that measuring distances using the speed of light requires very fast and precise devices. As an example, by using a counter with a clock frequency of ν = 1 GHz, the theoretically achievable distance resolution is ∆D = c/(2ν) = 15 cm. This example shows the high demands placed on time of flight measurements. By implementing more sophisticated approaches than a simple counter, the requirements can be lowered and thus higher distance resolutions can be achieved. In this paper, we present the approach developed at CSEM, based on the phase measurement of an incoming wave front1 . An intensity-modulated signal is emitted by the system, reflected by the objects in the scene and returns to the phase measuring detector. The phase of the incoming wave represents a direct measure of an object’s distance in the scene. This paper introduces the current implementation of the TOF measurement principle in a combined CMOS-CCD technology. Theoretical background information is presented and the physical limits to distance resolution using the phase measurement algorithm are sketched. Three pixel versions with different functional integration levels are discussed in detail.
2. THEORETICAL BACKGROUND OF THE PHASE MEASUREMENT TECHNIQUE The camera system emits a sinusoidally modulated light wave. This light impinges on the objects in the scene and is reflected back to the camera system. Through an appropriate optical system, the electromagnetic wave front is imaged onto a demodulating sensor. The emitted light can be modelled as ideally sinusoidal ³ ³ ν ´´ e(t) = e0 1 + sin t , (1) 2π Corresponding author,
[email protected]
with ν being the modulation frequency and e0 the emitted mean power. The wave front impinging on the sensor appears as ´´ ³ ³ν t−ϕ s(t) = B(t) + e0 k 1 + sin 2π
,
(2)
with B(t) as the background illumination power, k the attenuation factor including the target (distance, reflectivity) as well as the optics (lens, filter) and ϕ the phase shift arising from the wave front’s time of flight. In conventional environments, the background illumination changes with much lower frequencies than the emitted light and, therefore, can be approximated by a constant B(t) = B. If we demodulate the incoming signal by sampling four times (A0 , A1 , A2 , A3 ), where the four samples are equally distributed over the period, the incoming signal can be computed according to the following formulae: ¶ µ A3 − A 1 (3) ϕ = − arctan A2 − A 0 3
B
=
A
=
1X Ai 4 i=0 1p (A3 − A1 )2 − (A2 − A0 )2 2
(4) ,
(5)
where ϕ is the phase shift, B the offset and A the mean amplitude. These three parameters allow a complete reconstruction of the signal s(t). The offset B represents the intensity information, similar to conventional black and white images. The amplitude A is a means of quantifying the incoming light derived from the illumination unit. Finally, the distance D from the target can be calculated directly by equation 6. D=
ϕ c 2π 2ν
,
(6)
with c being the speed of light and ν the modulation frequency. The term c/2ν represents the non-ambiguity range of the phase measurement. Because the distance accuracy of the measurement depends on the acquired amplitude A and offset B, these two parameters provide a very fast and cost-efficient way to qualify the distance measurements. The ultimate physical limitations of such a distance measuring device are described in more detail in2 and3 .
3. IMPLEMENTATION 3.1. Concept of Charge Separation The demodulation pixels have to solve the task of sampling the reflected intensity-modulated wave at four distinct sampling points. In order to do so, first of all an efficient electronic shutter is needed. The photocharges for each of the sampling points have then to be efficiently separated and stored inside the pixel, since the sensor can not be read-out with a frame-rate of the order of the modulation frequency. Furthermore, with a modulation frequency of ν = 20 MHz less than one photon per pixel and per period arrives statistically at the sensor surface. Therefore, it is essential to successively add photo-charges over many periods as noise-free as possible. All these tasks can excellently be realised with the CCD principle. The CCD allows for a nearly noise-free addition and fast, directed transport (to implement a shutter) of charges. The parallel integration of all four sampling signals A0 ...A3 is outlined in figure 1. First, the photons are absorbed in the photosensitive CCD area. Then, the photogenerated charges are separated with a suitable CCD gate drive and stored below an integration gate (a so called ”tap”, which is displayed as a bucket in figure 1). This process is then repeated over several hundred periods. The next subsection deals with three different architectures in order to implement the mechanism described above.
+ Detection of light
A0 =
a0,i
PIXEL
Fast Separation
Repeated addition In-pixel storage
Figure 1: The process of charge creation, separation, storage and addition implemented in each pixel.
3.2. How Many Taps? All our system designs so far were based on the so-called four-buckets algorithm 4 . This means, that in total four samplings are performed (A0 , A1 , A2 , A3 ) to derive the phase / distance of the incoming wavefront. The four-buckets algorithm is less sensitive to non-linearities than the three-buckets algorithm, and it results in a simpler calculation of the phase, distance and offset (equations 3, 4 and 5). Based on the four-buckets algorithm, this section compares the three different pixel architectures, which were successfully implemented for the distance / phase measurement. All sensors were based on a 0.8µm CMOS / CCD technology from ZMD5 . The in-pixel demodulation is performed using the key advantage of almost noise-free summing of charges in a CCD; the complete readout structure is implemented with CMOS technology resulting in a freely addressable sensor. Below, the different pixel architectures (1-tap, 2-tap, 4-tap) are described and the advantages and drawbacks of each architecture and the corresponding algorithms are listed. Further multi-tap lock-in pixel approaches are presented in3 .
3.3. Comparison 1-tap / 2-tap / 4-tap 1-tap The first successful implementation of a lock-in pixel has been designed based on a 1-tap architecture. The number of taps represents the number of storage sites within one pixel. In a 1-tap pixel architecture, only one sample signal can be stored. Therefore, four consecutive measurements need to be performed in order to finally obtain the phase ϕ, according to equation 3. The requirement of four consecutive exposures renders the distance measurement based on the 1-tap pixel architecture vulnerable to dynamic objects in the scene. Also changing illumination conditions during the four integrations may cause unpredictable distance errors, because any change in the scene during any sampling can happen. A second drawback of the 1-tap pixel architecture is the loss of photo-generated electrons outside the sampling duration. The sampling time in the implemented 3D-system amounts to exactly half of the modulation period. In the case of the 1-tap pixel, during half of the modulation period the incoming signal is captured and stored, while during the other half the incoming signal is dumped, and thus lost. The key advantage of the 1-tap pixel design is that every sample uses exactly the same storage-site and read-out path. Therefore, matching errors of the transistors and capacitances are no key issues of the design. All samplings are controlled by the physically identical structures. This includes the control signals, the pixel demodulation properties and the output path. The timing of a 1-tap architecture sensor is sketched in figure 2. A complete camera has been developed based on the 1-tap pixel architecture and is presented in more detail in3 . The resolution amounts to 25 × 64 pixels. 2-tap The key advantage of the 2-tap pixel architecture is that all photo-generated electrons are exploited. Instead of dumping one half of the electrons such as within the 1-tap pixel, the dumped electrons represent just the opposite sampling signal. The sampling duration is set to half the modulation period: during the first half all electrons drift to one output, during the second half all electrons are transferred to the opposite output. Therefore, the samples A0 and A2 (A1 and A3 , respectively) are acquired simultaneously. But still, two consecutive exposures
need to be performed. That means fast changing scene-illuminations nevertheless may cause distance errors. These errors are not as severe as they are for the 1-tap sensor, because the frame rate is basically doubled. Therefore, the complete distance map is updated faster and a scene change has less influence. Frame Read out Acquisition A0
Acquisition A1
Data Read Proces out sing
Read out
Read out Acquisition A2
Acquisition A3
Figure 2: Timing diagram for the 1-tap algorithm.
t
Frame Data Read Proces out sing
Read out Acquisition A1 / A3
Acquisition A0 / A2
Figure 3: Timing diagram for the 2-tap algorithm.
t
Frame Data Read Proces out sing Acquisition of A0 / A1 / A2 / A3
Figure 4: Timing diagram for the 4-tap algorithm.
t A drawback of the 2-tap pixel architecture lies in the two physically separated output channels per pixel. Each output channel has its own response, and even by considering the conventional matching rules of ICdesign, some unequal behaviour of the two channels cannot be avoided. Certainly, this effect can be calibrated, however, it is not convenient. A camera based on the 2-tap pixel architecture and a resolution of 160 × 124 pixels is available2 and was awarded the European Information Society Technology Grand Prize 2004. 4-tap In order to avoid the requirement of more than one exposure to acquire a complete distance map, two taps on the pixel are not sufficient. In the following, the 4-tap pixel architecture has been successfully implemented and tested. As mentioned, the acquisition of the four samplings in parallel permits a very fast system response time (see the timing diagram in figure 4). Reasonably fast moving targets in the scene and changing illumination conditions become irrelevant. A further asset of the 4-tap pixel architecture is that the four samples per pixel are available at the same time. Directly after addressing the pixel, the data can be processed immediately in order to obtain distance, offset and intensity data. Intermediate storage of the samplings, such as it is necessary for
the 1- and 2-tap pixel architecture, is not required anymore. But again, the 4-tap pixel requires four physically separated output channels and thus, mismatch is a concern. The four individual read-out channels also require pixel-space, which reduces the fill-factor of the pixel. Table 1: Pixel characteristics of the four different tap-implementations. A of the new 0.6µm C7C technology from ZMD5 . ‡
2
Pixel size [µm ] Fill factor‡ [%] Miss-match problem Errors caused by changing light conditions during acquisition
‡
denotes an estimation based on parameters
1-tap 25 × 40 20 NO Strong
2-tap 25 × 55 15 YES Medium
4-tap 50 × 60 14 YES Irrelevant
4. TESTS OF THE LINE SENSOR BASED ON A TWO-TAP ARCHITECTURE For applications based on mobile robots a 3D-TOF line-sensor with 256 pixels using the 2-tap architecture was developed. In the following, tests performed with this line sensor are presented. First, the standard optical performance of the line-sensor was checked. As an example of these tests the linearity and sensitivity measurement results are displayed in figure 5. The line sensor was statically illuminated using an integrating sphere with a light source of 626 nm peak wavelength. The integration time was 2.15 ms. The fitted slope of 31.6 mV µW−1 cm2 is a direct measure of the sensitivity and is comparable to standard CCD devices. With a conversion capacitance of 40 fF this sensitivity translates to about 4 µV pro electron. This size of the read-out capacitance was selected as a good trade-off between sensitivity and full-well charge. 4 3.5
Ouput Voltage [V]
3 2.5 2 1.5 1
Figure 5: Sensitivity measurement of the line sensor.
fittet slope: 31.6 mV µW −1cm2
0.5 0 0
20
40
60 80 100 Optical Power [µW cm −2]
120
140
160
Then, a distance measurement example was taken in order to visualise the outcome of the 3D-TOF line camera. The left picture of figure 6 shows a photograph of the captured scene. The bright stripe in the picture represents the height of the line distance measurement. The labelled locations A, B and C are referenced to the corresponding distance plot acquired by the camera (on the right side of figure 6). It has to be mentioned that no data calibration or filtering was performed. The distance is scaled in centimetres on the y-axis. The following depth resolution test show the capabilities of the sensor as a distance measurement device. These tests were performed without background illumination. As described in detail in, 2 the depth resolution depends on the amount of light impinging on each pixel during the integration time. The number of electrons
Figure 6: Distance measurements with the TOF line sensor; left: picture of a scenery with referenced locations A, B and C; right: non-calibrated distance plot along the bright bar in the left picture.
stored in a pixel corresponds to the amount of light that reached the pixel during integration. It is therefore useful to analyse the depth resolution as a function of the number of electrons stored in one pixel. 200 consecutive measurements were performed with a static scene (white board) and the standard deviation of these 200 measurements was calculated for each individual pixel. The illumination source had a peak wavelength of 870nm. The result of these measurements is plotted in Figure 7. Due to the irradiance geometry of the illumination module used for the distance measurements, the central pixels of the line sensor are stronger illuminated than the pixels at the border. This results in strong intensityvariations over the line sensor. This effect, however, allows to cover most of the different illumination conditions with one single measurement. Figure 7 shows that a standard deviation better than 1 cm can be achieved if enough light is reflected back onto the sensor. The capture of 100’000 electrons (which corresponds to little reflected light) already guarantees a standard deviation of about 3 cm. Note, that the standard deviation is not calculated between the different pixels but over 200 measurements of the same pixel. As it can be seen in the right hand picture of figure 6, the difference between neighbouring pixels is slightly higher than the standard deviation of one single pixel itself. Therefore, off-chip calibrations become necessary in order to reach a reliable absolute distance accuracy.
5. APPLICATION IN ROBOTICS One of the many applications of the line sensor is in mobile robotic navigation. The most widely used sensor these days is the laser range scanner, which provides a 180◦ view in the plane of measurement. Two such sensors can be mounted opposite to each other to provide a full planar view. Elaborate Simultaneous Localization and Mapping (SLAM) algorithms have been demonstrated using this type of sensor 6 .
14
Standard deviation [cm]
12 10 8 6 4 2 0 0
1 2 3 Number of stored electrons per tap
4
5
Figure 7: Standard deviation of the distance measurements as a function of the number of electron stored per tap.
5
x 10
Table 2: Characteristics of the TOF Line Sensor and a typical Laser Range Scanner
Characteristic Range [m] Accuracy [mm] Horizontal fov [◦ ] Distance data points Intensity data points Measurement quality Speed [fps] Interface Weight [kg] Size [m3 ] Power consumption [W]
CSEM TOF Line Sensora 7.5 5b 120 256 256 Yes, for each pixel 70d USB 2 (480 Mb s−1 ) 0.3 0.145 × 0.032 × 0.04 12
SICK LMS 200 25 10 180 361 reflectivity datac No 50 RS422 (500 kb s−1 ) 4.5 0.155 × 0.156 × 0.21 20
a
Evaluation prototype Average for one pixel c Mainly used for on / off beacon extraction d Estimated value b
SLAM algorithms are probabilistic and require that each sensor input provides uncertainty information as well. In the case of the laser range scanners, a theoretical model is used to estimate the measurement accuracy. The new TOF line sensor not only delivers range and intensity measurements. It also delivers range accuracy (derived from equations 5 and 4). It has been demonstrated that for a large range of illumination levels the range accuracy is essentially only limited by the shot noise of the available light. 2 This allows to predict reliably the range resolution. In other words, for every measurement the measurement accuracy can be reliably predicted, which is necessary for state-of-the-art robot navigation. The new TOF line sensor presents advantages, in particular with respect to size, weight and power consumption, which are all key parameters for mobile applications. In order to better underline the capabilities of the line sensor especially for mobile robotics applications, it makes sense to compare it with a typical laser range scanner. Table 2 lists some of the key specifications of both sensors.
6. CONCLUSION AND OUTLOOK We presented a 3D measurement approach based on the time-of-flight measurement principle. It allows for a contact-free, simultaneous measurement of the phase, offset and amplitude of a radio frequency modulated light field that is emitted by the system and reflected back by the camera surroundings, without requiring any mechanical scanning parts. Three different pixel variations with different functional integration levels were introduced and their advantages and shortcomings with respect to speed, accuracy and their sensitivity to fast changing sceneries were analysed. An implementation as a line-sensor in an optimised CMOS/CCD technology was presented. With ideal illumination conditions, this sensor shows a distance resolution down to 5 mm. The line sensor will be incorporated on a mobile robot platform as a substitute for laser range scanners, which are traditionally used for navigation in dynamic and/or unknown environments. Expected advantages include a considerable reduction of system size, weight and power consumption, which are of utmost relevance for mobile robots. Currently, the CMOS/CCD TOF technology is transferred to a more aggressive 0.6 µm process, which will allow for the integration of even more functionality onto the sensor (system on chip) or for an increase of the sensor resolution. Moreover, new pixel structures are in development in order to further enhance the sensor performance.
REFERENCES 1. R. Lange and P. Seitz, “A solid state time-of-flight range camera,” IEEE Journal of Quantum Electronics 37, p. 390, 2001. 2. T. Oggier et al., “An all-solid-state optical range camera for 3d real-time imaging with sub-centimetre depth resolution (swissranger),” Proc. SPIE 5249, pp. 534–545, 2003. 3. R. Lange at al., “Demodulation pixels in ccd and cmos technologies for time-of-flight ranging,” Proc. SPIE 3965A, pp. 177–188, 2000. 4. K. Creath, “Phase-measurement interferometry techniques,” Progress on Optics XXVI, 1988. 5. www.zmd.de 6. E. N. J.E. Guivant, “Optimization of the simultaneous localization and map-building algorithm for real-time implementation,” IEEE Transaction on Robotics and Automation 17/3, pp. 242–257, 2001.
