Adv. Radio Sci., 5, 135–146, 2007 www.adv-radio-sci.net/5/135/2007/ © Author(s) 2007. This work is licensed under a Creative Commons License.
Advances in Radio Science
Multidimensional measurement by using 3-D PMD sensors T. Ringbeck, T. M¨oller, and B. Hagebeuker PMDTechnologies GmbH, Am Eichenhang 50, 57076 Siegen, Germany
Abstract. Optical Time-of-Flight measurement gives the possibility to enhance 2-D sensors by adding a third dimension using the PMD principle. Various applications in the automotive (e.g. pedestrian safety), industrial, robotics and multimedia fields require robust three-dimensional data (Schwarte et al., 2000). These applications, however, all have different requirements in terms of resolution, speed, distance and target characteristics. PMDTechnologies has developed 3-D sensors based on standard CMOS processes that can provide an optimized solution for a wide field of applications combined with high integration and cost-effective production. These sensors are realized in various layout formats from single pixel solutions for basic applications to low, middle and high resolution matrices for applications requiring more detailed data. Pixel pitches ranging from 10 micrometer up to a 300 micrometer or larger can be realized and give the opportunity to optimize the sensor chip depending on the application. One aspect of all optical sensors based on a time-of-flight principle is the necessity of handling background illumination. This can be achieved by various techniques, such as optical filters and active circuits on chip. The sensors’ usage of the in-pixel so-called SBI-circuitry (suppression of background illumination) makes it even possible to overcome the effects of bright ambient light. This paper focuses on this technical requirement. In Sect. 2 we will roughly describe the basic operation principle of PMD sensors. The technical challenges related to the system characteristics of an active optical ranging technique are described in Sect. 3, technical solutions and measurement results are then presented in Sect. 4. We finish this work with an overview of actual PMD sensors and their key parameters (Sect. 5) and some concluding remarks in Sect. 6.
Correspondence to: B. Hagebeuker (
[email protected])
1 1.1
Basic principle Time-Of-Flight basic principle
As the Time-Of-Flight principle has already been described in detail in many technical publications (Kraft et al., 2004; Ringbeck et al., 2003; Buxbaum et al., 2001), only the basic principles will be discussed here. Figure 1 shows the basic Time-Of-Flight principle. In its most simple form, a light pulse is transmitted by a sender unit and the target distance is measured by determining the turnaround time the pulse needs to travel from the sender to the target and back to the receiver. With knowledge of the speed of light the distance can then easily be calculated. However, the receiver needs to measure with picosecond-accuracy the delay between start and stop, if millimeter-precision is required. To realize such system solutions with discrete components, as is done in today’s TOF rangefinders, each component in the signal chain must have a very high system bandwidth. In contrast Figs. 1 and 2 show the basic working principle of a PMD based range imaging camera. Rather than using a single laser beam (which would have to be scanned over the scene to obtain 3-D) the entire scene is illuminated with modulated light. The advantage of PMD devices is that we can observe this illuminated scene with an intelligent pixel array, where each pixel can individually measure the turnaround time of the modulated light. Typically this is done by using continuous modulation and measuring the phase delay in each pixel. In addition to this robust method of obtaining 3-D without scanning, the realization of the phase measurement in a quasi optical domain offers huge advantages compared to the above mentioned discrete solutions. This is one reason, why we do not require an optical reference channel for most applications.
Published by Copernicus Publications on behalf of the URSI Landesausschuss in der Bundesrepublik Deutschland e.V.
We finish this work with an overview of actual PMD sensors and their key s.ding in Section 6. remarks in Section 6.
136
T. Ringbeck et al.: Multidimensional measurement by using 3-D PMD sensors
ciple
nciple of PMD eady been described
escribed
ns [6,7,8], only the he principle of a simplified PMD sensor realized only the ght principle. In its Because the complete mixing process of the smitted by a sender e. its place albyIntakes determining the within each pixel we call the PMD avelsender from the sender Figure 3 a)ofshows an illustration of a single pixel ning the With knowledge Figure 1: Time-Of-Flight measurement principle with pulsed light an optical input is be a calculated. five-terminal device with eteasily sender re with picosecondedge of Figure 1: Time-Of-Flight principle Fig. 1. Time-Of-Flight measurement principle with pulsed light. arent modulation in measurement the middle of the top, if millimeter-precision iselectrodes required. To realize such system solutions with culated. pulsed light y’s TOF rangefinders, each with component in the signal chain must have a very high htsecondsensitive Photogates are isolated from the ws the basicThe is emeter-precision layer. gates are To conductive transparent required. realize such and system solutions with ange imaging gefinders, each component in the signal chain must have very high On the left and the right there are readout adiodes beam (which ene to obtain pixel readout circuitry. In a PMD pixel the cthe th modulated charge carriers can be controlled by the reference gs that we can an intelligent h modulation gates. This way one can influence a individually ndulated light. eft or to the right side. The potential distribution d continuous nfluenced delay n in each by these push-pull voltages leading to a obtaining nteofgenerated charge carriers [6]. of the phase
y n offers huge Figure 2: Imaging Time-Of-Flight, measurement based on PMD et. mentionedFig. 2. Imaging Time-Of-Flight, measurement based on PMD. hy s we do not require an optical reference channel for most applications.
onstant and the modulation is a rectangular signal h50% the generated charge carriers within a g 1.2 Operation principle of PMD to the left and to the right equally. At the end of e This section describes principle of a simplified PMD sen-on Figure 3: Operation principle of PG-PMD ss the output voltages atthethe readout nodes are the 2: Imaging Time-Of-Flight, measurement based e Figure Fig. 3. Operation principle of PG-PMD. sor PMD realized in CMOS technology. Because the complete d Figure 3 mixing b). process of the electric and optical signal takes place an optical reference channel for most applications. tt require light is modulated as elements a rectangular signal), and there is no phase (phase = 0°) delay within each pixel(e.g. we callalso the PMD “smart pixels”. riers will be moved to one of both readout diodes (Fig. 3c). 3a shows an illustration of a single pixel PMD ight and Figure detector, then all charge carriers willsensor be moved to one of both readout diodes (Figure 3 For other phase delays, the modulation of the light and its element. It is a five-terminal device with an optical input winys, the modulation of themodulation light and its phase delay compared to the toelectrical signal phase delay compared the electricalreference reference signal redow, i.e. two transparent electrodes in the middle sults in a difference between the two output voltages. This etween the outputThese voltages. ThisPhotogates difference of thetwo illustration. light sensitive are iso-of both output nodes is directly dependent on difference of both output nodes is directly dependent on the lated from the substrate by a thin oxide layer. The gates are used n light and pixel modulation. This data can be todelay calculate theand distance from aThislightphase between light pixel modulation. data conductive and transparent for the received light. On the left ensor as and described in a later section. can be used to calculate the distance from a light-reflecting the right there are readout diodes which are connected to
sion of
object to the sensor as described in a later section. the pixel readout circuitry. In a PMD pixel the movement of generated charge carriers can be controlled by the reference Figure 4 gives an impression of how both readout nodes how both readout nodes will behave for different target distances. signal applied to the modulation gates. This way one can inwill behave for different target distances. fluence a charge transport to the left or to the right side. The potential distribution in the surface region is influenced by 1.3 Analysis of the ACF (autocorrelation function) these push-pull voltages leading to a “dynamic seesaw” for the generated charge carriers (Kraft et al., 2004). To calculate the distance between target and camera, the auIf the incident light is constant and the modulation is a tocorrelation function of electrical and optical signal is anarectangular signal with a duty cycle of 50% the generated lyzed by a phase-shift algorithm (Fig. 5). Using four samples charge carriers within a modulation period move to the left A1 , A2 , A3 and A4 – each shifted by 90 degrees – the phase, and to the right equally. At the end of such a modulation which is proportional to the distance, can be calculated using process the output voltages at the readout nodes are the same the following equation. as those shown in Fig. 3b. If, however, the incident light is modulated (e.g. also as a rectangular signal), and there is no phase (phase = 0◦ ) delay A1 − A3 ϕ = arctan (1) between modulation of light and detector, then all charge carA2 − A4 Adv. Radio Sci., 5, 135–146, 2007
www.adv-radio-sci.net/5/135/2007/
results in a difference between the two output voltages. This difference of both output nodes is directly dependent on the phase delay between light and pixel modulation. This data can be used to calculate the distance from a lightreflecting object to the sensor as described in a later section. T. Ringbeck et al.: Multidimensional measurement by using 3-D PMD sensors 137 Figure 4 gives an impression of how both readout nodes will behave for different target distances.
2.3. Analysis of the ACF (autocorrelation function)
To calculate the distance between target and camera, the autocorrelation function of electrical and optical signal is analyzed by a phase-shift algorithm (Figure 5). Using four samples A1, A2, A3 and A4 - each shifted by 90 degrees - the phase, which is proportional to the distance, can be calculated using the following equation.
⎛ A1 − A3 ⎞ ⎟⎟ A A − 2 4 ⎝ ⎠
ϕ = arctan⎜⎜
equation 2.1
In addition to the phase-shift of the signal, two other values can be extracted. At first, the signal strength of the received signal (amplitude):
a) Target distance: 0 m
( A1 − A3 ) 2 + ( A2 − A4 ) 2 a= 2.2 b) Target distance: 2equation m 2
And the second value, the offset b of the samples, represents the gray-scale value of each pixel.
b=
A1 + A2 + A3 + A4 4
equation 2.3
The distance d to the target is given by equation 2.4. At a modulation frequency of fmod=20MHz, for example, the wavelength λmod is 15 meters. The maximum distance for the target is dmax=λmod/2 because the active illumination has to cover the distance twice: from the sender to the target and back to the sensor chip.
d=
c ⋅ϕ 4π ⋅ f mod
c) Target distance: 5 m
equation 2.4 c) Target distance: 7.5 m
Figure 4: PMD operation with different target distances Fig. 4. PMD operation with different target distances.
Figure 5: ACF (autocorrelation function), SignalAmplitude Phase, Amplitude Fig. 5. ACF (autocorrelation function), Signal Phase, and Offset.and Offset
www.adv-radio-sci.net/5/135/2007/
Adv. Radio Sci., 5, 135–146, 2007
o generated electrons, both information holding electrons from the light source light. In most cases the accuracy can be improved linearly with aperture o generated electrons is a square function of the aperture. In very special cases, nt, quantization etc is dominant, the improvement of accuracy increases with theet al.: Multidimensional measurement by using 3-D PMD sensors 138 T. Ringbeck
nhanced by reflectivity signal, too. nt on the tial part of
prove the modulation ngth λmod). doubles the tations are e and the the contrast which will
etween the one. For the above were Figure 6: Calculated standard deviation versus measurement results ystem. The Fig. 6. Calculated standard deviation versus measurement results. fined environment conditions. A good match of both curves can be observed. ions can be easily estimated. on contrast ofIn PMD and light source, optimized filters,two highother modulation addition to the phase-shift ofoptical the signal, values and low noise floor are important design goals for PMD sensors and systems.
can be extracted. At first, the signal strength of the received signal (amplitude): p (A1 − A3 )2 + (A2 − A4 )2 is much brighter on a =than the active illumination. This has two negative impacts (2) 2 in distance noise) And the second value, the offset b of the samples, represents cs of the sensor is gray-scale occupied byvalue the sunlight (the sensor may saturate) the of each pixel. P P P P SUN 1+ 2+ 3+ 4 0.3 nm/Kb = (3) -40 °C 4 120 °C 160 K The distance d to the target is given by Eq. (4). At a mod48 nm ulation frequency of fmod =20 MHz, for example, the wave50 nm 98 nm length λmod is 15 m. The maximum distance for the target dmax =λmod /2 because the active illumination has to cover 20 deg is (diagonal) 40 m the distance twice: from the sender to the target and back to 1:1 10.0 m 99 m^2 the sensor chip. 100 LEDs c·ϕ 0.04 W/LED (4) 4W d = 4π · f mod 0.040 W/m^2 726 W/m^2 power density of sun
ed)
ght
0.039 5 20 0.20 0.19
W/m^2 Factor W W/m^2 W/m^2
2
83 W/m^2
power density of sun (filtered)
Challenges
2.1
Technical requirements: how much light for accurate measurement? ulation of the optical power of 100 LEDs in comparison to the sun for a typical
he ratio between sun and active light is 18,000=85db at a distance of 40m (20° It ishow obvious that can for be an improved active optical measurement on we will show this ratio by intelligent system system design. the reliability is the directly influenced the amount of (acally filtered. Although LEDs in near infrared have aby typical spectral width of motive temperature rangewhich requirements be taken account. tive) light arrives must at thealso sensor. In into Lange (2000)These and er width of approximately 100nmthe to dependency be used. In this the power density of Buxbaum (2002) ofcase, statistical measurement
uncertainty as a function of optical signal strength and system noise is derived. The measurement uncertainty dR of a PMD time of flight system can be calculated as: 1 1 dR = p · Nphase ktot ·
S N
λ mod ·√ 8·π
(5)
Where ktot is the modulation contrast, S is the number of (active) signal electrons, N is the equivalent number of noise Adv. Radio Sci., 5, 135–146, 2007
electrons (including shot noise of active light, background light and dark current as well as thermal noise, reset noise, system noise and ADC noise), Nphase is the number of measurements and λmod is the wavelength of the modulation sig0 nal in meters. An alternative notation of this product of ktot 0 0 and S is fully equivalent, where S represents the complete number of integrated electrons, including background light 0 represents the overall system conand dark current and ktot trast, also including dark current and background light offset. Please note that for both cases the product of ktot and S represents the “active signal share” after integration (demodulation amplitude). The helpful thing is that for each measurement, all parameters of the above equation are known. That means, that each single measurement more or less directly offers the information of its own reliability, without requiring the use of statistics. Subsequently, thresholds can be set to ignore measurement results with great uncertainty. One can also easily see from the equation above, what modifications of the system and sensor parameters can be made to improve the measurement accuracy. The following section describes the influence of the parameters mentioned above. High PMD contrast and high modulation contrast of the light sources improves the distance accuracy. Optimizing the contrast doesn’t influence the signal-to-noise ratio but rather decreases the inaccuracy by a factor of 1/ktot . The maximum value of kPMD and klight is 1. ktot = kPMD · klight
(6)
It is always desirable to lower the total noise without affecting the signal. This can be done by reducing the dark current noise. Therefore special opto-CMOS processes with low dark current should be used. Especially in automotive applications where the temperature can be up to 125◦ C, darkcurrent-optimized CMOS processes are essential for satisfactory performance because dark current doubles with every 8◦ C. Improvements are also achieved by low amplifier- and reset-noise on-chip and low system-noise, as well as decreasing background illumination noise. However, since background illumination is an environmental parameter and not a system parameter, it is difficult to decrease the background illumination in an observed scene. The number of photons reaching the sensor can be reduced though. This is done by proper band pass filtering of the incoming light. Increasing the signal by using stronger light sources increases the accuracy linearly if the photon shot noise of the uncorrelated light or other noise sources are dominant rather than the photon shot noise of the active illumination. Otherwise, if the photon shot noise Nactive of the active illumination is the most dominant part of the total noise N, the distance accuracy increases by the square root of the signal. √ S S, if Nactive > N ∝ (7) S, else N www.adv-radio-sci.net/5/135/2007/
For most outdoor applications sunlight is much brighter than the active illumination. This has two negative impacts on the TOF-system: • Shot noise is increased (results in distance T. Ringbeck et al.: Multidimensional measurement by noise) using 3-D PMD sensors 139 • A huge amount of the dynamics of the sensor is occupied by the sunlight (the sensor may saturate) LED LED Temp. Automotive min Temp. Automotive max Temp. Automotive range
LED Filter Viewing angle Range Image Format edge of image (object space) area of image (object space) #LEDS optical power per LED total optical power of LEDs (CW) power density in object space (CW) power density in object space (CW, filtered) Burst-Operation power desnisty in object space (Burst) power desnisty in object space (Burst, filtered)
SUN 0.3 -40 120 160 48
nm/K °C °C K nm
50 nm 98 nm 20 40 1:1 10.0 99 100 0.04 4 0.040 0.039 5 20 0.20 0.19
deg (diagonal) m m m^2 LEDs W/LED W W/m^2 W/m^2 Factor W W/m^2 W/m^2
726 W/m^2 83 W/m^2
power density of sun power density of sun (filtered)
Figure 7: Calculation sunlight vs. active light Fig. 7. Calculation sunlight vs. active light.
Figure 7 gives an overview of the calculation of the optical power of 100 LEDs in comparison to the sun for a typical application scenario. In this example the ratio between sun and active light is 18,000=85db at a distance of 40m (20° shows the comparison betweensystem the calculated It must be kept in mind, thatfollowing using more optical comeshow thisFigure viewing angle). In the section wepower will show ratio 6can be improved by intelligent design.acat aFirstly, price, and the overall of is a TOF camera systemAlthough can curacyinand measured the calculation, the paramthe optical inputcosts signal spectrally filtered. LEDs thethe near infraredone. haveFor a typical spectral width of etersrequirements described above identified for account. a 160×120 pixel significantly rise by adding moreand LED or lasers.temperature To avoid range 50nm, thermal drift of LEDs automotive must were also be taken into These camera system. measurement wellofdethis, burst operation be implemented. In burst facts allow only acan medium spectral filter widthoperation, of approximately 100nm to beThe used. In this case,was the done powerunder density fined environment conditions. A good match of both curves the light sources have the same average power and therecan be observed. Hence, the performance of new applicafore the same number of light emitting elements, but higher tions can be easily estimated. peak power than in continuous operation. The shorter exposure time leads to the same number of signal electrons but We can summarize that high modulation contrast of PMD less noise electrons due to less accumulated background light and light source, optimized optical filters, high modulation electrons. Thus, burst operation is effective under strong frequency, good fill factor, large pixels and low noise floor background light conditions, such as outdoor applications are important design goals for PMD sensors and systems. with sunlight. Another advantage of burst operation is the ability of detecting fast moving objects. 2.2 Influence of sunlight Large lens apertures lead to more photo generated elecFor most outdoor applications sunlight is much brighter than trons, both information holding electrons from the light the active illumination. This has two negative impacts on the source and unwanted ones from background light. In most TOF-system: cases the accuracy can be improved linearly with aperture enlargement since the number of photo generated electrons – Shot noise is increased (results in distance noise) is a square function of the aperture. In very special cases, when the noise floor due to dark current, quantization etc is – A huge amount of the dynamics of the sensor is occudominant, the improvement of accuracy increases with the pied by the sunlight (the sensor may saturate) square of the aperture. Sensitivity of the PMD sensor can be enhanced by high fill Figure 7 gives an overview of the calculation of the optical factor and micro lenses. Good reflectivity and close targets power of 100 LEDs in comparison to the sun for a typical apwill increase the signal, too. However, this is strongly depenplication scenario. In this example the ratio between sun and dent on the application and therefore not an essential part of active light is 18 000=85 db at a distance of 40 m (20◦ viewdesign. ing angle). In the following section we will show how this raAnother powerful method to improve the measurement actio can be improved by intelligent system design. Firstly, the curacy is to use higher modulation frequencies (lower modoptical input signal is spectrally filtered. Although LEDs in ulation wavelength λmod ). For example doubling the frethe near infrared have a typical spectral width of 50 nm, therquency doubles the measurement accuracy. Physical limimal drift of LEDs and automotive temperature range requiretations are the bandwidth of the light source and the bandments must also be taken into account. These facts allow width of the sensor. Furthermore the contrast may decrease only a medium spectral filter width of approximately 100 nm at high frequencies, which will lower the effectiveness of this to be used. In this case, the power density of the sun on the method. sensor is lowered and the ratio sun/active light goes down to
www.adv-radio-sci.net/5/135/2007/
Adv. Radio Sci., 5, 135–146, 2007
un/active light goes down to 2,000=66db. Further improvements can be the sun on the sensor is lowered and the ratio sun/active light goes down to 2,000=66db. Further improvements can be . A burst operationachieved of factor 5 results in a sunlight/active-light ratio of by using a burst mode for the LEDs. A burst operation of factor 5 results in a sunlight/active-light ratio of 140 400=52db. T. Ringbeck et al.: Multidimensional measurement by using 3-D PMD sensors
Figure 8: Optical spectrum of sunlight with spectral filtering and active light in burst-mode
Fig. 8. Optical spectrum of sunlight with spectral filtering and active light in burst-mode. ral filtering and active light8inshows burst-mode Figure the result of both enhancements in a diagram and Figure 9 shows the effects on an example Figure 8 shows the result of both enhancements in a dints in a picture.Spectral filtering and burst operation of the light agram and Fig. 9 shows the effects on an example picture. source are effective methods to improve the “optical” example SNR. The reasonable sunlight rejection of these Spectral filtering and burst operation of the light source are the light introduced methods is about -33db. But the remaining effective methods to improve the “optical” SNR. The reason“optical” ratio of sunlight /active signal is still poor: Even the able sunlight rejection of these introduced methods is about of these filtered sun is still a factor of 400 brighter than the maining −33 db. But the remaining ratio of sunlight/active signal is pulsed LEDs. Additional sunlight rejection methods Even the still poor: Even the filtered sun is still a factor of 400 brighter have to be found.
than the methods
ment are vity and sunlight
The key challenges for optical TOF measurement are sensors with a high bandwidth and high sensitivity and a system with high dynamics to achieve sunlight robustness.
than the pulsed LEDs. Additional sunlight rejection methods have to be found. The key challenges for optical TOF measurement are sensors with a high bandwidth and high sensitivity and a system with high dynamics to achieve sunlight robustness. 3 3.1
Figure 9: Usage of optical filters and burst mode Fig. 9. Usage of optical filters and burst mode.
Technical solutions Definition of contrast
Considering the output channels of the PMD sensor, two values are of importance: the difference of the readout channels delivering the phase information, and the sum of both chanFigure Usage of filters and burst mode light and the overnels9:which is optical proportional to the incident all photo generated charge carriers. As mentioned above, an important figure of merit for distance measurement is the ratio of the measured signal difference compared to the signal sum, the PMD contrast kPMD . kPMD =
2000=66 db. Further improvements can be achieved by using a burst mode for the LEDs. A burst operation of factor 5 results in a sunlight/active-light ratio of 400=52 db. Adv. Radio Sci., 5, 135–146, 2007
|QA − QB | |1VA − 1VB | = QA + QB 1VA + 1VB
(8)
The contrast has its maximum value when using phase shifts of 0◦ and 180◦ in terms of electrical and optical signal. www.adv-radio-sci.net/5/135/2007/
carriers contribute to the phase measurement since they are properly distributed to the corresponding readout ls (perfect device and system). Therefore, the contrast reflects the separation ability of the device in terms of carriers generated by correlated light. By definition the contrast is significantly affected by the signal sum, re determination of the contrast ismeasurement only validbywithout illumination, and so the T. Ringbeck et al.:device Multidimensional using 3-Dany PMDuncorrelated sensors 141 ement must be carried out with a modulated light signal only. ontrast of a PMD is mainly dependent el architecture and ry. The main focus pixel design is the active region of ice which is defined light sensitive area he light sensitive ation gates. Of other requirements lso be met, such as pixel fill factor and esponsivity. Our pment has led to pixels with scalable itch without a loss of mance. The active ize can be modified on application ments from 10µm x up to 300µm x or even larger. This is based upon a kind of pixel Fig. layout 10. Exemplary contrast showingcontrast high contrasts combined with high bandwidth. Figure 10: behaviour Exemplary behaviour showing high contrasts combined with D pixels with large high bandwidth ions can therefore Depending onasmodulation waveform, an idealare device can behaviour of a typical (beyond 2-gate PMD Theto contrast of 3he same performance small designs. The pixels optimized for high bandwidth 100pixel. MHz) have a maximum contrast of 100%, that is, all photo gengate pixels is typically higher (60%). It can be e pixel performance and accuracy. Figure 10 shows exemplary contrast behaviour of a typical 2-gate PMD pixel. seen that a eratedpixels chargeiscarriers contribute the phase measurement cancan be be attained overover several decades of modntrast of 3-gate typically higherto(60%). It can be seen that ahigh highcontrast contrast attained several since they are properly distributed to the corresponding readulation frequency which is mandatory for accurate distance s of modulation frequency which is mandatory for accurate distance measurements (equation 3.1).
out channels (perfect device and system). Therefore, the conmeasurements (Eq. 5). trast reflects the separation ability of the device in terms of 3.2 SBI – Suppression of background illumination charge carriers by correlated light. By definition SBI - Suppression ofgenerated background illumination the contrast is significantly affected by the signal sum, thereThe mentioned opticaldisturbing filtering anduncorrelated bursting of light sources entioned optical filtering andof bursting light is sources might not be sufficient to reduce fore determination the deviceof contrast only valid without might not be sufficient to reduce disturbing sunany uncorrelated illumination, and socan the measurement t. The remaining background illumination still saturatemust the sensor for typical integration times whichuncorrelated can light. The remaining background illumination can still satuout with a modulated light only. there are other causes for uncorrelated disturbances. The a decreasebeofcarried accuracy or pixel failures. Insignal addition, rate the sensorinfluence for typicalparticularly integration times which can lead to dable dark current of a semiconductor device especially, can have a considerable at higher a decrease of accuracy or pixel failures. In addition, The contrast of a PMD device is mainly dependent on atures. For example, an increase in temperature of 80 degrees leads to a dark current which can be 1000 times there are other causes for uncorrelated disturbances. pixel architecture anddecrease geometry. The main focus duringObviously and thus would significantly measurement accuracy. solutions must be found, in orderThe tounavoidable dark current of a semiconductor device especially, can have pixel design is always the active region of the device which the disturbing influence of any uncorrelated signal. a considerable influence particularly at higher temperatures. is defined by the light sensitive area and the light sensitive For example, an increase in temperature of 80 degrees leads modulation gates. Of course, other requirements must also to a dark current which can be 1000 times higher and thus be met, such as overall pixel fill factor and photoresponsivwould significantly decrease measurement accuracy. Obviity. Our development has led to PMD pixels with scalable ously solutions must be found, in order to reduce the disturbpixel pitch without a loss of performance. The active pixel ing influence of any uncorrelated signal. size can be modified based on application requirements from 10 µm×10 µm up to 300 µm×300 µm or even larger. This The possibility to combine the sensor with on-chip feature is based upon a special kind of pixel layout – PMD application-specific circuitry is fundamental for an enhanced pixels with large dimensions can therefore offer the same smart optical sensor. To improve the dynamic range of the performance as small designs. The pixels are optimized for PMD device, additional circuitry for the instantaneous suphigh bandwidth (beyond 100 MHz) to increase pixel perforpression of background illumination (SBI) has been intromance and accuracy. Figure 10 shows exemplary contrast duced. Using such a special readout technique allows precise www.adv-radio-sci.net/5/135/2007/
Adv. Radio Sci., 5, 135–146, 2007
measurements even in environments with huge uncorrelated signals at high temperatures. Therefore this principle is a key feature for a number of range imaging applications where the existence of disturbing uncorrelated signals is unavoidable. The working principle of PMDs patented SBI circuitry is illustrated in the following figure: 142
T. Ringbeck et al.: Multidimensional measurement by using 3-D PMD sensors
SBI
Figure Figure 12 shows the resulting standard deviation as a function of signal and background intensity for a for single PMD PMD 12 shows the resulting standard deviation as a function of signal and background intensity a single pixel with SBI. The overlaid saturation boundaries indicate the achieved dynamic range,range, limitedlimited by saturation pixeldisabled with disabled SBI. The overlaid saturation boundaries indicate the achieved dynamic by saturation Figure Figure 13 shows the increased dynamic range range with enabled SBI circuitry. The The of integration capacitors. In contrast, 13 shows the increased dynamic with enabled SBI circuitry. of integration capacitors. In contrast, dynamic range of a typical pixel can be can increased by a factor of 300of- that 50dB. Since Since qualityquality of SBIofperformance dynamic range of a typical pixel be increased by a factor 300 is, - that is, 50dB. SBI performance may vary pixeltodue to due technical fabrication tolerances, each single pixel may a slightly different mayfrom varypixel fromtopixel pixel to technical fabrication tolerances, each single pixeldeliver may deliver a slightly different no SBI SBIperformance dynamic range. Please note, that there are always pixels pixels in the in array higher SBI performance (factor(factor 1000 1000 and and dynamic range. Please note, that there are always the with arrayfar with far higher SBI even more). Therefore, in order yieldto ayield morea complete result, result, measurement shouldshould be extended from single pixel pixel even more). Therefore, in to order more complete measurement be extended from single Fig. 11. Working Principle of PMDTec’s patented SBI circuitry. characterisation to parallel characterisation of all pixels ofSBI a PMD characterisation to parallel of all pixels of a array PMD with arraySBI withcircuitry. SBI circuitry. Figure 11: Working Principle ofcharacterisation PMDTec’s patented circuitry
To eliminate uncorrelated signals which are distributed on both readout sides equally, additional compensation sources have been added. These sources inject additional charges in both readout terminals in order to instantaneously compensate for the saturation effects of uncorrelated signals during the integration process. This compensation means that the dynamic range of the following readout stage can be completely used by the correlated signal information only. In other words, the readout circuitry of the pixel ideally sees only the correlated signal and no unwanted background DC component. To offer a wide range of applications and flexibility this concept has been implemented in most of our PG-PMD arrays (Figure 16). An active SBI circuitry will reduce the background DC component, and so the scenes ambient light greyscale information will be lost. Alternatively a similar greyscale image can also be extracted out of the acquired amplitude data (active light greyscale image), even with active SBI. Also the SBI feature can be electronically deactivated or adjusted so that it is always possible to acquire both high-quality 3D measurement and 2D greyscale information, depending on the application needs.
4.3. Measurements
To measure the functionality of PMD sensors with SBI circuitry, a phase measurement procedure must be used. This is Fig.fact 12. Dynamic Measurements (SP, noofSBI). Fig.Figure 13. Dynamic Measurements (SP, SBI). due to the that the total number collected electrons is modified by the usage of an active SBI, and so Figure 12: Dynamic Measurements no SBI) and stored 13: Dynamic Measurements (SP, SBI) Figure 12: Dynamic Measurements (SP, no(SP, SBI) Figure 13: Dynamic Measurements (SP, SBI) the standard definition of contrast (equation 4.1) can no longer be used as characteristic device performance parameter. pletely used by thebackground correlated information only.a(sensor In To examine the achieved dynamic range (with regard to tolerable signal and intensity levels), real (TimeThe3-D diagram in Figure the distribution of phase value standard deviation for asignal complete PMD array The diagram in Figure 14 shows the distribution phase value standard deviation for a complete PMD array (sensor measurements even14in shows environments withof huge uncorreother words, the readout circuitry of the pixel ideally sees Of-Flight) phase measurement using a 4-phase algorithm is performed under controlled conditions (equation PhotonICs 1kS2) for fixed signal and background illumination levels. Statistical evaluation indicates that, 100% 4.2). of the
PhotonICslated 1kS2) for fixed and background evaluation indicates that,background 100% of the signals at highsignal temperatures. Therefore illumination this principle levels. onlyStatistical the correlated signal and no unwanted DC pixels a accuracy phase accuracy of lessimaging 4applications degrees for a signal photo current of 200pA, a background photo current is a achieve key feature for a number of range pixels achieve a phase of less than 4 than degrees for a signal photo current of 200pA, a background photo current component. To offer a wide range of applications and flexi⎛ ∑ u Int (ϕi ) ⋅ sin (ϕi ) ⎞ of 20nA and an integration time of 1ms. ⎜ ⎟ where the existencetime of disturbing signals is unof 20nA and an integration of 1ms. uncorrelated equation 4.2 in most of our PGbility this concept has been implemented ϕ = arctan ⎜ − i ⎟ avoidable. The working principle of TOF PMDs patented ∑ SBI cir( ) ( ) ϕ ⋅ cos ϕ u PMD arrays (Fig. 16). An active SBI circuitry will reduce Int i i ⎟ ⎜
⎝ i ⎠ cuitry is illustrated in the following figure: the background DC component, and so the scenes ambient ϕi = 0are , 12 πdistributed , π , 32 π To eliminate uncorrelated signals which on light greyscale information will be lost. Alternatively a simiboth readout sides equally, additional compensation sources lar greyscale image can also be extracted out of the acquired Using anhave automated measuring station, the intensities and background light required for even measurement can be been added. These sources inject additional chargesof in signal amplitude data (active light greyscale image), with acboth readout terminals in order to instantaneously compentive SBI. Also the SBI feature can be electronically deactiadjusted very reproducibly in our optical laboratory. To verify reliable phase measurement using subsequent statistic for the saturation of uncorrelated during or adjusted soof that it is always possible to acquire both analysis,sate a high number of effects measurements are signals made for severalvated combinations signal and background intensity. the integration process. This compensation means that the high-quality 3-D measurement and 2-D greyscale informadynamic range of the following readout stage can be comtion, depending on the application needs. Adv. Radio Sci., 5, 135–146, 2007
Figure 14: Statistic analysis (64x16 Matrix, SBI) Figure 14: Statistic analysis (64x16 Matrix, SBI)
www.adv-radio-sci.net/5/135/2007/
PhotonICs 1kS2) for fixed signal and background illumination levels. Statistical evaluation indicates that, 100% of the pixels achieve a phase accuracy of less than 4 degrees for a signal photo current of 200pA, a background photo current of 20nA and an integration time of 1ms. T. Ringbeck et al.: Multidimensional measurement by using 3-D PMD sensors 143
5.
Figure 14: Statistic analysis (64x16 Matrix, SBI) Fig. 14. Statistic analysis (64×16 Matrix, SBI).
For indoor and outdoor applications with different requirements, PMDTechnologies has developed sensors with specifi features, like resolution, optical format and SBI. A short overview of four sensors with optical and mechanica dimension is given in the following figure. The sensors can be ordered for evaluation purposes.
array , for a Hz, an tandard s to a tions of 99.3% ked in pixels, an 5°. n 95% ed red. ledger al ratio groundthat is % of all Figure 15: Dynamic range (64x16 Matrix, SBI) Fig. 15. Dynamic range (64×16 Matrix, SBI). rement tatistics and intelligent filter algorithms. These measurement results are based on
3.3
Measurements
To measure the functionality of PMD sensors with SBI circuitry, a phase measurement procedure must be used. This is due to the fact that the total number of collected and stored electrons is modified by the usage of an active SBI, and so the standard definition of contrast (Eq. 8) can no longer be used as characteristic device performance parameter. To examine the achieved dynamic range (with regard to tolerable signal and background intensity levels), a real (Time-Of-Flight) phase measurement using a 4-phase algorithm is performed under controlled conditions (Eq. 9). P ! uInt (ϕi )·sin(ϕi )
ϕTOF = arctan − Pi u
Int (ϕi )·cos(ϕi )
i
ϕi
= 0,
PhotonICs® PMD 768-S 3D Video Sensor Array with Active SBI • Array format 24(H) x 32(V) • Optical format 1/2’’, 16:9 image ratio • Effective image area 6.75mm (H) x 3.73mm (V) • Package Standard CLCC68 • Pixel size 210.8µm (H) x 155.3µm (V) • Fill factor (FF) 42% • 78dB dynamic range for active illumination • 50dB dynamic range increase by SBI circuitry PhotonICs® PMD 1k-S 2 3D Video Sensor Array with Active SBI • Array format 64(H) x 16(V) • Optical format 2/3’’, 3:1 image ratio • Effective image area 9.94mm (H) x 3.37mm (V) • Package Standard CLCC68 • Pixel size 155.3µm (H) x 210.8µm (V) • Fill factor (FF) 42% • 78dB dynamic range for active illumination • 50dB dynamic range increase by SBI circuitry PhotonICs® PMD 3k-S 3D Video Sensor Array with Active SBI • Array format 64(H) x 48(V) • Optical format 1/2’’, 4:3 image ratio • Effective image area 6.4mm (H) x 4.8mm (V) • Package Standard CLCC68 • Pixel size 100µm (H) x 100µm (V) • Fill factor (FF) 32% • 74dB dynamic range for active illumination • 50dB dynamic range increase by SBI circuitry PhotonICs® PMD 19k High Resolution 3D Video Sensor Array • Array format 160(H) x 120(V) • Optical format 1/2’’, 4:3 image ratio • Effective image area 6.4mm (H) x 4.8mm (V) • Package Standard CLCC68 • Pixel size 40µm (H) x 40µm (V) • Fill factor (FF) 31% • 65dB dynamic range for active illumination
Figure 16: PMD Sensor-Chips based on Photogate Technology
Fig. 16. PMD Sensor-Chips based on Photogate Technology.
All of these PMD Sensors are implemented in camera prototypes which shows the basic function principle o furthermore they are implemented in camera products like the PhotonICs® PMD 3k-S. In the next chapter, a camer system for pedestrian safety is introduced, which based on the PMD Sensor PhotonICs® PMD 1k-S.
(9)
1 3 2 π, π, 2 π
Using an automated measuring station, the intensities of signal and background light required for measurement can be adjusted very reproducibly in our optical laboratory. To verify reliable phase measurement using subsequent statistic analysis, a high number of measurements are made for several combinations of signal and background intensity. Figure 12 shows the resulting standard deviation as a function of signal and background intensity for a single PMD www.adv-radio-sci.net/5/135/2007/
TECHNICAL IMPLEMENTATION
5.1. Realized PMD Sensors
pixel with disabled SBI. The overlaid saturation boundaries indicate the achieved dynamic range, limited by saturation of integration capacitors. In contrast, Fig. 13 shows the increased dynamic range with enabled SBI circuitry. The dynamic range of a typical pixel can be increased by a factor of 300 – that is, 50 dB. Since quality of SBI performance may vary from pixel to pixel due to technical fabrication tolerances, each single pixel may deliver a slightly different dynamic range. Please note, that there are always pixels in the array with far higher SBI performance (factor 1000 and even Adv. Radio Sci., 5, 135–146, 2007
In Figure 17, a camera system for pedestrian safety is depicted. This A-Muster camera is designed to demonstrate solutions for several front view application like Stop&Go, PreCrash and Pedestrian Saftey therefore additional IRillumination sources are implemented inside the head etlights to increase transmitted powerbyand mearsurement range 144 T. Ringbeck al.: Multidimensional measurement using 3-D PMD sensors respectively. Other design spaces for illumination are determined and tested until beginning of 2007.
Ethernet
Sender-Modulation (LVDS) IR-Scheinwerfer
PMD-3D-Kamera compact-light-source 12-Volt
IR-Scheinwerfer
Figure 17: Test car from 2005 with an A-Muster PMD camera system for pedestrian safety Fig. 17. Test car from 2005 with an A-Muster PMD camera system for pedestrian safety.
An actual sensor is available with the specification depicted in Figure 18. The measurement range depends on illumination source. A range of 10m is realized an Illumination with 1W optical power to the camera more). Therefore, in order to yield a more completeby result, than 95%Source of all pixels. Please consider, thatnext the measurement system. A measurement range offrom above 35mpixel could be realized with illumination source in front the carand (i.g.intelligent inside the measurement should be extended single characcertainty can be further increased by of statistics head lights) with and optical powerofmore than of 8W. terisation to parallel characterisation all pixels a PMD filter algorithms. These measurement results are based on an array with SBI circuitry. integration Range time of 250 µs. Measurement 10m / 35m The diagram in Fig. 14 shows the distribution of phase 52° vertical value standard deviation for a complete PMD array (sensor Field of view 18° horizontal PhotonICs 1kS2) for fixed signal and background illumina4 Technical implementation tion levels. Statistical evaluation indicates that, 100% of the 10,4ms (96Hz) pixels achieve a phase accuracy of less than 4 degrees for a DataUpdate Rate 4.1 Realized PMD Sensors signal photo current of 200 pA, a background photo current Distance Resolution per Pixel ±10cm of 20 nA and an integration time of 1ms. For indoor and outdoor applications with different requireFigure 15 illustrates the SBI array performance for all ments, PMDTechnologies has developed sensors with spetested intensities, for a given modulation frequency of Operation Current of camera module Max. 600mA cific features, like resolution, optical format and SBI. A short 10 MHz, an integration time of 1 ms and a standard devioverview of four sensors with optical and mechanical Temperature Range -10° to 85°C dimenation of 5◦ , which corresponds to a distance resolution of sion is given in the following figure. The sensors can or(-40°to 105° inbe 2007) 20 cm. Combinations of intensities that result in at least dered for evaluation purposes. 99.3% working pixels (5σ -range) are marked in green. That Voltage 11 bis 18 DC All of these PMD Sensors are implemented in camera means for 99.3% of all pixels, the standard deviation is less Operating prototypes which shows the basic function principle or furthan 5◦ . Combinations that result in less than 95% working thermore they are implemented in camera products like the pixels (3σ -range) are coloured red. To facilitate measureFigure 18: Technical Specification of A-Muster PhotonICs® PMD 3k-S. In the next chapter, a camera sysment results, ledger lines of constant background-to-signal tem for pedestrian safety is introduced, which based on the ratio are overlaid. As can be seen, a background-to-signal ratio up to a factor of 400, that is 52 db, is achieved for more PMD Sensor PhotonICs® PMD 1k-S. Adv. Radio Sci., 5, 135–146, 2007
www.adv-radio-sci.net/5/135/2007/
An actual sensor is available with the specification depicted in Figure 18. The measurement range depends on illumination source. A range of 10m is realized by an Illumination Source with 1W optical power next to the camera T. Ringbeck et al.: Multidimensional measurement 3-Dwith PMDillumination sensors 145 system. A measurement range of above 35m couldby beusing realized source in front of the car (i.g. inside the head lights) with and optical power more than 8W. Measurement Range 10m / 35m Field of view
52° vertical 18° horizontal
DataUpdate Rate
10,4ms (96Hz)
Distance Resolution per Pixel
±10cm
Operation Current of camera module
Max. 600mA
Temperature Range
-10° to 85°C (-40°to 105° in 2007)
Operating Voltage
11 bis 18 DC
Figure 18: Technical Specification of A-Muster Fig. 18. Technical Specification of A-Muster.
4.2
Pedestrian safety camera system
In Fig. 17, a camera system for pedestrian safety is depicted. This A-Muster camera is designed to demonstrate solutions for several front view application like Stop&Go, PreCrash and Pedestrian Saftey therefore additional IRillumination sources are implemented inside the head lights to increase transmitted power and mearsurement range respectively. Other design spaces for illumination are determined and tested until beginning of 2007. An actual sensor is available with the specification depicted in Fig. 18. The measurement range depends on illumination source. A range of 10 m is realized by an Illumination Source with 1 W optical power next to the camera system. A measurement range of above 35 m could be realized with illumination source in front of the car (i.g. inside the head lights) with and optical power more than 8 W. 5
Conclusion
Time-Of-Flight systems based on the PMD-principle give the possibility of fast 3-D measurement with customizable resolutions depending on the application. High bandwidth and high sensitivity of the PMD pixels are the fundamentals for enhanced predictable measurement accuracy. Many applications (like outdoor) require robust operation even under very bright ambient light conditions. Spectral filtering combined with an active illumination in burst-mode improve the signal to noise ratio and are effective methods to reduce the impact of ambient light. However, for typical outdoor applications, additional sunlight rejection methods are necessary. PMDTechnologies has realized an outstanding ambient light rejection method, the active SBI circuitry realized as compact in-pixel electronics. This SBI circuitry removes all uncorrewww.adv-radio-sci.net/5/135/2007/
lated signals instantaneously during the integration process. It is approved in the 6th sensor generation and works very reliable. The combination of all the presented techniques provides robust 3-D measurement under full sunlight conditions.
References Buxbaum, B.: Optische Laufzeitentfernungsmessung und CDMA auf Basis der PMD-Technologie mittels phasenvariabler PNModulation, Ph.D. dissertation, University of Siegen, Germany, 2002. Buxbaum, B. and Hagebeuker, B.: Dreidimensionale UmfeldErfassung, Elektronik Automotive, Nr. 5, pp. 77, Sep., 2005. Buxbaum, B., Schwarte, R., Ringbeck, T., Grothof, M., and Luan, X.: MSM-PMD as correlation receiver in a new 3D-ranging system, SPIE – Remote Sensing, Laser Radar Techniques: Ranging and Atmospheric Lidar, Toulouse, 2001. Kraft, H., Frey, J., Moeller, T., Albrecht, M., Grothof, M., Schink, B., and Hess, H.: Universit¨at Siegen, B. Buxbaum, PMDTechnologies GmbH Siegen, 3D-Camera of High 3D-Frame Rate, Depth-Resolution and Background Light Elimination Based on Improved PMD (Photonic Mixer Device)-Technologies, OPTO 2004, AMA Fachverband, N¨urnberg, 2004. Lange, R.: 3D Time-of-flight distance measurement with custom solid-state image sensors in CMOS/CCD technology, Ph.D. dissertation, University of Siegen, Germany, 2000. Lange, R., Seitz, P., Schwarte, R., et al.: Time-of-flight range imaging with a custom solid state image sensor, Proceedings of the SPIE, vol. 3823, pp. 180–191, Munich, 1999. Ringbeck, T., Albrecht, M., Frey, J., Grothof, M., Heß, H., Kraft, H., M¨oller, T., Mosen, J., and Schink, B.: Time-of-Flight 3Dcamera for autonomous navigation and industrial automation, N¨urnberg, Sensor, 2003. Schwarte, R., Xu Z., Heinol H., et al.: A new active 3D-Vision system based on rf-modulation interferometry of incoherent light,
Adv. Radio Sci., 5, 135–146, 2007
146
T. Ringbeck et al.: Multidimensional measurement by using 3-D PMD sensors
SPIE – Intelligent Robots and Computer Vision XIV, vol. 2588, p. 126–134, Philadelphia, 1995. Schwarte, R., Buxbaum, B., Heinol, H., Xu, Z., Schulte, J., Riedel, H., Steiner, R., Scherer, M., Schneider, B., and Ringbeck, T.: New Powerful Sensory Tool in Automotive Safety Systems Based on PMD-Technology, AMAA – Advanced Microsystems for Automotive Applications 2000, Berlin, 2000.
Adv. Radio Sci., 5, 135–146, 2007
Spirig, T., Marley, M., Seitz, P., et al.: The multitap lock-in CCD with offset subtraction, IEEE Transactions on electron devices, vol. 44, no. 10, pp. 1643–1647, 1997.
www.adv-radio-sci.net/5/135/2007/
