Comparative Evaluation of Range Sensor Accuracy in Indoor Environments Todor Stoyanov, Athanasia Louloudi, Henrik Andreasson and Achim J. Lilienthal
Center for Applied Autonomous Sensor Systems (AASS) Örebro University, Sweden
[email protected]
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
1 / 15
Range Sensor Accuracy 3D range sensors are becoming widely available ... but how well do the new sensors perform?
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
2 / 15
Range Sensor Accuracy 3D range sensors are becoming widely available ... but how well do the new sensors perform?
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
2 / 15
Outline
1
Evaluation Methodology
2
Evaluation Results Sensor Setup Evaluation with Known Ground Truth Evaluation in an Uncontrolled Environment
3
Conclusions and Future Work
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
3 / 15
Evaluation Methodology
Outline
1
Evaluation Methodology
2
Evaluation Results Sensor Setup Evaluation with Known Ground Truth Evaluation in an Uncontrolled Environment
3
Conclusions and Future Work
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
4 / 15
Evaluation Methodology
Accuracy Evaluation — Overview Range sensor accuracy reported by manufacturer in strictly controlled environments Often combinations of factors influence accuracy Evaluations usually done in restricted scenarios simple, usually flat targets easier to model and describe error sources beneficial for calibration and error correction
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
4 / 15
Evaluation Methodology
Accuracy Evaluation — Overview Range sensor accuracy reported by manufacturer in strictly controlled environments Often combinations of factors influence accuracy Evaluations usually done in restricted scenarios simple, usually flat targets easier to model and describe error sources beneficial for calibration and error correction
Motivation: Develop a method to compare the performance of 3D range scanners in uncontrolled environments
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
4 / 15
Evaluation Methodology
Evaluation Methodology A simple 2D example will be used to illustrate the evaluation procedure Given a set of points collected by a reference range device . . . and a set of points collected by a test range sensor construct the ground truth model M Generate positive and negative samples from the test point set Evaluate occupancy probability, classify based on threshold
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
5 / 15
Evaluation Methodology
Evaluation Methodology Given a set of points collected by a reference range device . . . and a set of points collected by a test range sensor construct the ground truth model M Generate positive and negative samples from the test point set Evaluate occupancy probability, classify based on threshold
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
5 / 15
Evaluation Methodology
Evaluation Methodology Given a set of points collected by a reference range device . . . and a set of points collected by a test range sensor construct the ground truth model M Generate positive and negative samples from the test point set Evaluate occupancy probability, classify based on threshold
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
5 / 15
Evaluation Methodology
Evaluation Methodology Given a set of points collected by a reference range device . . . and a set of points collected by a test range sensor construct the ground truth model M Generate positive and negative samples from the test point set Evaluate occupancy probability, classify based on threshold
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
5 / 15
Evaluation Methodology
Evaluation Methodology Given a set of points collected by a reference range device . . . and a set of points collected by a test range sensor construct the ground truth model M Generate positive and negative samples from the test point set Evaluate occupancy probability, classify based on threshold
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
5 / 15
Evaluation Methodology
Evaluation Methodology Given a set of points collected by a reference range device . . . and a set of points collected by a test range sensor construct the ground truth model M Generate positive and negative samples from the test point set Evaluate occupancy probability, classify based on threshold
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
5 / 15
Evaluation Methodology
Evaluation Methodology Given a set of points collected by a reference range device . . . and a set of points collected by a test range sensor construct the ground truth model M Generate positive and negative samples from the test point set Evaluate occupancy probability, classify based on threshold
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
5 / 15
Evaluation Methodology
Evaluation Methodology Given a set of points collected by a reference range device . . . and a set of points collected by a test range sensor construct the ground truth model M Generate positive and negative samples from the test point set Evaluate occupancy probability, classify based on threshold Each test scan corresponds to a point on a Receiver Operating Characteristic (ROC) plot
pos’ neg’
T. Stoyanov et. al. (AASS)
pos TP=24 FN=24
neg FP=13 TN=13
European Conference on Mobile Robots(ECMR11)
tpr=TP/(TP+FN)=0.63 fpr=FP/(FP+TN)=0.34
5 / 15
Evaluation Methodology
What Spatial Model to Use?
(a) Point Cloud
(b) Occupancy Grid
(d) Elevation Grid T. Stoyanov et. al. (AASS)
(c) Triangle Mesh
(e) Multilevel Surface Map
European Conference on Mobile Robots(ECMR11)
6 / 15
Evaluation Methodology
The Normal Distributions Transform The Normal Distributions Transform originally developed for 2D scan registration Points grouped into cells Several possible ways to discretise space A Gaussian pdf used to represent space in each cell Extension to 3D is expressive and space efficient Each cell represented by Covariance matrix C and mean µ
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
7 / 15
Evaluation Methodology
The 3D Normal Distributions Transform - Applications 3D-NDT has already been used in a variety of applications: 3D scan registration place recognition change detection path planning
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
8 / 15
Evaluation Methodology
The 3D Normal Distributions Transform - Applications 3D-NDT has already been used in a variety of applications: 3D scan registration place recognition change detection path planning
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
8 / 15
Evaluation Methodology
The 3D Normal Distributions Transform - Applications 3D-NDT has already been used in a variety of applications: 3D scan registration place recognition change detection path planning
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
8 / 15
Evaluation Methodology
The 3D Normal Distributions Transform - Applications 3D-NDT has already been used in a variety of applications: 3D scan registration place recognition change detection path planning
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
8 / 15
Evaluation Results
Outline
1
Evaluation Methodology
2
Evaluation Results Sensor Setup Evaluation with Known Ground Truth Evaluation in an Uncontrolled Environment
3
Conclusions and Future Work
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
9 / 15
Evaluation Results
Sensor Setup
Sensor Setup
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
9 / 15
Evaluation Results
Sensor Setup
Sensor Setup
Fotonic B70 — range: 0.1-7 meters, ∼ 19.2k points, FOV: 70o x 50o T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
9 / 15
Evaluation Results
Sensor Setup
Sensor Setup
Actuated Laser — range 0.1-8 meters, ∼ 13.5k points, FOV: 180o x 45o T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
9 / 15
Evaluation Results
Sensor Setup
Sensor Setup
SwissRanger SR4000 — range: 0.5-5m, ∼ 25k points, FOV: 43o x 34o T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
9 / 15
Evaluation Results
Sensor Setup
Sensor Setup
Microsoft Kinect — range: 0.5-3.5 meters, ∼ 225k points, FOV: 57o x 43o T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
9 / 15
Evaluation Results
Evaluation with Known Ground Truth
Evaluation with Known Ground Truth — Setup
Fourteen scans from the same position/orientation Precisely known environment Ground truth model created by hand
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
10 / 15
Evaluation Results
Evaluation with Known Ground Truth
Evaluation with Known Ground Truth — Results ROC Plot for Ground Truth Evaluation 1
0.99
0.98
True Positives Rate
0.97
0.96
0.95
0.94
0.93
0.92
0.91
Laser Kinect Swiss Ranger Fotonic
0.9
0.05
0.1
0.15
0.2
0.25
0.3
False Positives Rate T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
11 / 15
Evaluation Results
Evaluation with Known Ground Truth
Evaluation with Known Ground Truth — Results ROC Plot for Ground Truth Evaluation (Laser) 1
0.99
0.98
True Positives Rate
0.97
0.96
0.95
0.94
0.93
0.92
Laser Kinect Swiss Ranger Fotonic
0.91
0.9 0
0.05
0.1
0.15
0.2
0.25
0.3
False Positives Rate T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
11 / 15
Evaluation Results
Evaluation in an Uncontrolled Environment
Evaluation in an Uncontrolled Environment — Setup
Uncontrolled indoor environment Fifty different poses used Scenes of varying complexity, different maximum distance
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
12 / 15
Evaluation Results
Evaluation in an Uncontrolled Environment
Evaluation in an Uncontrolled Environment — Results ROC Plot for all Scans 1
0.95
0.9
True Positives Rate
0.85
0.8
0.75
0.7
0.65
0.6
Laser Kinect Swiss Ranger Fotonic
0.55
0.5
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
False Positives Rate T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
13 / 15
Evaluation Results
Evaluation in an Uncontrolled Environment
Evaluation in an Uncontrolled Environment — Results ROC Plot for Low Range Scans 1
0.95
0.9
True Positives Rate
0.85
0.8
0.75
0.7
0.65
0.6
Laser Kinect Swiss Ranger Fotonic
0.55
0.5
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
False Positives Rate T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
13 / 15
Evaluation Results
Evaluation in an Uncontrolled Environment
Evaluation in an Uncontrolled Environment — Results ROC Plot for Mid Range Scans 1
0.95
0.9
True Positives Rate
0.85
0.8
0.75
0.7
0.65
0.6
Laser Kinect Swiss Ranger Fotonic
0.55
0.5
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
False Positives Rate T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
13 / 15
Evaluation Results
Evaluation in an Uncontrolled Environment
Evaluation in an Uncontrolled Environment — Results ROC Plot for High Range Scans 1
0.95
0.9
True Positives Rate
0.85
0.8
0.75
0.7
0.65
0.6
Laser Kinect Swiss Ranger Fotonic
0.55
0.5
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
False Positives Rate T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
13 / 15
Evaluation Results
Evaluation in an Uncontrolled Environment
Evaluation in an Uncontrolled Environment — Results ROC Plot for Full Evaluation with Constrained Range 1
True Positives Rate
0.95
0.9
0.85
0.8
Laser Kinect Swiss Ranger Fotonic
0.75
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
False Positives Rate T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
13 / 15
Conclusions and Future Work
Outline
1
Evaluation Methodology
2
Evaluation Results Sensor Setup Evaluation with Known Ground Truth Evaluation in an Uncontrolled Environment
3
Conclusions and Future Work
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
14 / 15
Conclusions and Future Work
Summary Proposed a method to compare the accuracy of range sensors. Advantages: Easy to interpret statistics No need for hand-crafted geometric models Evaluation on data from real environments aLRF performed the best, on short range Kinect is comparable work in progress
Perform an evaluation in different environments and under sensor motion
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
14 / 15
Conclusions and Future Work
Thank You!
Thank you for your attention. Questions?
T. Stoyanov et. al. (AASS)
European Conference on Mobile Robots(ECMR11)
15 / 15
