Detection of Obstacles in Monocular Image

Detection

of Obstacles

in Monocular

Image

Sequences

..a_

Final Technical

Report

"A Vision-Based

for NASA Co-Operative Number NCC 2-916

Obstacle

Period

Detection

of Grant:

System

August

Research

for Aircraft

Agreement Navigation"

1, 1995 to July 31, 1997

i

Submitted NASA .2,.

Ames

Technical Flight

Research

Officer:

Management

to Center

Albert

Ahumada

and Human

Factors

Division

Mail Stop 262-2 Moffett Field, CA 94035 I

i by Rangachar

Kasturi Principal

Departments

of Computer

Science

and Octavia

Camps

Investigators and Engineering

and

Electrical

The Pennsylvania State University University Park, PA 16802 Tel: (814) E-Mail:

863-4254

{kasturi,camps}

i _z._

i

(814)

865-3176

@cse.psu.edu

Graduate

Students:

Sadashiva Tarak

Devadiga Gandhi

November i

Fax:

26, 1997

Engineering

Detection

of Obstacles

in Monocular

Image

Sequences

Preface

This research

was initiated

development

of a High

Center.

Primary

to detect from

two

camera.

the

developing runway

Speed

goal of the research of sensors,

report a

sequence

into

approach of

images

aircraft

in monocular

Millimeter two

parts.

for detecting obtained

Wave The

features

corresponding

to the

at NASA

runway

markers.

histogram-based thresholding is used to detect second part addresses the problem of detection obtained from an on-board video camera.

image

(PMMW) part

runways/taxiways from

a

(SVS)

image

first

approximate runway model and the position information Global Positioning System (GPS) is used to define the image

System

here is to develop

obstacles

a Passive

Vision

(HSCT)

reported

and

is divided

a model-based from

of the Synthetic

Civil Transport

runways/taxiways

types The

as a part

moving

project Ames

Research

analysis

algorithms

sequences sensor

of this

obtained

and

report

and PMMW

for the

objects

a video aims

at

on the

sensor.

An

of the camera provided by the region of interest to search for Once

the

runway

is identified,

a

obstacles on and outside the runway. The of objects in monocular image sequences A recursive motion-based segmentation

algorithm identified

based on planar motion model is used. The background motion due to camera is as the dominant motion and the outliers which do not follow this motion are

identified

as obstacles.

ii

Abstract

The ability on-board

to detect sensors

takeoff,

and

different

weather

modalities.

and locate

is an essential

taxiing

phase

and lighting

mounted

such as detection and landing

In this report, sequences sensor their the

and a video

camera

same,

type.

resolution, different

These

goal

moving poor,

for the image identified,

outside

a single

real PMMW

alternative

on the

camera

runway

landing,

functions

under

sensors

of different

sensors

of different

systems

in functions

surface

navigation,

the sensors

differ

using

these

depending

regions

is tested

sensors

is not

on the

sensor

a method

of images

for

camera

provided

Once obstacles

using image

plane

the runway on

the

sequences

a

is very

We use

in the image

markers.

from

quality

runways/taxiways. of the

detecting

obtained

and the image

of interest

to detect

in

of this report.

is low

information

to the runway

This algorithm

Since

obstacles

for detecting

thresholding

(PMMW)

aircraft. obtained

image

Wave

is to develop

resolution

in monocular

Millimeter

in a sequence

position

to define

corresponding

airport

of objects

in two parts

report

approach

(GPS)

the runway.

a landing

for detecting

the sensor

use histogram-based

regions

The

System

collisions,

of the images

of the

and the

with

ground-based

a Passive

separately

part

Since

model

features

we

first

a model-based

runway

Positioning

on-board

are used

objects

sensor.

we propose

approximate Global

and

PMMW

mounted

(SVS),

of detection

of sensors,

are described

of the

runways/taxiways

the problem types

approaches

runway

flight,

of these

by using

using

conditions.

and the quality

approaches

The

two

System

captured

of low-altitude

be facilitated

the current

of potential

we address from

spatial

complements

in images

Automation

Vision

in all weather

obtained

can

Synthetic

and prevention

and takeoff

navigation.

situations,

on-board,

and obstacles

first step in the automation

of aircraft

An aircraft-based

modalities

runways/taxiways

the

by the to search region

runway

simulated

is and

from

image.

position

is to estimate

information

the camera

provided

position

using

iii

by

the

image-based

GPS

is not features,

accurate. such

An

as points

and

lines.

sensor

The

of sensor model

resolution

for three

due

required

The

aircraft.

plane

part

image

using a corner

planar

and propose model

recovered

the dominant

actual

flight

In this report

position

tested

a video

problem camera

at coruer-like

using

this

are presented.

is useful

We

camera

in deciding

background

both synthetic

mounted

the

points

is either

regions. due

planar

as due to the obstacles.

on the

or is piecewise

based Model

in

a landing

detected

to camera

and real image

of objects

on-board

algorithm

motion

are identified

of detection

feature

that the runway

into separate

are considered

we have plane

tested

using

plane

containing

plane)

for a given

explored

parameter

and velocity

the plane

the

flow vectors

using

obtained

in terms

in the estimated

analysis

segmentation

and the outliers

results

the error

to compute

for accuracy

of the sensor

motion-based

The

model

of the

of SVS for navigation.

addresses from

resolution

on the planar parameters

are

is identified Various

sequences

as

stages

obtained

from

test.

Line features

the runway

positions

theoretical

is computed

of the region.

were

Analytical

for computing

We assume

for segmenting

of the algorithm

in 3-D.

Such

obtained

detector.

motion

position

approach

a recursive

for each

equations

and develop

at six different

flow

on the

position

of this report

the optical

depends

an analytical

for use in the design

sequences

image

however,

in 3-D. We propose

quantization.

resolution

second

First,

motion

sensors

a new algorithmic

to image

monocular

camera

and the sensor

different

camera

estimates,

of the sensor

of the estimated

also propose

could

of such

and the position

the accuracy

pose

accuracy

using

sequences. runway

line segment

parameter

not be properly

could

runway

Since

marker

markers, obstacles

the

in two frames

and hence

and using

Though good

not be obtained.

iv

object

features.

(1) estimating

and

estimating

the

lines. The

algorithm

was

the projecting line

planes

segment

small to resolve,

line features

estimate

than point

purposes:

(2) tracking

corresponding was too

reliable

for two

the angle between

and

not be obtained.

located

and more

the use of line features

of the ground-based

real image the

to be prominent

were

for the motion

good tracked,

(i.e., the

in the

image

estimate

for

end points

of the object

could

Table

of Contents

ii

Preface

.°°

Abstract

Ul

1

1. Introduction

3

1.1 Overview 1.2 Review

of Literature

on Obstacle

Detection

Systems

for Navigation

9

1.3 Organization

2. Sensor

Sensitivity

2.1 Sensor

Evaluation Positional

Sensitivity

Imaging

2.1.3

Quantitative

11

Evaluation

15

Geometry Results

Calibration

2.2.1 Error

11

and Calibration

2.1.1

2.2 Camera

due to Image of Image

2.2.3 Experimental

21

and Discussion Features

24

Quantization

25

from Image-based

2.2.2 Analysis

Plane Plane

27

Quantization

38

Results

2.2.3.1

Data

2.2.3.2

Results

38

Simulation

40

and Discussion

45

2.3 Summary

3. Detection

of Objects

3.1 Model

in Passive

Millimeter

Wave

47

Images

48

Transformation

3.1.1 Defining

Regions

3.1.2 Defining

Search

3.2 Feature

6

Localization

3.2.1

Runway

3.2.2

Object

3.3 Experimental

of Interest Space

and Object Localization

Detection

Results

for Runway

for Horizon Detection

Line

Edges

49 51 52 52 52 53

3.3 ExperimentalResults

53

3.4 Summary

54

4. Multiple Motion Segmentation andEstimation 4.1 Methodologiesfor Motion DetectionandEstimation

58 59

4.1.1Feature-based Motion DetectionandEstimation

60

4.1.2OpticalFlow-basedMotion Estimation

62

4.1.3Multiple Motion SegmentationandEstimation

64

4.2 ProblemStatementandthe ProposedApproach

66

4.3 OpticalFlow Computation

68

4.3.1FeatureDetection

75

4.3.2HierarchicalApproachfor OpticalFlow Computation

77 85

4.4 Motion-basedSegmentation 4.4.1PlanarMotion Model

86

4.4.2RecoveringModel Parameterfrom OpticalFlow

91

4.4.2.1LeastSquareModel Fitting

91

4.4.2.2DetectingOutliers

93

4.4.2.3Algorithm for Recoveryof SingleMotion Model

94

4.4.3SegmentationandEstimationof Multiple Motion

95

4.4.3.1Split andMergeAlgorithm

97

4.4.3.2ExperimentalResults

99

4.5 PerformanceCharacterization

100

4.6 Summary

106

5 StructureandMotion Estimationfrom Line Features 5.1 Estimationof PlaneParameters

115 116

5.1.1Analysis

116

5.1.2FeatureDetectionandMatching

120

5.1.3Discussion

125

5.2Estimationof 3D PositionandVelocity

vi

128

5.2.1Analysis

130

5.2.2ExperimentalResults

134

5.3 Summary

136

6 Conclusions

137

6.1PrimaryContribution

138

6.2 FutureResearch

140

References

143

vii

1

Introduction

The

ability

on-board

sensors

takeoff

and

igates the

to detect

the

(ILS), the

phase

aircraft and

up the

Microwave

human

pilot

about

landing

if the

visibility

information darkness, based

from

navigation,

such

and

Implementing IIIa 1 or better, Another

capability

step

and

in the

height

of the

ground

(MLS)

etc.)

assumes

control

of the

aircraft.

Crew

and

System

detection

landing an SVS which

motivating

are

in order bad

would

factor

Hence,

the

equipped

the

landing

with

development

runway

CAT

System

which

decisions

fusion

of

conditions

of

for an aircraft-

airport

in various surface

[17, 107].

of over

1100

II 2 or CAT

of a SVS

and

systems

collisions,

on

or delayed

under

ground-based

conditions

capabilities

aborted

need

nav-

after

makes

combination

is a strong

the

Landing

pilot

mission

landing,

installed

landing

is often

their

of potential

in all weather

for the

point,

there

to complement

upgrade

currently

The

during

using

system

system

as Instrument

this

to accomplish

prevention

takeoff

At

flight,

autopilot

autopilot

the runway

captured

altitude

the

on a sophisticated

weather.

(SVS)

and

and

rely

such

above

visibility.

members

sensors

visibility Vision

external

present

on the

systems

in images

of low

At

(depending

System

on the

obstacles

automation

of aircrafts.

Landing

multiple

as

first

to a certain

is poor.

Synthetic

runways/taxiways

of navigation

depending

reduced

functions

locate

is an essential

taxiing

aircraft

and

is the

US airports

13 capability design

to CAT [24,

107].

of a supersonic

1The minimum vertical visibility ceiling is 12-35 feet with runway visual range of 100-300 meters. A fully automatic landing system with automatic flare and a failure survival autopilot system with a probability of catastrophic failure leas than 10 -T per hour is required. Pilot assumes control at touchdown. 2The minimum required vertical visibility ceiling is 100 feet and runway visual range is 400 meters. A fail passive autopilot is required. Pilot takes over landing at a height of 100 feet. 3There is sufficient vertical visibility at a height of 200 feet with a runway visual range of at least 800 meters for the pilot to carry out a safe landing under manual control. A simplex autopilot system is acceptable. Pilot assumes control at a height of 200 feet.

commercial

aircraft.

a Concorde-like weight[107,

It is argued

drooped

116].

nose

Without

the pilot has to depend

and

locate

the

sound

cockpit

aircraft

sensor's

information

(HSCT) in the

sensor

to see through

external

System

(GPS)

to detect

determine

visibility

is designed

of sensors. sensor

to be operated

Outputs

information

of these

(yaw, pitch, Navigation

the runways/taxiways conflicts,

and

due to fog is known

and the Inertial

potential

takeoff

Wave (PMMW)

fog [132].

and orientation

without

gross

from combination

Milli-Meter

A PMMW

position

database

the airport,

we address

for obstacle

the problem

obtained

detection

navigation,

use high resolution

images

are often used to estimate

are developed

and

vehicle

and high quality

with having accurately.

of ground-based

and

issue

and

System obstacles,

advisories,

Most of the algorithms evaluated

and

systems

obtained

to an obstacle.

such a system there

in monoc-

available

for applications

The use of stereo

Hence,

obstacles

for use in an SVS. Any object

and highway

images

the distance

and poor quality.

for SVS due to the difficulty the 3D position

sensors

to be an obstacle.

and intelligent

ever, axe of low resolution

of detection

from on-board

cations

to estimate

reduction has limited

with a Passive

Positioning

geometry

within

sequences

vehicle

derives

the ability

with aircraft

field of view is considered

literature

the cockpit

Transport

alarms.

In this research,

road

thus providing

an airport

image

nose,

by the Global

(INS),

ular

a drooped

Civil

in significant

and video cameras.

will be integrated

roll) provided

result

Speed

(e.g., 94 GHz), at which point the energy attenuation

to be at a minimum, sensors

could

to be equipped

to the infra-red

at lower frequencies

a High

on an SVS which

The SVS is envisioned in addition

that

PMMW

with a large

is a strong

enough

These

camera.

sensor

is not currently

in the

in robotics,

(IVHS).

using a video

in the

images,

appliStereo how-

recommended base distance

need to design and evaluate

computervision algorithmsfor usein SVSfor obstacledetectionusing monocularimage sequences.

1.1

Overview

This report

is about

moving

sensor.

analyze

image

on-board

The initial sequences

the

PMMW

detection

aircraft

sensors

obtained to detect

orientation

of aircraft,

The position is provided accuracy. such

data

in the

to guide

of the aircraft

by the INS. For example,

updates

could

a few hundred

is the key to detecting in this information

obtained

form of the our system

could

vision

and

instruments

runway.

sensors

is poor.

approximate

are known

position

of the camera

motion

information

in an error

Although contrast However,

position

position

of the aircraft

only up to a certain

data

position

a few

to be as much from

and velocity.

in the estimated

and

image.

and it is likely that

camera

to

mounted

in the low resolution

once every second

their

system

and radiometric

causing

estimating

from a

system

from the GPS, and the orientation

potentially

result

using these

objects

are updated

and

on the

resolution

to locate

GPS data

objects

obtained

a computer

objects

model

from these

Knowledge

sequences

Wave imaging

airport

The data

feet off.

and

in fog, their spatial

is available

be missed,

Millimeter

runways/taxiways

of the images

image

was to design

from a Passive

have good response

additional

in monocular

goal of this research

are very low, and the quality we have

of objects

the

as

GPS/INS

Any inaccuracy

and

velocity

of the

objects. An alternate

approach

objects

with

known

world

sections

of runway/taxiways,

to obtain coordinates corners

an improved and

their

of buildings,

estimate position etc.).

of camera in the

image

This requires

position plane

is to use (e.g.,

an analytical

interstudy

of the

relationship

among

distances

between

accuracy

of the estimated

sensor

the aircraft

resolution

for computing

the camera

and

and objects.

camera

sensor

position

are useful

resolution

equations

in 3-D. We also propose

camera

images,

model

and

for accuracy

quantization.

resolution

the

to compute

the

in terms

a new algorithmic

pose due to plane

the required

of the

an analytical

and develop

camera

in deciding

the

We propose

position

error in the estimated

ical analysis

parameters,

of

approach

Such theoret-

for use in the design

of

SVS for navigation. Initial camera

experiments located

at a fixed

50 x 150 pixel sequence our

image.

not be continued

an aircraft,

• Camera

Hence, objects that

position

obtained

mechanically information

were encouraging,

scanning and

Although

further

from

the

pixel

it to obtain airport

results

research

a single

using

model,

obtained

a a

using

PMMW

sensor

reasons:

of pixels

could

not be developed,

such a sensor could

be safely

mounted

on-board

and

from image-based

was not better

in monocular

We describe

image

for this experiment.

at this time whether

the remaining

the objects

a test

and then

camera

with an array

pose estimation

sor images

in space

the

images

using

for the following

camera

• It is not clear

point

was simulated

on these

• A practical

conducted

Using

of 30 frames

algorithms

could

were

part

images

were moving

a new recursive

than

the position

of this research obtained

features

information

is focused

by a moving

on a background motion-based

which

video

provided

on detecting camera.

is either

segmentation 4

using low resolution

sen-

by the GPS.

independently In this work,

planar

algorithm

PMMW

moving

we assume

or is piecewise for segmenting

planar. images

into regionscorresponding to

independently

moving

objects

and

estimating

their

motion

parameters. Apparent first.

motion

Since the

the image. grouped

camera

Planar

square

model

as outliers

hypothesis

surface

motion

Line features types board

a landing

to the 3-D objects least

the estimated in motion features

and estimate

This

solved tration requires

parameters,

the

problem

using

digital

knowledge

to be moving

from the runway

a single

for the runway

violating

planar

is computed

the planar

motion

and more reliable

sequences

plane Using

obtained

in are

surface

in

using a least

model

are detected

filter-based

and velocity

and

are computed

features.

corresponding

Two

mounted

on-

line features

due

by matching

motion

approach

of the object

point

from a camera

camera

in the image

Kalman

than

markers

parameters the known

line features

the position

information

at and

to the 3°D object

is used to track

in 3-D assuming

that

the

line

the object

runway.

addresses

segmentation,

from

flow, is computed

appears

resulting

due to the runway

Runway

A recursive

on the planar research

- - line features

in two frames.

are tracked.

is moving

based

plane

are resulting

parameters

in image

in the scene.

two line features

flow vectors

to be prominent

are detected

aircraft

the optical

as obstacles.

are considered

of line features

they

Flow vectors

and are identified

called

(i.e., the runway)

motion,

that

model

approach.

in images,

background

of camera

on the

fitting

patterns

is moving,

Using knowledge

based

motion

of brightness

feature

many

tracking,

of motion-based warping

difficult

dynamic

scene

segmentation

and other

of the warping

problems

statistical

parameters.

5

in computer

vision,

analysis,

In the past,

etc.

using

the Hough

methods.

The

It is mainly

method,

digital

used

such

as motionresearchers image

warping

to compensate

regis-

approach for the

dominantmotion; all featuresviolatingthe dominantmotionare detectedas outliers or obstacles.This resultsin warpingnoisein additionto the digitization noise.The Hough methodcanbeusedto segment andestimatemultiplemotionssimultaneously, althoughit is computationallyexpensive andits success andaccuracyaremainlydependenton knowledge of the rangeandresolutionof the parameterspace. Unlike the Hough-based approach,our approachdoesnot requireprior knowledgeof the rangeof parametervaluesfor the motionmodel.Instead,parametervaluesaxedirectly estimated.This estimationis refinedby removingthe outliersat everyiteration. Usingour algorithm,basicsegmentationcanbe donewithout knowledgeof the motionparameters, as opposedto the warpingtechnique,which requiresthe cameramotion parametersto computethe warpingmatrix. Todealwith the computationof multiplemotions,wepropose incorporationof split and mergetechniqueto the motion-basedsegmentation algorithm. The algorithmsdevelopedin this work havebeentestedusingboth real andsynthetic images.The syntheticimagesweregeneratedby the programdevelopedfor this report, aswell as a simulationsoftwaredevelopedat the NASA AmesResearchCenter. Several real imagesequences usedin this workwereobtainedby sensorsmountedon-boarda landing aircraft, and wereprovidedby the NASA AmesResearchCenterand NASA Langley ResearchCenter.

1.2

Review

of the

Literature

on

Obstacle

Detection

Systems

for Naviga-

tion

In recent

years, considerable

vision algorithms

effort has been put into evaluating

for these tasks.

This section

the feasibility

gives a brief overview

of research

of computer work done

at NASA The

and other detection

to the runway the runway

institutions of runway

where

the camera

an image

model

database.

during

flight,

but

they

from the captured

defining

difference model

image

A cost function minimized

reasonably

good

the runway

range

models

accurate

the

map

image

acquired

using

the sensor

buildings,

etc.).

landing

the position the position

- - namely

is expected

sensor

sensor

input.

of of a

by the

input

and

The choice

and speed of convergence edge-based

the viewed

of

model,

scene

image

of such an

area-based

model,

with an expected with the camera

scene. image

is

position.

detection

is supplied

with a moving

for all points

is segmented A Kalman

that

from the actual

of the expected

camera

of obstacle

if either

estimation

scene, as well as the choice of a cost function

to the accuracy

the matching

the problem

runway

automatic

to determine

and the expected

- - are used to compare

more

position

of

obtained

airport

are used to determine

is computed

the actual

different

quantifying

on a model image

the

to permit

have attempted

of the sensor

between

solved

sky, runways,

and

specific

[4, 61, 69].

of fit, axe crucial

model

to obtain

In principle,

camera

enough

lines, etc.,

of authors

for generating

In [102] three

and texture-based

accurate

runway

have been based on some model of environment

the goodness

algorithm.

of the

properties

in [30] is based

with the expected

position

such as points,

[5, 60, 73]. The position

a reasonable

is compared

for navigation.

in [55, 77] using

Work described

axe not considered

in [4, 61, 70]. A number

some perceived

lines).

the known

detection

attempted

INS and GPS can provide

features

Most approaches sensor

image

using

On-board

[101]. Image-based

sensor

lines has been

(e.g., the use of parallel

by generating

a sensor

in the area of obstacle

and

in front

recognized

filter-based

camera

recursive

can be completely

of the

camera,

into constituent estimation

or the

parts procedure

(i.e., for

estimatingrangeto featurepointsin the imageis explainedin [95,108,110,111]. Image regionsof size11 x 11pixelswith high variancearedetectedasfeatures.The initial range of thesefeaturesis computedusingmotionstereo.The featurepositionsin the subsequent framesare predicted,and Kalman

filter.

and images

The

estimation

least squares state

method

parameters.

[120] an optical Image

regions

background optical are

camera flow-based

over

an extended

and velocity

obstacle

runway

is a planar

motion

using

the camera

moving originally

few frames using

of the moving

incremental

tracking

in the world

weighted

using known analysis

linear.

moving

by Nelson These

in In

objects. from the

[87] on the

detected

algorithm

coordinate

camera

described

are segmented

proposed

to

in [109].

independently objects

images

are assumed

to be piecewise

of the sequence.

a model-based

objects

image

motion

indoor

a

in [97]. Motion-based

objects

plane

using

regions

described

is estimated

in

using

filter.

detection

image

of stationary

filtering

from the initial

images

are compared

is used for detecting

the constraint

frames

stereo

the epipolar

by assuming

are refined

for both

in [118] uses a simple

to the independently

subsequent

Kalman

extends

are reported

estimation

the position

approach

corresponding

[56]. Position

helicopter

motion

range

points

In this work, all obstacles

to handle

described

for estimating

flow computed

The

for passive procedure

by applying

tracked

and the results

was modified

This algorithm

[14, 8] to general

to the feature

from the actual flight test.

algorithms

range

of ranges

is tested

The algorithm

and stereo-based The

algorithm

acquired

be stationary.

the estimates

surface

algorithm

described

and computes

warping.

The

in [115, 44] assumes

the residual

algorithm

flow by compensating

is tested

flight.

8

that

using

images

the obstacle-free for the camera obtained

during

a

1.3

Organization

This section in this

provided

report.

Even

in monocular

image

a brief introduction

to the research

though

is based

this report

sequences,

the

total

from two different

sensors.

Detecting

modalities

different

processing

require

using PMMW

sensor

images.

on the main

research

obstacles

dealt

in images

techniques.

Subsequent

problem

theme

obtained

describe

addressed

of object

with processing

The next

sections

and objectives

detection

images

from sensors

two sections

obtained of different

explain

our work

the work done using video

image

sequences. Section

2 describes

orientation

of the

described

camera

parameters. - - the

Passive

(FLIR)

sensor,

and

quantitative

in Section

array

approach

Section into regions

for error

in camera

Wave

Definition

error

features.

of its position

for detecting

were obtained

by the NASA

was not completely

using this sensor

image-based

the

in the

An analytical

and orientation,

_parameters

(PMMW)

sensor,

Television

(HDTV)

position

and

model other

sensor

Looking

is

sensor

are also derived.

the Forward

and

Three Infrared

- - are evaluated

and

are presented.

3. Results

provided

High

using

for computing

in terms

Millimeter

the

results

A model-based

sensor

Equations

sensors

model

estimated

for an on-board

internal

camera

the analytical

modality.

4 describes corresponding

our

in a PMMW

using an image

Langley

developed

objects

Research

by NASA's

The remaining motion-based to different

sequence

Center. commercial

sections

obtained

and

image

partner,

algorithm estimating

is described

from a single

Since a practical

of this report

segmentation motions

sensor

PMMW

we could

use video

sensor

not continue

images.

for segmenting their

pixel

motion

images

parameters.

An algorithmfor computingthe optical flow from imagesequences is described.A planar motionmodelanda linearalgorithmfor recoveringthe modelparametersfrom vectors

are presented.

dent motions

A recursive

is described.

algorithm

The results

for estimating

obtained

using

single

both

and

synthetic

optical

multiple

and

flow

indepen-

real images

are

presented. Algorithm 5.

Details

image runway.

are given

sequence

future

the plane parameters

for a Kalman

and estimating

The feasibility

axe presented suggests

for estimating

the position

of these methods

in this section. research

filter-based

Section

using line features approach

and velocity

was tested 6 contains

topics.

10

is described

for tracking of 3-D objects

using real image a summary

Of the

in Section

line features

in the

moving

on a planar

sequences;

the results

entire

research

and

2

Sensor

Sensitivity

A Synthetic

Vision

craft during

landing

acquired

System

sensors

navigation (INS)

instruments

are known

used),

are more camera

accurate

position.

landmarks sponding requires (i.e.

2.1

An alternative

positions

in the

and orientation

the sensor

Sensor

computers.

Positional

represent

data

data

from these

a better

from

on-board

relationship

among

instruments

information

of the

their

corre-

position.

This

positional

and

accurate of certain

and

camera

the camera

characteristics,

navigation

more

of buildings

estimate

Naviga-

the type of the instru-

for obtaining

corners

cameras.

by the Inertial

upon

they are useful

in 3-D), the sensor

Sensitivity

of mapping in both

The

resolution

limited

the scene

The

the air-

and images

and low light TV

is provided

Since

database

is to use the 3-D location

to obtain

is quantized

to an M × N array

infrared

and control

parameters

the relative

distance

and objects.

is a process

plane

second.

data,

airport

(depending

of runways/taxiways,

of the

to navigate

(GPS).

accuracy

approach

image

study

wave,

System

every

the GPS-based

such as intersection

position

Imaging image

than

once

ability

by the system

only to a certain

an analytical

between

needed

Positioning

axe updated

Calibration

uses an on-board

such as millimeter

and Global

and

and

the pilot's

operations

information

tion System

ments

for enhancing

and taxiing

by external

Additional

Evaluation

of points introduces

Evaluation

a large spatial

called

3-D

scene

directions

of the sensor pixels.

significant

Such

amount

11

onto a small

to facilitate requires

image

being

plane.

processing

that the entire

an array of error

2-D

scene

too small

into computations

This

2-D

by digital be mapped

to adequately involving

the

locationsof imagepointsandfeatures.Anothernatural outcomeof the imagingprocessis the lossof depthinformationresultingfrom mappingof an infinite numberof pointson a line-of-sightontoa singlepoint in the imageplane.As a result,reconstructionof a sceneby computingthe depthof scenepointsusingimage-based featuresis a challengingproblem. The inverseprocess,knownas the cameracalibrationproblem,is to computethe camera positionby triangulationbetweenknownscenepointsin the world andtheir corresponding positionsin the imageplane. Nearly all of the methodsfor solvingthis problemrequire only onemonocularimagewith specialmarkswhichcanbeman-madepoint featuresor line features[42,46, 54,64,69,113].Theaccuracyof suchcomputationalapproaches, however, dependsuponthe camerageometryandthe quantizationof the imageplane. Most analysisfor obtainingthree-dimensional positioninformationof the scenepoints are basedon the triangulationsystem[13,76]. To computethe depth to a scenepoint, triangulation must be performedusing its projectiononto two images,captured either simultaneously,by two camerasseparatedby a basedistance(asin the caseof stereo),

or separately,

stereo). other the

This requires image.

the spatial

computation

Assuming

3-D information

correct

using

quantization

Equations vision

by a single camera

established

percentage

error

occupying

the

shift

a correspondence

in the

the next step is to recover

of such a computation

depends

on

plane.

in [76] present

between

(as in the case of motion

and

been found,

The accuracy

error in distance

in distance

positions

in one image

have already

triangulation.

for calculating

system

of features

matches

of the image

at two different

binocular

measurement

measurement

a worst

case error

is inversely

the two images

and

12

between

analysis,

proportional

directly

an object and

and a stereo show

to the number

proportional

to the

that

the

of pixels object

dis-

tance. BlostienandHuang[13]havedeveloped equationsto determinethe probabilitythat a certainestimateis within a specifiedtolerancegiventhe camerageometryof a stereo setup. Error equationsdevelopedin [31,32] for determiningthe optimumline width for visualnavigationof anautonomous mobilerobot givethe percentage oferror for anysensor geometry,line width, and error conditions. Even thoughtheir analysislookssomewhat similar to our work,the basisfor their analysiswasmeasurement of line width. In this research, weestimatedthe aircraftpositionby trackingknownstaticscenepoints in the imageplane. Sincethe accuracyof this estimationdependson varioussensorparameters,wewantedto developan analyticalmodelthat could relatethe accuracyof the estimatedsensorpositionto the sensorpositionalparametersandattributesofthe captured image.Thesensorpositionalparametersincluderange,crossrange,altitude andpitch, roll, and yawangles.Sensorimagingattributesincludethe numberof pixelsin the imageand the optical angularview (measuredin degrees). Weassumea pin-holecameraand,therefore,ignorecameralensdistortion andoptical non-linearities. We alsoassumethat the problemof featurecorrespondence was solved usingsomecorrespondence algorithmandthat the correspondence wasaccurateto a single pixel. Hence,the analysisdoesnot dependon the type of algorithmusedfor establishing featurecorrespondence. Any errorin computationwasbasicallydueto spatialquantization of the imageplane. In this analysis,weprojectedthe imageplaneonto the ground,and the pixel (p,q) area

represented

the minimum next

in the image

plane

by the pixel). change

pixel in the image

in a camera

was modeled

We defined parameter

as a patch

the sensitivity that

plane.

13

would

on the ground or the error

plane

(ground

in computation

move a fixed ground

point

as to the

s

i

,/

Z

Ceater of gravity

¥ s

I/

I

/J

i i

/s

¢ PX Lateral World

Coordinate

System

pitching

Figure

2.1.1

Imaging

Throughout

center

analysis,

scribe

gravity. dicular

of reference

The position coordinate

axes

(pitch,

of the aircraft

with pitch

axes

we assume

that

axis

roiling

an imaging

the

sensor

sensor

position

or

axis

and aircraft

of notations

in Fig. 1. The figure shows

roll, and yaw) is defined

passing

with respect

and horizontal

is located

at the position

of the earth.

the basis of the system

axes X, Y, and Z. The image

axes of the airplane,

Fig. 2 shows

forms

is shown

to the rolling axis, with its vertical pitching

Longitudinal or yawing

body

the effect of curvature

axis that

of the sensor

perpendicular

world

1: Airplane

Hence, we can use the terms

We also neglect

the position

mutually

Vertical

for convenience,

of gravity.

interchangeably. The system

_

axis

Geometry

the

aircraft's

or

through

an aircraft

with three

the aircraft's

center

to the three

plane is assumed

axes coinciding

used to de-

mutually

of

perpen-

to be perpendicular

with the yawing

and

the

respectively. situation

during

landing,

angle 8, zero yaw, and zero roll angle.

14

where

the aircraft

is at (Xc, Yc, Zc),

Let c_ = 90 - 8. The

field of view of the

camerais determinedby two viewingangles:Aa

defined

right

extent

angles

extent.)

to Aa.

Even

(Aa

though

determines

the

image

area

captured

by the sensor

Aa,

A/3, and

various

pixel

in the image

pixel-patch

2.1.2

the vertical

obtained

sensor

Consider

a point

coordinates

whose

a rectangle, and

orientation,

on the ground

plane.

the ground

area etc.

depend Note

on

that

a

We refer to this as a

Analysis

feature

which

of this feature

has been detected

in its position

feature

at the same pixel (p, q). Hence,

by a certain

will always

give the

same

the camera

position

is large enough

We define

this minimum

Note that

this is a measure

position,

image

(p, q) in the image Let Nx and

at some pixel (p, q). Let the actual

be (P, Q, 0). Since a pixel represents

could change

camera

change

amount

a camera

pose for nearby for the feature

in camera

of camera

size in number

of pixels,

unless

of the

triangulation the change

in the neighboring

as the sensitivity

the

in

pixel.

of the camera.

estimate

and is a function

resolution,

and the pixel

of the location

plane.

Ny represent pixels

the number

are numbered

of pixels

N___ 1 in the horizontal 2

direction.

the bottom

of the patch

right corner

--_,...

in the vertical

-_

pass through

positions,

position angular

the image

by passive

to be observed

The

0,

camera

world

on the ground,

while still retaining

displacement

of accuracy

a patch

pose estimation

respectively.

' ......

and Aft its horizontal

side length

like position,

to a patch

camera

2

is always

as 0, and Af_ at

(See Fig. 3).

Sensitivity

camera

ABCD

parameters

plane corresponds

of the image

by the sensor

is a trapezoid

other

in the same plane

0,...

15

-_ -

The rolling

and

horizontal

directions,

1 in the vertical

direction

axis of the plane

is assumed

on the ground

plane,

which

and

corresponds

to

Z

C

X

A

D

Figure

2: Image

obtained

by the sensor

16

is projected

towards

the

ground

,Ep

(X3. r3)

r.q

.41.w

Figure 3: Ground area covered in the actual image

to the

center

pixel

The coordinates estimated

in the

by the sensor.

image

of the reference

by the following

plane. corner

Other

X

=

Xc + Zctan(a

Y

=

Yc+

plane are obtained

Since a pixel-patch corners

become

the

pixels

corresponds

are referenced area

covered

to a pixel

in a similar by a pixel

manner.

(p, q) can be

relations:

Z_

rolling angle ¢, the ground

(p, q) in the image

small trapezoid

of the ground

cos(_

For a non-zero

Each

is referenced reference

+p-_-) hc,

+ p_)

coordinates

with replacing

=

pcos¢-qsin¢

q'

=

psin¢+qcos¢

of its three

(X', yI)

which

correspond

to a pixel

by (pr, q_), where

(2)

right corner

neighboring

17

(1)

(p, q) in the equation

pr

by the bottom

A_

tan(q-KT--)

pixel

of the pixel, the other patches,

as shown

three

in Fig. 4.

p

(X3 '. 1"3)

T

(._'. r2'_

(p+ l. q+ l)

I

(p+ l, tl)

(XI" YI') (X_', r_ ')

(p,0)

(p, q+ l)

Figure 4: A pixel

Thus,

the

1, where

four corners of this pixel-patch (p, q) are replaced

by (p',q'),

p_ !

qi

where (Pl,ql)= Eq.

1 gives

ground

point

(P, q), (P2,q2)

corresponding

of the

parameter

that

imagery

parameter.

(X_, Y/),

the ground

i = 1, 2, 3, 4 are obtained

=

pi cos ¢ - qi sin ¢

=

pi sin ¢ + qi cos ¢

between to a pixel

sensor.

the partial

by using

Eq.

such that

the camera (p, q).

This

would move a fixed ground

this by taking

towards

(3)

--- (P+ 1,q), (P3, q3) -- (P+ 1,q+

the relationship

sensitivity

obtain

(p, q) projected

derivative

parameters

We are

is defined point

1), and

--- (P,q+

1).

(Xc, Yc, Zc, O, ¢) and

the

now interested

as the

minimum

(P4,q4)

in computing change

in a camera

to the next pixel in the image

of X{ and Y{ with respect

the

plane.

We

to the corresponding

For example, Dxx

= OX_ y OX'----_' Dx-

18

OY_ = OXc

(4)

This derivationis an approximationto the amountof Xc.

Thus,

we estimate

to Y_ (which

that

the amount

define the corners

of change

of adjacent

pixels)

Note that

SY xc = oc, as expected.

in a similar

manner.

in the direction image

sensitivity

sensor

plane

horizontal

function

is a function

is inclined

directions,

Equivalently,

in Table

positional

the

in

X{ to X_ or YI'

(5)

--Dx_ to other

1. In general,

parameter

ground

area

accuracy

of estimation

represented

of estimation

angle to the number

plane,

parameters

S_ stands

j computed

the border

parameters

is defined

for sensitivity

at pixel

and, hence,

position

parameters.

pixels.

of the sensor

is a function

of sensor

varies

is a function using

For a given

at the top half of the sensor by these

and sensor

the sensitivity

of sensor

as well as other

are observed

as we move towards

sensor

the sensor plane

accuracy

that

of various

to the ground

along

of pixel position

using features large

i due to the sensor

to change

change

as

with reference

are summarized

in X_ for unit

(p, q) in the

plane.

Sensor the

These

Sensitivity

in Xc in order

Sxo-

'

change

of pixels in the image.

19

in the vertical

ground

truth

range,

are less accurate

in the horizontal

Since

of pixel number

Also, for a given p, the

characteristic

attitudes.

data

and (p, q). is a

the estimation because

accuracy

of the

decreases

direction.

In summary,

the

and the ratio

of the sensor

view

Sensor

Sensitivity 2Zc sin(cos

cos(2a+

_

at (p, q)

Sensitivity

_b_)

((2p-{- 1) cos _b-2q

at

sin(_) sin ¢))+1

2Zc

cos(2a+

_

sin(2a+

_)

)+ i

CO oo

O0

S Y

Y_

z_

S xZc SY Zc

Table

1: Sensor

positional

2O

sensitivity

equations

(0, 0) ¢ = 0

2.1.3

Quantitative

The

sensitivity

Results

analysis

sensors

at six different

tivities

SxXc, S_,

are plotted

proaching feature Thus,

6, and

section

for landing).

range

applied

to three

different

are given in Table

2. Sensi-

(i.e., p = 0, q = 0) for various In all of the

is 0 feet (typical

Note that

horizontal

was

of the sensors

7, respectively.

is 0 °, and cross

the

velocity

stationary The

sensitivity

also improves that

for the objects

that

objects above

than

are located are located

values

sXc is larger

pixel before

above

than

when

sensor cases,

positions

pitch

angle

an aircraft

is ap-

Szr at (0, 0) and, hence,

it moves

to the next

GPS

number

In addition

the accuracy correspond

of camera

for the

sensor

is moved

unless

that that

vertical

a

pixel.

to the

highest

frames,

estimation

pixel

ground.

resolution.

It becomes

poor

axis (top of the sensor As expected,

accurately

that

of camera

sensor

by knowing

or by using more

i.e.,

the position the position

21

estimation Note

by motion

feature

this analysis

by any algorithm

scene points.

state

is employed.

can be obtained

to being useful in sensor design,

to known

the

of

are closer to the aircraft.

the accuracy

improvements

state

with

closer

more

a high resolution

of image

sensor

at the far end of the vertical

can be computed

indicate

potential

is best

at the far end of the runway).

on the ground

results

the

a large

minimum.

as the

of the aircraft

do not consider

that

previous

Characteristics

move to the next

for the features

using

in the

only S Zc x is important.

Sensitivity

better

5,

the runway would

Discussion

SzX, at the aim point

in Figures

As expected,

and

described positions.

and

is -3.0 °, roll angle

and

that

points

that

would these

stereo than

be no results

techniques the required

will also help us evaluate uses image-based

features

Sensor

Positional

Parameter

Sensor

Location

Range

Altitude

Threshold CAT II-DH

in ft. 0.0 908.1

in ft. 50.0

HDTV

100.0 200.0

FLIR MMW

CAT I-DH Middle Marker 1000' Altitude Outer Marker Table

2816.2 4500.0 18081.1 29040.1

Sensor

Tgpe

Characteristic Pixel

Field

(H x V) 1920 x 1035 512x512

(H x V)deg 30x24 28x21 27x22

80x64

288.2 1000.0 1574.3

2: Sensor positional

parameters

and sensor

characteristics

100f

5OO

0.05

Figure

0.1

, 0.15 Reso_u_en

5: Sensitivity

0.2 in degre4Vl_xe4

0.2S

in the direction

22

i 0.3

0.35

of range

of View

180 t60

120

i: 80 4C 2O

o, O._

Figure

6: Sensitivity

0.15 0,2 Reso_utt_ in _xal

0-245

0.3 0.35

in the direction

of cross range

20O

..E

L

0_

Figure

0,(_

0.1

7: Sensitivity

0.15 Ruo_on in _

o2

0.25

in the direction

23

0.3 0_._

of altitude

2.2

Camera

Camera and

Calibration

calibration

orientation

extrinsic tation include

They

is the problem

- from

parameters,

can be divided

provide

These

and

placed

[130],

scene

points

camera

Fischler

and

of rectangles

of pixels

and

pose

position

intrinsic

in the image are

location

into two categories:

the camera

and

for the purpose

and

and orien-

parameters,

which

plane.

available

in the

literature.

Tsai

their

[39], and

of the rotation [23] used house

of unknown using

parallelepiped

in [113]. For calibration

image

the scene points, research

and translation corners

as marks

with known dimensions mounted

24

or from certain

landmarks

points,

triangulation

or by developing

Liu et al.

vector

of the

object,

Haralick

as the calibration

points, by Wolf that

are separable.

[46] used the

whereas

and

[70] showed

camera

view angle

on an autonomous

image

was done

[43].

the camera

as a calibration

of a camera

man-made

the corresponding

and

and

from certain

on this approach

Ganapathy

size to determine a cube

arise either

is done either by applying

corresponding

Earlier

pose has been to use point

of calibration

Estimation

that relate

Bolles

the camera

point or line features

parameters.

et al.[21] proposed rectangular

system,

the camera

exist in the 3D scene.

the computation Chou

in units

- including

can be grouped

regarding

to estimating

in the scene

a set of equations

the

information

parameters

into two classes:

line correspondences.

solving

parameters

for estimating

approach

to known

camera

world coordinate

most common

that already

Features

of determining

and scale factors

of methods

objects

Image-based

Camera

to a reference

focal length

• The

images.

which

with respect

A number

from

corners

parameters. a method object

Chen that

uses

is proposed

land vehicle

running

on an outdoor road,Liu andDeng[66]usedroad boundariesas calibrationobjects. Fukuchi[42]usedspecialman-mademarkswith certainconstraintsfor determining the positionof a robot usingstandardshapedmarks. • Objectswith curveshavebeenusedfor cameracalibration,suchasa planewith conic or polygonarcsas in [48]and semi-circlein [65]. Mageeand Aggarwalproposeda methodthat usesa calibratedsphere[75]. All ofthe aboveprocedures involvetwo steps.Thefirst stepis to locatefeaturepointson the imageplanethat correspond to the known3-Dpoints,andthe secondstepis to formulate and solvea set of equationsthat relatethe scenepoints and the imagepoints, thereby satisfyingcertainconstraints. Thesecomputationalapproachesassumean ideal pinhole cameraandmodelthe l_ixelsaspointsof insignificantdimension,largelyignoringthe error introducedby the imageplanequantization.Moreaccuratecalibrationof the camera,or a priori knowledgeabouttheerrorintroducedby theimageplanequantization,isquitecritical in variousaspectsof computervision like objectrecognition,scenereconstruction,robot navigation,etc. A brief reviewof someof the pastwork in error analysisof triangulation to imageplanequantizationis givenin the next section.

2.2.1

Error

Quantization outcome problems sensors.

due

to Image

of the image

of various

impact

Quantization

plane as well as the intensity

computational

are particularly The

Plane

approaches

levels have significant

to solving

important

when

the

of quantization

error

on computer

25

images

computer are

captured

vision

vision using

impact

on the

problems.

The

low resolution

was addressed

as early

as

1969.A report by Hart on stereo-scopic calculationsdiscusses sensitivityof triangulation processto pan, tilt, and quantizationerrors[49]. McVeyand Leedevelopedequationsfor measuringthe worstcaseerror in calculatingthe distancebetweenan object anda stereo vision system[76]. The navigationsystemreportedin [31,32]usesa singlecontinuousstrip paintedon the floor markingthe robot'sroute. Forsuccessful navigationof the robot, an importantdesign considerationwasthe width of conditions

on the

line width

on the floor.

The

effect

stereo

set-up

equally

three

was

in [13, 85].

method,

in this section,

points

and the corresponding

consider

four

scene

points

focal

point

These

the effects

plane,

works

of object

assumed

formed

the probability

of various

to determine

determination

the volume

a new approach

is based on the fact that

size of the

image

pixel,

lines-of-sight

pixels

that

by the

the

distribution

error

the optimal

location scene

lateral

using

point

was

[13] or axial of the

through

and their

corresponding

image

is expected

to lie. The

pose.

of proper

We propose

for determining

errors

[85] in all

the error in the camera

two or more lines-of-sight meet at a single point

for each scene

passing

in the camera selection

on the

derived

we describe

Our approach

to finite

and

analyzed

directions.

state.

Due

within

study

in the image

quantization

to be everywhere

component

Later

plane

analyzed

triangulation

That

of the line, as seen

of image

likely

stereo

width

the line.

volume

to minimize

point

the four pixels

and

scene points - and also the number

26

pixel, Two

within

is proportional

by considering

the scene

as the focal point.

of each pixel.

in a polyhedron

of this polyhedron this volume

known

the corresponding

corners

result

connecting

we can or more

which

to the error

more points.

of scene points - reduces

the

the region

The of

uncertaintyin the determinationof the camerapose.This analysiscanbeusedto determine goodcalibratingpoints,their distribution,andthe numberof scenepointsrequiredto stay within the allowedrangeofcomputationalerrorfor aparticularvisionproblem.In situations whereit is not possibleto determinethe calibratingpointsin advance,this analysiscanbe usedto computethe error in the estimatedcameraposition. Our analysiscan be usedto designalgorithmto selectgoodfeaturepointsfor calibrationfrom availablepool of feature points dynamically.

2.2.2

Analysis

Determination minimum

of Image

of camera

number

(I1, I2, I3,..., requires

Under

ideal

Basic

all the

finite

scene points a polyhedron

as shown

for camera

state

world

coordinate

effects

number

within

in Fig. 8. The

parameters system

the

in the

within

which approach

its orientation

focal

image image

coordinate

presented

with respect

image lens.

and

lens

there

can

Due

to the

onto the image any two such

lines of sight form

of the camera

of the center

points,

When

of these

here computes

points

camera

plane

system.

the finite sized pixel. the intersection

image

of the

can be projected

the focal point

27

of the

a certain

corresponding

point

corresponding world

requires

corresponding and the

of quantization

(i.e., the 3-D coordinates and

points

through

onto the image plane,

volume

scene

a given scene point

of points

features

Sn) and their

and their

for the camera

pixel, however,

are projected

anywhere,

the

set of scene points

an infinite

of finite

of these to pass

Error

using point-based

($1, $2, $3,...,

triangulation

position

size of an image

plane through

parameters

(ignoring

for a given

be only one unique

Quantization

lines-of-sight

conditions

distortion),

external

of scene points

In).

points

Plane

is expected the range

of the image

to the reference

to lie

of values

plane in the

axes that

would

w v

SL &?: $¢¢_t Pinata & tZ' t.mte._,.ts Xc , Y¢.Z¢ : I_

plar_ t_a tstv

itc

Figure

satisfy

8: Lines of sight

the constraint

As explained of known volume

scene within

extrinsic

point

points which

is located

image

the

Hence,

concept,

The problem of error

there

the camera

position

and

distance and that are ranges

image

system.

f (focal length)

quantization

of values

of values

for each camera

as a result,

28

where

plane

that

parameter problem

a stereo the region

system

this

the focal along

point

the

Ii in the

would

the mathematical

from the well known

and,

camera's

that

to image

parameter

of finite

the

clear

of the image

we present

using

Since

it is known

Si corresponds

of the objects

a polyhedron

it is not

However,

for each camera

In the next two sections

basic triangulation

to lie.

in front

the scene point

is fixed and known

forms

are uncertain,

coordinate

here is different

points

lens is expected

orientation)

the 3-D position

plane

due to image plane quantization,

of the

to find the range

addressed

in computing

point

due to image

of the focal point).

corresponding

in the world

the above two constraints. of the above

focal

(position

axis of the camera, plane.

section,

.and their

is at a perpendicular

optical

a polyhedron

for the valid location

in the previous

parameters

polyhedron

forming

satisfy

treatment

separately. of determination [13, 85], where

of uncertainty

for the

object locationis well definedby the camerageometry.However,in our problemscenario the location of itself

is uncertain.

problem

being

approach

Fig.

9

systems

addressed

shows

roll angles

problem

their

discussed

analysis

not make

geometry

the coordinate

a priori,

any

since the camera

in [31, 32] is somewhat

is limited

used

to a particular

assumptions

about

of the

camera.

position"

is in front

the coordinate

application. the

kind

system

in the image plane.

the position

image

plane.

given

of the center

In the above

numbered

Z = 0 to be the

Let ] be the focal length

the u and v directions

as (0, 0). We make

• This work analyzes scene points

of the image

geometry,

• A pinhole

the following

camera

of error

model

plane,

assumptions

in obtaining

on a single quantized

image

plane,

"camera and the

the gimbal

the height

of the

respectively,

M × N be the number

respectively,

with the

center

of

pixel

in this analysis:

Image

pose by triangulation

plane

of the camera

[59], thereby

29

and

and

the tilt, yaw,

we assume

Therefore,

the camera

plane.

in computation

is assumed

plane.

of the camera

of the image

the accuracy

to be the only source

ground

coordinate

in [45]: the term

is Zc. Let A u and A v be the size of a pixel along the u and v directions,

along

of outdoor

system,

and (8, ¢, ¢) are, respectively,

sensor

plane.

The

(Xc, Yc, Zc) is the

offset

pixels

to be zero and

system

coordinate

center

on the image

position

close to the

Two separate

the world

We follow the convention means

of the

in this analysis.

(X, Y, Z) represents

in the world coordinate

or "sensor

focal point

system

(u, v) represents

position

position"

here,

is not defined

for calibration.

the

are shown:

camera

the

in this work does

available

the coordinate

of uncertainty

Although

taken

scene points

and

the region

ignoring

quantization

of 3D

is assumed

position.

camera

lens distortion

and

¥

Zv

% /°

Yc

X

Figure

other

• The

optical

The

of detecting

to have been

calibration

process

focal point

Assuming

formed

Analysis

solved.

- especially

• Assume

the

plane

calibration

when

corresponds

coordinates

to a given

scene

of the scene points

plane,

point

is

used in the

as is the case in any practical

to be in front of the lens center introduce

significant

the scene points

position

coordinates

that

to be accurate.

out in the following

camera

for the world

The world

is in front of the image

since this might

is carried

the image point

are assumed

the image

analysis,

geometry

non-linearities.

problem

assumed

9: Camera

vector

error

is not appropriate

in the volume

camera. for this

of the polyhedron

are close to the camera.

four steps:

to be (Xc,Yc, Zc, 8, ¢, ¢), and

of the corners

of the image

process.

3O

pixels that

develop

equations

are to be used in the

* Formthe planeequationsthat containthe scenepoint andthe two adjacentcorners of the pixel corresponding to the scenepoint. • Computethe coordinatesof the focal point in the world coordinatesystemfor the assumedcamerapositionvector. Verify that the focalpoint lieswithin the polyhedronformedby the intersectionof the aboveplanes. The mathematicaltreatmentof the abovefour stepsaredescribedin detail below.

A.

Computing

We assume

the world coordinates

the

image

plane

of the corners

to be at the origin

of image

of the world

tilt, roll, and yaw, as shown in Fig. 10. In this situation, XZ When

plane

and the world

the camera

coordinates

is rotated,

tem can be computed

of a pixel

the coordinates

by applying

necessary

(i,j)

pizels coordinate

the image

in the image

of the pixel

system,

plane coincides plane

are (iAu,

(i, j) in the world

point transformations

with zero with the O, jay).

coordinate

[45] to the original

syspoint

as given below. Xij

iAu

o

=R

(6)

jay k

1

J

k

31

1

¥

P

x

Figure

10: A point

(i,j)

in the image

plane

rotated

about

(X, Y, Z)

where Too

rOl

to2

to3

rio

rll

r12

r13

T20

r21

r22

r23

r30

r31

r32

r33

(7)

R = RcP_R_ =

The

individual

following

three

elements rotation

of the

rotation

vector

where

=

by premultiplying

/cos0//loo0//c o0/

C stands

0

0

0

S_

0

C_

0

0

-Se

Ce

0

0

0

0

1

0

0

0

1

plane,

plane by applying

S#

1

for cosine

- S_

=

R_

and S for sine of the respective with computing

we can think proper

Re

Ce

0

Since we are concerned in the image

be computed

the

vectors.

0

R_

can

rotation

the world

of the imaging vectors,

process

32

0

0

0

0

1

0

0

0

0

1

(8)

angles.

coordinates as rotating

with the center

C_

=

of the corners every

point

of the image plane

of any pixel in the image

as the origin

of

the world

coordinate

system

coordinate

system

by applying

the camera

position

[45], and then translating

vector,

proper

translation

the world coordinate

them

vector.

to a new position

Therefore,

of a pixel

in the world

for nonzero

(i, j) in the image

values

plane

of

is given

by Xij

iAu

., c _C

=R

(9)

+

1

1

where

-

The

3D coordinates

coordinate

system

_ dt are

considered

to be outliers,

where

dt is a threshold

value

decided

experimentally.

4.4.5

Algorithm

From

the above

for Recovery

analysis,

from a single planar the motion

model

model

parameter

small,

then

If there

surface

accuracy

outliers,

need

the motion

points

1. Compute

iterative

3. Compute

data

consider

at that

point,

the

the point

points

These

on the least

_ using N data

pixel using

points

the

theoretical

distance a threshold

for refining

flow the recovered

in the optical

flow is

of the parameter

noise

data

points,

before

reliable

square

fit method

vector. effect

called

estimates

in the least square (x, y) and

the

can be

for recovering

in the following

plane

optical

result

can have significant

is described

the gradient-based

in the previous

If d is above

these

of a pixel in the image

compute

Mahalanobis

If noise

systematically

based

optical

a good estimate

then

all flow vectors

can be easily used to recover

parameters.

parameters.

removed

algorithm

_ computed

flow.

errors,

model

the coordinate

(up, vp) computed

parameter

fit approach

can provide

with large

the model parameter

where

is no noise in the estimated

estimate

and

Model

ideal situations

from a set of N noisy flow vectors

include

2. For each

under

to the actual

of the recovered

model

model

If there

to be detected

Motion

a least square

will be identical

A simple

points

motion,

parameter.

are few data

obtained.

it is clear that

the least square

on the

of Single

four steps.

sense.

the optical

approach

described

flow (ut, vt)

using

Data flow

earlier.

the

model

stage.

d between

the practical

dr, consider

the estimate

94

in future

the data iterations.

flow and point

the theoretical

as an outlier,

else

.

If there

are no outliers

to converge parameter

and

the

data

4.4.6

these

which

and

described The

data

the

obtained

surface

least square

in this iteration

motion.

points

algorithm is the

If one or more outliers

and

repeat

steps

is said

best

model

are detected

1 - 4 for the

fit algorithm

breakdown

moving

in different

directions.

the optical

flow vectors

not follow the planar In [20], the plane containing

point

of flow vectors,

of these vectors

from a differently

of Multiple

remaining

could

object.

fails if more than

2) The

computed

using

that

an optical

are in

This can happen

different

although

by matching

a portion

in

resulting

objects

may not be perfectly

for

surface

to the few flow vectors

flow algorithm,

were first computed

was used to warp

the second

image to the first.

using the warped

image to detect

flow at other

points

points

from a single planar

come from

the scene

region

flow was computed.

points

of good

moving

planar; correct,

thus, may

model.

parameters

camera

in addition

data

the planar

Using the known

method.

axe resulting

be in error

number

50% of the data

for a least square

though

It is also possible

surface

Motion

can be used only when a sufficient

least square

1) Majority

many

Estimation

above

is the actual

two reasons:

image

parameter

the planar

ignore

Segmentation

are available.

motion,

in this iteration,

points.

The algorithm

error,

model

describing

in this iteration,

detected

motion

obstacles.

image.

Model

The residual

parameter

optical

In [115], the runway

and the plane parameters,

Large residual

pixels was assumed

to the current

flow vectors to be caused

95

by errors

thus recovered

flow was then computed

was assumed

images

due to obstacles

of a previous

to be planar.

were warped

were removed.

in the initial

model

and residual The residual

parameter

and

was used to improve the known moving

the model's

camera

average

motion

and

of the optical

from the available

computing earlier. and

that

a more general

mainly

problem.

motion

due

were computed

were tracked

We computed

the outliers

to obstacles.

Our

parameter

in the set of data

was due to the

model,

planar

algorithm

a

as obstacles.

the model

motion

using

by computing

flow were detected

flow from the estimated

the dominant

were

Features

and identified

of the optical

that

the outliers

parameters.

flow vectors,

the deviation We assumed

plane

In [44], warping parameters

flow. Pixels with large residual

In this work, we considered vector

accuracy.

points

by

as explained

surface

in motion,

can be used

for two

applications:

• By computing addition

the dominant

to detecting

for the planar

• Planar

outliers,

surface

motion

to describe Each

motion

optical

[53] and values expected

to vote

the object

if the range

approaches

in which

uses Hough

votes

between

optical

with the into the

bin.

space

resulting

from

the background

assumption

can be used.

uses the afflne motion the

six-parameter using

in the

multiple Hough

an optical

defined

a single

in clusters

96

model parameter

however,

to detect

for the model

All flow vectors

same bin,

planar

flow value computed

flow expected

In

to the scene is large compared

[1], his study

to a set of bins in the

the optical

the motion

the ground,

case piecewise

at any

and

in the form of outliers.

is moving.

objective

point,

are detected

also provides

work had a similar

the theoretical

associated

on which

planar,

flow vector

on the difference

our algorithm

As the camera

may not be perfectly

Adiv's

obstacles

model is a good assumption

to the focal length.

Although

motion,

rigid

model

motions.

space

based

flow algorithm

by the

six parameter

object

in motion

Hough

space,

where

are each

clustercorresponds to separatemovingobjects. knowledge

about

the range

parameter

space.

The higher

computationally

is then

4.4.7

Split

We assume

initially, derived

and

that

to be very

Merge

the runway

or the terrain

planar.

We started

applied

motion

model

failed then models diagram

we hypothesized

The

starts

model

A higher

the camera the runway

of the although

Hough

approach

resolution

Hough

space.

is moving

The outline

is either

to be perfectly

flow vectors.

to be piecewise

with the hypothesis

parameter

parameter The

the algorithm

converges

the deviation

algorithm

the ego-motion

1_, and k can be computed objects.

is computed

is refined

flow vectors.

moving

in which

to all of the available the runway

by computing

model

represents

resolution

If the

planar,

of the algorithm

planar,

initial

and applied

planar and

hypothesis

planar

is shown

motion

in the block

in Fig. 32.

are detected The

is used.

in the first Hough

by hypothesizing

to each of the pieces separately.

The algorithm surface.

detected

on the

the segmentation,

In [122], an adaptive space

depends

Algorithm

or is piecewise planar

Hough

the clusters

and the

the more accurate

expensive.

a low resolution around

of this method

for each of the parameters

the resolution,

it is going

is used where, space

of values

The success

that using

of each

by iteratively

of the camera from _. The

If the outliers

the least

then

and the structure outliers

are distributed

97

and

outliers

computed

model.

from

the initial

outliers

plane

of the background.

they

due to noise

set of

are detected.

the computed

are either

randomly,

from the

no more

to a single planar

approach,

the outliers

when

50% outliers,

belong

square

of the points

removing

is said to converge

with less than

all flow vectors

If

parameter

Note

that

54

or independently

can be simply

ignored.

If the

Feature detection

Optical

flow

computation

square

I

fit

Repeat for every region

Reject outliers 0

Split region No No For outliers

Merge

Figure

regions

32: Split and merge

98

algorithm

I

optical

flow vectors

object;

a separate

If the lution

least

are close together, motion

square

is completely

Piecewise

entire

image

The regions This

model

distributed

The

based

in this study plane

because

and,

we split

the image

to be piecewise

constraint

local regions

into four parts. region.

recursively

until

square

algorithm

merged

motion-based

the least

if the error after

Experimental

the

so-

planar.

local regions.

and the merging

error.

points

are

may not be well defined

merging

The

Where succeeds

is below

image segmentation.

iterative necessary,

least the

square regions

for a given

region.

In this algorithm were

split

Regions

are

a threshold.

Results

segmentation

algorithm

for detection

for the background/runway

is tested

3 of an image

then

for each of these

used for intensity

to each

technique

50% outliers,

it is not clear how the feature

therefore,

independently

model

sets of outliers.

local regions of size N × N over the

on the spatial

is applied

motion

to a single

distributed.

application,

4.4.8

than

can be computed

2. We follow the split and merge algorithm

then

from these

is assumed

to non-overlapping

parameters

on the image

or evenly

more

to belong

in two ways.

can be then merged

is not practical

with

The background/runway

can be applied and

can be considered

can be recovered

converges

can be applied

algorithm

they

parameters

fit algorithm

dropped.

planarity

1. The

model

then

using both

sequence

simulated

obtained

using

with

the

incorporation

and real image the

NASA

99

of obstacles

sequences.

simulation

based of the

on the split

Fig. 33(a)

software.

Fig.

and

planar merge

shows frame 33(b)

shows

the featuresdetectedin frame 3, and Fig. 33(c)showsthe optical flow computedfor the featurepointsusing our hierarchicalframework. Fig. 33(d) showsflow vectorsviolating the planarmotionmodelconstraint,whichhavebeenidentifiedaspotentialobstacles.The planecrossingthe runwayis very clearlydetectedas an obstaclein the images.(Fewfalse alarmsaredueto noisein the estimatedoptical flow.) Fig.34(a)showsframe50ofthe realimagesequence runway_crossing_new a camera

mounted

in Fig. 34(b). and

34(d)

4.5

on-board

Fig

shows the set of feature

In this section, that

algorithms. then

image

optical

flow vectors

the approach of the

in this image

are shown

flow algorithm,

as obstacles.

and

function

for which

it is being

the optical

planar

surface

of this motion

model

using

recovery

in motion.

from a sequence

The

the planar

estimation

contains

optical

motion

algorithm

algorithm,

proposed

of images

the gradient-based

are used to compute

system two sub-

flow method,

model

requires

as-

parameters.

determining

the

sub-aigorithms.

six components

propagating,

of our motion

parameters

described

dating,

by applying

detected

flow using our optical

identified

flow is computed

of each of these

Unlike

a single

model

of the performance

performance

optical

the performance

contains

the motion

optical

Evaluation

in terms

we evaluate

First,

these

vectors

Features

from

Characterization

the

for recovering

aircraft.

34(c) shows the computed

Performance

suming

a landing

obtained

in [47] for characterizing of protocol

optimizing), used.

flow constraint

the computer

(i.e., modeling,

we evaluate For example, equation

i00

the

within

annotating,

algorithm

the optical

vision

estimating,

in terms

flow algorithms

a local region.

algorithms

Thus,

of the

valigeneral

are developed the ideal input

(a)

(b)

•,-_--- ..... _"_--- --_,-- ."..'_'.,'V.'.:':;:', __.'_.-:_'_.:.-::._,:'_'= :__':11 .-_

'

II (c)

(d)

Figure 33: Results obtained for a simulated landing sequence Landing_normal_32L (b)Detected Features flow vectors detected as due to obstacles

101

image sequence: (a)Frame 3 of image (c)Computed optical flow (d)Optical

.-

(a)

:

...

:.

-

(b)

i -i-

!

/ (c)

(d)

Figure 34: Results obtained for a real image sequence: way_crossing_new (b)Detected Features (c)Computed detected as due to obstacle

102

(a)Frame 60 of image sequence runoptical flow (d)Optical flow vectors

to this algorithmrequiresgenerationof data accordingto the followingconstraintequation within a localneighborhood. ,rxu+

where

Ix, I v, and

images.

The output

the local noise

It are the spatial

region.

models

for error

parameters

of the algorithm. flow algorithms

in [47], partly the

noise

because

functions

or gathering for optical

a representative

data

to the 2-D projection

characterize input

motion

of brightness

flow algorithms

and standard

and output

datasets.

computation velocity

pattern

five or more

data

model

with proper computation are the tuning

a protocol

as suggested

or proper

modeling

or perhaps

in annotating

underlying

the algorithm

smoothness,

single

in real imaging

in the image

within

and

of

motion,

constant

situations.

Optical

is only approximately

of 3-D velocity. were evaluated,

deviations

in [47] for characterizing the algorithms

The

ideal

for the

is applied,

computation,

to a large extent,

equivalent

described

the

values

masks

by designing

are violated,

apparent

mean,

in creating

set.

are gradient

the constraint

about

flow is the

density,

which

using

to the above equation

The gradient

assumptions

etc.,

optical

inputs

with the derivative

illumination,

Various

size within

difficulty

associated

which

computation.

computed

solution

have never been evaluated

of the

flow involves

gradients

to [47], model

in gradient

and the local region

(53)

is the least square

according

of gradient,

Optical

and temporal

of the algorithm Hence,

= -xt

in error

proper

quantitative

were reported

the computer

by selecting

and

in [10]. Unlike

vision algorithms, model for input,

They used a set of image sequences

103

measures

Barton

output,

such as flow the approach

et al. [10] do not

and perturbation

to compare

the performance,

in

and used angular performance Our

deviation

the computed

and the true

has been

together,

a given

sequence

images

which

to evaluate

in terms

of images.

are input

the

of their Hence,

optical

ability

to recover

the input

to the optical

flow algorithm

flow algorithm.

the estimated

parameter

vector

was moving

the known camera model

the image.

motion

The optical optical

We evaluated motion.

vector

this weighting

flow and the estimated

to the

of a planar

algorithm surface.

we computed

computed

the optical

is the sequence

of

is motion

we assumed

and square

the actual

optical

from

ap and the covariance

sequences,

with sinusoidal

stage,

flow algorithm

using

of the estimated

optical

we feel that

function.

that

texture.

(theoretical)

of the

Using planar

flow (ut, vt) at every point

on

flow (up, vp) and the covariance

optical

quantitative

square

(Up -

7zt) _1

sequences and

evaluation distance measure,

(Up -

of planar

surfaces

of the optical between which

texture

in

function

flow should

the true optical

is given

by

(54)

l_t) T

obtained

sinusoidal

104

images

flow will be used as weighting

flow as the performance

will be image We used

synthetic

We use the Mahalanobis

E u =

Input

surface

parameter

flow _u.

Since the covariance

consider

as the

estimation

of this system

vector

image

at and the theoretical

flow algorithm

our optical

in the segmentation

the

and plane parameter,

parameter

of the estimated

To create

in front of a planar

model

The output

in the form of an eight-parameter

motion

vector

motion

to this whole vision system

parameters

Ea.

and

the motion

and structure

camera

displacement

metric.

objective

algorithms

between

using a camera for the

plane.

moving

in front

A sequence

of

15 imageswasobtainedfor differentvelocitiesof the camera. Figures frames

1, 5, 10, and

velocity

15 of a 15-frame

image

of (15, 0, 0) in front of a planar

feet from

the planar

These

image

model

and

sequences known

this parametric optical

surface

form,

geometry,

optical

optical

form for optical

plane. motion

where

distance

a

of 100

the planar

at pixels

flow at pixel

with

flow was derived.

The Mahalanobis

optical

moving

to the ground

Using

flow was computed

36 show

was at a height

respect

flow algorithm.

was available.

flow and the practical

a camera

the camera

of 60 ° with

a parametric

the theoretical

simulating

where

to the optical

flow using the flow algorithm

the theoretical

surface,

with an inclination

were input

camera

sequence

35 and

Using

practical

Eu between

(x, y) on the image

plane

was computed. Figures

37(a)

and

as a percentage

38(a)

of the

show the probability

total

number

of available

continuous

function,

the Eu axis has been

and

show

cumulative

38(b)

the