andFittingof Mapping byDragging Models Wire-Frame GeorgeVosselmanand HenriVeldhuis.
Abstract
Semi-automatic meosurement of objects with regular shapes can be performed efficiently in three steps:(1) selection of an object model and approximate alignment of its wire frame by an image andyst, (2) precise dignment to the image by a fitting algorithm, and (3) correction of fitting etors, again by the image analyst. This paper presents a new apprcach to perform these three steps using the same princip)e in all three steps.The developedapproach allows the image analyst to drag both points and lines of the proiected wire frame, including cunred edges and contour edges,in order to align thesefeatures with the image contents. Using the described algorithm, there is no need for the image analyst to specify which parameters of the object models are to be odapted in order to improve the dignment. The performance of the fitting step is analyzed and compared with an alternative approach.
lntroduction Future (geographic)information systemswill contain threedimensional (so) and highly structured information. The developmentof proceduresfor the extractionof sn objectmodels from digital imageryis, therefore,receivingmuch attention at researchinstitutes.Whereaseffortsto fully automatethe processofbuilding extractionshow good progress(Henricsson 1997;Steinhage,1997),it is clearthat under and Baltsavias, many circumstancesautomation is extremely difficult to achieve.For this reason,semi-automaticapproachesarebeing developedwhich allow interactionbetweencomputer algorithms and an imageanalyst,thus taking advantageof the excellent interpretationand evaluationskills of the imageanalyst. The measurementof obiectscan be dealt with most efficientlv if the obiectscanbe modeledby a combinationof a few primiiive shapes.Thesecompositeobiectmodels can be conitructed prior to the measurementprocess.The model-based semi-aut6maticmeasurementof obiectscan then be seenas a procedure:(t)the selectionof an objectmodel from a three-step library and the approximatealignmentof the obiectmodel with the objectappearancein the imagesby an imageanalyst,(2) the precisealignmentof the objectmodel to the imageby a fitting algorithm, and (3) the correction of ertoneousfitting results, againby an imageanalyst. In this paperwe will presenta new approachin which we usethe sameprinciple for modifying the poseand shapeof the objectmodelsin all three steps. The next sectionfirst reviewssomeof the relatedwork presentedin the last few years.In sectionsthree to five the three
Delft University of Technology,Faculty of Civil Engineering Thijsseweg1.1.,2629JA Delft, The Netherand Geosciences, .nl). Iands (
[email protected] H. Veldhuis is currently with the Grontmij Geogroep,Bovendonk 29, 4700 BS Roosendaal,The Netherlands (Henri.Veldhuis@grontmij. nl). & REMOTE SENSING ENGINEERING PHOTOGRAMMETRTC
stepsof the measurementapproachwill be elaborated.The fitting algorithm is describedin sectionthree.This algorithm is similarto the one describedby Lowe (1991).Successrates,precision, speed,and pull-in rangeare analyzedin a mapping experimentinvolving buildings in a residentialand an industrial area.The resultsare comparedwith thoseobtainedwith the snake-likealgorithm describedby Fua (i996). In section four, the designedalgorithm is applied to a different set of observationssuch that it becomessuitablefor correctingerroneousfitting results.Sectionfive demonstratesthat the same algorithm can also be used for approximatealignment of the objectmodels by an imageanalyst.The main featuresof the developedmethodsaresummarizedin the final section.
Work Related Representation of 0bjectModels Objectmodels can be describedin various ways (Mortenson, 1997).The two most importanttypesarebriefly discussed, becausethe way ofrepresentingan objectmodel can have an impact on the way in which the parametersof the model can be manipulated. ConJtructiveSolid Geometry{csc) is widely usedfor computer aided design(cAD).With this techniquean objectis composedby taking unions and intersectionsof severalprimitive shapessuch as rectangularboxes,spheres,cylinders, cones, and tetrahedra.Such an obiectmodel is describedby specifying for eachprimitive the values of the shapeparametersand the are six pose parameters.Often the absolutepose pa-rameters specifiedfor one primitive only and the poseparametersof the other primitives are describedrelative to the first primitive. Boundary representations(B-rep)ofobjects describethe geometryof the points, edgelines, and surfacesof the obiect boundariestogetherwith the topologicalrelationshipsbetween thesepoints, lines, and surfaces.A polyhedral objectcan therefore be describedby the coordinatesof the cornerpoints and the point numbersthat makeup the lines and faces. ofWireFrames Manipulation The functions availablein cRo packagesto changethe position and shapeof objectmodels are often fairly limited. Common approachesare the editing of numeric values of the obiect parametersand slide barsfor rotations around a specifiedaxis. Thesetools are primarily designedfor the constructionof c,tn models.Whereasthey can be used for approximatealignment of an objectmodel with an image,they arenot very userfriendly
PhotogrammetricEngineering& RemoteSensing Vol. 0s, No. 7, luly 1999,pp. 769-776. 0 0 9 9 - 11 2 l 9 S / 6 5 0 7 - 7 6 S $ 3 . 0 0 / 0 O 1999 American Society for Photogrammetry and RemoteSensing
luly te99
769
for the purposeof model-basedmeasurement,Therefore,new tools needto be develooed. An approachdediiated to measurementwith parameterized models is describedby Lang and Forstner(19S6).By dragging corner points of an objectmodel to the corresponding position in an image,one or two parametersof the objectmodel can be adapted.Which parametersare adapteddependson the chosenreferencepoint. the point that is moved,and the entries of a so-calledasso-ciation ta6le.A table has to be made for each objectmodel availablein the objectmodel library. The tables can be edited in order to storethe preferencesofthe imaee analyst. FittingAlgorithms After the approximatealignment of the objectmodel by an image analyst,more precise estimatesof pose and shape parameterscan be obtained by fitting algorithms. Several approachesto optimize the alignment of an objectmodel can be found in the literature. Lowe (tggt) describesa least-squares algorithm that fits the edgesof the projectedwire frameto edgepixels. Theseedge pixels arepixels with a greyvalue gradientabovesomepre-set threshold.Startingwith an approximatealignment,the errors in the alignmentare quantified by the perpendiculardistances of the edgepixels to the nearestedgeof the wire frame.The task of the fitting algorithm is to estimatethe changesto the values of the poseand shapeparametersthat have to be applied in order to minimize the squaresum of thesedistances.This is accomplishedby settingup the observationequation r=K
r--
E I A u I: ) 9 a A o , r--- opi
(1)
for eachedgepixel. In this equationAu is the perpendiculardistanceof an edgepixel to the nearestedgeof the wire frame,p; arethe objectparameters,and K is the number of parameterl. An iterative least-squares adjustmentresults in an optimal estimationof the objectparametervalues.The required partial derivativesareobtainedby numerical simulation. Sesterand Fcjrstner(1989)describea clusteringalgorithm followed by a robust estimationof poseand shapeparamerers. Both algorithmsarebasedon correspondences betweenedges of the wire frame model and straightedgesthat are extracted from the image.Whereasthe clusteringalgorithm is not suitable for the simultaneousdeterminationof a largenumber of parametervalues,the robust estimationis applicableto measurementof polyhedral obiects. Fua (1996)usesa snake-likeapproachto fit a polyhedral objectmodel to an image.An energyfunction is defined as the negativesum ofthe absolutegrey value gradientsalong the edgesof a projectedwire frame.With a steepestgradienl algorithm the positions of the cornersof the polyhedlal model ire adaptedsuch that the value of the energyfunction is minimized. Insteadof the typical regularizationcomponentin the energyfunction used for road mapping, Fua employs constraintson the direction in which the coordinateJof the obiect cornersmay be adapted.Theseconstraintsare necessaryto ensurethatthe geometricrelationshipsbetweenperpendicular and parallel lines of the objectmodel aremaintained. Further Automation in Mapping Buildings Fitting methods require approximatevalues of the object parameters-. The derivation of an approximatelyalignedobject model can be done completelymanually,but, in order to fuither speedup the mapping process,(partial)automationis required.Giilch et o1.(1998)presentan approachfor the measurement of buildings with gableroofswhich usually only requires the imageanalystto point out both ridge points in one image. 77O
luly 1999
Basedon this knowledge,a searchand matching processis sta-rtedto find additional roof edgesand reconstructthe ao building model. A very similar approachis presentedby Li ef 01.(1gs8).In their paper,the number of manual measurements rangesfrom one to three for flat roofs and two to four for gable roofs. The number of measuredpoints per model dependson the successofthe hypothesisgenerationconductedaftereach measuredpoint. Hsieh (1996)developedanother interactive modeling environment in which user-delineatedroof boundariesin one imageareused as cuesin a matching algorithm to determinethe ground level and roof level. Due to the complexity of aerial imageunderstanding,the automationprocessesdescribedabovearecommonly restricted to simple building types such as flat roof or gableroof buildings. In this paper we will not focus on a high degreeof automationfor the measurementsof a few soecificmodels.but. rather,pursue the developmentof efficient tools for the measurementof the largeclassof arbitrarily shapedparameterized models.
Fitting0biectModels to lmages In this section, we first outline some drawbacksof the describedfitting methods.We then show how thesedrawbacks can be eliminated and describe several implementation aspects.In the last paragraphthe performanie characteristics of the new fitting procedureare analyzedand comparedwith thoseofthe snake-likeapproachby Fua (1S96). Analysis of FittingMethods The purposeof a fitting algorithm is to determinethe poseand shapeparametersof an objectmodel such that the edgesof the wire frame,asprojectedinto the image(s),areoptimally aligned with the pixels with high gradients.It is, of course,assumed that thesepixels correspondto the projection ofthe oblect's edges.This objectiveis most clearly visible in the energyfunction of the snakesapproachby Fua (1996).However,the optimization algorithm usedto find the bestparametersis computationally-expensive, becausethe derived energygradientsonly indicate the direction in which the vector of parametervalues should be changed.It takesa substantialnumter of iterationsto accuratelydeterminethe amount of changerequired to obtain the best fit. Computationally,the least-squaresapproachesby Lowe (1991)and Sesterand Fcirstner(1989)are much faster.Due to the edgedetection,the direct relationship betweenparameter ^Weak valuechangesand gradientvaluesis, however.Iost. edge pixels will not be used in the parameterestimationif their eridients arebelow the threshold for edgedetection. Modification of Lowe's FittingMethod A direct relationship betweenthe estimationof the pose and shapeparametersand the gradientsof the pixels cin be achievedby modifying the approachby Lowe (rggr). Lowe introducesan observationequationfor eachpixel which fulfils two conditions: (t) the grey value gradient should be above somethreshold and (2) the pixel should be within somerangeof a projectedwire frame edge.We proposeto drop the first condition:i.e., for all pixels within somerangeof a wire frame edge, the observationequationis introduced. In order to ensurethat the pixels with the higher gradientsdominate the parameter estimation,the squaredgrey value gradientsof the pixels are used asweightsto the observationequations:*i.e., *This is similar to the estimation of subpixel coordinatesfor points found with the interest operator described in Frjrstner and itilch (1982).In this paper the coordinatesof all pixels within a selected window are used to estimate the interest point by using the squared grey value gradientsas weights.
PHOTOGRAMMETRIC ENGINEERING & REMOTESENSING
E [ A u :]
: i g;oo, wra,r(r,s)
(2)
In this way, we avoid the problem of selectinga threshold for edgedetectionand at the sametime improve the accuracyof thJfitting becausethe strongeredgeswill havemore weight in the estimationwhereasall edgepixels have unit weight in Lowe'sapproach. The giey value gradientused in the weight function is calculatedai the partial derivativeof the greyvalue g in the dire-ction u perpendicularto the edgeof the wire frame.This has the advaniagethat gradientscausedby backgroundobjectswith perpendiculardirectionswill not interferewith the parameter estimation.
Analysis Perfomance To assessthe performance of the designed fitting algorithm and to compareit with the snake-likealgorithm by Fua (1996),several experimentshavebeenperformed,The imageryand referencedita were taken from the Avenchesdata setprovided for the first AsconaWorkshop (Gruenet al., 1.995).The images were recordedat a scaleof 1:5000and scannedwith a pixel size of 15 pcm(z.s cm on the ground). After a description of the used object models the set-up of the experimentsis outlined. Finally, the performanceof both fitting-algorithmsis analyzedwith respectto successrates, speed,precision,and pull-in range.
Roof Models All measurementswere performedwith a standardgableroof model. This roof model can be describedby sevenparameters: the three coordinatesof a roof point (X Y, Z); Ihe rotation Aspects lmplementation around the Z-axis (r);and the length (1),width (w),and height Gradientpixels of other objectswill disturb the parameter-esti- (fi) of the roof. The height is defined as the height difference matesif they arewithin somedistanceof the wire frame edges betweenthe roof ridge and the gutter.Note that the wall height of the obiecito be measured.In order to reducethe biasby other is not estimatedin the fitting experiments. objects,it would be bestto only use pixels within a very small The snake-likeapproachused a polyhedral boundary repbuffer aroundthe edges.Using a small buffer,however,implies resentation.In this model the three coordinatesof all six roof that the imageanalystneedsto supply an accurateapplox-iooints are treatedas unknown parameters.To make surethat matepositioning of the wire frame,This would reducethe benthe coordinatesestimatedby the fitting algorithm still form a efits oJa fitting method. It is thereforebetter to start with a Iarge gableroof, elevenshapeconstraintsareused (Veldhuis,1998). buffer and then to reducethe buffer size after eachiteration of estimation. the least-squares A largebuffer will contain a largenumber of pixels. Setting Expeilmental PrctocoL A total of ten buildings with gableroofs was selectedfrom two up an obs;rvation equation for each pixel would make the fitstereopairs. For all buildings, the wire frame of the roof model ting algorithm time ionsuming. In the first few iterations with a was approximatelypositionedby an imageanalyst.-Usingthese Iargerbuffer size,we thereforesubsamplethe buffer with proapproximatepositions,15 measurementswerepe-rformedwith file:sperpendicularto the edges(seeFigure 1)' Observation U-oinfitting methodsfor eachbuilding. Thesedifferent meaequafionsale setup only for points on theseprofiles.The point surementswere usedto study the effectsof the subsamplingof densityis increasedeachtime the buffer sizeis reduced.In this the buffer onto the successratesand the computationtime of *ay iu.g" initial buffer size can be combinedwith a fastand " fitting. the fitting. The subsamplingis used for both the least-squares precise and the snakesapproach.For the distancesbetweenthe profiles The partial derivativesof the distanceswith re,spectto D" and the initial profile length Iu, 5, 10, or 20 pixels were changesin the parametersare estimated numerically, as in used.The initial number of points on a profile N, was either 5 Lowe-'sapproach.A more detailed explanationof the estimaor 10. All combinationswere made,however,with the restriction canbe found in Vosselman(1998). tion that the minimum distancebetweenpoints in a profile is one pixel. Therefore,l, : 5 was not combinedwith N, : 10' In the hst iteration,the number of points in a profile is setto three and the distancebetweenthe points is setto one pixel. The analysesof the precision and the pull-inrange were done with the subsamplingwhich resultedin the highestsuccessrates.For the analysisofthe pull-in range,anotherset of experimentswas performedin which the errorsin the approximate position and rotation of the roof models were systematically increaseduntil the fitting algorithm could no longer make a correctestimate.
to a proFigure1. Profilesperpendicular jected wire frame edge.The point density bythe distancebetween can be controlled the profilesDu,the lengthof the profiles 1,, and the numberof pointson a profile N,. The distancebetweenpointswithin a profileis definedbYDu: L,/Nu.
&,REMOTESENSING ENGINEERING PHOIOGRAMMETRIC
Success Rates Looking at the fitting results, it was found that many of the fit ted modelswere noffully aligned.Often one or two edgeswere wrong due to the presenie of other objectsin the-image'Most of the edges,however,were positioned correctly'The successrate was th-ereforedefined as the percentageof wire frame edgesthat required no correctionby the imageanalyst.The resultsare vizualized in Figure 2. For many combinationsof subsampling parameters,both methodsachievedsuccessratesover 90 perapproachseemsto be-alittle more ient. The least-squares robustwith respeit to changesin the subsampling.Both meth-ods show a lower successrate with an initial profile length of onlv 5 pixels. For both methods,bestresultswere obtainedwith an initial profile length of t0 pixels, 5 points per profile, and a distancebetweenthe profiles of 5 pixels, luly 1999
77L
!
Snakes
Least squares
I
100 90 80 70 a
60 50 N - d - N - d + N i o o a n h n o o a 6 h r n
N N N N N N
Figure2. Percentages successfullyfitted wireframeedgesfor both methods parameters(1, - N, - D"). with varyingsubsampling
TngrE2. StcNonRo Devrlrrolrs oFrHEESTMATEo PosEnruoSHlpe PnnnvrreRs. rs ExpRessro rru TneSrnruonRo DEVIATToN or tnE r-Rornrrolr VnruesAnr m MrreRsoRPrxgrs. Drcnres.ALLorHER
Speed The resultson the cPU-timeused for fitting are shown in Figure method is much fasterthan 3. It is clearthat the least-squares the snake-likemethod . This is due to the comparativelylarge number of iterationsrequiredby the last method. Also clearly visible is the dependencyof the computationtime on the number ofpoints for which observationequationsare derived. Another critical factor with respectto the computation time is the hidden line analvsis.The performanceof hidden line algorithmsis no bottlen-eckwherrworking with polyhedral objectsbut doesbecomeone when working with curved objectssuch as cylinders (Ermesand van den Heuvel, lgg8).
A
Least squares(m.) Snakes
(pixel)
Least squares (m) Snakes
0,o7 1.0 0.12 1.6
0.22 0.22 0.22 0.22
0.06 0.8 0.08 1.0
0.09 1..2 o,o7 0.9
0.09 1,.2 0.09 1.3
Pull-In Ranse An importait reason for using fitting techniques is the reduction in mapping time. The image analyst can work faster if the object models only need to be positioned approximately. Therefore, the benefits of a fitting method increase when the estio . 1 4 mated parameters are correct even though the approximate 1.9 positioning was bad. The pull-in range is defined as the range o.L2 of differences between the correct and approximate parameter 1.6 values for which the measurement still succeeds. This oull-in
Esrrvnreo RoorcoRrurR
0.09 '1.2
Ipixel) (m) (pixel)
0.06 0.05 0.8 o,7 0.09 0.07 1) 1.0
Z-coordinate of the roof ooints and the house width were better estimated by the snake-like approach. Considering the expected accuracy ofthe reference data (0.1 m), it is difficult to assessthe accuracy potential of the fitting methods. It can, however, be concluded that the reference data probably had a standard deviation below 0.1 m and that both fitting methods are capable of obtaining the same order of precision.
Precision For the evaluationofthe orecision. standarddeviationswere computedfor both the coordinatesof the roof corners(Table1) and the pose and shapeparameters(Table2). Both fitting methods obtain approximately the sameresults. The new least-squares fitting method is often slightly better,but the
rraLe1. Srnruonno "*B#J;ff;:E
(pixel) Im]
0.09 I,2 0.10 L.4
o.72 1.6
I
l5
lt;
1)
I
Least squares
f1,
L
o
'E i6 D
F+
0
Snakes
ft tt
L il: IL:
t
T'l n
I l:,, I
l
I l"'
IF
n tl il* I !'i IT
It
It
lt
:l L
IL
T1
l-
Qo6OonoO6oOhOOn qi-N-d-NiN-
ddo;;;ddd;,;;;;,A ddodd4doddd3,A,A; N N d d N d
parameFigure3. Fittingspeedfor both methodswith varyingsubsampling ters (Lq- N, D,).
Iuly 1999
PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING
rangewas investigated for a shift of the obiect model in the XYdirection (Figure 4a) and a rotation Elroundthe Z-axis (Figure 4b)by systematicallyincreasingthe errorsin the approximate positioninguntil the fitting failed. The experiment-wasp_eriormed for all ten buildings. The results show that the pul]-in rangefor the least-squaresmethod is a bit larger than for the snake-likeemethod. Of course,the pull-in rangewill also dependon the sizeof the buffers around the wire frame edgesand the presenceof other obiectedges.The subsamplingparameterswere, however,not varied in this experiment.
ofFittingResults Conection EnorcInFittingResults Resultsof fitting algorithms usually contain errors. However, oftenthe maioriiy oTthemodel edgeswill be correctlyalignedby a fitting aigorith-, whereasonly a few lines aremisaligned due to, foiexample, interferingneighboringobjects. Figure 5 shows a typical exampleof builjling extraction from a6rial imagery.the two imagei on the Ieft side are patches of the two photographsof a stereopair overlaid with an approximately alignedwire frameof the housemodel. The two images in the secondcolumn show the result of the fitting algorithm' Most edgesarealignedcorrectly,but the high contrastbetween the houi's shadowand the yard attractedthe lower horizontal edgein the first image.The correctpositiol would havebeen a fJw rows higher.Due to this mismatch,the slopedlines of the roof in the secondimagealso show errors,The slope is underestimated. forConection of FittingErrors 0bjective Becausemost of the edgesare alignedcorrectly,measurements by the imageanalystonly haveto deal with one or more incorrectly aligned edges.However, in aparameterizedobject.model,th"epositidn of edgescan not 6e adaptedindividually' Instead,we needto find the changesto the valuesof the pose and shapeparametersthat correctthe errors' Fofan imageanalyst,this may be a difficult task to perone parameterneedsto be changed,this can be form. If only -easilv. However, often a combination of two or more done fairlv
g
Snakes
I
Leastsquares
l5 810
gs tro
E"
-) .= -! ^-
-lu
-15
l)
3 ro g5 e0 F
