Robot Tool Center Point Calibration using Computer ... - DiVA Portal

Robot Tool Center Point Calibration using Computer Vision Master’s Thesis in Computer Vision Linköping Department of Electrical Engineering by

Johan Hallenberg

LiTH - ISY - EX - - 07/3943 - - SE Department of Electrical Engineering Linköpings University, SE-581 83, Linköping, Sweden

Linköping February 2007

Printed by: UniTryck, Linköping, Sweden Distributed by: Linköpings University Department of Electrical Engineering SE-581 83, Sweden

Robot Tool Center Point Calibration using Computer Vision Master’s Thesis at Computer Vision Laboratory University of Linköping by Johan Hallenberg Reg nr: LITH - ISY - EX - - 07/3943 - - SE

Supervisors: Klas Nordberg ISY, Linköping University Ivan Lundberg ABB Corporate Research Center, Sweden Examiner: Klas Nordberg ISY, Linköping University Linköping 2007 - 02 - 05

Abstract Robot Tool Center Point Calibration using Computer Vision By: Johan Hallenberg MSc, Linköping University, February 2007 Examiner: Klas Nordberg Supervisors: Klas Nordberg and Ivan Lundberg Today, tool center point calibration is mostly done by a manual procedure. The method is very time consuming and the result may vary due to how skilled the operators are. This thesis proposes a new automated iterative method for tool center point calibration of industrial robots, by making use of computer vision and image processing techniques. The new method has several advantages over the manual calibration method. Experimental verifications have shown that the proposed method is much faster, still delivering a comparable or even better accuracy. The setup of the proposed method is very easy, only one USB camera connected to a laptop computer is needed and no contact with the robot tool is necessary during the calibration procedure. The method can be split into three different parts. Initially, the transformation between the robot wrist and the tool is determined by solving a closed loop of homogeneous transformations. Second an image segmentation procedure is described for finding point correspondences on a rotation symmetric robot tool. The image segmentation part is necessary for performing a measurement with six degrees of freedom of the camera to tool transformation. The last part of the proposed method is an iterative procedure which automates an ordinary four point tool center point calibration algorithm. The iterative procedure ensures that the accuracy of the tool center point calibration only depends on the accuracy of the camera when registering a movement between two positions.

v

Acknowledgements I have had the opportunity to write my Master’s Thesis at ABB Corporate Research Center, Mechatronics department, in Väster˚as. This period has been a great experience for me and I have really enjoyed working with the project. I want to thank my supervisor Development Engineer Ivan Lundberg at ABB Corporate Research Center for his great enthusiasm, all the conversations, and for the extremely good supervision throughout the project. I also want to thank my supervisor and examiner Associate Professor Klas Nordberg at Linköping University, for all the great phone conversations, the good tips and all the support and guidance throughout the project. Special thanks also to PhD. Mats Andersson at the Center for Medical Image Science and Visualization for his great interest in the project and helpfully ideas. Linköping, February 2007 Johan Hallenberg

vii

Notation Symbols • f (x, y) denotes a 2D function or an image. • x, X denote scalar values. • x, X denote vectors or coordinates. • X denotes a matrix.

Operators • f (x, y) ∗ g(x, y) denotes the convolution between the image f (x, y) and the image g(x, y). • f (x, y) ? g(x, y) denotes the cross correlation between the image f (x, y) and the image g(x, y). • XT denotes the transpose of X. • X† denotes the pseudo inverse of X.

Glossary TCP Jog Base frame Robtarget DOF Q.E.D

Tool Center Point. Manually move the robot with joystick. The robot’s base coordinate system. Cartesian target. Degrees of freedom. ”Quod Erat Demonstrandum” a latin phrase meaning ”which was to be demonstrated”.

ix

Contents 1

2

3

Introduction 1.1 Background . . . . . . . . . . . 1.2 Problem Specification/Objectives 1.3 Delimitation . . . . . . . . . . . 1.4 Thesis Outline . . . . . . . . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

1 1 2 2 2

Robotics 2.1 Overview . . . . . . . . . . . 2.2 Coordinate Systems . . . . . . 2.2.1 Tool Center Point, TCP 2.2.2 Base frame . . . . . . 2.2.3 World frame . . . . . 2.2.4 Wrist frame . . . . . . 2.2.5 Tool frame . . . . . . 2.3 TCP Calibration algorithm . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

3 3 4 4 4 5 5 6 7

. . . . . . . . . . . . . .

9 9 9 9 9 10 10 11 11 12 12 13 14 16 16

. . . . . . . .

Theory 3.1 Image Processing Theory . . . . . . . 3.1.1 Threshold . . . . . . . . . . . 3.1.2 Morfological operations . . . 3.1.3 Structure Element . . . . . . . 3.1.3.1 Erosion . . . . . . . 3.1.3.2 Dilation . . . . . . 3.1.3.3 Opening . . . . . . 3.1.3.4 Closing . . . . . . . 3.2 Computer Vision Theory . . . . . . . 3.2.1 Camera System . . . . . . . . 3.2.2 Pinhole Camera Model . . . . 3.2.3 Camera Calibration . . . . . . 3.3 Transformation Theory . . . . . . . . 3.3.1 Homogeneous Transformation

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

xi

xii

CONTENTS

3.3.2 3.3.3 4

5

3.3.1.1 3D translation . . . . . . . . . . . . . 3.3.1.2 3D Rotation . . . . . . . . . . . . . . 3.3.1.3 Homogeneous Transformation Matrix Screw Axis Rotation representation . . . . . . . Rodrigues´s formula . . . . . . . . . . . . . . .

Determination of Tool Center Point 4.1 Overview of the system . . . . . . . . . . . . . 4.1.1 Equipment . . . . . . . . . . . . . . . 4.2 Determine T2 . . . . . . . . . . . . . . . . . . 4.3 Finding X1 and X2 transformation matrixes . . 4.4 Importance of measure A and B with 6 DOF . 4.5 Iterative Method for increasing the accuracy . . 4.6 Alternative method for rotation symmetric tools

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

16 16 16 17 18

21 . . . . . 21 . . . . . 22 . . . . . 22 . . . . . 24 . . . . . 29 . . . . . 31 . . . . . 33

Image Segmentation of Robot Tool 5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.1 Open Computer Vision Library (OpenCV) . . . . . 5.2 Creation of binary mask image . . . . . . . . . . . . . . . 5.2.1 Image registration . . . . . . . . . . . . . . . . . 5.2.2 Background subtraction . . . . . . . . . . . . . . 5.2.2.1 The Hough filling method (HFM) . . . . 5.3 Edge Detection . . . . . . . . . . . . . . . . . . . . . . . 5.3.1 Canny’s Edge Detection . . . . . . . . . . . . . . 5.4 Contour Retrieving . . . . . . . . . . . . . . . . . . . . . 5.4.1 Freeman Chain Code Contour representation . . . 5.4.2 Polygon Representation . . . . . . . . . . . . . . 5.4.3 Freeman Methods . . . . . . . . . . . . . . . . . . 5.4.4 Active Contour (The Snake algorithm) . . . . . . . 5.5 Finding the contour matching the tool shape . . . . . . . . 5.5.1 Convex Hull and its defects . . . . . . . . . . . . 5.5.2 Polygonal Approximation of Contours . . . . . . . 5.5.2.1 Douglas-Peucker Approximation Method 5.6 Geometric constraints . . . . . . . . . . . . . . . . . . . . 5.7 Corner Detection . . . . . . . . . . . . . . . . . . . . . . 5.7.1 Harris corner detector . . . . . . . . . . . . . . . 5.7.1.1 Sub pixel Accuracy . . . . . . . . . . . 5.7.2 Fast radial . . . . . . . . . . . . . . . . . . . . . . 5.7.3 Orientation Tensor . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

35 35 35 35 36 38 39 40 40 41 41 42 42 42 44 44 45 46 46 47 47 48 50 50

CONTENTS 6

xiii

Results 6.1 Repetition Accuracy of the Camera . . . . . . . . . . . . . . . . . 6.1.1 Averaging to achieve higher accuracy . . . . . . . . . . . 6.1.2 How does the number of points affect the repetition accuracy 6.2 Camera versus Robot measurements . . . . . . . . . . . . . . . . 6.2.1 Rotation measurements . . . . . . . . . . . . . . . . . . . 6.2.2 Translation measurements . . . . . . . . . . . . . . . . . 6.2.2.1 Averaging to achieve higher accuracy . . . . . . 6.3 Accuracy of the TCP Calibration method . . . . . . . . . . . . . 6.3.1 Averaging to obtain a higher accuracy . . . . . . . . . . . 6.4 Repetition Accuracy of the TCP Calibration method . . . . . . . . 6.4.0.1 How does the number of points affect the repetition accuracy of the TCP calibration method .

55 55 56 57 59 59 61 64 67 69 69

Discussion 7.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.1 Accuracy . . . . . . . . . . . . . . . . . . . . . . 7.1.2 Major Problems/known difficulties . . . . . . . . . 7.1.2.1 Finding distinct points . . . . . . . . . . 7.1.2.2 Light reflections in the tool . . . . . . . 7.1.3 Fulfillment of objectives . . . . . . . . . . . . . . 7.2 Future Development . . . . . . . . . . . . . . . . . . . . 7.2.1 Rotation symmetric tools . . . . . . . . . . . . . . 7.2.2 Light reflections in the tool . . . . . . . . . . . . . 7.2.3 Image Segmentation controlled by CAD drawings 7.2.4 Neural network based methods . . . . . . . . . . . 7.2.5 Online correction of the TCP calibration . . . . . . 7.2.6 Other image processing libraries . . . . . . . . . .

. . . . . . . . . . . . .

73 73 73 74 74 74 75 75 75 76 76 76 76 77

A Appendix A.1 Finding intrinsic parameters . . . . . . . . . . . . . . . . . . . .

79 79

Bibliography

87

7

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

72

Chapter 1 Introduction This chapter begins with a short background, then the problem is specified and the objectives and delimitations are determined. Finally the disposition of the thesis is presented.

1.1

Background

All robots delivered by ABB are calibrated in the production. When the calibration procedure is done the robot is calibrated up to the last axis and is thereby able to calculate the positions of all axes during a movement. When a tool is mounted on the last axis (the tool flange) the robot needs to know the actual position of the active point of the tool, the tool center point, which for instance can be the muzzle of a spot welding tool. For this reason a tool center point calibration has to be performed every time the tool is changed. Today the TCP calibration is done manually by moving the robot, letting the TCP brush against a fixed point in the environment. The fixed point is typically the tip of a nail. The tool center point needs to brush against the tip of the nail with great precision and from at least four different angles. Then the coordinate of the tool center point can be calculated by the robot in relation to the robot’s tool flange coordinate system. The method is time consuming and the result may vary due to how skilled the robot operator is. The resulting accuracy of todays method is approximately ±1 mm. Products for automating the TCP calibration exist, for instance BullsEye for calibration of arc welding tools and TCP-Beam for calibration of spot welding tools. The disadvantage with these methods are that they are specialized to one single type of robot tools. 1

2

1.2

CHAPTER 1. INTRODUCTION

Problem Specification/Objectives

The purpose of this thesis is to investigate whether or not a computer vision method can be used to automate the tool center point calibration process. If it is possible to find a method then the accuracy of the method is also of great interest. The objective is to find a method with an accuracy of at least ±1 mm, that automates the tool center point calibration procedure. For development of the method C#, C/C++ and Matlab are allowed to be used, but the final demonstration software shall be programmed in C# and C/C++.

1.3

Delimitation

To deliver a method with the ability to calibrate all kinds of robot tools during a 5 months Master’s thesis is of course impossible. Instead the method is of importance, and any kind of tool is permitted to be used to demonstrate the method.

1.4

Thesis Outline

Chapter 1

Chapter 2

Chapter 3

Chapter 4 Chapter 5 Chapter 6 Chapter 7

Appendix A

Chapter 1 describes the background to the project. The problem specification, delimitations and the thesis outline are also presented. Chapter 2 gives a short introduction to robots and the chapter concludes with a description of a four point tool center point calibration algorithm. In chapter 3 some theory needed to understand the rest of the thesis are presented. The Chapter is divided in image processing theory, computer vision theory and transformation theory. Chapter 4 describes the proposed method in detail and the configuration of the devices is presented. Chapter 5 describes in detail how the image segmentation of the robot tool was done. Chapter 6 presents the results of several tests performed to investigate the accuracy of the proposed method. Chapter 7 presents the conclusions and difficulties of the project. The chapter concludes with a discussion of future development. Appendix A presents some results from the camera calibration, performed with the Camera Calibration Toolbox in Matlab.

Chapter 2 Robotics This chapter will give an overview of robots and their coordinate systems. The chapter concludes with a description of a four point tool center point calibration algorithm used in the robots today.

2.1

Overview

Traditionally a robot consists of a mechanical arm and a computer controlling its position and movement. In this project only serial kinematics manipulators will be concerned. The serial kinematics manipulators consist of several axes connected by joints see figure 2.1.

Figure 2.1: The ABB Robot IRB 6600 (Courtesy of ABB). 3

4

CHAPTER 2. ROBOTICS

Usually the mechanical arm (the manipulator) is divided into an arm, a wrist and an end-manipulator (the tool). All the manipulators used during this project have got six degrees of freedom, making it possible to position the tool anywhere in the robot workspace with a given orientation. Internally the robot end effector orientation is represented by quaternions. The ABB robots can be manually controlled by using the flexpendant. The flexpendant is a hand held controller connected to the computer controlling the robot. A robot program telling the robot how to move can be written in the program language RAPID. The RAPID programs can be written directly in the flexpendant but it is also possible to develop the RAPID program on a PC and then transfer the program (by a TCP-IP connection or a USB connection) to the computer controlling the robot.

2.2

Coordinate Systems

2.2.1

Tool Center Point, TCP

When a robot movement is programmed by specifying a path of robtargets for the robot to follow, then all the robot movements and robot positions are relative to the Tool Center Point (TCP). Normally the tool center point is defined to be the active point of the tool e.g. the muzzle of a spot welding tool or in the center of a gripper. Several tool center points can be defined e.g. one for each tool, but only one tool center point can be active at a given time. When the robot is programmed to move along a given path, it is the tool center point that will follow the actual path. It is possible to define other coordinate systems and program the robot to move according to these coordinate systems. The tool center point is then expressed in relation to the coordinate system used in the program [1].

2.2.2

Base frame

The robot base coordinate system is located on the robot base. The origin of the base frame is located at the intersection of axis 1 and the robot’s mounting surface. The x axis is pointing forward, the y axis points in the robot’s left side direction and the z axis coincides with the rotational axis of axis 1. Figure 2.2 illustrates the base frame [1].

2.2. COORDINATE SYSTEMS

5

Figure 2.2: The base coordinate system (Courtesy of ABB).

2.2.3

World frame

If several robots are working in the same area, a world frame can be set up and the robots base coordinate systems can be expressed in relation to the world coordinate system. It is then possible to make RAPID programs telling the robot to move to a certain position in the world frame. If one of the robots is mounted up side down, the definition of a world frame will of course simplify the programming of the robot. Please, see figure 2.3 [1]

Figure 2.3: The world coordinate system (Courtesy of ABB).

2.2.4

Wrist frame

The wrist frame is fixed to the tool flange/mounting flange (the surface where the tool is to be mounted). The origin of the wrist frame is positioned in the center

6

CHAPTER 2. ROBOTICS

Figure 2.4: The wrist coordinate system (Courtesy of ABB). of the tool flange and the z axis coincides with axis six of the robot. There is a calibration mark at the tool flange, see figure 2.5. The x-axis of the wrist frame is always pointing in the opposite direction of the calibration mark and the y-axis is achieved by constructing an orthogonal coordinate system axis to the x and z axes. [1]

2.2.5

Tool frame

To be able to define a tool center point, a tool frame is needed. The tool frame is a coordinate system on the tool, and the origin of the tool frame coincides with the tool center point. The tool frame can also be used to obtain information about the direction of the robot movement. [1]

Figure 2.5: The tool coordinate system (Courtesy of ABB).

2.3. TCP CALIBRATION ALGORITHM

2.3

7

TCP Calibration algorithm

Today, the TCP calibration is done by jogging the robot making the TCP brush against a fixed point in the close surrounding to the robot. The fixed point is for instance the tip of a nail. By jogging the robot, making the TCP brush against the tip of the nail from at least four different orientations, the coordinate of the TCP in relation to the robot base frame is calculated. Let T1 be the position of the tool flange in relation to the robot base frame, e.g. the robot’s forward kinematics. Assume N different positions of the robot are known, making the TCP brush against the fixed point in the environment. Then N different transformations T1i are also known, where i ∈ [1...N ]. Let h iT T CPx T CPy T CPz be the translation from the origin of the tool flange coordinate system to the tool center point. Let Q be the homogeneous transformation from the tool flange to the tool center point:   1 0 0 T CPx  0 1 0 T CP   y   Q=  0 0 1 T CPz  0 0 0 1 It is obvious that equation 2.1 is satisfied, due to the fact that the tool center point is at the same coordinate independent of which of the N robot positions that is examined. (2.1) T1i Q = T1j Q Where i, j ∈ [1...N ] and i 6= j. Denote: 

 

T1i

a11 a12 a22 a32 a41 a42

a13 a23 a33 a43

a14 a24 a34 a44

b11 b21 b31 b41

b13 b23 b33 b43

b14 b24 b34 b44

 a 21 =    a31

   

T1j = 

b12 b22 b32 b42

   Ra     =    0 0 

 ta  

0

  1  

(2.2)





tb  

Rb

    =  

 

0

0

0

(2.3)

1

Examine one row of equation 2.1 gives a11 T CPx + a12 T CPy + a13 T CPz + a14 = b11 T CPx + b12 T CPy + b13 T CPz + b14 ⇔

8

CHAPTER 2. ROBOTICS

(a11 − b11 )T CPx + (a12 − b12 )T CPy + (a13 − b13 )T CPz = −(a14 − b14 ) (2.4) For every equation 2.1 a system of three equations like equation 2.4 is retrieved. The system of three equations can be rewritten as  h

Ra − Rb

i



T CPx h i    T CPy  = − ta − tb T CPz

(2.5)

By using all distinct combinations i, j of the N robot positions, a system of h iT is then easily calculated equations 2.5 is retrieved. T CPx T CPy T CPz as the linear least square solution of the system of equations. As the rotation part of the transformation from the tool flange coordinate system to the tool coordinate system is of no importance for determine the tool center point, all information needed for determine the coordinate of the tool center point is hereby achieved.

Chapter 3 Theory This chapter will describe theories necessary for understanding the rest of the thesis. The chapter is divided in image processing theories, computer vision theories and transformation theories.

3.1 3.1.1

Image Processing Theory Threshold

The threshold operation is a grey level/one color channel image processing method resulting in a binary image. Let the image be f (x, y), the resulting binary image b(x, y) and the threshold value T then (

b(x, y) =

3.1.2

1 if f (x, y) ≥ T 0 if f (x, y) < T

Morfological operations

Morfological operations are methods working on binary images. During this section the following terminology will be used. f (x, y) is the binary image. r(x, y) is the resulting binary image. a(x, y) is a structure element. See [2] for more information.

3.1.3

Structure Element

A structure element is a binary image a(x, y) that defines the erosion or the dilation operations. The structure element is used as a convolution kernel in the 9

10

CHAPTER 3. THEORY

dilation operation and a cross correlation kernel in the erosion operation. 3.1.3.1

Erosion

The erosion operation is accomplished by setting all object pixels within a certain distance from a background pixel to 0. In practice a structure element is used. The origin of the structure element is then translated to every object pixel. If all of the structure element is accommodated in the object at a certain position the pixel is set to 1 in the resulting image, otherwise 0. r(x, y) = a(x, y) f (x, y) Where is the erosion operator. The erosion operation can also be seen as a cross correlation between the structure element a(x, y) and the image f (x, y). (

r(x, y) =

1 if a(x, y) ? f (x, y) = A 0 if a(x, y) ? f (x, y) 6= A

Where A is the number of pixels in the structure element.

Notice, a(x, y) f (x, y) 6= f (x, y) a(x, y) due to the fact that correlation does not fulfill the commutative law

Figure 3.1: Original image before morfological operation.

Figure 3.2: Erosion operation applied to figure 3.1. A 8 × 8 structure element was used.

3.1.3.2

Dilation

The dilation operation is the opposite of the erosion operation. It is accomplished by setting all background pixels within a certain distance from an object point to

3.1. IMAGE PROCESSING THEORY

11

1. In practice a structure element is used. The origin of the structure element is then translated to every object pixel. In every position a pixel wise OR operation is carried out. r(x, y) = a(x, y) ⊕ f (x, y) Where ⊕ is the erosion operator. The dilation operation can also be seen as a convolution between the structure element a(x, y) and the image f (x, y). (

r(x, y) =

1 if a(x, y) ∗ f (x, y) ≥ 1 0 else

Figure 3.3: Dilation operation applied to figure 3.1. A 4 × 4 structure element was used.

3.1.3.3

Opening

The opening operation consists of one erosion operation followed by one dilation operation, where the distance is the same in both of the operations. When implemented with a structure element the same structure element is used in the two operations. The opening operation will split two objects which border on each other. 3.1.3.4

Closing

The closing operation consists of one dilation operation followed by one erosion operation, where the distance is the same in both of the operations. When implemented with a structure element the same structure element is used in the two operations. The closing operation will merge two objects which border on each other.

12

3.2 3.2.1

CHAPTER 3. THEORY

Computer Vision Theory Camera System

The lens inside the camera refracts all rays of light from a certain object point to one single point in the image plane. If the lens is thin implying the distortion can be neglected, the lens law is valid. 1 1 1 + = α β f

(3.1)

Where α is the distance between the lens and the object, β is the distance between the lens and the image plane and f is the focal length. Figure 3.4 illustrates the lens law.

Figure 3.4: Illustration of the lens law. By the lens law it is obvious that an object at the distance α from the lens will be reproduced with complete sharpness on the image plane. If the distance between the object and the lens differs from α, the reproduction on the image plane will be more or less blurred. How blurred the image will be, is determined by the depth of field s. ! f s = 2λ (3.2) D Where D is the diameter of the aperture and λ is the wavelength of the incoming ray of light. The depth of field can also be defined as the interval where the

3.2. COMPUTER VISION THEORY resolution is greater than

1 2d

13

where the resolution

1 d

is defined as following

1 1D = d λf

(3.3) [3]

3.2.2

Pinhole Camera Model

One common way of modeling a camera is to use the pinhole camera model. The model performs well as long as the lens is thin and no wide-angle lens is used. In practise the image plane is located behind the lens, but to simplify calculations and relations between the coordinate systems, the image plane can be put in front of the lens. Figure 3.5 illustrates the pinhole camera model with the image plane located in front of the lens.

Figure 3.5: Illustration of the pinhole camera model, image plane in front of the lens to simplify calculations (Courtesy of Maria Magnusson Seger). Equation 3.4 shows the relation between the coordinate systems. 









un U h i      W  vn  =  V  = R t   1 W Where

h

U V

W

iT

are the camera coordinates,

malized image coordinates and

h

X Y

Z

iT

h

X Y Z 1

    

un vn

(3.4)

iT

are the ideal nor-

are the world coordinates.

14

CHAPTER 3. THEORY h

i

Matrix R t is the extrinsic parameters and describes the translation and the rotation between the coorinate systems, e.g. how the camera is rotateted and translated in relation to the origin of the world coordinate system. The real image plane usually differs from the ideal normalized image plane. Equation 3.5 describes the relation between the two planes. 







u un     v   = A  vn  1 1

(3.5)

Where A defines the intrinsic parameters. 



α γ u0   A =  0 β v0  0 0 1 "

#

u0 Where α and β are scaling factors for the u and v axes. are the image v0 coordinates of the intersection between the image plane and the optical axis. γ describes the skewing between the u and v axes. Observe, the relation in equation 3.5 implies equation 3.4 can be rewritten as  X u h i Y      W  v  = kA R t    Z  1 1 



Where k is an arbitrary constant. Let s = as 

u 1

h

(3.6)

implies equation 3.6 can be rewritten 



  s v  = A

W k





i R t   

X Y Z 1

    

(3.7)

[3]

3.2.3

Camera Calibration

Equation 3.7 describes how a point in the environment maps to the image plane up to a scale factor s. Let: h i C=A R t (3.8)

3.2. COMPUTER VISION THEORY

15 iT

h

h

and the world Xi Yi Zi If N corresponding points in the image ui vi are found and i ∈ [1 . . . N ] then C can be determined to up a scale factor. 



C11 C12 C13 C14   C =  C21 C22 C23 C24  C31 C32 C33 C34

(3.9)

By setting C34 = 1 in equation 3.9 the scale factor is determined. Let: c=

h

C11 C12 C13 C14 C21 C22 C23 C24 C31 C32 C33

iT

To determine c (determine the intrinsic and extrinsic parameters) a system of equations given by the corresponding points can be used. Dc = f    D=  

X1 Y1 Z1 0 0 0 .. .. .. . . . 0 0 0

(3.10)

1 0 0 0 0 −u1 X1 −u1 Y1 −u1 Z1 0 X1 Y1 Z1 1 −v1 X1 −v1 Y1 −v1 Z1 .. .. .. .. .. .. .. .. . . . . . . . . 0 XN YN ZN 1 −vN XN −vN YN −vN ZN    f =  

u1 v1 .. .

     

     

vN c is then given by

c = DT D

−1

D T f = D† f

(3.11)

Note, at least six corresponding points are needed to determine the intrinsic and extrinsic parameters.

[3]

iT

16

3.3

CHAPTER 3. THEORY

Transformation Theory

3.3.1

Homogeneous Transformation

3.3.1.1

3D translation

Let point P =

h

x y z

translation vector t =

h

iT

0

be translated to a point P =

tx ty tz

iT

h

x0 y 0 z 0

. Then P can be expressed as

0

(3.12)

3D Rotation

Let point P = h

by the

0

P =P+t 3.3.1.2

iT

x0 y 0 z 0

iT

h

x y z

iT

0

be rotated θ◦ around the Z axis to a point P =

0

. Point P can be expressed according to [4] as x0 = x cos θ − y sin θ x0 = x sin θ + y cos θ z0 = z

Let:



(3.13) (3.14) (3.15) 

cos θ − sin θ 0   Rz =  sin θ − cos θ 0  0 0 1 0

Then P can be written as

0

P = Rz P 3.3.1.3

(3.16)

Homogeneous Transformation Matrix

Translation and multiplicative terms for a three dimensional transformation can be combined to a single matrix. Expand the three dimensional coordinates P and P 0 in section 3.3.1.1 and section 3.3.1.2 to four element column vectors as e P =

h

xh yh zh h

f0

h

x0h yh0 zh0 h

P =

iT iT

Where h is the nonzero homogeneous parameter. x = xhh , y = yhh , z = zhh x0 y0 z0 x0 = hh , y 0 = hh , z 0 = hh

3.3. TRANSFORMATION THEORY

17

For a geometric transformation the homogeneous parameter h can be set to any nonzero value. Suitably h is set to 1 implying e = eh where e ∈ {x,y,z}. The translation in equation 3.12 can be represented by a homogeneous matrix M as f0 e P = MP (3.17)    

M=

1 0 0 0

0 1 0 0

0 0 1 0

tx ty tz 1

    

The rotation in equation 3.16 can be represented by the homogeneous matrix M if   cos θ − sin θ 0 0  sin θ cos θ 0 0     M=  0 0 1 0  0 0 0 1 A complete transformation of a rigid body e.g. a transformation between two different coordinate systems can be expressed as a homogeneous transformation matrix T.   r11 r12 r13 tx  r  21 r22 r23 ty  T=    r31 r32 r33 tz  0 0 0 1

3.3.2

Screw Axis Rotation representation 



a11 a12 a13   A rotation matrix R =  a21 a22 a23  is completely defined by the axis of a31 a32 a33 rotation

h

nx ny nz

iT

a11 a12 a13 a21 a22 a23 a31 a32 a33

and the rotation angle θ as = = = = = = = = =

(n2x − 1)(1 − cos θ) + 1 nx ny (1 − cos θ) − nz sin θ nx nz (1 − cos θ) + ny sin θ ny nx (1 − cos θ) + nz sin θ (n2y − 1)(1 − cos θ) + 1 ny nz (1 − cos θ) − nx sin θ nz nx (1 − cos θ) − ny sin θ nz ny (1 − cos θ) + nx sin θ (n2x − 1)(1 − cos θ) + 1

(3.18) (3.19) (3.20) (3.21) (3.22) (3.23) (3.24) (3.25) (3.26)

18

CHAPTER 3. THEORY

Equation 3.26 is called the screw axis representation of the orientation. [5] h iT Given a rotation matrix R, the angle of rotation θ and the rotation axis nx ny nz can be obtained by the following equations. a11 + a22 + a22 − 1 (3.27) 2 a32 − a23 nx = (3.28) 2 sin θ a13 − a31 ny = (3.29) 2 sin θ a21 − a12 nz = (3.30) 2 sin θ This representation makes it possible to compare two different measuring units which uses different coordinate systems to measure the rotation. In this project the method was used to ensure the camera measured the same rotation angle as the robot, when the robot was told to perform a rotation. θ = cos−1

3.3.3

Rodrigues´s formula

One way of finding the rotation matrix R in section 3.3.2 is to make use of the Rodrigues’s formula for a spherical displacement of a rigid body. Let point P1 rotate around the rotation axis n, resulting in a new position P2 of the point. See figure 3.6. The Rodrigues’s formula is then defined according to equation 3.31

Figure 3.6: Definition for Rodrigues’s formula. r2 = r1 cos θ + n × r1 sin θ + n(rT 1 n)(1 − cos θ)

(3.31)

3.3. TRANSFORMATION THEORY

19

According to [5] equation 3.31 can be rewritten as equation 3.32 r 2 = 1 R2 r 1 Where 1 R2 is the rotation matrix fulfilling the equations 3.18 - 3.26

(3.32)

Chapter 4 Determination of Tool Center Point This chapter describes how the problem of finding the tool center point was solved. The chapter starts by describing the proposed method for determination of the camera to tool, tool to robot wrist and camera to robot base coordinate system transformations respectively. The chapter also describes the proposed iterative method for finding the tool center point. The iterative method is used to remove uncertainties due to camera calibration errors and robot calibration errors.

4.1

Overview of the system

Figure 4.1: Overview of the system. Figure 4.1 shows the arrangement of the system. A computer is connected to a camera by a USB cable and to the robot by a TCP-IP cable. 21

22

4.1.1

CHAPTER 4. DETERMINATION OF TOOL CENTER POINT

Equipment

The robot used during the project was an IRB 1400 ABB robot controlled by an IRC5 computer. A laptop computer with an Intel PentiumM 1600Mhz processor and 1Gb RAM was connected to the IRC5 computer by a TCP-IP cable. A program was then developed in C# and C/C++ making it possible to control the robot movements directly from the laptop. The laptop computer was also used for image segmentation of the tool. The images were retrieved at 50 frames per second by an uEye (UI221x-M V2.10) CCD grayscale USB camera with a resolution of 640 × 480 pixels. Except for the equipment, four different transformations T1 , T2 , X1 and X2 are displayed in figure 4.1. By defining a tool coordinate system TF (tool frame) with its origin at the tool center point, the problem of finding TCP actually becomes equivalent by finding the transformation X1 from the tool frame TF to the TFF (Tool Flange Frame) coordinate system fixed at the last axis of the robot manipulator. To make it possible to determine transformation X1 , a closed loop of homogeneous transformation matrixes can be written as an equation. T2 X1 = X2 T1

(4.1)

Transformation matrix T1 in Equation 4.1 is the robot’s forward kinematics. Consequently T1 is known, making the problem of finding X1 become equivalent by determine X2 and T2

4.2

Determine T2

By calibrating the camera using the method proposed by Zhengyou Zhang [6] the intrinsic parameters of the camera were determined and the lens distortion coefficients were retrieved, see Appendix A. h iT h iT Let m = u v denote a 2D image point and M = X Y Z denote a 3D point. The augmented representations of these vectors are achieved h iT f f = u v 1 = by adding 1 as the last element of the vector. m and M h

iT

X Y Z 1 . By modeling the camera as a perfect pinhole camera, the relation between a 3D point M and its projected image point m is fulfilling equation 4.2 f f = ATM sm (4.2) Where A is the intrinsic parameters, s is a scale factor and T is the extrinsic parameters. h i T = r1 r2 r3 t

4.2. DETERMINE T2

23

By setting up a plane in the world, the homography between the plane and its image can be determined. Let the plane fulfilling the constraint of Z = 0. Equation 4.2 is then rewritten as       X u h h i Y  i X      s  = A r1 r2 t  Y   v  = A r1 r2 r3 t   0  1 1 1 h

i

i

h

Let the homography H = h1 h2 h3 = A r1 r2 t . According to [6] this implies the extrinsic parameters can be determined as r1 = λA−1 h1 r2 = λA−1 h2 r3 = r1 × r1 t = λA−1 h3 Where λ = || A−11 h || = || A−11 h || 1

2

By letting the plane be expressed in the tool coordinate system TF, finding transformation T2 in figurei4.1 becomes equivalent of finding the extrinsic parameters h T = r1 r2 r3 t . From 4.2 it is obvious that T is totally determined by the intrinsic parameters A and the homograpghy H. The camera calibration toolbox for Matlab was used to determine the intrinsic parameters, A, for the camera. See Appendix A. This toolbox uses theh same technique i as mentioned in this section and in [6]. The homography H = h1 h2 h3 can be obtained according to [6] by minimizing 

 P

i

mi −

1 T h3 Mi

T

T



 h1 Mi  (σ 2 I)−1 mi − T i h2 M i

 1 T h3 Mi

T



 h1 Mi  T h2 M i

Where hj is the j:th row of H, I is the identity matrix and σ is the standard deviation of the Gaussian noise that is assumed to affect mi . The problem can be solved as a non-linear least square problem.  P minH i mi −

 1 T h3 Mi

T

 2

 h1 Mi  T h2 M i

Due to the fact that a plane is by definition defined of at least three points, at least three corresponding points mi and Mi belonging to the plane needs to be determined. An image segmentation method for finding three distinct points in the plane lying in the tool coordinate system TF, is described in chapter 5. The tool used for the segmentation was a calibration tool with an awl looking shape. The TCP of the tool was the tip of the awl, and the TCP was one of the points determined by the method.

24

4.3


Finding X1 and X2 transformation matrixes

When the transformations T1 and T2 were found only X2 remained to be determined. Jintao and Daokui proposed a method in [7] where the transformation X2 was determined by moving the robot to a fixed position e.g. the tip of a nail with known relation to a calibration pattern. Zhuang et al. [8] proposed a method that solved X1 and X2 simultaneously by applying quaternion algebra to derive a linear solution. Fadi Donaika and Radu Horaud [9] proposed one closed form method and one method based on non-linear constrained minimization for solving the equation 4.1 of homogeneous matrixes. The method implemented was first described by Roger Y. Tsai and Reimar K. Lenz [10]. By moving the robot to a second position finding T1 and T2 at the new location of the robot, a system of two equations 4.3 and 4.4 can be achieved, please see figure 4.2 T21 X1 = X2 T11

(4.3)

T22 X1 = X2 T12

(4.4)

By multiply the inverse of equation 4.4 by equation 4.3 equation 4.6 will be obtained. (T22 X1 )−1 T21 X1 = (X2 T12 )−1 X2 T11

(4.5)

⇔ −1 = T−1 12 X2 X2 T11 ⇔ −1 T−1 T X 22 21 1 = X1 T12 T11 ⇔

−1 X−1 1 T22 T21 X1

AX1 = X1 B

(4.6)

−1 Where A = T−1 22 T21 and B = T12 T11 are known. Observe, A and B are the transformations from position 1 to position 2 of the tool flange frame and the tool frame respectively. See figure 4.2 The transformation matrixes A,B,X1 are all homogeneous matrixes. A homogeneous matrix consists of one rotation matrix R and one translation vector t.

4.3. FINDING X1 AND X2 TRANSFORMATION MATRIXES

25

Figure 4.2: Transformations AX1 = X1 B. RB is the robot base frame, CF is the camera frame, TF is the tool frame and TFF is the tool flange frame. A and B are the transformations from position 1 to position 2 of the tool flange frame and the tool frame respectively.

X1 =

RX1 tX1 0 0 0 1

!

!

A=

RA tA 0 0 0 1

!

B=

RB tB 0 0 0 1

The rotation matrix R can be written as T





n21 + (1 − n21 ) cos θ n1 n2 (1 − cos θ) − n3 sin θ n1 n3 (1 − cos θ) + n2 sin θ    n22 + (1 − n22 ) cos θ n2 n3 (1 − cos θ) − n1 sin θ  R =  n1 n2 (1 − cos θ) + n3 sin θ  n1 n3 (1 − cos θ) + n2 sin θ n2 n3 (1 − cos θ) + n1 sin θ n23 + (1 − n23 ) cos θ (4.7) h i Where n1 n2 n3 is the rotation axis and θ is the rotation angle. Obviously it is possible to specify R in equation 4.7 by specifying θ.

h

n1 n2 n3

i

and

26


By using a modified formula, see section 3.3.3, a funch version of Rodrigues i tion Pr depending on n1 n2 n3 and θ can be defined as Pr = 2 sin

i θh n1 n2 n3 , 2

0≤θ≤π

(4.8)

Let: PA , PB , PX1 be the rotation axes defined according to equation 4.8 for RA , RB , RX1 respectively.

1 0 PX1 = q 4 − |PX1 |2

(4.9)

For a vector v = [vx vy vz ] let 



0 −vz vy  0 −vx  Skew(v) =  vz  −vy vx 0

(4.10)

0

By setting up a system of linear equations according to equation 4.11, PX1 can be solved by using linear least square techniques. 0

Skew(PB + PA )PX1 = PA − PB

(4.11)

Proof: 0

PA − PB ⊥ PX1

(4.12)

PA − PB ⊥ PA − PB

(4.13)

By equation 4.12 and 4.13 follows 0

PA − PB = s(PA + PB ) × PX1 Where s is a constant 0

PA − PB and (PA + PB ) × PX1 has got the same length implying s = 1.

(4.14)

4.3. FINDING X1 AND X2 TRANSFORMATION MATRIXES

27

0

Let α be the angle between PX1 and (PA + PB ). 0

0

|(PA + PB ) × PX1 | = |PA + PB ||PX1 | sin α = ,

=

,

Use equation 4.9

=

1

|PA + PB |2 sin 2θ (4 − 4 sin2 2θ )− 2 sin α = |PA + PB | tan 2θ sin α = |PA − PB | This implies equation 4.14 can be written as 0

PA − PB = (PA + PB ) × PX1

(4.15)

By equation 4.15 and the relation a × b = Skew(a)b follows 0

Skew(PB + PA )PX1 = PA − PB Q.E.D Skew(PB + PA ) is always singular and therefor at least two pairs of positions are needed to create the system of equation 4.11 e.g. three different positions for 0 the robot are needed. When PX1 is determined PX1 can be retrieved by equation 4.16. The rotation RX1 is then determined by equation 4.17 0

2PX1

PX1 = q 0 1 + |PX1 |2 RX1

q |PX1 |2 1 T = (1 − )I + (PX1 PX1 + 4 − |PX1 |2 Skew(PX1 )) 2 2

(4.16)

(4.17)

Let:    

TX1 = 

1 0 0 1

0 1 0 0

0 | 0 tX1 1 | 0 1

    

(4.18)

28

CHAPTER 4. DETERMINATION OF TOOL CENTER POINT    

TA =   

 TB =  

1 0 0 1

0 1 0 0

0 | 0 tA    1 |  0 1

(4.19)

1 0 0 1

0 1 0 0

0 | 0 tB 1 | 0 1

(4.20)



    

To find the translation tX1 of the homogeneous transformation X1 , a linear system of equations 4.21 can be solved by a linear least square technic. (RB − I)TX1 = RX1 TA − TB

(4.21)

Proof: AX1 = X1 B ⇔ (TB + RB − I)(TX1 + RX1 − I) = (TX1 + RX1 − I)(TA + RA − I) ⇔ T B T X 1 + T B RX1 − T B + RB T X1 + RB RX1 − RB − T X1 − RX1 + I = −RX1 − TA + RX1 TA + RX1 RA + TX1 + TA − I ⇔ T B + T X1 − I + T B + RX1 − I − T B + RB T X1 + RB RX1 − RB − T X1 − RX1 + I = −RX1 + RX1 TA + RX1 RA + TX1 − I ⇔ TB − I + RB TX1 + RB RX1 − RB = −RX1 + RX1 TA + RX1 RA + TX1 − I ⇔ RB TX1 + RB RX1 − RB − TX1 =−RX1 + RX1 TA + RX1 RA − TB ⇔ ,

,

Remove all terms not having a translation ⇔ RB T X1 − T X 1 = RX1 T A − T B ⇔ (RB − I)TX1 = RX1 TA − TB Q.E.D. At least three different positions of the robot are needed to calculate the equations 4.11 and 4.21 by a linear least square technique. In the implementation, four

4.4. IMPORTANCE OF MEASURE A AND B WITH 6 DOF

29

different positions were used. The result varied much due to how the positions were chosen. The best estimation of X1 was achieved when the positions were fulfilling the constraint of completely exciting all degrees of freedom. In practise this constraint is equivalent of ensuring the three first positions differ in both a linear movement along orthogonally axes x, y, z and rotations around each axis. The h iT h iT = 1 1 1 fourth position was chosen to be a movement in x y z and a rotation around the same axis.

4.4

Importance of measure A and B with 6 DOF

Actually, if only five degrees of freedom can be measured in T1 or T2 , no unique solution X1 exists to the equation AX1 = X1 B. This situation occurs for instance if the tool has an axis of symmetry. Proof: First four lemmas have to be defined.

Lemma1 : Matrixes A and B are similar if a matrix X exists such as B = X−1 AX ⇔ AX = XB If A and B are similar then A and B has got the same trace, determinant and eigenvalues.[11]

Lemma2 : The eigenvalues of a rotation matrix R are λ1 = 1 λ2 = cos θ + i sin θ λ3 = cos θ − i sin θ Where θ is the angle of rotation specified by equation 4.7 [12]

30


Lemma3 : A rotation matrix R fulfills according to [13] the orthogonality conditions R−1 = RT and RT R = I

Lemma4 : For a matrix R fulfilling the ortogonality conditions, the trace of R is the sum of the eigenvalues of R according to [12]. P T r(R) = i λi Lemma 2 and 3 in Lemma 4 gives T r(R) = 1 + cos θ + i sin θ + cos θ − i sin θ ⇔ T r(R) − 1 θ = arccos 2

!

(4.22)

If the robot is told to perform a rotation θ around the axis of symmetry, the robot will measure a rotation θB = θ. However the camera will not be able to distinguish any difference between the orientation before and after the rotation around the axis of symmetry. This implies the camera will measure a rotation θA = 0. It is then obvious by equation 4.22 that in general T r(A) 6= T r(B) implying according to Lemma1, there is no solution X1 to equation AX1 = X1 B. Q.E.D

Observe, once transformation X1 is determined, the transformation X2 between the camera coordinate system and the robot base frame can be obtained as X2 = T2 X1 T−1 1 . The calibration tool described in chapter 5 had an axis of symmetry making it impossible to find a solution X1 to equation AX1 = X1 B. Instead, the non symmetric chess board in figure 4.3 was used as a robot tool. The TCP was defined as the upper left corner of the chess pattern. Even though it is impossible to find a solution X1 to equation AX1 = X1 B if the robot tool has an axis of symmetry, it is still possible to determine X1 by using another approach, see section 4.6.

4.5. ITERATIVE METHOD FOR INCREASING THE ACCURACY

31

Figure 4.3: Chessboard pattern.

4.5

Iterative Method for increasing the accuracy

Although the transformation X1 is known and the coordinate of the tool center point is found, the accuracy of the method is under influence of errors in the camera calibration and errors in the robot calibration. To ensure a high accuracy an iterative method had to be used. Denote the tool center point coordinate found by the method as TCPguess and the correct coordinate of TCP as TCPcorrect . Assume the robot reorient around TCPguess . If TCPguess = TCPcorrect the TCPcorrect would stay at the same point TCPguess during the reorientation, while TCPcorrect would move away from TCPguess if TCPguess 6= TCPcorrect . This phenomena can be used to achieve a great accuracy of the tool center point calibration. Denote the error vector between TCPguess and TCPcorrect after a reorientation around TCPguess as . By measure the coordinate of TCPcorrect after the reorientation around TCPguess , can be retrieved as = TCPcorrect − TCPguess

(4.23)

Of course the new measurement of TCPcorrect is only a guess, retrieved in the same way as TCPguess and with the same accuracy. This implies e.g. = TCPguess2 − TCPguess

(4.24)

The robot can then be told to perform a linear movement of −. Ideally this would

32


imply TCPguess = TCPcorrect but in most cases TCPcorrect 6= TCPguess2 implying TCPguess 6= TCPcorrect . Instead TCPcorrect is measured again by the camera and a new is calculated. This procedure is done iteratively until || < β, where β is the desired accuracy. Observe, to retrieve a faster convergence of the iterative algorithm the robot can be told to move −α, where α is a constant < 1. When the desired accuracy β is retrieved, two positions of the tool are known where TCPcorrect = TCPguess . The orientation of the tool differs between the two positions, because only linear movements are used when moving the robot −. By performing reorientations around TCPguess in six different directions and using the iterative procedure to linearly move TCPcorrect back to TCPguess , six different positions are obtained. All of the six robot positions ensure TCPcorrect is at the same certain point TCPguess , but with six different orientations. Let T1i , i ∈ [1...6] be the homogeneous transformations from the robot base frame coordinate system to the tool flange coordinate  system e.g.  the robot’s forT CPx   ward kinematics for the six different positions. Let  T CPy  be the translation T CPz from the origin of the tool flange coordinate system to the tool center point. Let:    

Q=

1 0 0 0

0 1 0 0

0 T CPx 0 T CPy    1 T CPz  0 1 

Due to the fact that the tool center point is at the same coordinate independent of which of the six robot positions that is examined, the algorithm described in section 2.3 on page 7 can be used to determine the true position of the tool center point. By using the iterative procedure and calculating the tool center point as a least square solution described in section 2.3, the accuracy of the TCP calibration method becomes independent of the camera calibration errors and the robot calibration errors. Instead, only the accuracy of the camera when registering a movement between two different positions, affects the final accuracy of the tool center point calibration achieved by the iterative method. This can be assured by considering that the camera is an external observer during the iterative procedure, and the method will iterate until the camera measures (with the desired accuracy) the true tool center point is at the same location as the initial guess point.

4.6. ALTERNATIVE METHOD FOR ROTATION SYMMETRIC TOOLS

4.6

33

Alternative method for rotation symmetric tools

If the transformation between the camera and the robot’s base frame, X2 , is known, the iterative method for increasing the accuracy will work even if the tool is symmetric. During the iterative method, the robot performs a reorientation and the camera only needs to measure the translation error between the positions before and after the reorientation. This assures the transformation, T2 between the camera frame and the tool frame only needs to be determined with five degrees of freedom during the iterative procedure. Although, the direction of the movement - must be expressed in relation to the robot’s base frame. Therefore the rotation part of the transformation, X2 between the camera frame and the robot base frame needs to be determined. The rotation part, RX2 , of the homogeneous transformation X2 can be obtained according to [4] by moving the robot a certain distance in the X,Y and Z directions respectively in the robot’shbase frame. iT h iT h iT and w = wx wy wz Let u = ux uy uz , v = vx vy vz be the vectors measured by the camera when the robot moves a specified distance in the X, Y and Z directions respectively. According to [4] the rotation part RX2 , of the homogeneous transformation X2 is then obtained as 

RX2



ux vx wx  =  uy vy wy   uz vz wz

(4.25)

The translational scaling factor between the two coordinate systems is also possible to determine, because the distances the robot moves in the X, Y and Z directions are known and the distances of the same movements measured by the camera are also retrieved. This assures it is possible to find the translation error between TCPguess and the true coordinate of the tool center point TCPcorrect , even if the robot tool is rotation symmetric. It is thereby possible to perform a tool center point calibration, even if the robot tool is rotation symmetric.

Chapter 5 Image Segmentation of Robot Tool This chapter will describe the methods evaluated during this project for image segmentation of the robot tool.

5.1

Overview

Since one of the goals of this thesis was to deliver a software program made in C/C++ and C# , all of the image processing had to be written in either of these program languages. The HMI part of the program was written in C# and the image processing functions were written in C/C++. To keep the production costs as low as possible only open source libraries were allowed to be used during this project. Due to this constraint the image processing functions in Open Computer Vision Library 1.0 (OpenCV 1.0) were mainly used for image segmentation.

5.1.1

Open Computer Vision Library (OpenCV)

OpenCV 1.0 is an open source project written in C consisting of a large collection of image processing algorithms. The library is also compatible with Intel’s IPL image package and has got the ability to utilize Intel Integrated Performance Primitives for better performance. [14] [15]

5.2

Creation of binary mask image

To retrieve a rough approximation of the image region where the object is located numerous approaches can be used. In this section two different methods will be evaluated. 35

36

CHAPTER 5. IMAGE SEGMENTATION OF ROBOT TOOL

5.2.1

Image registration

The idea of image registration is to find the transfer field v(x) : R2 → R2 , x = " # x making a reference image I2 (x, y) fit as good as possible in a target image y I1 (x, y) e.g. finding v(x) minimizing in equation 5.1 2 = ||I2 (x + v(x)) − I1 (x)||

(5.1)

The method outlined in this section is described in more detail in [16]. By letting the reference image be a picture of the desired tool and the target image be the image achieved by the camera, this method should iteratively be able to find the location of the tool in the target image. One constraint of the target image have to be fulfilled.

ConstraintI : The image can locally be described as a sloping plane. I(x, t) = I(x, t) − ∇I T (x, t)v(x) + ∆t ∆t = I(x,"t + 1) − I(x, t) # ∇x I(x, t + 1) ∇I(x, t) = ∇y I(x, t + 1)

ConstraintI can be rewritten as I(x, t) = I(x, t) − ∇I T (x, t)v(x) + ∆t ⇔ I(x, t) = I(x, t) − ∇I T (x, t)v(x) + (I(x, t + 1) − I(x, t)) ⇔ I(x, t) = −∇I T (x, t)v(x) + I(x, t + 1) ⇔ ∇I T (x, t)v(x) = I(x, t + 1) − I(x, t)

(5.2)

5.2. CREATION OF BINARY MASK IMAGE

37

Let: I1 = Ix,t+1 I2 = Ix,t v(x) = B(x)p "

B(x) =

1 0 x y 0 0 0 1 0 0 x y 

#



p1  .   p =  ..   p6 Then 5.2 can be written as ∇I2T B(x)p = I1 − I2 Let:

(5.3)

∇I2 (x1 )T B(x1 )   ..  A= .   ∇I2 (xn )T B(xn ) 







I1 (x1 ) − I2 (x1 )   ..   b= .  I1 (xn ) − I2 (xn ) Where n is the number of pixels in I2 . Equation 5.3 is then rewritten to equation 5.4. Ap = b (5.4) This implies p and v(x) are determined as p = (AT A)−1 AT b = A† b v(x) = B(x)p

(5.5) (5.6)

The new location I2new of the reference image is then obtained by interpolating I2 from v(x). The method is then iterated with I2new as I2 until |v(x)| < β, where β is the desired accuracy. The method described in this section was implemented and evaluated in Matlab. Unfortunately the method was not reliable. The resulting position of the

38


reference image I2 was heavily affected by the light condition. Although, when using the method on synthetic binary test images the method performed very well. Of course it would be possible to threshold the image I1 retrieving a binary image Ibinary , and then apply the image registration method on the binary image. Although, finding the threshold value that separates the background completely from the tool is a hard problem to solve. Of course there might exist image registration methods, better suited for solving the problem. A phase based image registration method would perhaps give a better result than the method described here. However, due to the fact that it is possible to move the robot to different positions, the background subtraction method was instead evaluated.

5.2.2

Background subtraction

If several image frames can be acquired and the object is moving between each frame the background subtraction method can be used to retrieve a smaller search area for the object. The background subtraction is done by subtracting two images from each other and perform a threshold. If the scene is completely static except for the moving object, all of the background will be eliminated resulting in a binary mask image where background pixels are set to 0 and object pixels are set to 1. Let • f1 (x, y) be the image acquired at the start position. • f2 (x, y) be the image acquired at the final position. • b(x, y) be the binary result image. • T be threshold value. (

b(x, y) =

1 , kf1 (x, y) − f2 (x, y)k ≥ T 0 , kf1 (x, y) − f2 (x, y)k < T

The resulting binary mask image will of course have its pixels set to one in both the start location and the final location for the object. To distinguish between the object’s true location and the last location, the object can be moved to a third position and a new image f3 can be retrieved. By applying the background subtraction method to f1 − f3 and f2 − f3 separately and applying a pixel wise logical AN D operator to the resulting binary images, a new binary mask image only giving the last location of the object will be retrieved. In practise the resulting binary mask image was not perfect, several holes occurred in the object region see figure 5.1.

5.2. CREATION OF BINARY MASK IMAGE

39

Figure 5.1: Binary mask image from background subtraction. To fill the holes in the mask image, morphological operations were applied to the binary image. By gradually apply dilate and erode operators (cvDilate and cvErode in OpenCV) on the mask image the holes were filled. However, due to the quadrangular structure element used by the morphological operators the tip structure of the object mask image was deteriorated. Later on during the segmentation process the damaged tip structure showed to be an obvious drawback. Instead of using the morphological operators, a new method referred to as the Hough Filling Method (HFM) was invented. 5.2.2.1

The Hough filling method (HFM)

By applying the Progressive Probabilistic Hough Transform (PPHT) described in [17] to the binary mask image in figure 5.1, all probabilistic lines were found. If the permitted gap constraint was set to a suitable value and the minimum distance between pixels which were to be considered as a line was set to zero, all holes were perfectly filled when the lines found by PPHT were drawn on the binary mask image. The HFM method will not increase the mask image boundary as long as the curvature of the boundary remains low. The PPHT algorithm is implemented in the OpenCV function cvHoughLines2 [18]. By applying the background subtraction method and filling gaps and holes with the Hough filling method, almost perfect binary mask images were achieved for all three locations of the tool, see figure 5.2. After the binary mask was used only a small search area was left in the image.

Figure 5.2: Binary mask image from background subtraction after applying the HFM.

40

5.3


Edge Detection

An edge can be treated as a locally odd function in an image. f (−x, −y) = −f (x, y) And such a function will always have a gradient. The easiest way of finding the gradient is probably to convolve the image with the sobel operators.









1 0 −1 −1 2 −1     Gx =  2 0 −2  , Gy =  0 0 0  1 0 −1 1 2 1 The edge image may then be retrieved as fedge =

q

(f ∗ Gx )2 + (f ∗ Gy )2

Unfortunately this method won’t, according to [2], work properly in noise affected environments. Irregular light conditions will also decrease the functionality of this method.

5.3.1

Canny’s Edge Detection

1986 J.Canny presented a more robust technique in [19] for edge detection. In contrast to the previous method, Canny´s edge detection algorithm makes use of the second derivates, which are zero when the first derivates reach their maxima. be the unit vector in the gradient direction. Where fi (x, y) Let n = √ 2 ∇f (x,y)2 fx (x,y)+fy (x,y)

is the image f (x, y) derived in the i direction. The edge image fcanny can be found by deriving the absolute value of the gradient image in the direction of the gradient. 2 2 y (x,y)fxy (x,y)+fy (x,y) fyy (x,y) fcanny = n · ∇k∇f (x, y)k = fx (x,y)fxx (x,y)+2fx (x,y)f fx2 (x,y)+fy2 (x,y) To be able to find the contour of the tool, the Canny edge detection algorithm [19] was used on the masked image (

f (x, y)masked =

f (x, y) , if b(x, y) = 1 0 , if b(x, y) = 0

(5.7)

Where b(x, y) is the resulting image given from the background subtraction and HFM methods. To ensure that all of the tool was left in the masked image fmasked , the dilation operator was applied to b(x, y) before calculating fmasked according to equation 5.7.

5.4. CONTOUR RETRIEVING

41

After applying the Canny edge detection algorithm on the masked image fmasked , the binary edge image bedge was obtained. The Canny algorithm is implemented in the OpenCV library function cvCanny [18]. The threshold values in the cvCanny function had to be determined according to the current environment. If the image contained a lot of noise, the threshold had to be set to a high value to ensure the elimination of the noise. Unfortunately the high threshold sometimes resulted in discontinuities of the tool edges in the edge image bedge . To solve the problem of discontinuities of the tool edges the Hough Filling Method (see section 5.2.2.1 on page 39) was used with an excellent result. Another disadvantage was the presence of a light reflection in the metallic surface of the tool. The contour of the reflection had the same shape as the actual tool, resulting in a possible false hit later in the segmentation procedure. To avoid the influence of light reflections the tool was painted in a non-reflecting colour.

5.4

Contour Retrieving

To retrieve the contours from the binary edge image, bedge , two different methods were evaluated. This section will illuminate the two methods, but first two different contour representations will be described. Observe, it would be possible to retrieve an approximation of the tool contour from the binary mask image, see figure 5.2. In practise the result became more accurate by retrieving the tool contour from the binary edge image bedge given by the Canny edge detection algorithm.

5.4.1

Freeman Chain Code Contour representation

The freeman chain code representation is a compact representation of contours. By denoting the neighbours of the current pixel with different digits, see figure 5.3, the contour can be stored as a succession of digits. Each digit then contains all necessary information for finding the next pixel in the contour.[18] Please, look at the example in figure 5.4

Figure 5.3: The Freeman representation.

42


Figure 5.4: An example of the Freeman Chain Code.

5.4.2

Polygon Representation

The polygonal representation of a contour is according to [18] a more suitable representation for contour manipulations. In the polygonal representation the sequence of points are stored as vertices e.g. the pixel coordinates are stored.

5.4.3

Freeman Methods

The four scan line methods described in [20] are implemented in the OpenCV function cvFindContours [18]. The scan line methods scan the image line by line until it finds an edge. When an edge is found the method starts a border following procedure until the current border is retrieved in the Freeman representation. The first method described in [20] only finds the outer most contours, while the other methods discover contours on several levels. The methods were evaluated on the binary edge image with satisfactory results. The methods with the ability to find contours on different levels had a disadvantage because unnecessary contours were found, increasing the risk of mixing the true object with a light reflection of the same shape.

5.4.4

Active Contour (The Snake algorithm)

Active contours or Snakes are model-based methods for segmentation. The active contour is a spline function v(s) with several nodes. v(s) = (x(s), y(s))

(5.8)

Where x(s), y(s) are the x, y coordinates along the contour. s ∈ [0, 1] Each node has got an internal energy and the algorithm is trying to minimize the total energy, E, of the spline function. E=

Z 0

1

Esnake (v(s))ds

(5.9)

5.4. CONTOUR RETRIEVING

43

The spline function is affected by internal forces Eint (v(s)), external forces Eimg (v(s)) and Econ (v(s)). Esnake = Eint (v(s)) + Eimg (v(s)) + Econ (v(s)) The internal forces affecting the active contour are divided in tension forces and rigidity forces. The tension forces imply spring behaviour of the spline function, while the rigidity forces make the spline function resist bending. dv(s) 2 d2 v(s) 2 + β(s) Eint (v(s)) = α(s) ds ds2

Where α(s) specifies the elasticity and β(s) specifies the rigidity of the spline function. The external forces consist of image forces Eimg and user specified constraint forces Econ . The image force is an image, where each pixel value defines a force. The constraint forces can be used to guarantee that the active contour is not getting stuck at a local minima. The constraint forces might be set by a higher level process.[21] One can compare the snake algorithm to a rubber band expanded to its maximum when the algorithm is initialized. The rubber band then iteratively contract until the energy function in each node has reached equilibrium. By using the binary edge image as the external force and defining the initial contour to be the result of applying the Freeman scan line method to the binary mask image, the active contour will successively contract until it perfectly encloses the object, see figure 5.5

Figure 5.5: The snake contour at initial state to the left and at the final state to the right. The contour is randomly color coded from the start point to the end point of the contour. In fact the mask image and the initial active contour have to be larger than the object itself to be able to enclose the object. If there are structures or patterns

44


behind the object, the active contour might get stuck due to forces in the edge image belonging to these structures see figure 5.6. This drawback together with the fact that the algorithm is iterative and therefore time consuming made the Freeman method better suited for retrieving the contour.

Figure 5.6: The snake contour could get stuck at the background pattern. The contour is randomly color coded from the start point to end point of the contour. The snake algorithm is implemented in the OpenCV function cvSnakeImage. [18]

5.5

Finding the contour matching the tool shape

When all contours were retrieved by the Freeman method, a process of logics had to be applied for actually finding a contour or a part of a contour matching the wanted tool object. Two different methods were implemented and evaluated.

5.5.1

Convex Hull and its defects

A widely used methodology for finding objects with a shape reminding of a hand or a finger is to analyse the convex hull of the object. The awl shape of the tool object does remind much of a finger shape. In the OpenCV library a function named cvConvexHull2 exists for finding the convex hull of a contour. There is also a function for finding the deepest defects of the convex hull e.g. the points most far away from every line segment of the convex hull. The two functions were used to retrieve all points outlining the convex hull and the points constituting the deepest defects of the hull. All five points defining the tool were found by these functions. By finding the two deepest defect points, two of the five desired points could be determined (the two points closest to the robot’s wrist), see figure 5.7. To determine the three remaining points of the tool, lines were drawn between these points and their nearest neighbours, resulting in four different lines. The four lines

5.5. FINDING THE CONTOUR MATCHING THE TOOL SHAPE

45

were then compared two by two to find the two lines being most parallel to each other. See figure 5.8. By this method four out of five of the desired points were easily determined. Finding the last point (TCP) of interest, was then becoming easy while it was the only point lying on the complex hull fulfilling the constraint of being a nearest neighbour of two of the points already found. The method was fast and performed well as long as the robot only made reorientations around axis five during the background subtraction method. When more complex robot movements were used, a huge number of defect points in the convex hull were found resulting in that the method became time consuming. Instead the polygonal approximation method was evaluated.

Figure 5.7: The green lines is the convex hull found by the method, and the red dots are the points defining the convex hull and the defect points.

Figure 5.8: The four lines are illustrated in this figure. The green lines are the two lines being most parallel and found by the logics. The red lines are not parallel and will therefore be neglected.

5.5.2

Polygonal Approximation of Contours

To further compress the retrieved contour several methods can be used such as Run Length Encoding compression and polygonal approximation. In this implementation the Douglas-Peucker polygonal approximation was used (cvApproxPoly in

46


the OpenCV library). The reason of using a polygon approximation method was due to the possibility to set the approximation accuracy to a level where the tool object only consisted of five points see figure 5.9

Figure 5.9: Five points polygonal approximation of the tool.

5.5.2.1

Douglas-Peucker Approximation Method

The method starts in the two points p1 and p2 on the contour having the largest internal distance. The algorithm loops through all points on the contour to retrieve the largest distance from the line p1 p2 . If the maximum distance is lower than the desired accuracy threshold, the process is finished. If not the point p3 at the longest distance from the line p1 p2 is added to the resulting contour. The line p1 p2 is then split into the two line segments p1 p3 and p3 p2 . The same methodology is then applied recursively until the desired accuracy constraint is fulfilled.[18]

5.6

Geometric constraints

When the five points were found some logics were applied to make sure the five points fulfilled the geometry of the tool shape. The angles α1 , α2 and α3 , between the four different lines L1 , L2 , L3 and L4 of the shape, illustrated in figure 5.10, were determined and the area A of the tool was calculated. The longest sides L1 and L2 of the tool had to fulfill the constraint of being parallel and the length of the long sides L1 and L2 had to be at least twice the length of the short sides L3 and L4 . If all these constraints were fulfilled, the five points were classified as part of the tool shape. To ensure only one tool was to be found, the constraints were set to be very tough. If the method did not find a tool when the toughest constraints were used, the constraints were gradually slackened.

Figure 5.10: Definition of geometric constraints of the tool.

5.7. CORNER DETECTION

5.7

47

Corner Detection

When the tool contour was found by the polygonal approximation algorithm, five points defining the object were obtained. To use the camera as a measurement unit, at least three points had to be found. Unfortunately only three of the five points defining the contour of the tool could be found with high accuracy. Due to occlusion the two points closest to the robot wrist were hard to detect. Instead the tool center point (the tip of the awl) and the two points where the cylinder part of the awl starts were used. See figure 5.11

Figure 5.11: Calibration tool with the three points found. All three points had one thing in common, their locations were at corners. By matching the three points given by polygonal approximation algorithm with the closest points of high certainty being a corner, the three points in the figure 5.11 was found. To find the three points of high probability being corners in the neighbourhood of the points given by the polygonal approximation, three different methods were evaluated.

5.7.1

Harris corner detector

The idea of Harris corner detection algorithm described in [22] is to use a function E(x, y) =

X

w(x, y)(I(x + u, y + v) − I(x, y))2

(5.10)

where w(x, y) is a window function, typically a Gaussian function, and I(x + u, y + v) is a shifted version of the intensity image I(x, y). At a corner pixel the difference between I(x + u, y + v) and I(x, y) would be high resulting in a high value of E(x, y). For a small shift (the u and v are small) a bilinear approximation can be used. E(x, y) ∼ =

h

u v

iX x,y

w(x, y)

Ix2 Ix Iy Ix Iy Iy2

!"

u v

#

(5.11)

48


Where Ix is the x-derivate and Iy is the y-derivate of the image I. Let M =

X

w(x, y)

x,y

Ix2 Ix Iy Ix Iy Iy2

!

(5.12)

Equation 5.11 and equation 5.12 give E(x,y) ∼ =

h

u v

i

"

M

u v

#

(5.13)

The eigenvalues λ1 and λ2 of M then include the information of the directions of the fastest and the slowest changes. As mentioned earlier in this section a corner has fast intensity changes in all direction, resulting in high λ1 and λ2 values. λ1 ≈ λ2 = high values λ1 < λ 2 λ1 > λ 2 λ1 ≈ λ2 = small values

Corner Edge Edge Flat

Table 5.1: Interpretation of the eigenvalues. The Harris corner detection algorithm makes use of the information stored in the eigenvalues of M. R = det(M) − k(trace(M)) = λ1 λ2 − k(λ1 + λ2 ), where k is a parameter. (5.14) R |R| > 0 Large < 0 Large − Small

Result Corner Edge Flat

Table 5.2: Interpretation of R. The Harris corner detection algorithm was implemented in Matlab. Figure 5.13 shows the R image when the algorithm was applied to figure 5.12. By thresholding the R image, only points with very high certainty for being a corner were left. The Harris corner detection algorithm is also implemented in the OpenCV function cvGoodFeaturesToTrack.


49

Figure 5.12: Image of robot tool used for Harris corner detection.

Figure 5.13: The R image in Harris corner detection algorithm. 5.7.1.1

Sub pixel Accuracy

By using the positions of the corners found by the Harris detector as input to the OpenCV function cvFindCornerSubPix, the positions of the corners could be retrieved with sub pixel accuracy. Let i be a minimization function, pi is a point in the neighborhood of q, ∇Ii is the gradient in the point pi , q is the corner with sub pixel accuracy, see figure 5.14. (5.15) i = ∇IiT • (q − pi )

Figure 5.14: Accurate sub pixel corner detection. The blue arrows are the image gradients and the green vectors are the vectors from q to pi .

50


Observe, all vectors qpi are orthogonal to ∇Ii . To retrieve the corner point q with sub pixel accuracy, i in equation 5.15 was minimized for several points pi resulting in a system of equations giving q. q=(

X

∇Ii ∇IiT )−1 (

i

X

∇Ii ∇IiT p1 )

(5.16)

i

Please, refer to [18] for more information about sub pixel accuracy.

5.7.2

Fast radial

G. Loy and A. Zelinsky proposed a method in [23] based on Radial Symmetry for finding points of interest. A Matlab implementation by the authors was found. See the result image in figure 5.15 when the algorithm was applied to figure 5.12.

Figure 5.15: Result of the Fast Radial algorithm. The fast radial algorithm resulted in more points of interest in relation to the Harris corner detector. In this particular case the algorithm found many false hits. Due to this effect, the fast radial algorithm was rejected.

5.7.3

Orientation Tensor

To be able to find the three points of the tool with sub pixel accuracy the orientations of the tool sides can be used. The orientation of every pixel at the tool’s contour was found in double angle representation [24]. To find the double angle representation the orientation tensor was determined. Several ways of determine the orientation tensor have been proposed. In [24] G. Granlund and H. Knutsson propose a method based on quadrature filter responses. C. Mota, I. Stuke, and E. Barth describe a method in [25] for creating the orientation tensor as an outer product of derivatives. G Farnebäck proposed another method for retrieving the orientation tensor in [26] based on polynomial expansion.


51

The method based on outer products of derivatives was implemented in Matlab. T11 T12 T21 T22

T=

!

2 Ixx Ix Iy 2 Ix Iy Iyy

=

!

(5.17)

Ixx ,Ixx and Ix Iy were retrieved by 2D convolutions of image I(x, y) and derived 2D Gaussian functions. Let:

−X 2

g(x) = e 2σ2

−Y 2

g(y) = e 2σ2

∂g(x) −X −X 2 = 2 e 2σ2 ∂x σ ∂g(y) −Y −Y 2 = 2 e 2σ2 ∂y σ ∂g(x) ∗ g(y) ∂x ∂g(y) h2 (x, y) = ∗ g(x) ∂y h1 (x, y) =

Figure 5.16 shows h1 (x, y) and h2 (x, y). By performing a 2D convolution of h1 (x, y) and image I(x, y) the image Ix was retrieved. Observe, a faster way to implement the function is to perform a sequence of 1D convolutions on image I(x, y). ∂g(y) Ix = I(x, y) ∗ g(x) ∗ ∂y The double angle representation of orientation can be retrieved from the orientation tensor T. Let v be the double angle orientation vector. v=

v1 v2

!

=

T11 − T22 2T12

!

=

Ix2 − Iy2 2Ix Iy

!

To ensure v is the double angle representation let Ix = cos(φ) and Iy = sin(φ) v=

Ix2 − Iy2 2Ix Iy

!

=

cos(φ)2 − sin(φ)2 2 cos(φ) sin(φ)

!

=

cos(2φ) sin(2φ)

!

The advantage of using a double angle representation of orientation is the possibility to avoid branch cuts in the representation. If only the angle between the

52


Figure 5.16: Gaussian derived in x and y directions respectively. x-axis to an image line is used as orientation information, no average between neighboring points can be done. For instance a line with an angle close to 0 rad and a line with an angle close to π rad will have very different orientations. When using the double angle representation these two lines will instead have orientations close to each other. This implies the possibility of counting an average of orientations when using the double angle representation. By color coding the double angle and letting the intensity be |v| an orientation image can be retrieved, please refer to [24]. Figure 5.18 shows the orientation image of 5.17

Figure 5.17: Test image for double angle representation.


53

Figure 5.18: Orientation image of figure 5.17 By finding the orientation tensor to the image achieved by the camera and calculating the orientation image, figure 5.19 was retrieved. For every pixel a line was drawn in the direction specified by the orientation image. If all lines were accumulated the three points could be found by applying a high threshold. Observe, by calculating the histogram of the orientation image it is possible to tell if the tool was found in the image. Figure 5.20 shows the histogram. Three peaks are observed, the highest peak is the orientation of the two longest sides of the tool, while the two lower peaks are the orientations of the two shortest sides of the tool. The relations between these three orientations are specific for the tool. Of course the tool has to be segmented before calculating the histogram, otherwise the tool specific orientations would be impossible to determine among all the orientation information in the image. The histogram can also be used to find the orientation of the tool (the highest peek in the histogram), after the tool has been segmented.

Figure 5.19: Double angle representation of the robot tool.

54


Figure 5.20: Histogram of orientations.

Chapter 6 Results In this chapter the results of different tests will be presented. The tests were performed to determine the accuracy of the method.

6.1

Repetition Accuracy of the Camera

Figure 6.1 shows the configuration of the camera in relation to the robot tool flange coordinate system. The optical axis of the camera is directed in -X direction, where X is the X axis of the robot base coordinate system.

Figure 6.1: Configuration during the test. To be able to determine the repetition accuracy of the camera the start position h iT of the TCP found by the method was stored in a vector POS1 = x1 y1 z1 . The robot was then randomly moved to a second position. The new position of h iT TCP was measured by the method several times and stored as POSi = xi yi zi . 55

56

CHAPTER 6. RESULTS

Figure 6.2: Movment mi plotted for all i. Where i is the estimation number. The tool used during this test was the chess board in figure 4.3 and 24 points of the chess board pattern were used to determine the transformation q between the camera and the tool (chess board). For all i the movement mi = (x1 − xi )2 + (y1 − yi )2 + (z1 − zi )2 was calculated. Figure 6.2 shows the result of mi . By the graph it is possible to determine the difference between the maximum and minimum interval to approximately 0.03 mm, resulting in that the camera is not able to measure distances shorter than 0.03 mm.

6.1.1

Averaging to achieve higher accuracy

By letting every estimation mj be the average of several mi see equation 6.1, a test was performed to investigate whether a better accuracy could be obtained. P10

mj =

i=1

10

mi

(6.1)

During the test every mj was the average of ten different mi . The result is illustrated in figure 6.3. The difference between the maximum and minimum interval was still determined to 0.03 mm. It is then obvious that averaging the result does not increase the repetition accuracy of the camera when 24 points are used for measuring the transformation between the camera and the tool (chess board).

6.1. REPETITION ACCURACY OF THE CAMERA

57

Figure 6.3: Movment mj plotted for all j.

6.1.2

How does the number of points affect the repetition accuracy

To investigate how the amount of points in the chess board pattern affects the repetition accuracy of the camera, the same test (not with averaging) was done again. Although, this time only the three most upper left points of the chess board pattern were used to measure the transformation between the camera and the tool (chess board). Observe, three points are minimum for finding the transformation between the camera and the tool. This is due to the fact that a plane is needed to be observed by the camera to determine the homography, see section 4.2. A plane is by its definition defined of at least three points. The result of the new test is illustrated in figure 6.4 The difference between the maximum and minimum interval was determined to approximately 0.58 mm. The greater difference between the maximum and minimum interval indicates the number of points used during the test affects the repetition accuracy. Although, still using only three points the camera repetition accuracy is better than 1 mm.

58

CHAPTER 6. RESULTS

Figure 6.4: Movment mi plotted for all i when only 3 points of the chess board were observed.

6.2. CAMERA VERSUS ROBOT MEASUREMENTS

6.2

59

Camera versus Robot measurements

Both the camera and the robot measurements are affected of errors during the camera calibration and the robot calibration respectively. To investigate the effect of these errors different tests were performed. The configuration of the devices is illustrated in figure 6.1. All twenty four points of the chess board were observed during these tests.

6.2.1

Rotation measurements

The robot was told to perform a rotation of θ = 20◦ around a random axis and the forward kinematics transformation matrix T1 and the measured transformation matrix between the camera and the tool T2 were saved before and after the rotation, see figure 4.1 at page 21. The transformation A from the start position to the final position measured by the camera is then A = T−1 2af ter T2bef ore and the transformation B from the start position to the final position measured by the robot is then B = T−1 1af ter T1bef ore , see figure 4.2 at page 25. Due to the fact that A and B are homogeneous matrixes it is obvious that the rotation part of the transformations A and B can be obtained as the upper left 3 × 3 sub matrixes of A and B respectively. Denote these sub matrixes RA and RB respectively. The angle of rotation θrobot measured by the robot and the angle of rotation θcamera measured by the camera were then obtained by using the screw axis rotation representation described in section 3.3.2 on page 17 and equation 3.27 on page 18. The difference γ between θrobot and θcamera was then calculated accorded to the following equation. (6.2) γ = |θrobot | − |θcamera | The test was done several times and the resulting γ:s were plotted in figure 6.5. The difference between the maximum and minimum interval was determined to 0.06◦ and the maximum deviation was 0.04◦

60

CHAPTER 6. RESULTS

Figure 6.5: Difference of rotation measured by the robot versus the camera.


6.2.2

61

Translation measurements

The robot was told to perform linear movements of 10 mm along the three orthogonal axes of the robot base coordinate system. The positions of the robot measured by the robot and the camera respectively were stored after every movement. h iT Denote the positions of the robot measured by the robot as robposi = rxi ryi rzi and the positions of the robot measured by the camera and transformed to the h iT where i ∈ robot’s base frame by applying X2 as camposi = cxi cyi czi [1,N ] and N is the number of movements. Let: robdiff i = robposi+1 − robposi camdiff i = camposi+1 − campos i   dif fxi   diff i = robdiff i − camdiff i =  dif fyi  dif fzi Figure 6.6 illustrates |diff | during movements in the X direction of the robot base coordinate system. Figure 6.7 and Figure 6.8 illustrates |diff | during movements in Y direction and Z direction of the robot base coordinate system respectively. Table 6.1 shows the resulting differences between maximum and minimum intervals and the maximum deviations. Observe, the accuracy is higher in the Y and Z directions in relation to the X direction.

Figure 6.6: Difference of translation in X direction measured by the robot versus the camera.

62

CHAPTER 6. RESULTS

Figure 6.7: Difference of translation in Y direction measured by the robot versus the camera.

Figure 6.8: Difference of translation in Z direction measured by the robot versus the camera.


63

Movement in direction

Difference between maximum and minimum interval [mm]

Maximum deviation [mm]

X direction

2.72

1.45

Y direction

1.49

1.16

Z direction

1.66

1.05

Table 6.1: Table showing the differences between maximum and minimum intervals and the maximum deviations.

64 6.2.2.1

CHAPTER 6. RESULTS Averaging to achieve higher accuracy

The same procedure was redone, but this time every robot position, camposj , measured by the camera was the average of ten measurements. P10

camposj =

i=1

camposi 10

The results are illustrated in figure 6.9, figure 6.10 and figure 6.11. Table 6.2 shows the resulting differences between maximum and minimum intervals and the maximum deviations. Note, the accuracy has increased in all directions in relation to the test when no averaging was used. Although, the accuracy of movements in the X direction is still lower than the accuracy in Y or Z directions. Observe, the accuracy of these tests are approximately ten times lower than for the camera repetition accuracy. This shows the importance of using the iterative method for increasing the accuracy of the TCP calibration method, since the iterative method is independent of the robot calibration error, see section 4.5 on page 31.

Figure 6.9: Difference of translation in X direction measured by the robot versus the camera, averaging has been done.


65

Figure 6.10: Difference of translation in Y direction measured by the robot versus the camera, averaging has been done.

Figure 6.11: Difference of translation in Z direction measured by the robot versus the camera, averaging has been done.

66

CHAPTER 6. RESULTS

Movement in direction


Maximum deviation [mm]

X direction

0.56

0.33

Y direction

0.26

0.49

Z direction

0.21

0.22

Table 6.2: Table showing the differences between maximum and minimum intervals and the maximum deviations. Averaging has been done.

6.3. ACCURACY OF THE TCP CALIBRATION METHOD

6.3

67

Accuracy of the TCP Calibration method

The accuracy of the proposed method is limited to the least distance between two points which the camera is able to register, see section 4.5. Due to the fact that the repetition accuracy of the camera is 0.03 mm, see section 6.1, there is no need of trying to register a shorter distance. A test was performed to find the shortest distance between two points which the camera is able to register. Denote the initial position of the tool center point, determined by the camera and transformed to the robot base coordinate system as p1 . This position was calculated as the mean value of 100 different measurements. The robot was then told to move a distance d and the new position, p2 , was measured by the camera and transformed to the robot base coordinate system. The test was performed until the smallest distance d between p1 and p2 was found, still making the camera able to register a movement. To ensure the camera registered a movement, the distance, dcamera , measured by the camera had to be greater than the repetition accuracy of the camera (of course dcamera was also expressed in the robot base coordinate system). Figure 6.12, figure 6.13 and figure 6.14 illustrate the distance dcamera when the robot was told to move a distance d = 0.10 mm in the X, Y and Z directions respectively. By these figures it is possible to tell that the accuracy in the X direction limits the total accuracy of the method. When the robot was told to move a distance shorter than 0.1 mm some of the measurements, dcamera , became lower than the camera repetition accuracy, resulting in that the system was not able to determine a significant movement. Therefore the accuracy of the proposed method is assumed to be 0.1 mm.

Figure 6.12: Distance dcamera registered by the camera when a movement of 0.1 mm in X-direction was performed by the robot. The dashed line is the camera repetition accuracy.

68

CHAPTER 6. RESULTS

Figure 6.13: Distance dcamera registered by the camera when a movement of 0.1 mm in Y-direction was performed by the robot.

Figure 6.14: Distance dcamera registered by the camera when a movement of 0.1 mm in Z-direction was performed by the robot.

6.4. REPETITION ACCURACY OF THE TCP CALIBRATION METHOD 69

6.3.1

Averaging to obtain a higher accuracy

The same tests were performed once again, but this time both positions of the tool, p1 and p2 , were determined as an average of 100 measurements. Table 6.3 shows the results of the distance dcamera measured by the camera, when a movement d = 0.06 mm was performed by the robot. If the distance d was chosen to d < 0.06 mm no significant movement was registered by the camera. The result of this test assures an increased accuracy of the method, if the average of several measurements are used. d

dcamera

The robot moves 0.06 mm in X direction

0.04 mm

The robot moves 0.06 mm in Y direction

0.04 mm

The robot moves 0.06 mm in Z direction

0.04 mm

Table 6.3: Significant movements registered by the camera.

6.4

Repetition Accuracy of the TCP Calibration method

To test the repetition accuracy of the TCP calibration method presented in this thesis, the method was executed several times for the same tool. The tool used was the chess board pattern and all 24 points of the chess pattern were observed. The desired accuracy of the iterative method was set to 1 mm. Denote  the translation  T CPx  from the tool flange to the TCP found by the method as TCP =   T CPy . The T CPz result of T CPx , T CPy , T CPz for the different runs of the method are illustrated in figure 6.15, figure 6.16 and figure 6.17 respectively. Table 6.4 shows the differences between the maximum and minimum interval for each of the results. By observing the information in table 6.4, it is obvious the desired accuracy of 1 mm was obtained.

70

CHAPTER 6. RESULTS

Figure 6.15: Repetition accuracy of T CPx measured by the proposed TCP calibration method.

Figure 6.16: Repetition accuracy of T CPy measured by the proposed TCP calibration method.

6.4. REPETITION ACCURACY OF THE TCP CALIBRATION METHOD 71

Figure 6.17: Repetition accuracy of T CPz measured by the proposed TCP calibration method.

q

Element of TCP


T CPx

0.97

T CPy

0.52

T CPz

0.48

T CPx2 + T CPy2 + T CPz2

0.66

Table 6.4: Difference of max and min interval of |TCP| and for the elements of TCP respectively.

72 6.4.0.1

CHAPTER 6. RESULTS How does the number of points affect the repetition accuracy of the TCP calibration method

To investigate how the number of points affect the repetition accuracy of the proposed TCP calibration method, the same procedure was executed but only the three most upper left points of the chess board pattern were observed. The method managed to give a satisfying result most of the times, but the method became unreliable and very time consuming. Sometimes the robot diverged from the correct TCP point during the iterative phase. This implies more than three points have to be observed to make the method reliable. Although, the reason why the robot sometimes diverged during the iterative procedure might be that the wrong solution to the three points was found. If three points with almost the same distances to each other are observed, it might be difficult for the algorithm to distinguish between the points. This results in three possible solutions, but only one of the solutions is correct. If the mirrored solutions are concerned, six different solutions are possible to find. The mirrored solutions were removed during the test by only permitting solutions with Z > 0, where Z is the camera coordinate axis coinciding with the optical axis.

Chapter 7 Discussion This chapter presents the conclusion of this Master’s thesis. Problems and difficulties are described and the chapter concludes with a discussion of future development.

7.1

Conclusion

This thesis has proposed a new method for tool center point calibration. The method can be split into three different parts. 1. Calculation of a good guess of the Tool Center Point, by solving a closed loop of transformations resulting in a homogeneous equation AX1 = X1 B. 2. Image Segmentation of the tool. 3. Iterative method for increasing the accuracy. The method proposed is completely automated, resulting in no variations in accuracy, due to how skilled the robot operator is. The method is also very easy to configure, only a USB camera connected to a laptop computer with the software developed during this project is needed. No fixed devices with known relations to the robot’s base frame are needed, implying the calibration set up used in this method is completely portable.

7.1.1

Accuracy

Tests have been performed with regard to investigate the accuracy of the proposed method. The tests showed the need of the iterative method to achieve a satisfying accuracy. Due to the fact that only accuracy of the camera when registering a movement between two different positions limits the accuracy of the 73

74

CHAPTER 7. DISCUSSION

iterative method, an accuracy of approximately 0.1 mm is possible to achieve by this method if a camera of the same model is used, see section 6.3. By that remark, this method is able to perform a tool center point calibration with equivalent accuracy or even better in relation to the manual four point TCP calibration method.

7.1.2

Major Problems/known difficulties

During the project two main problems have occurred. 1. Finding distinct points 2. Light reflections in the tool

7.1.2.1

Finding distinct points

To be able to determine the transformation between the camera and the tool, at least three distinct points on the tool have to be observed by the camera. In this thesis results have been presented, showing that it is possible to perform a calibration by only observing three points. However, to make the method reliable and fast, more points are needed. If the tool has a very simple geometry it might be difficult to find several points and assure it is the same points found in every frame. This problem can be solved by for instance painting the tool with a certain pattern. Another possibility would be to press a matrix pattern on the tool. The matrix pattern could then also store information about what kind of tool it is. The best idea though, would probably be to ensure that the tool has some kind of structure e.g. not a simple geometric shape.

7.1.2.2

Light reflections in the tool

A major problem during the development of the image segmentation part of the method, was the influence of light reflections in the tool. This is a well known problem within the field of image segmentation. Several methods exist to make image segmentation robust against reflections, but no suitable functions for solving this problem exist in the Open Computer Vision library. Instead the problem was solved by painting the tool in a non reflecting color. Before using the method in an industrial environment, this problem must be solved.

7.2. FUTURE DEVELOPMENT

7.1.3

75

Fulfillment of objectives

The objectives of the thesis is to investigate whether or not it is possible to create an automated method for tool center point calibration, by making good use of computer vision and image processing technologies. To consider the method as useful; an accuracy of ±1 mm had to be obtained. In this thesis a method has been proposed and a software has been delivered with the ability to successfully perform tool center point calibrations. The acuracy of the proposed method has been analyzed, showing the method’s ability of performing TCP calibrations with an accuracy of approximately 0.1 mm, see the discussion in section 6.3. Therefore all objectives are considered to be fulfilled.

7.2

Future Development

In this section some future development will be presented. 1. Rotation symmetric tools 2. Light reflections in the tool 3. Image Segmentation controlled by CAD drawings 4. Neural network based methods 5. Online correction of the TCP calibration 6. Other image processing libraries

7.2.1

Rotation symmetric tools

One problem of the proposed method is to calculate a good initial guess of the tool center point, if the robot tool has an axis of symmetry. The symmetry makes it impossible to measure the transformation between the camera and the tool by 6 degrees of freedom. A mathematical proof, showing it is impossible to measure a good guess of TCP by using this method if the tool is rotation symmetric, is published in the thesis. Although, the iterative part of the method still works for rotation symmetric tools, if the transformation between the camera and the robot coordinate system is known and an initial guess of the tool center point is obtained by another procedure. The alternative method for finding the transformation between the camera frame and the robot’s base frame presented in section 4.6 should be implemented and evaluated.

76

CHAPTER 7. DISCUSSION

Observe, other methods for solving this problem exist. For instance the rotation symmetry can be removed by painting the tool with a non symmetric pattern. Attaching a blinking light emitting diode (LED) to the tool, easy to find by image segmentation, can also remove the symmetric geometry. By fasten a calibration pattern, for instance a chess board pattern, at the tool it is possible to determine the transformation between the camera frame and the robot’s base frame. The iterative method then works even though the tool is rotation symmetric. Instead of fasten a calibration pattern at the tool, a matrix pattern can be pressed on the tool. Pressing a pattern on the tool will also solve the problem of finding distinct points, see section 7.1.2.1.

7.2.2

Light reflections in the tool

Before using the method in an industrial environment, the problem of handling light reflections in the tool, see section 7.1.2.2, must be solved.

7.2.3

Image Segmentation controlled by CAD drawings

To make the method general it needs the ability of calibrating all kinds of tools. Therefore the next step is to control the image segmentation part of the method by data from for instance a CAD drawing of the actual tool. Developing a software with the ability to read 3D drawings of tools and analyze the drawings to find distinct points or lines to search for during the image segmentation would ensure that the method would be able to handle different kind of tools. It would be even better if only one general 3D model for each kind of tool would be enough to control the image segmentation method. Then the user would only need to choose what kind of tool he/she wants to calibrate.

7.2.4

Neural network based methods

When the method is able to handle different kinds of tools, it would be possible to make the system able to distinguish between different tools by itself e.g. classifying tools. This can be done by making use of a neural network approach such as principle component analysis (PCA). As soon as a robot tool is located in front of the camera, the system would then be able to classify the tool and start the calibration procedure.

7.2.5

Online correction of the TCP calibration

There are several events which can cause misplacement of the tool center point during a robot program. The method proposed in this thesis could easily be re-

7.2. FUTURE DEVELOPMENT

77

configured for correction of the current TCP calibration. By placing the camera at a fixed position and once and for all determine the relation between the camera and the robot base frame, only the iterative part of the method is needed to determine the correct placement of the tool center point. The initial guess of the TCP is then of course chosen to be the current tool center point.

7.2.6

Other image processing libraries

The image processing library (Open Computer Vision Library) used during this project, does not include suitable functions for handling light reflections in metallic tools. Therefore it could be of great interest to try other image processing libraries, for instance the library developed by Cognex or the library developed by Halcon.

Appendix A Appendix This appendix will present the results of the camera calibration done during this project.

A.1

Finding intrinsic parameters

The camera calibration toolbox for Matlab by Jean-Yves Bouguet was used to perform the camera calibration. Fifty two images of a chess board pattern were achieved by the camera. The chess board was rotated and translated between every frame. Figure A.1 illustrates the positions of the chessboard. The camera calibration toolbox was then used to determine the intrinsic parameters presented in table A.1. Figure A.2 illustrates the reprojection errors in pixels after the camera calibration was done. Also the distortion coefficients were calculated and the result can be seen in table A.1. Figure A.3 and figure A.4 illustrates the radial and the tangential lens distortion respectively. By the figure it is possible to tell that the radial distortion is greater than the tangential distortion. Figure A.5 illustrates the complete lens distortion.

79

80

APPENDIX A. APPENDIX

Calibration results (with uncertainties): "

Focal Length: fc =

615.17386 615.69958 "

Principal point: cc =

#

337.27613 233.77556

"

±

3.32053 3.41099

#

"

±

#

4.32134 4.42747

#

Skew: alpha c = [0.00000] ± [0.00000] => angle of pixel axes = 90.00000 ± 0.00000 degrees     Distortion: kc =    

−0.20033 0.25346 0.00009 −0.00143 0.00000

"

Pixel error: err =

0.06992 0.07343





      ±      

0.00863 0.02484 0.00164 0.00166 0.00000

       

#

Note: The numerical errors are approximately three times the standard deviations (for reference). Table A.1: The camera calibration result given by camera calibration toolbox for Matlab.

A.1. FINDING INTRINSIC PARAMETERS

Figure A.1: Illustration of the chess board positions.

81

82


Figure A.2: Reprojection errors in pixels.


Figure A.3: Radial lens distortion.

83

84


Figure A.4: Tangential lens distortion.


Figure A.5: Complete lens distortion.

85

Bibliography [1] ABB, Rapid reference manual Industrial robot controller system. ABB. [2] P.-E. Danielsson., “Bildanalys,” in Bildanalys, pp. 52–63, 2005. [3] M. M. Seger, “Lite om kamerageometri.” September 2005. [4] Donald Hearn, M.Pauline Baker, Computer Graphics with OpenGL. Pearson Education, Inc. Upper Saddle River: Pearson Prentice Hall, third ed., 2004. [5] Lung-Wen Tsai, Robot Analysis The Mechanics of Serial and Parallel Manipulators. 605 Third Avenue, New York, N.Y. 10158-0012: John Wiley & Sons, INC., 1998. [6] Z. Zhang, “A flexible new technique for camera calibration,” technical report, Microsoft Research, http://research.microsoft.com/ zhang, 8 2002. [7] Jintao Wang, Daokui Qu and Fang Xu, “A new hybrid calibration method for extrinsic camera parameters and hand-eye transformation,” International Conferance on Mechatronics and Automation Niagara Falls, Canada, July 2005. [8] H. Zhuang, Z. Roth, and R.Sudhakar, “Simultaneous robot/world and tool/flange calibration by solving homogeneous transformation of the form AX=YB,” IEEE Transaction on Robotics and Automation, vol. 10, pp. 549– 554, August 1994. [9] Fadi Dornaika and Radu Horaud, “Simultaneous robot-world and hand-eye calibration,” IEEE Transaction on Robotics and Automation, vol. 14, August 1998. [10] Roger Y. Tsai and Reimark K.Lenz, “A new technique for fully autonomous and efficient 3d robotics hand/eye calibration,” IEEE Transaction on Robotics and Automation, vol. 5, June 1989. 87

88

BIBLIOGRAPHY

[11] “Matrix reference manual: Matrix relations.” Internet link. http://www.ee.ic.ac.uk/hp/staff/dmb/matrix/relation.html#similarity. [12] “3d rotations (wrf).” Internet link. http://www.ecse.rpi.edu/Homepages/wrf/Research/ Short Notes/rotation.html#find. [13] “Rotation matrix.” Internet link. http://mathworld.wolfram.com/RotationMatrix.html. [14] “Sourceforge OpenCV.” Internet link. http://sourceforge.net/projects/opencvlibrary/. [15] “Intel OpenCV.” Internet link. http://www.intel.com/technology/computing/opencv/. [16] Björn Svensson, Johanna Pettersson, Hans Knutsson, “Bildregistrering geometrisk anpassning av bilder.” Mars 2004. [17] C. Galambosi J. Matastsand J. Kittlert, “Progressive probabilistic hough transform for line detection,” Computer Vision and pattern Recognition, vol. 1, pp. 733–737, June. [18] Intel Corporation, http://surfnet.dl.sourceforge.net/sourceforge/opencvlibrary/ OpenCVReferenceManual.pdf, Open Source Computer Vision Library Reference Manual, first ed., 12 2000. [19] J.Canny, “A computational approach to edge detection,” IEEE Transaction on Pattern Analysis and Machine Intelligence, vol. 8, no. 6, pp. 679–698, 1986. [20] K. A. S. Suzuki, “Topological structural analysis of digital binary images by border following,” CVGIP, pp. 32–46, July 1985. [21] F. Albregtsen, “Active contours and region based segmentation,” INF 269 Digital Image Analysis, p. 25, 10 2002. Lecture 6. [22] C. Harris and M.J. Stephens, “A combined corner and edge detector,” Alvey Vision Conference, pp. 147–152, 1988. [23] G. Loy and A. Zelinsky, “Fast radial symmetry for detecting points of interest,” IEEE Transaction on Pattern Analysis and Machine Intelligence, vol. 25, pp. 959–973, August 2003. [24] Gösta H. Granlund, Hans Knutsson, Signal Processing for Computer Vision. P.O Box 17, 3300 AA Dordrecht The Netherlands: Kluwer Academic Publishers, 1995.

BIBLIOGRAPHY

89

[25] C. Mota, I. Stuke, and E. Barth, “Analytic solutions for multiple motions,” International Conference on Image Processing, 2001. Institute for Signal Processing, University of Lübeck Ratzeburger Allee 160, 23538 Lübeck, Germany. [26] G. Farnebäck, Polynomial Expansion for Orientation and Motion Estimation. PhD thesis, Linköpings University, Department of Electrical Engineering Linköpings University, SE-581 83 Sweden, 2002.

På svenska Detta dokument hålls tillgängligt på Internet – eller dess framtida ersättare – under en längre tid från publiceringsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och tillgängligheten finns det lösningar av teknisk och administrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsmannens litterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida http://www.ep.liu.se/ In English The publishers will keep this document online on the Internet - or its possible replacement - for a considerable time from the date of publication barring exceptional circumstances. The online availability of the document implies a permanent permission for anyone to read, to download, to print out single copies for your own use and to use it unchanged for any non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional on the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its WWW home page: http://www.ep.liu.se/ © [Johan Hallenberg]

Robot Tool Center Point Calibration using Computer ... - DiVA Portal

Robot Tool Center Point Calibration using Computer ... - DiVA Portal

Suggest Documents

Quantum point contact - DiVA portal

Using Computer Vision Technologies to Make the Virtual ... - DiVA portal

Using Computer Vision Technologies to Make the Virtual ... - DiVA portal

Tool Support for Language Extensibility - DiVA portal

Automated Computer Systems for Manufacturability ... - DiVA portal

Lecture Notes in Computer Science - DiVA portal

Developing educational computer-assisted simulations - DiVA portal

Students' Conceptions of Computer Programming ... - Diva Portal

lidar system calibration using point cloud - asprs

lidar system calibration using point cloud - asprs

Mirror Based IMU-Camera and Internal Camera Calibration - DiVA portal

Calibration of the Accelerometer Triad of an Inertial ... - DiVA portal

Simple Non-Iterative Calibration for Triaxial Accelerometers - DiVA portal

Contour Reconstruction using Recursive Smoothing ... - DiVA portal

Using CO in Supermarket Refrigeration - DiVA portal

Terrain Navigation using Bayesian Statistics - DiVA portal

Formations and Obstacle Avoidance in Mobile Robot ... - DiVA portal

Designing and Evaluating Human-Robot Communication - DiVA portal

Formations and Obstacle Avoidance in Mobile Robot ... - DiVA portal

Robot Collection and Transport of Objects: A Biomimetic ... - DiVA portal

Vision Based Grasp Planning for Robot Assembly - DiVA portal

human robot interaction solutions for intuitive industrial ... - DiVA portal

Non-orthogonal tool/flange and robot/world calibration

snowbots; a mobile robot on snow covered ice - DiVA portal