High Speed Reactive Collision Avoidnace on an Autonomous Unmanned Surface Vehicle A System using Pushbroom Stereo Vision for Marine Navigation
Prepared by:
Nathan Pilbrough PLBNAT001 Department of Electrical Engineering University of Cape Town
Prepared for:
R.A Verrinder Department of Electrical Engineering University of Cape Town
October 2015 Submitted to the Department of Electrical Engineering at the University of Cape Town in partial fulfilment of the academic requirements for a Bachelor of Science degree in Mechatronics.
Key Words: collision avoidance, autonomous, stereo vision, stereo calibration, Pushbroom algorithm, high speed, remote controlled boat
i
ii
Declaration 1. I know that plagiarism is wrong. Plagiarism is to use another's work and pretend that it is one's own. 2. I have used the IEEE convention for citation and referencing. Each contribution to, and quotation in, this final year project report from the work(s) of other people, has been attributed and has been cited and referenced. 3. This final year project report is my own work. 4. I have not allowed, and will not allow, anyone to copy my work with the intention of passing it off as their own work or part thereof
Name:
Nathan Pilbrough
Signature:
Date:
iii
13 October 2015
Terms of Reference The agreed terms of reference are discussed in this section. A description of the project and the deliverables for the project are shown below.
Description Obstacle detection and avoidance is an important feature for robust long--‐term navigation of an environment using an autonomous robot, as the trajectory of a robot must be updated if an obstacle is detected in its path. This is particularly important in environments, which are cluttered and contain a number of obstacles. These techniques are often implemented on platforms with high processing power, large and expensive sensors. Small low--‐cost platforms often do not have these capabilities and alternative approaches need to be investigated. One approach is to use off--‐ the--‐self cameras and single board computers, such as the BeagleBone Black or Raspberry PI. This project will investigate collision avoidance strategies for an existing micro unmanned surface vehicle (USV) platform equipped with a stereo--‐camera system and a BeagleBone Black. Real--‐time performance and vehicle dynamics must be considered. Obstacle detection techniques for the device must also be investigated.
Deliverables Objectives The main objectives of this project are: I. II. III. IV. V. VI. VII. VIII.
Understand the requirements of the project Conduct a literature review of previous work in this field and critically evaluate current technology/research Design a reactive collision avoidance controller for the USV, taking limited processing power and algorithm speed in to account. Collect a test data set which can be used for algorithm evaluation Simulate the algorithm under a variety of different conditions Transfer the algorithm to the real--‐world platform Test and evaluate the systems performance based on a performance metric Discuss the performance of the system, draw conclusions and make recommendations for future work
Deliverables and expectations: I. II. III. IV. V. VI. VII. VIII. IX.
A literature review report (Hand--‐in 2weeks into the project) Presentation of project background, scope and proposed methodology to the Mechatronics Research Group Weekly attendance of individual meeting with supervisor Simulations of the algorithm in MATLAB or equivalent program Testing of the algorithms on a real--‐world platform (model sailboat) A project report Satisfactory completion of all ECSA ELOs Departmental seminar at the end of the project Poster and demonstration at the Departmental Open day
iv
Acknowledgements “If I have seen further it is by standing on the shoulders of giants.” – Isaac Newton
The names listed below are of those people upon whose shoulders I have stood, without whom this project would not have been a success. I extent my most sincere gratitude to you all for the help and assistance that you have given throughout the course of this investigation.
Robyn Verrinder – for allowing me to embark on this adventure and her support and advice throughout the duration of this project. Andrew Barry – for giving me full access to the Pushbroom algorithm which resulted in the success of this investigation. Also for his advice and willingness to help at all times. The Firm – for four years of putting up with my hair twirling and battling through sleepless nights together. Steven Kimmel – for his help with the control algorithm for the boat. Jonty Kruger and Sarah Henderson – for their assistance during testing and holding the boat straight for hours on end. My Digs – Christopher Taylor, Alice Huang, Sarah Henderson and in particular Scott Stainbank for putting up with my rants and patiently listening to why the boat wasn’t working. Stephen Pilbrough – for designing the art work on the boat. Last, by certainly not the least. My Mother – for taking the hours and days out of her time to proof reading this project.
v
Abstract Marine environments introduce unique challenges with regards to the automation of marine vessels. This work sought to investigate the implementation of real time collision avoidance on an unmanned surface vessel. The main objective of the investigation was to implement an obstacle detection system which did not limit the performance of the test platform. Stereo vision was chosen as the sensing modality due to its high information to weight ratio. The system utilised a pair of calibrated cameras to implement a simplified, stereo, block matching algorithm in order to sense obstacles. The algorithm was able to run considerably faster than conventional disparity algorithms by limiting the search through space to a single depth. Despite this trade-off of information for speed the algorithm was able to effectively detect obstacles. All the sensing and processing for the collision avoidance system was done on-board the test platform. The algorithm was tested over the course of six testing sessions during which a full characterisation of the performance of the system was conducted. The system exceeded expectations and was limited by the dynamics of the test platform rather that the collision avoidance algorithm itself. This performance represents an advancement in the field of high-speed autonomous marine navigation and a novel method of reactive collision avoidance.
vi
Table of Contents Declaration .................................................................................................................................................................. iii Terms of Reference ................................................................................................................................................... iv Description .................................................................................................................................................................................. iv Deliverables ................................................................................................................................................................................ iv Objectives................................................................................................................................................................................ iv Deliverables and expectations: ...................................................................................................................................... iv Acknowledgements..................................................................................................................................................... v Table of Contents ...................................................................................................................................................... vii List of Figures ............................................................................................................................................................. xii Glossary .................................................................................................................................................................... xviii 1.
2.
3.
Introduction ......................................................................................................................................................... 1 1.1
Background to the study .......................................................................................................................................... 1
1.2
Objectives of this study............................................................................................................................................. 1
1.2.1
Problems to be investigated ......................................................................................................................... 1
1.2.2
Purpose of the study ........................................................................................................................................ 2
1.3
Scope and limitations ................................................................................................................................................ 2
1.4
Plan of development .................................................................................................................................................. 2
Literature Review .............................................................................................................................................. 3 2.1
Background and development of marine vessels .......................................................................................... 3
2.2
Robotic architecture .................................................................................................................................................. 4
2.3
Obstacle detection ...................................................................................................................................................... 4
2.3.1
Active sensing modalities .............................................................................................................................. 5
2.3.2
Passive sensors .................................................................................................................................................. 8
2.3.3
Object detection modality comparison ................................................................................................. 11
2.3.4
State-of-the-art stereo vision implementations ................................................................................ 14
2.3.5
Choice of implementation. .......................................................................................................................... 16
2.4
Previous implementation ..................................................................................................................................... 16
2.5
Comparison between the previous and proposed algorithms .............................................................. 17
2.6
Literature review summary................................................................................................................................. 17
Theoretical Development ............................................................................................................................. 19
vii
3.1
Epipolar geometry ................................................................................................................................................... 19
3.2
Stereo calibration theory ...................................................................................................................................... 20
3.2.1
Undistortion ..................................................................................................................................................... 20
3.2.2
Rectification ...................................................................................................................................................... 20
3.3 4.
Camera Mount Development ....................................................................................................................... 24 4.1
5.
Camera mount analysis ......................................................................................................................................... 24
4.1.1
Baseline trade-off ........................................................................................................................................... 24
4.1.2
Resolution calculations ................................................................................................................................ 26
4.2
Camera model upgrade ......................................................................................................................................... 27
4.3
Reversion to the original cameras .................................................................................................................... 27
4.4
Camera servo motor redundancy...................................................................................................................... 28
Stereo Calibration ............................................................................................................................................ 29 5.1
Three modality calibration .................................................................................................................................. 29
5.2
AprilCal......................................................................................................................................................................... 30
5.3
Calibration procedure ............................................................................................................................................ 32
5.3.1
AprilCal calibration........................................................................................................................................ 32
5.3.2
Matlab calibration .......................................................................................................................................... 33
5.3.3
OpenCV calibration ........................................................................................................................................ 36
5.4 6.
The Laplacian edge detection function ........................................................................................................... 22
Accuracy test .............................................................................................................................................................. 36
Algorithm Development ................................................................................................................................ 38 6.1
Pushbroom algorithm analysis and modification....................................................................................... 38
6.1.1
Overview of the Pushbroom algorithm ................................................................................................. 38
6.1.2
Fundamental principles of the algorithm............................................................................................. 39
6.1.3
Horizontal invariance check ...................................................................................................................... 39
6.1.4
Algorithm flow diagrams ............................................................................................................................ 40
6.1.5
Last valid pixel explanation ....................................................................................................................... 42
6.1.6
Modification of the Pushbroom algorithm........................................................................................... 43
6.2
Image capture synchronisation.......................................................................................................................... 44
6.3
Algorithm acceleration analysis......................................................................................................................... 45
6.3.1
Parallel processing......................................................................................................................................... 45
6.3.2
ARM Neon: algorithm accelerator ........................................................................................................... 45
viii
6.4
Development of an object verification system............................................................................................. 46
6.4.1 6.5 7.
Development of the collision avoidance algorithm ................................................................................... 47
Algorithm Implementation........................................................................................................................... 51 7.1
Computer based implementation...................................................................................................................... 51
7.1.1 7.2
8.
Windows based implementation ............................................................................................................. 51
Debian based implementation............................................................................................................................ 52
7.2.1
Virtual Box......................................................................................................................................................... 52
7.2.2
Cross Compilation .......................................................................................................................................... 52
7.3
CMAKE program compilation ............................................................................................................................. 53
7.4
Duel boot implementation ................................................................................................................................... 53
Algorithm Anomaly ......................................................................................................................................... 54 8.1
9.
Choice of object verification method...................................................................................................... 47
The lag effect .............................................................................................................................................................. 54
Test Platform Development ......................................................................................................................... 57 9.1
Previous test platform ........................................................................................................................................... 57
9.2
Test platform upgrades ......................................................................................................................................... 58
9.2.1
Battery upgrade .............................................................................................................................................. 58
9.2.2
Power regulator upgrade ............................................................................................................................ 58
9.2.3
MOSFET upgrade ............................................................................................................................................ 59
9.3
Build overview .......................................................................................................................................................... 60
9.4
Communication......................................................................................................................................................... 61
9.5
Circuit diagram ......................................................................................................................................................... 62
9.6
Test platform development summary............................................................................................................. 64
10. Testing and Results ......................................................................................................................................... 65 10.1
Performance tests .................................................................................................................................................... 65
10.1.1
Expected performance ................................................................................................................................. 65
10.1.2
Computer performance test at the original resolution ................................................................... 66
10.1.3
BeagleBone performance test at the original resolution ............................................................... 67
10.1.4
Image resolution reduction ........................................................................................................................ 67
10.1.5
Computer performance test at the reduced resolution .................................................................. 68
10.1.6
BeagleBone performance test at the reduced resolution .............................................................. 68
10.1.7
Complete speed performance characterisation................................................................................. 69
ix
10.2
Indoor Control tests ................................................................................................................................................ 70
10.2.1
Testing conditions.......................................................................................................................................... 70
10.2.2
Data analysis..................................................................................................................................................... 72
10.2.3
Computer tests ................................................................................................................................................ 72
10.2.4
BeagleBone tests............................................................................................................................................. 75
10.2.5
Obstacle test summary ................................................................................................................................. 77
10.2.6
Speed test summary ...................................................................................................................................... 78
10.3
Pool testing ................................................................................................................................................................. 80
10.3.1
Pool testing day one ...................................................................................................................................... 80
10.3.2
Pool testing day two ...................................................................................................................................... 82
10.3.3
Pool testing day three ................................................................................................................................... 83
10.4
Open water testing .................................................................................................................................................. 88
10.4.1
Open water testing day one ....................................................................................................................... 88
10.4.2
Open water testing day two ....................................................................................................................... 93
11. Discussion ........................................................................................................................................................... 97 11.1
Object detection capability .................................................................................................................................. 97
11.2
Speed of the algorithm ........................................................................................................................................... 97
11.3
Speed of the test platform .................................................................................................................................... 97
11.3.1
Analysis of the effect of speed on detection capability ................................................................... 97
12. Conclusions ........................................................................................................................................................ 98 12.1
Test results summary............................................................................................................................................. 98
12.1.1
Conclusions from the indoor test ............................................................................................................ 98
12.1.2
Conclusions from the pool test ................................................................................................................. 98
12.1.3
Conclusions from the open water test ................................................................................................... 98
12.1.4
Conclusions from the performance of the test platform ................................................................ 98
12.2
Limitations .................................................................................................................................................................. 99
12.3
Significance of research and success of the project ................................................................................... 99
13. Recommendations ........................................................................................................................................ 101 13.1
Larger test platform implementation ............................................................................................................101
13.2
Full Pushbroom algorithm implementation ...............................................................................................101
13.3
Control scheme .......................................................................................................................................................101
13.4
Hardware upgrade ................................................................................................................................................101
x
13.5
Duel modality system ...........................................................................................................................................101
14. List of References .......................................................................................................................................... 102 15. Appendices ...................................................................................................................................................... 106 15.1
Raw data ....................................................................................................................................................................106
15.2
Arduino code............................................................................................................................................................106
15.3
BeagleBone code ....................................................................................................................................................108
16. EBE Faculty: Assessment of Ethics in Research Projects ................................................................ 124
xi
List of Figures Figure 2-1: Differentiation between active and passive sensing modalities. Active sensors generate the energy which is reflected off an object and then reabsorbed by the sensor. Passive sensors analyse the information generated by an external source that is reflected off an object. ...... 5 Figure 2-2: Image showing the Xbox Kinect module. The sensing is performed by the device by using the IR Emitter, Colour Sensor and IR Depth Senor. Image adapted form [25] .................................... 7 Figure 2-3: Diagrammatic explanation of the triangulation principle. The distance Z to an object P can be determined using the known parameters of the cameras and the law of similar triangles. Image adapted from [33] ................................................................................................................................ 11 Figure 2-4: Flow chart showing the difference between the algorithm implemented by de Klerk and that proposed for this investigation. The proposed algorithm is a modification of that developed by Barry et al that excludes the 3D point cloud feature but adds a collision avoidance system. .................................................................................................................................................................................... 17 Figure 3-1: Diagrammatic explanation of epipolar geometry. Every point in an image frame has a corresponding epipolar line it is associated with in the other image frame. Image adapted from [33] ................................................................................................................................................................ 19 Figure 3-2: Image showing the calibration procedure in a graphical form. Image adapted from [1]. A) shows the raw uncalibrated images. B) shows the undistortion process whereby linear features in reality are mapped to linear lines in image. C) shows the rectification process whereby the images are adjusted to be coplanar and row aligned. D) finally the images are cropped such that there is a common field of view between the two cameras. ....................... 21 Figure 3-3: Image of the 10x7 checkerboard used for OpenCV and Matlab calibration. The sharp change in contrast is easily identifiable by computer vision algorithms.................................................... 22 Figure 3-4: Image demonstrating the edge detection capability of the Laplacian function in conjunction with the Sobel or first derivative of the image. Image adapted from[46]................................... 23 Figure 4-1: Image of the original camera mount constructed by de Klerk. The baseline distance was 89mm for this stereo rig. ................................................................................................................................. 24 Figure 4-2: Image showing the relationship between the baseline distance of a stereo rig and its effect on depth calculations. D1 and D2 are the image disparities for the short and long baseline stereo rigs respectively. In the same manner M1 and M2 are the minimum distances that can be measured by the respective stereo rigs. The view lines converge at the focal point of each respective camera. ................................................................................................................................... 25 Figure 5-1: Flow chart showing the relationship and dependencies between the three calibration modalities. The parameters derived from AprilCal where used for the OpenCV calibration. Matlab was used as a means of verifying the quality of the calibration procedure. ............... 29 Figure 5-2: Image of the AprilCal calibration pattern. Unlike a checkerboard each individual square is unique and identifiable from the rest. This enables greater flexibility with respect to the calibration procedure since the entire pattern does not need to be visible. ............................. 30 Figure 5-3: Image showing the reprojection method. The solid, coloured squares are superimposed over the AprilTags. This enables easy identification of the squares. The suggested configuration
xii
of the pattern is then suggested and shown as hollow squares. The user is required to line the solid squares up with the hollow squares, at which point the program takes a snapshot. .................................................................................................................................................................................... 31 Figure 5-4: Images used to calculate the intrinsic parameters for the right camera. ..................................... 32 Figure 5-5: Images used to calculate the intrinsic parameters for the left camera. ........................................ 33 Figure 5-6: Image showing a typical pair of stereo frames used for calibration .............................................. 33 Figure 6-7: Image showing the Matlab calibration procedure whereby corners are identified by a green circle and the reprojection points are depicted by a red cross. The yellow square shows the checkerboard origin but in both cases in obscured by the camera label. ................................... 34 Figure 5-8: Image showing the Matlab calibration procedure whereby corners are identified by a green circle and the reprojection points are depicted by a red cross. The yellow square shows the checkerboard origin but in both cases in obscured by the camera label. ................................... 34 Figure 5-9: Graph showing the mean reprojection error for the 25 image pairs used for calibration. The overall mean was just over 0.15 pixels ..................................................................................................... 34 Figure 5-10: Visualization of the calibration environment showing the 25 positions of the checkerboard used for calibration............................................................................................................................................ 35 Figure 5-11: Image showing rectified image pairs. The coloured lines are the epipolar lines which were manipulated into horizontal scanlines by moving the epipoles of the cameras to infinity . 36 Figure 5-12: Graph showing the distance accuracy of the stereo rig. A target was set-up at 1m, 2m and 3m respectively. The distance to the target was then estimated by the stereo vision system. .................................................................................................................................................................................... 37 Figure 6-1: Image depicting the principle of the Pushbroom algorithm. A detection boundary moves in front of the test platform. The algorithm is able to detect obstacles as they break the boundary. ............................................................................................................................................................... 38 Figure 6-2: Image showing the effect of the horizontal invariance check. Erroneous detections on the horizon occur due to repetitive patterns in the environment. ........................................................ 40 Figure 6-3: Flow diagram depicting the ProcessImages function and in particular the initialization of the multithreaded processing. .............................................................................................................................. 41 Figure 6-4: Flow diagram depicting the single disparity function which computes the SAD and checks for horizontal invariance to determine if a stereo hit is valid or not. ........................................... 42 Figure 6-5: Image depicting the reasoning behind using a last valid pixel constraint. The frame to the left is the view of the camera and the image to the right shows a bird’s eye view of the scene. .................................................................................................................................................................................... 43 Figure 6-6: Image showing the time difference between image frames. The time difference above of 31ms was the maximum difference recorded between frames. ..................................................... 44 Figure 6-7: Image depicting the Neon algorithm accelerator functionality. Many elements of the source registers are combined in parallel by the operation and stored in the destination register.[51] .......................................................................................................................................................... 46
xiii
Figure 6-8: Flow diagram showing the basic high level structure of the collision avoidance algorithm. Two infinite while loops are utilised, one without the stereo vision enabled and the other with stereo vision enabled. ............................................................................................................................ 48 Figure 6-9: Flow diagram showing the collision avoidance algorithm. The box with the dashed lines was added after an observation made during testing. ................................................................................. 49 Figure 7-1: Flow diagram showing a graphical representation of the development tool chain used to compile and run the Pushbroom algorithm on the BeagleBone ..................................................... 53 Figure 8-1: Image showing the effect of relative motion between the obstacle and the test platform. The solid circle represents the object before it has moved and the perforated ring represents the object in the next image frame. The last sequence shows how the disparity appears to change due to the image frames being unsynchronised. ................................................................... 54 Figure 8-2: A graphical representation of the lag effect. The perceived distance of an object was affected by which image frame was captured first and in which direction the object was moving in relation to the boat. ........................................................................................................................................... 55 Figure 9-1: Image of the boat hull and the fixed components including the motor and rudder servo motor .................................................................................................................................................................................... 58 Figure 9-2: Image showing the top side of the control board: 1) BeagleBone Black 2) battery connector, 3) Lexma USB hub 4) main power switch 5) 10A fuse 6) SMPS regulator switches 7) GPS module 8) Edimax Wi-Fi router 9) Arduino Mega 10) motor connector ................................... 61 Figure 9-3: Image showing the bottom side of the control board: 1) battery connector, 2) main power switc, 3) 10A fuse 4) SMPS regulator switches 5) Opamp circuitry 6) SMPS regulators 7) IRF540N MOSFET 8) 6A flyback diode 9) motor connector .......................................................... 61 Figure 9-4: Control board system block diagram showing the communication connections between the components. Image adjusted from [1]. ...................................................................................................... 62 Figure 9-5: Circuit diagram showing all power connections and voltage levels between components. 63 Figure 9-6: Image showing complete test platform. Cling wrap was used to cover the electronics in order to protect them during water testing. ........................................................................................................ 64 Figure 10-1: Graph showing the time taken to process each image frame in a set of 1000 frames. The average for the test was 2029µs and is shown in orange. ................................................................. 66 Figure 10-2: Graph showing the time taken to process each image frame in a set of 100 frames. The average time for the test was 168349µs and is shown in orange. ................................................. 67 Figure 10-3: Image showing the image resolution reduction in terms of size and pixel count. ................. 68 Figure 10-4: Graph showing the time taken to process each image frame in a set of 1000 frames. The average for the test was 820µs and is shown in orange..................................................................... 68 Figure 10-5: Graph showing the time taken to process each image frame in a set of 400 frames. The average time for the test was 28278µs and is shown in orange. .................................................... 69 Figure 10-6: Pie chart showing the total time taken to perform sections of the code. The other section consists of all command signalling, data recording, feedback and other general administration. .................................................................................................................................................... 69
xiv
Figure 10-7: A longboard was used as the test platform for the indoor control tests. This resulted in smooth continuous motion simulating the motion of the boat. ...................................................... 70 Figure 10-8: A panoramic view of the indoor testing scene depicting the test platform and the object being detected...................................................................................................................................................... 71 Figure 10-9: A schematic representation of the testing scene showing the distances between the object and the test platform......................................................................................................................................... 71 Figure 10-10: showing the camera view at three positions during the 320x240 resolution testing sequence. The images are shown at 100% scale. Frame A corresponds to the starting position for the test, 220cm away from the obstacle, where no hits were detected. Frame B corresponds to the detecting range of the algorithm, 175cm away from the obstacle, where a large number of hits are detected. Frame C corresponds to the finishing position of the test, 135cm away from the obstacle, where residual hits are detected....................................... 73 Figure 10-11: Graph showing the typical hit clustering for tests performed with the computer running the algorithm at 30 FPS at a resolution of 320x240 pixels. The obstacle was placed directly in front of the camera. The letters in sections A, B and C on the graph correspond to the image frames A, B and C in the figure above. .......................................................................................... 74 Figure 10-12: Image showing a typical frame captured during the reduced resolution tests. The image is shown at 100% scale. ....................................................................................................................................... 74 Figure 10-13: Graph showing the typical hit clustering for tests performed with the computer running the algorithm at 30 FPS at a resolution of 176x144 pixels. The obstacle was placed directly in front of the camera. ...................................................................................................................................... 75 Figure 10-14: Graph showing the typical hit clustering for tests performed with the BeagleBone running the algorithm at 3.5 FPS at a resolution of 320x240 pixels. The obstacle was placed directly in front of the camera. ...................................................................................................................................... 76 Figure 10-15: Graph showing the typical hit clustering for tests performed with the BeagleBone running the algorithm at 14 FPS at a resolution of 176x144 pixels. The obstacle was placed directly in front of the camera. ...................................................................................................................................... 76 Figure 10-16: Graphs summarising the obstacle tests performed on the computer and the BeagleBone at both the original and reduced resolutions ......................................................................................... 77 Figure 10-17: Graphs summarising the speed tests performed on the computer and the BeagleBone at both the original and reduced resolutions............................................................................................... 79 Figure 10-18: Image showing how the black flag is camouflaged against the dark background .............. 81 Figure 10-19: Image showing the contrast of the bucket against the dark background ............................... 81 Figure 10-20: Graph showing a summary of the pool test results obtained on the second day of testing. The algorithm was able to identify the object in both the left and right image frames. ....... 82 Figure 10-21: Image showing the path of the boat during the no obstacle test. The boat did not divert off course in an attempt to avoid a phantom obstacle and therefore passed the no obstacle test. .................................................................................................................................................................................... 83 Figure 10-22: Graph showing the hit chart for the no obstacle pool test. The maximum number of false negative hits remained below the avoidance threshold. ................................................................... 84
xv
Figure 10-23: Image showing the path of the boat when aimed at the right hand side of the obstacle. The boat automatically avoided the obstacle. ................................................................................................. 84 Figure 10-24: Graph showing the hit chart for the obstacle avoidance pool test. The object was detected in the left hand plane of the image and the boat turned right accordingly. ............................... 85 Figure 10-25: Image showing the frame capture at the moment the object was detected .......................... 85 Figure 10-26: Image showing the path of the boat when aimed at the left hand side of the obstacle. The boat automatically avoided the obstacle. ................................................................................................. 86 Figure 10-27: Graph showing the hit chart for the obstacle avoidance pool test. The object was detected in the right hand plane of the image and the boat turned left accordingly. ............................... 86 Figure 10-28: Image showing the frame capture at the moment the object was detected ......................... 87 Figure 10-29: Graph showing a summary of the data collected during the speed tests................................ 87 Figure 10-30: Image showing the testing location. The size of the dam allowed the boat to be tested at greater speeds. The buoy was held in place by two 20m long ski ropes anchored on either side of the dam..................................................................................................................................................... 88 Figure 10-31: Graph showing a summary of the dam speed test............................................................................ 89 Figure 10-32: Image showing the frame captures at the moment of detection. The frames show the variable nature of the algorithm as no hit pattern is the same as another. The speed ratings for each frame are as follows: A = 0.5m/s B = 0.75m/s C = 1m/s D = 1.25m/s E = 1.5m/s F = 1.75%....................................................................................................................................................................... 90 Figure 10-33: Graph showing the hit chart for the failed collision avoidance run. The maximum number of valid hits was equal to the threshold level and therefore the obstacle was not considered valid. ......................................................................................................................................................................... 91 Figure 10-34: Graph showing a summary of the dam speed test using an extended detection range. The data did not follow the expected trend that greater speed results in a reduced number of hits. ........................................................................................................................................................................... 92 Figure 10-35: Image showing the frame captures at the moment of detection. The speed ratings for each frame are as follows: A = 0.75m/s B = 1.25m/s C = 1.5m/s D = 1.75m/s E = 2m/s. .............. 92 Figure 10-36: Image showing the test location for the second open water test. The location above was more easily accessible that the previous test location........................................................................ 93 Figure 10-37: Summary of the results obtained from the second day of open water testing. Twelve tests were conducted and eight out of the twelve tests resulted in successful avoidance of the obstacle. The failed attempts are depicted by the red avoidance threshold bars which are higher than the maximum number of true correspondence hits for the respective run. ..... 94 Figure 10-38: Screenshots captured at the moment the object was detected. The images are displayed in increasing speed order as per the figure above. A = 0.75m/s B = 1.25m/s C = 1.5m/s D = 1.75m/s E = 2m/s F = 2.5m/s G = 2.75m/s H = 3m/s. ........................................................................ 95 Figure 10-39: Image showing the effect of the boat's pitch with regards to object detection. The pitch of the boat could not formally be quantified due to the lack of an on-board accelerometer or gyroscope. The image is merely an illustration of the effect. ........................................................... 96
xvi
List of Tables Table 2-1: Showing the advantaged and disadvantages of each object detection modality that was investigated........................................................................................................................................................... 12 Table 2-2: Showing the results of the criteria cost analysis of the modalities under investigation. The weighting multiplier was determined according to the importance of the criteria to the system. .................................................................................................................................................................... 13 Table 2-3: Comparison between the current state-of-the-art implementations .............................................. 14 Table 4-1: Previous implementation, stereo rig characteristics.............................................................................. 26 Table 4-2: Properties of the Logitech C210 webcams. Table adapted from [1] ................................................ 27 Table 4-3: Properties of the PS3 eye webcam. ................................................................................................................ 27 Table 5-1: Showing the strengths and weaknesses of the three calibration modalities ............................... 29 Table 7-1: Showing the specifications of the computer used for developing, debugging and testing the Pushbroom algorithm....................................................................................................................................... 51 Table 8-1: Showing the characteristics of the lag effect for different scenarios when there is relative motion between the boat and an object. ................................................................................................... 55 Table 9-1: Original test platform component list........................................................................................................... 57 Table 9-2: Current component list ....................................................................................................................................... 59 Table 9-3: Electrical characteristics of the components used on the current system. ................................... 60 Table 10-1: Calculation of the expected decrease in performance due to hardware restrictions............. 65 Table 10-2: Weather conditions on the first day of testing. ...................................................................................... 89 Table 10-3: Weather conditions on the second day of testing. ................................................................................ 93
xvii
Glossary The list of abbreviations listed below are used throughout this document.
TOF
Time Of Flight
FOV
Field Of View
AIS
Automatic Identification System
UAV
Unmanned Aerial Vehicle
MAV
Micro Aerial Vehicle
FPGA
Field Programmable Gate Array
FPS
Frames Per Second
CLB
Configurable Logic Blocks
IOB
Input/Output Block
SAD
Sum of Absolute Difference
SSD
Sum of Squared Difference
SURF
Speeded Up Robust Features
MRE
Mean Reprojection Error
ERE
Expected Reprojection Error
IDE
Integrated Development Environment
SMPS
Switch Mode Power Supply
Opamp Operational Amplifier
xviii
1. Introduction 1.1
Background to the study
Boats and marine vessels have existed since the dawn of time and have provided a means by which humans are able to move across the surface of water. Throughout the centuries, boating technology has been developed and refined to the point of saturation. As a result new areas of development are being investigated. One such area is that of automation and enabling marine vessels to navigate through an environment without assistant. In order to enable this capability, vessels are required to have situational awareness at all times. One of the fundamental building blocks of automation is that of obstacle detection and avoidance. Without this ability higher functionality is not possible. The focus of this investigation is on the development of these fundamental building blocks. Oceans cover 71% of the earth’s surface and as such the development of a fast and reliable collision avoidance system would have a significant impact on marine transportation and research. Removing the human element form the control and navigation of these vehicles would allow the far reaches of the ocean to be explored and documented, providing valuable research information. This investigation builds on the knowledge base developed by de Klerk [1] in the previous iteration of this investigation into autonomous collision avoidance in a marine environment. The future of marine vehicle development is based on automation. This investigation represents a small step forward, towards this ultimate goal.
1.2
Objectives of this study
The objective of this investigation is, therefore, to increase the performance of the previous implementation, such that the limiting factor of the collision avoidance system is not the object detection algorithm but rather the dynamics of the marine vessel. In this regard, the aim is to develop a system which does not limit the performance of the boat in anyway, such that the boat is able to detect and avoid obstacles at high speed. The objective of the investigation is to also determine the capabilities of the BeagleBone with regards to the implementation of a state-of-the-art computer vision algorithm. The final objective is to upgrade the previous test platform and, in particular, increase the efficiency and decrease the weight of the system. 1.2.1
Problems to be investigated
The following questions will be considered throughout the course of this investigation:
Is stereo vision the fastest and most reliable method available for the implementation of an effective collision avoidance system? Is the combination of feature detection and range estimation algorithms the fastest method of implementing an effective collision avoidance system? What frame rates are achievable with alternative algorithms? What is the maximum speed achievable by the collision avoidance system? Is the system reliable enough for remote, autonomous deployment?
1
1.2.2
Purpose of the study
As a means of increasing the knowledge in the field of autonomous object detection and collision avoidance, the purpose of this investigation is to improve and enhance the previously implemented system. In this regard, the purpose of this study is to provide, by proof of concept, an effective and reliable method of implementing a collision avoidance system on an unmanned surface vehicle which is able to avoid objects in a marine environment at high speeds.
1.3
Scope and limitations
The scope of this investigation primarily encompasses the computer vision and algorithm implementation aspect of a collision avoidance system. The main focus is developing a system which is able to very quickly and reliably identify an obstacle. The test platform is required to effectively avoid the obstacle at high speed, however, the control system and dynamic modelling of the avoidance manoeuvre is not the primary focus of this investigation. Focus is placed on the vessel being able to avoid a single obstacle at high speed rather than multiple obstacles in a marine environment. Due to the inherent constraints of using cameras as a means of object detection, the proposed system is intended for use during the day only. In order to implement a complete collision avoidance system, a duel sensing modality is required wherein daylight is not a prerequisite for object detection. The investigation is limited in terms of resources, cost and time available. The hardware used for the system is limited to that which was used for the previous implementation. A budget of R1000 is granted for the investigation. Time is always a valid constraint and as such, improvements can be made to the system with more time and resources.
1.4
Plan of development
Firstly an in depth scrutinization of the relevant literature is presented in order to give an indication of the available modalities of object detection and to develop a deep understanding of these modalities. Subsequently, a theoretical development of the chosen modality is covered such that the method is fully understood. The process of implementation of the chosen algorithm using the available hardware is then explained, and the test platform development and construction is presented. Subsequently, the testing and results section fully characterises the performance of the system on the available hardware. Finally, conclusions are drawn based on the observations made during testing and recommendations are made for future iterations of this investigation.
2
2. Literature Review 2.1
Background and development of marine vessels
Marine transportation is an integral part of the history of mankind. The origin and inventors of the first boat are shrouded in mystery since boat building predates human drawings. Sailing was the first mode of aquatic transportation that was not directly driven by manpower. This enabled man to travel and explore the earth throughout the centuries, running on the limitless power of the wind. Boats have seen the rise and fall of empires and were the dominant form of transportation until the invention of the stream engine and aeroplane. This ancient discipline has stood the test of time and boats are still used today for leisure, transportation, fishing and research [2]. The automation of marine vessels is of particular interest within the area of marine research. In most instances the limitation of a research project is the endurance of the humans conducted the investigation. Removing the human element would increase the data capture process significantly. To this end, research groups have been conducting investigations in order to determine the most effective methods of automation of marine vessels. The first significant advancement in this field was the development of The DOLPHIN (Deep Ocean Logging Platform with Hydrographic Instrumentation and Navigation) in the year 1983. It featured a full scale, 7m long hull and was used for scanning the ocean floor [3]. Although The DOLPHIN was developed for underwater locomotion, which is outside of the scope of this investigation, it was significant because it kick-started research within the field. Since its development numerous other autonomous marine vessels have been developed which include; ACES (Autonomous Coastal Exploration System) developed at Cambridge in 1997 [4], the AutoCat developed at Cambridge in 2000 [5], the SCOUT (Surface Craft for Oceanographic and Undersea Testing) developed by MIT in 2005 [6]and lastly HUSCy (Hydrographic Unmanned Survey Craft) which was developed by SeaRobotics in 2005 [7]. There has also been a more recent surge in the development of autonomous, small scale sail boats. The majority of this research has occurred post 2005 as observed by Stalzer [8]. This can be attributed to the initiation of the Microtransat Challenge, a competition in which a fully automated sailboat is required to cross the Atlantic. The attraction of a sail boats, from a research point of view is that no fuel is required to power the vessels and they could therefore, theoretically, remain at sea indefinitely. A number of noteworthy autonomous implementations include the canvas wing vessel developed by Stelzer [9], the fuzzy logic control implementation developed by Abril [10] and the fixed wing vessel developed by Kilpin at the University of Cape Town, which was not part of the Microtranstat challenge but was a successful implementation nevertheless [11]. The scope of this investigation focuses on the development of a collision avoidance system on a motorized autonomous vessel. However the knowledge developed throughout this investigation could be adapted such that it could be used on an autonomous sail boat. All of the systems mentioned above were predominantly focused on the long term navigation and control of the vessel. As such there is a lack of significant research into the area of unknown obstacle detection and avoidance on marine vessels. This investigation sought to bridge this gap in automation.
3
2.2
Robotic architecture
There are two primary methods of obstacle avoidance - deliberate and reactive. Deliberate control involves the process of map making in which all movements and actions are planned before hand and executed in a predictable manner. A reactive system is indicative of the subsumption architecture proposed and developed by Brooks [12]. This architecture utilises a hierarchy of layers of competence in order to form ever increasingly complex robots. Each of the layers are independent of each other such that they can function independently. It is the process of “slicing the problem on the basis of desired external manifestations” [12] as opposed to slicing the problem into a layer of linearly executed functions. The first and most basic layer of automation is that of obstacle avoidance which was the primary objective of this investigation. The advantage of a reactive system is that no mapping of the environment is necessary which is computationally burdensome. The system can therefore be used to navigate in completely unknown environments.
2.3
Obstacle detection
Obstacle detection is the ability of a system to differentiate an object from the environment in which it is situated. Object detecting sensors can be split into two categories - active sensors and passive sensors. Active sensors function through the generation of energy which is transmitted into an environment, reflected off objects and subsequently reabsorbed by the sensor. The reflected energy is then analysed in order to determine information about the environment. Passive sensors function simply through the analysis of energy generated by an independent source. The defining factor that differentiates the two sensing systems is that active sensors function through the self-generated transmission of energy, whereas passive sensors rely on an independent energy source. This is not to say that the analysis of energy does not require energy itself. Passive sensing systems need energy to receive and process information. The image below shows the difference between passive and active sensing.
4
Figure 2-1: Differentiation between active and passive sensing modalities. Active sensors generate the energy which is reflected off an object and then reabsorbed by the sensor. Passive sensors analyse the information generated by an external source that is reflected off an object.
A list of the most commonly used obstacle detection modalities is shown below: Active sensing:
Electromagnetic Time of Flight o Radar o Lidar Acoustic Time of Flight o Sonar o Ultrasound Infrared computer vision: 3D sensors
Passive sensing:
2.3.1
AIS Computer Vision: o Optical flow o Monocular vision o Stereo vision o Trinocular vision
Active sensing modalities
Within the active sensing category, the majority of the modalities incorporate a time of flight (TOF) calculation in order to detect objects. There are two forms of TOF sensing - electromagnetic and acoustic.
5
i.
Electromagnetic time of flight
The electromagnetic TOF category showcases two commonly used modalities of obstacle detection radar and lidar. Radar Radar is an acronym for Radio Detecting And Ranging. The first Radar system was implemented in 1935 in Britain. This modality utilises the time of flight principle to detect objects by emitting electromagnetic energy and detecting the echo as the energy is reflected off an object. Radar operates in the electromagnetic frequency range of a few megahertz through to the gigahertz range[13]. One of radar’s most advantageous features is its ability to function in all weather conditions due to the penetrative nature of the radio waves, as well as the fact that radar is an active sensing system. It is a reliable and accurate modality with implementations that have a high enough resolution for wire detection [14]. Traditionally, these sensors were expensive and complex systems, implemented on a large scale since their original purpose was to detect full scale aircraft. Small implementations of the radar systems do exist and have been used in object detection applications such as those used for adaptive cruise control [15], however, these systems suffer from a narrow field of view (FOV) as well as low accuracy in lateral directions[16]. Although radar systems have been successfully implemented on large vessels, the inherent form factor of the system renders them incapable of being scaled to a size that is viable on micro sized vehicles[14]. Radar also struggles to detect objects at a close range. In general, the expense and complexity of a radar system overshoot the needs and recourses of the current investigation. Lidar Lidar functions according to the same fundamental principles as radar except that the frequency range of electromagnetic energy emitted is higher than that of radar. Lidar utilises lasers with a wave length of approximately 1 µm which is near the infrared and visible light band of the electromagnetic spectrum. Due to this small wavelength, lidar has a greater range resolution and higher accuracy in comparison to that of radar[17]. However, unlike its radio wave equivalent, lidar is very sensitive to the effects of vision obscuring weather such as rain and fog [16]. Another downfall is that lidar systems can interfere with one another, producing erroneous signals. The sensors also consume a large amount of power and are relatively expensive [18]. Similar to the radar system, a limiting factor of these devices is their size and weight relative to micro vehicles [19]. The systems also suffer from a limited perception horizon[20]. Successful lidar systems have been implemented by Brock et al[21] and Soumare et al [22], however these systems ran at slow processing rates on a land based vehicle at slow speeds unacceptable for the high speed nature of this investigation.
ii.
Acoustic time of flight
This category is comprised of the modalities which utilise the propagative nature of acoustics in order to calculate the distance to an object. As opposed to electromagnetic energy TOF calculations, sound is used in order to calculate the distance to an object. Electromagnetic energy propagates at the speed of light while sound waves travel at the speed of sound which is dependent on the medium through which the waves are travelling. This is the significant difference between the electromagnetic and acoustic TOF calculations. Other than this difference, the two methods are fundamentally the same. Within this category, there are two object detection modalities - sonar and ultrasound.
6
Sonar Sonar is an acronym for Sound Navigation And Ranging. The range to an object is determined by the time of flight calculation of sound waves to and from an object. Sonar uses sound waves that are relatively low frequency such that they are audible to the human ear [23]. Although sonar does have terrestrial applications, it is predominantly used as an aquatic range finder. This is due to the fact that the sound propagates far more effectively in water due to its density with respect to air. A sonar obstacle avoidance system was success implemented by Ulrich et al [24], however, sonar equipment is bulky and multiple sensors are needed in order to generate a detailed depth map. Ultrasound Similar to sonar, ultrasound uses sound waves in order to calculate range. The difference between the two modalities is that the ultrasound waves are of a higher frequency such that the sound is inaudible to humans. An advantage of both sonar and ultrasound is that the systems are not dependent on lighting conditions and can therefore be used at night. Ultrasound is also a simple system to implement. A shortcoming of the system is that is has a short detection range with a direct line of sight FOV. Due to this, the modality suffers when it comes to detecting complex 3D shapes [18]. This issue can be resolved through the use of an ultrasonic sensor array in order to generate greater depth perception. This is an expensive solution however, with high power requirements [22].
iii.
Infrared computer vision: 3D sensors
This active sensing modality does not use time of flight as a means of determining range information but rather uses the principle of disparity and triangulation. The most widely used 3D sensor is that of the Xbox Kinect. The system consists of an infrared (IR) projector, IR camera and a standard camera. An image of the system is shown below:
Figure 2-2: Image showing the Xbox Kinect module. The sensing is performed by the device by using the IR Emitter, Colour Sensor and IR Depth Senor. Image adapted form [25]
The Kinect functions according to the following principles: hundreds of infrared dots are projected onto the scene by the IR projector, this is called a speckle pattern. The IR camera captures the scene and the speckle pattern is analysed. Disparity is calculated by comparing the captured speckle pattern to the pattern that is expected by the system. From this disparity, the depth of each pixel in the scene can be calculated using triangulation techniques.
7
After its release in 2010 the Kinect gained popularity within the computer vision community for its ease of use and accuracy. It is considered the go to device for applications such as object detection and other applications that require depth information within a controlled environment [25]. Despite this popularity the use of IR for depth estimation has one significant shortcoming in that the system is only suitable for indoor implementation. In an outdoor environment, the IR light produced by the sun washes out the scene causing saturation which renders the system incapable of determining disparity [18] [19] [26]. The range of the Kinect is also limited and as such it is not the most effective outdoor implementation [27]. 2.3.2
Passive sensors
i.
Automatic identification system
The Automatic Identification System (AIS) utilises GPS data in order to transmit a ship’s location to other vessels. Introduced by the International Maritime Organization, the system allows ships to automatically communicate with other ships such that collisions are avoided [28] [1]. The AIS modality differs from all other sensors in that data is acquired through communication rather than detection. Nevertheless, provided a GPS signal can be acquired, this is essentially a foolproof system for marine collision avoidance between ships. Unfortunately, other vessels are not the only obstacle that must be avoided, and as such this system has no means of detecting unforeseen obstacles in unfamiliar environments. For this reason, AIS cannot be considered as a viable solution for object detection.
ii.
Electro optical computer vision
Within this category, the modalities have the common trait of gathering data through the use of digital cameras. A digital camera is fundamentally a two dimensional light intensity sensor which represents information as a two dimensional array commonly known as a picture. It is this detection of light that defines digital cameras as passive sensors since the light is generated by an external source, the sun. Depth estimation is calculated by active sensing modalities most commonly through the TOF principle. However, this principle cannot be used for passive sensors since no prior information is available regarding the detected energy. For this reason, electro optical devices utilise processor intensive computer vision algorithms in order to recover depth information. In general, electro optical modalities have the advantage of a wide field of view and a high information to size and weight ratio. The disadvantages of these modalities is that they are sensitive to weather conditions, cannot function at night and they require a high computational effort. In TOF systems, the range information is inherent in the modality whereas with passive computer vision methods, depth estimation must be calculated using alternative means. There are three categories of cues that can be used for depth perception from two dimensional images:
Monocular cues Motion parallax: correspondence in time Stereopsis: correspondence in space
These cues correspond to their own individual modalities which are described below:
8
Monocular vision Monocular vision is the subset of computer vision in which a single camera is utilised. Unlike stereo vision wherein depth can be determined by triangulation, a single camera must rely on other methods of extracting depth information. These are commonly referred to as monocular cues. These cues include: Defocus: A lens is able to focus on an object within a certain range, but outside of this range, the object becomes blurred. This information can therefore be used to estimate depth [29]. Haze: The further an object is away from a point of view, the more obscured it is due to atmospheric interference [29]. Texture variation: Textures become smoother the further away the point of view is. This is due to a loss of detail with depth[30]. The above mentioned techniques are but a few methods from which depth information can be extracted from 2D images. There have been a number of monocular vision based collision avoidance implementations including high speed robotic cars [31] and unmanned aerial vehicles (UAVs) [20]. The robotic car implementation utilised a machine learning strategy which is not suitable for navigation through an unknown environment. The UAV implementation on the other hand ran at a rate of 10 FPS which is not as fast as other methods of electro optical sensing. Overall monocular vision is not reliable or fast enough to justify its use over other techniques such as stereopsis which has greater obstacle detection capabilities. Optical flow Optical flow functions according to the principle of motion parallax, and is the method of generating a vector field which describes the motion of pixels from one frame in time to the next. It is one of the oldest methods of computer vision. Pixel vectors are generated by finding correspondences between pixels. A dense point correspondence can then be generated over the entire image in order to generate a vector field. This vector field can then be used to derive the relative motion between the camera and the scene as well as objects within the scene. Optical flow is usually performed using greyscale video such that each pixel is represented by one light intensity value. The process can be extended to RGB video, however, this adds to the already burdensome computational process. The first formalized optical flow algorithm was proposed by Horn and Shunck [32] in which they characterised a spatio-temporal vector which defined the movement of pixels as a function of space and time. This global method relies on the brightness constancy assumption, in which it is assumed that the intensity level of each pixel remains the same over time. As an added constraint, this method was further refined by the addition of a smoothness factor in order to create a realistic vector field and fill in areas of ambiguity. The Lucas-Kanade method alternately uses a windowed approach which assumes that the pixel movement is contained within a small area. This adaptation is well suited to overcoming the ambiguities inherent in global systems, however, it cannot determine flow in low textured images. [33] There have been a number of successful robotic obstacle avoidance implementations that have utilised obstacle flow. The platforms used for these implementations include an omnidirectional robot [34] and micro aerial vehicles (MAVs) [35] [36]. These systems are, however, generally constrained to low frame rate applications with limited resolution. Such a constraint limits the systems, and in highly complex environments, detection may not be precise enough to avoid small objects.
9
Stereo vision Stereo vision is the computer vision technique whereby the use of two cameras is used to give a system the view of a scene from two angles. This binocular vision can be used in order to extract depth information of objects within the scene. This is done through a four point process of undistortion, rectification, correspondence and triangulation. In order to understand the process, triangulation is first explained. It is initially assumed that an ideal stereo camera rig has been setup such that the following assumptions are true:
The focal points of both cameras are situated on the same horizontal plane. The focal length (F) of both cameras is the same. The principle point1 of each camera is at the centre of the image plane of each camera. The principle rays2 of the camera are parallel to each other.
If these assumptions are true, the following distance calculation can be applied. With reference to the diagram below, the distance (Z) to an arbitrary point (P) can be calculated if the focal length (F) is known and the baseline distance (T) separating the centre of each camera is known. The last piece of information that must be known is that of the horizontal pixel coordinates corresponding to the object P in both the left and right image frames (xl and xr ). The difference between these two quantities is called the disparity: 𝑑 = (𝑥𝑙 − 𝑥𝑟 ) If the above quantities are known then the rule of similar triangles can be applied in order to determine depth. From the diagram, it can be seen that triangle xl – P – xr is similar to triangle Ol – P – Or such that the following relationship can be derived: 𝑇 − (𝑥𝑙 − 𝑥𝑟 ) 𝑇 = 𝑍−𝑓 𝑍 ∴𝑍=
1
𝑓𝑇 𝑑
The principle point is defined as the point where the principle ray intersects the image plane The principle ray is an imaginary line that extends outward from the focal point at an angle perpendicular to the image plane 2
10
Figure 2-3: Diagrammatic explanation of the triangulation principle. The distance Z to an object P can be determined using the known parameters of the cameras and the law of similar triangles. Image adapted from [33]
In order to use triangulation to compute the distance to an object, correspondence points must be found. A correspondence point is defined as a common point in a scene that is observed by both cameras. This point will have different coordinates in the left and right image planes respectively. For example, the face in the above diagram has the horizontal coordinate xl in the left image and the horizontal coordinate xr in the right image. The correspondence problem is the core issue of stereo vision around which the majority of the research is based. [33] Stereo vision has been used extensively as a means of obstacle detection and collision avoidance for a number of applications. These include: adaptive cruise control on motor vehicles [16] [37] and navigation of small robots [18] [38] [39] and UAVs [14] [19] [26]. The popularity of stereo vision implementations can be attributed to the excellent weight to computational output ratio of digital cameras. All the systems mentioned above showed promising results due to the implementation of stereo vision with regards to object detection. A criticism of stereo vision is the computational burden of determining correspondence and disparity, however, this can be overcome with epipolar geometry and other such constraints as mentioned in the Epipolar geometry section. The UAV implementations present the most promising results due to the speed at which the algorithms are required to run in order to be used in real time on aerial vehicles. Trinocular vision As an extension to stereo vision, a third camera has been added in some collision avoidance applications [40]. The additional camera is then able to determine disparity in the vertical plane. This added information is necessary in some applications but is a redundancy for the purposes of this investigation. 2.3.3
Object detection modality comparison
The table below summaries the advantages and disadvantages of each modality mentioned above.
11
Table 2-1: Showing the advantaged and disadvantages of each object detection modality that was investigated.
Radar Advantages Disadvantages
Lidar Advantages Disadvantages
Sonar Advantages Disadvantages
•Excellent weather penetration • Reliability
• Very high accuracy • Fast data acquisition
• Excellent performance in water • Can Work at night
• Expensive •Low accuracy over short distances
Ultrasound Advantages Disadvantages • Can work at night • Simple to implement
• Unable to detect complex shapes •Low information feedback Electro optical computer vision Advantages Disadvantages • Excellent weight to information ratio • Data appealing to human interpretation
• Affected by the weather •Limited perception horizon
3D sensors Advantages Disadvantages • Excellent object detection attributes • High accuracy
• Does not work outdoors well • Limited detection range
• Limited performance through the air • High complexity AIS Advantages Disadvantages
• 100% accurate localisation •Reliability
• Cannot detect unknown obstacles • Cannot work without GPS
• Compuation –ally burdensome • Sensitive to the weather
The table above shows the trade-offs and compromises that are inherent in each sensing modality. None of the modalities provides a perfect solution to the problem and as such it was necessary to choose a modality that best suited the application. To this end a criteria cost analysis was performed. The areas of highest priority with regards to the sensing modality were; cost, weight/size, ease of implementation, range, field of view and the ability to detect unknown obstacles. Each of these criteria were given a weighing corresponding to their importance with respect to the investigation. The table showing the results of the criteria cost analysis is shown below.
12
Table 2-2: Showing the results of the criteria cost analysis of the modalities under investigation. The weighting multiplier was determined according to the importance of the criteria to the system.
Weighting multiplier
Radar
Lidar
Ultrasound
3D sensors
AIS
Computer Vision
Cost
20
1
2
1
8
5
6
5
Weight/size
25
3
3
5
8
6
5
7
Ease of implementation
10
1
2
1
7
8
6
4
Range
15
9
7
9
1
3
9
5
Field of view
10
4
3
5
2
7
9
6
Ability to detect unknown obstacles
25
9
9
9
7
8
0
9
505
495
565
640
645
530
675
Criteria
Total
Sonar
Radar, lidar and Sonar all scored poorly in the areas of cost and ease of implementation due to the inherent complexity of these modalities. AIS scored highly in most areas except for that of the ability to detect unknown obstacles, which was an essential component of this investigation. The sum total of the weighted scores showed that computer vision modalities were the most applicable for this investigation. As a result further research was conducted in this area. As previously discussed there are three primary sensing methods which fall under the computer vision modality. Of the three, stereo vision was chosen to be implemented for the following reasons:
Real time depth information: optical flow depth estimation is based on the analysis of monocular cues which require relative motion between the camera and the environment. Alternatively stereo vision depth calculations are available with or without relative motion of the cameras and the environment. Reliability: monocular cues are by their nature, unreliable which in turn results in unreliable depth accuracy. Alternatively the depth accuracy of stereo vision applications is dependent on the internal properties of the cameras and the quality of calibration rather that external influences. High frame rates achievable: after a comprehensive review of the literature it was found that stereo vision algorithms were able to achieve frame rates much higher than those achieved by monocular vision and optical flow algorithms.
Based on the aforementioned reasoning, further investigation was conducted into the implementation of a stereo vision algorithm.
13
2.3.4
State-of-the-art stereo vision implementations
The current state-of-the-art implementation for stereo vision obstacle detection was that implemented by Honegger et al [41]. The stereo correspondence algorithm was optimised by a Field Programmable Gate Array (FPGA) module such that the computation of a dense disparity map was performed at 127 frames per second (FPS) at an image resolution of 376x240 pixels. This implementation optimised stereo correspondence in order to create a full disparity map such that it ran at low latency and in real time on a MAV at high speed. As an alternative to this method of significantly increasing processing speed, Barry et al [19] implemented the Pushbroom stereo algorithm. This implementation significantly reduced the computational load by searching for objects at a single disparity. This in turn allowed the system to run in real time at 127 FPS and at a resolution of 376x240 pixels, therefore matching the FPGA system performance as mentioned above. The Pushbroom algorithm was complemented by state estimation feedback which was used to record past detections and superimpose them on the current image frame.
i.
Comparison between FPGA implementation and the Pushbroom algorithm
The difference between the two methods of object detection was that the FPGA system significantly reduced computational time whilst retaining all disparity information. The Pushbroom system on the other hand was able to run at the same rate and resolution by sacrificing a significant amount of depth calculations. Despite this reduction in computational effort, the two systems showed similar testing results [26]. The table below shows a comparison between the two implementations. Table 2-3: Comparison between the current state-of-the-art implementations
Hardware Algorithm Implementation Data output
Honegger et al FPGA Semi global matching stereo Dense correspondence
Speed (fps) Resolution (pixels) Latency (ms) Power Consumption (watts)
127 376x240 2 No. of right hits?
No
Activate avoidance mode
No No
Avoidance mode activated? Yes X frames elapsed?
Yes
Recorrect course
No Write hit count to a .csv file
No
Exit?
Yes End
Figure 6-9: Flow diagram showing the collision avoidance algorithm. The box with the dashed lines was added after an observation made during testing.
49
When implemented on the BeagleBone, the algorithm above ran at approximately 14 Hz. The sequence begins by receiving frames from the two cameras. The synchronisation of image frames is of significant importance, and is explained in the Image capture synchronisation section. Once the frames have been captured, they are converted from three channel RGB images to 8-bit greyscale images such that they can be processed. The images are then processed according to the Pushbroom algorithm as explained in the Algorithm Development section. This function returns a vector containing the coordinates of the pixels, for which valid stereo hits are recorded. The number of hits determines whether a collision with an obstacle is imminent or not, therefore if the number of hits is above a certain threshold, the BeagleBone sends a signal to the Arduino, telling it to turn either left or right. The two conditional blocks, outlined by the dashed line, were added subsequent to observations made during the dam test in the Open water testing section. If an obstacle is detected the boat enters into a collision avoidance mode during which the turning manoeuvre is performed, and the boat’s speed is reduced (later iterations of the algorithm did not cause the boat to slow down during an avoidance manoeuvre). After a prescribed, x, number of frames have elapsed, the boat’s rudder is automatically straightened. At this stage, the hit count for each half of the image frame is written to a csv file for offline analysis. The loop is then restated if an exit command is not received.
50
7. Algorithm Implementation The implementation of the Pushbroom algorithm took longer than initially expected due to unforeseen issues. This section explains how those issues were overcome and the way in which the algorithm was implemented on both a personal computer and a BeagleBone Black.
7.1
Computer based implementation
The computer used for the testing and verification stages of this investigation had the following specifications:
Table 7-1: Showing the specifications of the computer used for developing, debugging and testing the Pushbroom algorithm
Model Operating system
Toshiba Satellite
Windows 7 Home Premium Linux Debian (8) Jessie (via Virtual box) Linux Debian (8) Jessie (via duel boot system)
Processor RAM System type
7.1.1
Intel Core i5 (2.4GHz) 8.00 GB 64-bit
Windows based implementation
Initial attempts utilised the default operating system, Windows, in order to implement the Pushbroom algorithm. A key aspect of the investigation was the use of the OpenCV library for both calibration and stereo vision, and as result, the first order of business was installing the OpenCV library on Windows. This proved to be a difficult task. Since OpenCV is an open source computer vision library, it is a culmination of knowledge from the general public. This has significant advantages in terms of pooling knowledge together, but has the disadvantage of not being as neatly packaged as other independent libraries. The result of this was that all attempts to install the C++ version of the library on Windows, failed. The first attempt to install the library utilised the latest package – OpenCV 3.0.0 . Virtual Studio 2015 was chosen as the IDE for integration with the OpenCV library due to its popularity within the computer vision community. Simple code examples worked as expected with this implementation, however, the introduction of more complex functions caused errors, and the programs would not run due to missing dynamic-link library (.dll) files. After a period of investigation, it was believed that the problem was based on the fact that OpenCV 3.0.0 had been recently released. As a result, it was thought that it did not have all the features that OpenCV 2.4 had and was, therefore, prone to errors. However, after uninstalling OpenCV 3.0.0 and reinstalling OpenCV 2.4.11, the same missing .dll file error occurred.
51
Due to the time constraints and availability of other implementation methods, integration of OpenCV with Visual Studio on Windows was abandoned.
7.2
Debian based implementation
The operating system of choice for many engineers is that of Linux. This is due to the low level, uncluttered and extremely powerful nature of the Unix based system. Many useful tools are seamlessly integrated into the Linux kernel, such as the package manager, which allows a high level of customisation and modification. The entire Linux ecosystem is also based on open source development perfectly suited to the integration of OpenCV and other such libraries. From the numerous operating systems available under the Linux name, Debian 8 (Jessie) was chosen for this investigation. This was based on the fact that Debian is the operating system of choice on the BeagleBone. The use of the same operating system on both platforms simplified the implementation process. Debian is also a very lightweight system which was able to run all the necessary applications. 7.2.1
Virtual Box
Debian was first installed through the Virtual Box program which is a system emulator that allows an operating system to run simultaneously through another operating system. This has its respective advantages and disadvantages. Advantages: Virtual Box allows access to both operating systems, therefore enabling Windows and Linux based programs to run simultaneously. Disadvantages: A significant disadvantage of using Virtual Box is that it must share available hardware resources with the host operating system, and even if no programs are running on the host, the virtual operating system will run at a slower rate than if it had been run independently. Another disadvantage of Virtual Box is that it does not have necessary firmware and drivers to run all USB devices, and while a mouse and keyboard operated as expected, the Logitech cameras returned a black image frame. This problem was addressed and is elaborated on in the next section. Despite the lack of a live camera feed, Debian was used via Virtual Box for the duration of the investigation for tasks which included: camera calibration, secure shell communication and secure file transfer with the BeagleBone as well as offline Pushbroom algorithm implementation from a still image. The Integrated Development Environment (IDE) used with Debian was that of Eclipse Luna due to its popularity, availability and cross compilation attributes. Integration of the OpenCV library with Eclipse was far easier than earlier attempts with Visual Studio, and the system ran smoothly after following a simple tutorial [52]. Once the OpenCV library had been integrated into the Eclipse environment, the Pushbroom algorithm was opened in a new project. The program did not compile initially, as expected, due to compilation errors and missing header files. Once these issues had been resolved, the program ran without error. 7.2.2
Cross Compilation
Personal computers run on the AMD64 software architecture which is different to that of the BeagleBone Black which runs on the ARMv7-A software architecture. As a result, code compiled on a computer will not run on the BeagleBone and vice versa. Due to the differential processing power of the two devices, code is compiled at a slower rate on the BeagleBone Black.
52
As a result, the most efficient method of compiling code for the BeagleBone is to setup a cross compilation environment. This allows the computer to compile code in the ARMv7-A architecture such that it can be run on the BeagleBone. A cross compilation environment was subsequently set up using Eclipse. This method worked well for simple code that did not require additional libraries. However, attempts to integrate OpenCV code into cross compiled programs failed. Further investigation revealed that the OpenCV library installed on the computer was AMD64 specific, and therefore, could not be compiled in the ARMv7-A architecture. As a result an alternative method of compilation was required for the BeagleBone.
7.3
CMAKE program compilation
After exhausting all attempts to perform cross compilation, alternative methods of compilation were investigated. Linux systems are well suited to the use of MAKE, a program compilation manager. Despite its intended ease of use, MAKE can be a complicated development tool to use for beginners. To resolve this issue, CMAKE was developed. CMAKE is one level of abstraction higher than that of MAKE and was purpose-built to create the MAKE files necessary to compile a specific program. The shallow learning curve of CMAKE is a testament to its simplicity, and code was successfully compiled on the same day of CMAKE’s discovery. The flow diagram below shows the development tool chain used to compile code onto the BeagleBone.
Eclipse: C++ File
SFTP
BeagleBone: C++ File
CMake
BeagleBone: Make File
Make
BeagleBone: Executable File
Figure 7-1: Flow diagram showing a graphical representation of the development tool chain used to compile and run the Pushbroom algorithm on the BeagleBone
7.4
Duel boot implementation
In most cases, computer vision algorithms require some means of visual inspection in order to verify that the program is running correctly. The Pushbroom algorithm is no different, and a large part of the code is dedicated to a human interface in order to effectively debug the program. As a result of the Virtual Box not being able to capture a live image feed, it became a necessity to search for an alternative means by which this could be achieved. The solution was to setup a duel boot system such that Debian could run independently of the Windows operating system. After following a number of online tutorials, a duel boot was successfully installed and the cameras worked as expected. Eclipse and the OpenCV library where subsequently installed such that the development environment mimicked that of the Virtual Box configuration. Another advantage of the duel boot system is that the performance of the Pushbroom algorithm was much greater than that achieved by the Virtual Box Debian setup.
53
8. Algorithm Anomaly An anomaly was observed during testing as a result of relative motion between the test platform and an object. Its effects and the solution to the problem are described below.
8.1
The lag effect
Any movement of the test platform resulted in a change in the properties of the Pushbroom algorithm. Due to the fact that the images were not perfectly synchronised, the environment changed between grabbing an image from the first camera and grabbing an image from the second camera. As a result, the disparity between objects was affected according to the speed of image capture, the speed of the test platform and the order in which the images were taken. This anomaly was named “the lag effect”. The image below graphically depicts the lag effect.
Figure 8-1: Image showing the effect of relative motion between the obstacle and the test platform. The solid circle represents the object before it has moved and the perforated ring represents the object in the next image frame. The last sequence shows how the disparity appears to change due to the image frames being unsynchronised.
54
The disparities of the object in the stationary platform sequence and the moving synchronised camera sequence are equal. This is because the image does not move during image capture. The same cannot be said for the unsynchronised camera sequence. The effect of a change in disparity is that the object will appear to be further away or closer than it actually is. This effect occurred when the boat was moving towards the object or when the cameras were panning with respect to the object. The table below summarises the characteristics of the lag effect when there is relative motion between the object and the boat.
Table 8-1: Showing the characteristics of the lag effect for different scenarios when there is relative motion between the boat and an object.
Frame capture order
Object position
Effect on Disparity
Object appears…
Left frame first
Passing from right to left Passing from left to right
Decrease Increase
Further away Closer
Right frame first
Passing from right to left Passing from left to right
Increase Decrease
Closer Further away
The image below shows a graphical representation of the information in the table above:
Figure 8-2: A graphical representation of the lag effect. The perceived distance of an object was affected by which image frame was captured first and in which direction the object was moving in relation to the boat.
The lag effect influenced the disparity in a manner which was dependent on the scenario and which side the object was with respect to the boat. Due to this, it was impossible to formulate a strategy which fully negated its effect. However, as a form of mitigation, a strategy was implemented when the boat was told to turn.
55
The safest scenario with respect to collision avoidance was to change the frame capture order such that any potential objects appeared closer than they actually were. As a result, a conditional statement was added to the algorithm whereby if the boat was ordered to turn to the left, such that the object passed from left to right, the frame order was automatically set to capture the left frame first. If the boat was ordered to turn right, the frame order would subsequently switch to capture the right frame first. The frames were captured 20-30 ms apart from each other as mentioned in the Image capture synchronisation section. The lag effect did not have significant repercussions on the performance of the algorithm due to the speed of the boat and the time between frame grabs. However, this anomaly must be taken into account for future iterations and cases where the speed of the boat is increased significantly.
56
9. Test Platform Development This section describes the upgrades made to the test platform, how they were implemented, and what the effect was, of the upgrades on the performance of the system.
9.1
Previous test platform
The test platform in the previous iteration of this investigation was a remote controlled speed boat with a shallow vee, 75cm long hull. It was designed and constructed by de Klerk [1]. The boat was driven by a BCL51G DC motor, and steered by a balanced rudder operated by a JR 591 servo motor. In terms of processing power, the boat is equipped with a BeagleBone Black for the computer vision processing and an Arduino Mega which directly controlled the boat. The cameras used to implement the computer vision algorithm were Logitech C210 webcams. The boat was powered by two battery packs, the first was a 4500mAh NiMH battery and the second was a 2200mAh LiPo battery. Auxiliary devices included a USB bus, a Wi-Fi router, an independent GPS module and a second servo motor to turn the stereo rig. A full list of components is shown below:
Table 9-1: Original test platform component list
BCL51G DC motor JR NES 591 servo motor x2 BeagleBone Black Arduino Mega Logitech C210 Webcams x 2 Ansmann-racing 4500mAh NiMH battery Power Tiger 2200mAh 3 cell LiPo battery Lexma 4 port USB hub Edimax BR-6258 Wi-Fi router GlobalTop FGPMMOPA6H GPS module 7805CT linear regulator x3
The image below shows the hull of the boat, the motor and the rudder servo motor as they were received.
57
Figure 9-1: Image of the boat hull and the fixed components including the motor and rudder servo motor
These components, as well as the camera mount, remained unaltered between the original and current implementation. The rest of the components were upgraded, rewired or repositioned in order to improve the system.
9.2
Test platform upgrades
In accordance with the objective of this investigation, certain upgrades to the test platform where deemed necessary. These improvements included: 9.2.1
Battery upgrade
NiMH batteries have a low power to weight ratio. As a result, the original test platform was very heavy and sat low in the water. The previous author recognised this issue but was unable to source a sufficiently large LiPo battery. A duel battery system was also necessary because of brown out issues, and this significantly added to the weight of the system. In response to these issues, the latest implementation used four, 1000mAh, three cell LiPo batteries connected in parallel. The justification for connecting four smaller batteries together was that they were readily available for use. Although the battery capacity was reduced from 6700mAh to 4000mAh, the weight of the system was also reduced, and battery life was not an issue throughout the test period. The use of LiPo batteries also solved the brown out issues of the previous system. 9.2.2
Power regulator upgrade
The linear regulators of the previous implementation where replaced with switch mode power supply (SMPS) regulators as a means of increasing efficiency. The SMPS used was that of the LM2576T-5. This component is a high frequency buck converter that steps the supply voltage down to 5V. However, as the current load increases, the voltage drops from this nominal value. If this drop is too severe, it can cause the on-board devices to malfunction or power down. The voltage output to current rating for the LM2576T-5 is shown below: 𝑉𝑜𝑢𝑡 (0.5 ≤ 𝐼 ≤ 3𝐴) = 4.8 → 4.75𝑉4
4
From the LM2576T-5 datasheet
58
The current load for the on-board devices (as shown in Table 9-3 below) was split between two SMPS supplies, and therefore, the current through each unit was well within the above limit. The voltage tolerance of the BeagleBone Black (which was one of the most sensitive components) is ∓ 0.25𝑉 which therefore justified the use of SMPS because 𝑉𝑜𝑢𝑡 was above this minimum tolerance. This setup significantly increased the electrical efficiency of the system since linear regulators have an efficiency of 27.5% and SMPS regulators have an efficiency of 77% [53]. 9.2.3
MOSFET upgrade
The previous implementation used a HUF75339P3 N channel MOSFET to control the motor. This was replaced by the IRF540N high powered N channel MOSFET. The current rating of this device far exceeded that needed for the application but was chosen due to its availability. The previous author noted that the HUF75339P3 MOSFET became very hot due to the high current flow under a high torque demand. In order to rectify this heat issue, the author added more fly back diodes in parallel with the MOSFET. For the current implementation, a much larger flyback diode was used. The output voltage of the Arduino Mega was also stepped up from 5V to 10V through a LM358N duel operational amplifier (opamp) integrated circuit. This voltage drove the gate of the MOSFET. By increasing the gate voltage, the channel resistance of the MOSFET was significantly reduced. As a result, this solution resolved the issues of the previous implementation, and no heat problems were encountered throughout the duration of the investigation The table below shows the component list for the current implementation.
Table 9-2: Current component list
BCL51G DC motor JR NES 591 servo motor BeagleBone Black Arduino Mega Logitech C210 Webcams x 2 Zippy Compact 1000mAh 3 cell LiPo battery x4 Lexma 4 port USB hub Edimax BR-6258 Wi-Fi router GlobalTop FGPMMOPA6H GPS module LM2576-5 SMPS regulator x2 LM358N Duel Opamp
The difference between the original component list and the current component list was the use of the 1000mAh batteries, the use of only one servo motor, the replacement of the linear regulators for SMPS regulators, and finally, the inclusion of an opamp integrated circuit. The table below shows the electrical characteristics of the components used.
59
Table 9-3: Electrical characteristics of the components used on the current system.
Component BCL51G DC motor JR NES 591 servo motor BeagleBone Black Arduino Mega Logitech C210 Webcams Lexma USB hub Edimax BR-6258 Wi-Fi router GlobalTop GPS module Total
Voltage Rating (V) 4 -9 5 5 5 5 5
Average current (mA) 2000 200 300 60 100 Same as cameras
Max current (mA) 4000 1000 600 100 200 Same as cameras
5
500
700
3.3 -
20 3180
25 6625
The average current consumption expected from the system was 3A when the motor was running and the collision avoidance system had been activated. The maximum current expected was 6.625A, which justified the use of the 10A fuse as a safety feature. The current consumption of the USB hub was the same as the cameras because the USB hub consumed a negligible amount of power, and was merely a pathway for passing information between the cameras and the BeagleBone.
9.3
Build overview
The boat hull, drive motor and servo motor were used in the current implementation as they were found from the previous implementation. All of these components, and the boat hull in particular, were constructed and assembled very professionally, and no upgrades where required for the current implementation. In order to implement the upgrades listed above, the control board or ‘brain’ of the boat was redesigned. A key design upgrade of the new board was that it was completely detachable from the boat throughout the duration of the investigation. Another key feature was that all the electronic components, besides the drive and servo motors, were attached either on top or underneath the control board and not directly to the boat hull. This was done as a means of safeguarding the components from water that may have splashed into the boat during testing. As a result, the control board used for the previous test platform was modified using SolidWorks, and the design was cut out of a 3mm sheet of clear Perspex. The ability to see through the Perspex at all times assisted the development process. The images below show the control board.
60
Figure 9-2: Image showing the top side of the control board: 1) BeagleBone Black 2) battery connector, 3) Lexma USB hub 4) main power switch 5) 10A fuse 6) SMPS regulator switches 7) GPS module 8) Edimax Wi-Fi router 9) Arduino Mega 10) motor connector
Figure 9-3: Image showing the bottom side of the control board: 1) battery connector, 2) main power switc, 3) 10A fuse 4) SMPS regulator switches 5) Opamp circuitry 6) SMPS regulators 7) IRF540N MOSFET 8) 6A flyback diode 9) motor connector
9.4
Communication
The image below shows the system communication block diagram.
61
Wi-Fi Adapter Edimax BR-6258 150Mbps router
Right Camera Logitech C210 176x144 Resolution
USB 2.0
Ethernet
Left Camera Logitech C210 176x144 Resolution
Lexma USB hub 4 X USB 2.0 input Unpowered
USB 2.0
BeagleBone Black AM335X 1GHz ARM Cortex-A8 512 DDR RAM Debian Distribution UART
JR591 Servo (Rudder) Analogue 0.24 sec/60° 5.1 kg-cm
PWM
Arduino Mega Atmega2560 16Mhz
PWM
BCL51G DC Drive Motor 4.8-9.6V 64.9W 20000rpm Max
UART
GPS Module GlobalTop Stand alone GPS module FGPMMOPA6H
Figure 9-4: Control board system block diagram showing the communication connections between the components. Image adjusted from [1].
Communication on-board the test platform featured:
One Ethernet connection Three USB connections Two UART connections Two PWM connections
The UART and PWM communications were unidirectional as shown above. No feedback was necessary from the Arduino Mega, however, future iterations with state estimation and feedback would require this capability. Bidirectional communication was necessary between the cameras and BeagleBone in order for the BeagleBone to set image properties and for the camera to send through image frames. In the same manner, bidirectional communication was utilised between the BeagleBone and the Wi-Fi router in order for the BeagleBone to send and receive information. The Wi-Fi router was used as the communication link between the computer and the BeagleBone. The 150mbps bandwidth of the router was more than sufficient for passing information in real time between the two devices, and commands were received and performed by the boat at an imperceptible latency. This quick response of the system to input commands was well suited to high speed testing. It was initially thought that the frame rate of the cameras would breach the maximum bandwidth of the USB 2.0 connection between the USB hub and the BeagleBone. However, this was not the case, and the connectivity between devices and components was seamless throughout the testing process. No problems were encountered with regards to communication.
9.5
Circuit diagram
The circuit diagram below depicts the power connections between the batteries, regulators and electronic devices.
62
BCL51G DC Motor Li-Po battery bank 12.5V 4000mAh
LM328N Signal from Arduino 4 6 5 100KΩ
Flyback Diode IRF540N
100KΩ
12.5V
IN
LM2576-5 FB GND SMPS OUT
100µF
5V
BeagleBone 100µH
EN
1000µF 10A Fuse
Lexma USB hub Logitech cameras
12.5V
100µF
IN LM2576-5 FB GND
SMPS EN
OUT
Arduino Mega
5V
100µH 1000µF
3V3
GPS Module Wifi Module Rudder Servo
Figure 9-5: Circuit diagram showing all power connections and voltage levels between components.
The BeagleBone, USB hub and the cameras were run off a separate SMPS regulator due to the high current drawn from these components. Power for the USB hub and cameras was routed via the BeagleBone, and they did not require independent power sources. No brown out issues where encountered throughout testing. The Arduino, Wi-Fi module and rudder servo motor were run off the second SMPS regulator, and the GPS module was powered from the 3V3 regulated power supply available on the Arduino Mega. No power issues were encountered for these devices throughout the testing process. The image below shows the complete physical test platform with the control board and camera mount included.
63
Figure 9-6: Image showing complete test platform. Cling wrap was used to cover the electronics in order to protect them during water testing.
9.6
Test platform development summary
Certain aspects of the boat’s hardware were upgraded in order to make the boat lighter and more efficient. The result of these upgrades was evident during the water tests. The boat ran smoothly throughout the entire investigation and no problems of any form or nature were encountered with regards to the hardware of the boat.
64
10. Testing and Results This section describes the methods by which the collision avoidance algorithm was tested. Tests were conducted on and off the test platform during both outdoor and indoor tests respectively. A total of seven test procedures were conducted during the course of the investigation. The first test was a performance test conducted to determine the speed of the algorithm on different platforms. The second test was an indoor control test conducted to ensure the system functioned correctly and to gather control data. The third, fourth and fifth tests were conducted in a small swimming pool in order to test the algorithm on the boat test platform. The sixth test was an open water test conducted in a dam in order to simulate a real world scenario, and finally, the last test was performed in open water at high speeds to characterise the performance of the algorithm and test the system to the limit of its capability. The results of each test are integrated into each respective subsection for ease of reference.
10.1
Performance tests
In order to successfully implement a high speed collision avoidance system, the algorithm was required to run at a high frame rate. As a result, this section is dedicated to the analysis of the speed performance of the Pushbroom algorithm when implemented on the computer and BeagleBone platforms. 10.1.1
Expected performance
Barry et al were able to implement the Pushbroom algorithm at a rate of 127 FPS. This was through the use of specialised hardware such as the ODROID-U2 and the Grey Point Firefly MV camera. Due to the hardware restrictions of this investigation, the frame processing rate was expected to decrease. The two processes which require the most amount of time are that of image capture function and image processing function. The image capture function implemented by Barry et al utilised the libdc1394 camera library to capture frames at a very fast rate. The image processing function, therefore, required the most amount of time between the two functions. If the image capture function is assumed to have taken a negligible amount of time, then at very most, the image processing function can be assumed to have taken 8ms at most to process one frame. Using this value, it is possible to calculate the expected decrease in performance.
Table 10-1: Calculation of the expected decrease in performance due to hardware restrictions
Processing speed (GHz) Number of threads Total
BeagleBone 1 1
ODROID Multiplier 1.7 x1.7 8 x4 x6.8
The x4 multiplier was determined experimentally by tests conducted by Barry et al [49].
65
𝐸𝑥𝑝𝑒𝑐𝑡𝑒𝑑 𝑃𝑒𝑟𝑓𝑜𝑟𝑚𝑎𝑛𝑐𝑒 = 8𝑚𝑠 × 6.8 = 54.4𝑚𝑠
𝐸𝑥𝑝𝑒𝑐𝑡𝑒𝑑 𝐹𝑟𝑎𝑚𝑒 𝑟𝑎𝑡𝑒 = 18.38 𝐹𝑃𝑆
In reality, the algorithm took longer than expected as discussed below. 10.1.2
Computer performance test at the original resolution
i.
Virtual box
Due to the fact that Virtual Box shares the hardware resources of the computer between the host and guest operating systems, the results of the performance tests conducted on the virtual machine were ignored. A true representation of the performance of the algorithm on the computer is presented below.
ii.
Duel boot
The speed performance test was conducted at the same time as the indoor control tests. The first test was conducted on the computer platform at a resolution of 320x240 pixels, utilising eight threads. The algorithm ran at 30 FPS and was limited by the maximum frame rate of the cameras. The graph below shows the time taken to process each frame in microseconds.
Speed Performance Test on the Computer at a Resolution of 320x240 12000
Time (µs)
10000 8000 6000 4000 2000
1 31 61 91 121 151 181 211 241 271 301 331 361 391 421 451 481 511 541 571 601 631 661 691 721 751 781 811 841 871 901 931 961 991
0
Frame Number Average: 2029µs Figure 10-1: Graph showing the time taken to process each image frame in a set of 1000 frames. The average for the test was 2029µs and is shown in orange.
The processing speed observed on the computer was very erratic. The minimum time recorded was 899µs and the maximum time recorded was 10479µs. This can be attributed to the fact that many background processes were running during the test. Despite this, the average time to process an individual frame was a quarter of the maximum time taken when running at 127 FPS on the ODROIDU2 platform.
66
BeagleBone performance test at the original resolution
10.1.3
The algorithm was subsequently compiled and tested on the BeagleBone. The graph below shows the speed performance test for the duration of 100 frames.
Speed Performance Test on the BeagleBone at a Resolution of 320x240 350000 300000
Time(µs)
250000 200000 150000 100000 50000
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100
0
Frame Number Average: 168349µs
Figure 10-2: Graph showing the time taken to process each image frame in a set of 100 frames. The average time for the test was 168349µs and is shown in orange.
The speed of the algorithm was significantly reduced when run on the BeagleBone, and over shot the expected performance decrease by a factor of three. After further investigation, it was found that this was due to the inability of the BeagleBone to process commands in parallel. This notion is expanded upon further in the Algorithm acceleration analysis section. The frame capture function also suffered a significant speed reduction when the algorithm was run on the BeagleBone, and as a result, the total time taken to capture and process a single frame was approximately 285ms, resulting in a system frame rate of 3.5 FPS. This was a performance reduction of 97% from that obtained by Barry et al. The effect of the reduced frame rate is evident in the speed tests of the Indoor Control tests section x. 10.1.4
Image resolution reduction
The reduced frame rate was unacceptable for high speed collision avoidance. As a means of increasing the frame rate, the frame resolution was reduced from 320x240 pixels to 176x144 pixels. The new resolution necessitated recalibration of the system as mentioned in the Stereo Calibration section. The image below shows the relative size reduction of the image frame at 100% scale.
67
Figure 10-3: Image showing the image resolution reduction in terms of size and pixel count.
Computer performance test at the reduced resolution
10.1.5
The speed performance tests were subsequently repeated at the reduced resolution. The graph below shows the speed performance when the algorithm was run on the computer.
Speed Performance Test on the Computer at a Resolution of 176x144 8000 7000
Time(µs)
6000 5000 4000 3000 2000 1000
1 31 61 91 121 151 181 211 241 271 301 331 361 391 421 451 481 511 541 571 601 631 661 691 721 751 781 811 841 871 901 931 961 991
0
Frame Number Average: 820µs Figure 10-4: Graph showing the time taken to process each image frame in a set of 1000 frames. The average for the test was 820µs and is shown in orange.
The speed performance increased by a factor of 2.47 for the computer platform. Once again, a large standard deviation from the average was observed due to background processing. 10.1.6
BeagleBone performance test at the reduced resolution
The new parameters for the reduced resolution were transferred to the BeagleBone and the speed performance test was repeated. The graph below shows the results of the performance test.
68
Speed Performance Test on the BeagleBone at a Resolution of 176x144 70000 60000
Time(µs)
50000 40000 30000 20000 10000
1 13 25 37 49 61 73 85 97 109 121 133 145 157 169 181 193 205 217 229 241 253 265 277 289 301 313 325 337 349 361 373 385 397
0
Frame Number Average: 28278µs Figure 10-5: Graph showing the time taken to process each image frame in a set of 400 frames. The average time for the test was 28278µs and is shown in orange.
The speed performance increased by a factor of six on the BeagleBone due to the reduction in resolution. Tests conducted at the reduced resolution showed that the algorithm was able to perform at the same level as that observed at the original resolution. The results of the tests are further elaborated on in the next section. The reduced resolution, therefore, resulted in an increase in frame rate for a negligible decrease in detection ability. 10.1.7
Complete speed performance characterisation
A full analysis of the speed of the algorithm is shown below.
Time Taken to Process the Major Functions of the Pushbroom Algorithm
Image Processing: 28934 µs 41% Other: 185 µs 0% Image Capture: 41908 µs 59%
Figure 10-6: Pie chart showing the total time taken to perform sections of the code. The other section consists of all command signalling, data recording, feedback and other general administration.
69
Summing the values shown above gives the total time taken to complete one cycle of the collision avoidance algorithm: 𝑇𝑜𝑡𝑎𝑙 𝑡𝑖𝑚𝑒 = 41 908 + 28 934 + 185 = 71 027 µ𝑠 From this, the number of frames processed per second can be calculated: 𝐹𝑟𝑎𝑚𝑒𝑠 𝑝𝑒𝑟 𝑠𝑒𝑐𝑜𝑛𝑑 =
71 027 1 000 000
≈ 14.1
The Pushbroom algorithm, therefore, processed images at four times that reported for the previous implementation [1]. This increase in frame rate allowed the boat to avoid the obstacle at significantly faster speeds than those recorded for the previous implementation. Notably, the image capture function required nearly 60% of the total time per frame. This observation explained the dramatic increase in performance when the resolution was reduced from 320x240 to 176x144 pixels. This percentage also suggests that future methods of improvement should focus on reducing time taken to capture image frames from the cameras.
10.2
Indoor Control tests
In order to simulate the smooth continuous movement of a boat, the indoor control tests were conducted with the stereo rig attached to the front of a longboard. This method also simulated the low profile of the boat as the cameras were relatively close to the ground. The picture below depicts the testing rig.
Figure 10-7: A longboard was used as the test platform for the indoor control tests. This resulted in smooth continuous motion simulating the motion of the boat.
It must be noted that during the indoor tests, the speed at which the camera rig was moving could not be accurately quantified. This method of testing was justified by the fact that it was simply conducted to give an indication of what to expect during the water tests and to determine testing parameters in a controlled environment. Tests were, therefore, performed in order to determine the number of valid correspondence detections the algorithm could make whilst moving towards the obstacle. 10.2.1
Testing conditions
The tests were performed in a moderate to well-lit room at approximately midday. Whilst an accurate light reading was not necessary for quantifying the results of these tests, sufficient lighting was
70
necessary such that the cameras were not forced to reduce their frame rate to compensate for the bad lighting conditions. The images below depict the testing scene.
Figure 10-8: A panoramic view of the indoor testing scene depicting the test platform and the object being detected
Figure 10-9: A schematic representation of the testing scene showing the distances between the object and the test platform
The obstacles chosen to be detected for the indoor test were a hockey stick and a small bag. The bag was chosen due to its dark tone, which had a strong contrast against the lightly coloured walls, and a means of keeping the hockey stick upright. The hockey stick was chosen due to its excellent detection attributes. Previous tests performed whilst implementing the Pushbroom algorithm, led to the observation that the pattern on the hockey stick was easily detected, and therefore, a suitable object for control testing. The obstacle was placed 220cm away from the starting position of the camera rig and longboard. Once the test began, the longboard was moved a distance of 80cm towards the obstacle. The Pushbroom disparity was set such that detections would occur at 175cm away from the obstacle. This disparity setting was dependent on the image resolution being tested.
71
The indoor tests were performed on each platform at 320x240 and 176x144 resolutions, resulting in four data sets. The test was divided into two categories - obstacle detection and speed tests. During the obstacle detection tests, position was the variable under consideration. The hockey stick and bag were subsequently placed to the left, in front of and to the right of the camera rig. Three iterations of these tests were performed for each position in order to verify consistency. The speed of the camera rig was kept constant during these tests. During the speed tests, the velocity of the camera rig was the variable under investigation. As a result, the obstacle was placed in front of the camera, and the test was performed three times at different speeds. As previously stated, the speed of the longboard could not be quantified outside of the general terms of slow, medium paced and fast. Despite this lack of quantification, the results showed a clear trend, as highlighted in the section below, from which conclusions could be drawn. 10.2.2
Data analysis
A large amount of data was gathered during all the tests performed. The representation of which was, therefore, a difficult task. A summary of the data that was gathered is displayed in the following standardised manner. Firstly, a typical correspondence hit chart is shown for the specific test run. A summary of both the obstacle test and the speed test is then displayed. The obstacle tests display the following information:
10.2.3
The maximum number of false negative hits o This is the number of incorrect hits that were recorded as the stereo rig approached the object The maximum number of true correspondence hits o This value represents the maximum number of true correspondence hits that were recorded during the three iterations of each test The minimum number of maximum correspondence hits o This value represents the minimum number of maximum correspondence hits that were recorded over the course of the three obstacle tests.
Computer tests
The computer ran the algorithm at 30 FPS, as expected, whereby the limiting factor was the rate at which the cameras could capture image frames. The image sequence below shows the view of the camera at different points during the tests.
72
Figure 10-10: showing the camera view at three positions during the 320x240 resolution testing sequence. The images are shown at 100% scale. Frame A corresponds to the starting position for the test, 220cm away from the obstacle, where no hits were detected. Frame B corresponds to the detecting range of the algorithm, 175cm away from the obstacle, where a large number of hits are detected. Frame C corresponds to the finishing position of the test, 135cm away from the obstacle, where residual hits are detected.
The graph below shows the hit chart for a typical test performed by the computer running at 30 FPS at a resolution of 320x240 pixels.
73
Hit Chart: Computer Test at a Resolution of 320x240 20
B
A
C
18
Number of Hits
16 14 12 10
Left Hits
8
Right Hits
6 4 2
1 9 17 25 33 41 49 57 65 73 81 89 97 105 113 121 129 137 145 153 161 169 177 185 193 201 209 217 225
0
Frame Number Figure 10-11: Graph showing the typical hit clustering for tests performed with the computer running the algorithm at 30 FPS at a resolution of 320x240 pixels. The obstacle was placed directly in front of the camera. The letters in sections A, B and C on the graph correspond to the image frames A, B and C in the figure above.
The figure shows that there was a clear distinction between section B, during which the obstacle crosses the disparity boundary, and sections A and C when the obstacle was outside, and subsequently, inside the disparity boundary respectively. The image below shows the view of the camera at the reduced resolution.
Figure 10-12: Image showing a typical frame captured during the reduced resolution tests. The image is shown at 100% scale.
The graph below shows the hit chart for a typical test performed by the computer running at 30 FPS at a resolution of 176x144 pixels.
74
Hit Chart: Computer Test at a Resolution of 176x144 12
Number of hits
10 8 6 Left Hits 4
Right Hits
2
1 8 15 22 29 36 43 50 57 64 71 78 85 92 99 106 113 120 127 134 141 148 155 162 169 176 183 190 197 204
0
Frame Number Figure 10-13: Graph showing the typical hit clustering for tests performed with the computer running the algorithm at 30 FPS at a resolution of 176x144 pixels. The obstacle was placed directly in front of the camera.
As can be seen the hit cluster was very similar in shape to that recorded at a resolution of 320x240 pixels. 10.2.4
BeagleBone tests
Once the computer tests were complete, the algorithm was transferred to the BeagleBone for testing. As mentioned in the Performance tests section, the speed of the algorithm on the BeagleBone was significantly reduced in comparison to the speed at which it ran on the computer. As a result, the data set for the obstacle and speed tests was gathered over a smaller number of frames. At this stage, the BeagleBone was not capable of saving image frames, however, the algorithm was tested at the 320x240 and 176x144 resolutions, as shown in Figure 10-10 and Figure 10-11 in the same manner as the computer tests. The graph below shows the hit chart for a typical test performed by the BeagleBone running at 3.5 FPS at a resolution of 320x240 pixels.
75
Hit Chart: BeagleBone Test at a Resolution of 320x240 14
Number of Hits
12 10 8 6
Left Hits
4
Right Hits
2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Frame Number Figure 10-14: Graph showing the typical hit clustering for tests performed with the BeagleBone running the algorithm at 3.5 FPS at a resolution of 320x240 pixels. The obstacle was placed directly in front of the camera.
The data cluster recorded by the BeagleBone at 320x240 pixels was far more sparse than that recorded by the computer test due to the fact that the frame rate was reduced by a factor of ten. The graph below shows the hit chart for a typical test performed by the computer running at 14 FPS at a resolution of 176x144 pixels.
Hit Chart: BeagleBone Test at a Resolution of 176x144 10 9
Number of Hits
8 7 6 5
Left Hits
4
Right Hits
3 2
1 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
0
Frame Number Figure 10-15: Graph showing the typical hit clustering for tests performed with the BeagleBone running the algorithm at 14 FPS at a resolution of 176x144 pixels. The obstacle was placed directly in front of the camera.
The data cluster for the reduced resolution returned to a similar shape as recorded by the computer tests because of the increased frame rate.
76
Obstacle test summary
10.2.5
The figure below is a summary of the obstacle tests for both the computer and BeagleBone at both the original and reduced image resolutions.
Computer Obstacle Test at a Resolution of 320x240 19
No. of hits
20
16 13
15
13
11
10 10 5
1
1
1
0 Object Left
Object Front
Object Right
Computer Obstacle Test at a Resolution of 320x240 25
21
No. of hits
20
16
15
11
9
10 5
11
7 3
2
2
0 Object Left
Object Front
Object Right
BeagleBone Obstacle Test at a Resolution of 320x240 30
26
No. of hits
25
20
16
15
14
14
12 8
10 5
2
2
1
0 Object Left
Object Front
Object Right
BeagleBone Obstacle Test at a Resolution of 176x144 25
21
No. of hits
20
16
15
11
10 5
9
8
5 2
1
1
0 Object Left
Object Front
Maximum number of false negative hits
Object Right
Maximum number of true correspondence hits
Minimum number of true correspondence hits Figure 10-16: Graphs summarising the obstacle tests performed on the computer and the BeagleBone at both the original and reduced resolutions
77
The number of hits recorded by both platforms when the obstacle was on the right was significantly higher than those recorded for the other positions. This was due to the fact that more light was cast upon the obstacle when it was in this position. It was, therefore, to be concluded that the algorithm’s performance was largely dependent on the lighting conditions of the environment. The maximum number of false negative hits across both platforms and both resolutions was three hits. This showed that the algorithm had excellent disturbance rejection characteristics once the horizontal invariance check had been set up correctly. The least number of valid hits was recorded by the BeagleBone at a resolution of 176x144 where five hits were recorded on the obstacle. The difference between this value and the maximum number of false negatives was two hits. From this, it was concluded that the algorithm had a threshold of two hits between detecting a phantom obstacle and a valid obstacle. Most importantly, the maximum number of false negatives did not exceed the minimum number of true detections as this would have required a more stringent method of identification. 10.2.6
Speed test summary
The figure below is a summary of the speed tests for both the computer and BeagleBone at both the original and reduced image resolutions.
78
Computer Speed Test at a Resolution of 320x240 20
18
No. of hits
15
12
10 5
4
3 1
1
0 Slow
Medium
Fast
Computer Speed Test at a Resolution of 176x144 10
9 8
No. of hits
8 6
5
4 2 2
1
1
0 Slow
Medium
Fast
No. of hits
BeagleBone Speed Test at a Resolution of 320x240 16 14 12 10 8 6 4 2 0
15 13
3
3 1
0 Slow
Medium
Fast
BeagleBone Speed Test at a Resolution of 176x144 No. of hits
25
20
20
20 15 10 5
6
4
2
1
0 Slow
Medium
Maximum number of false negative hits
Fast
Maximum number of true correspondence hits
Figure 10-17: Graphs summarising the speed tests performed on the computer and the BeagleBone at both the original and reduced resolutions
79
The speed tests depict a clear trend that revealed an attribute of the algorithm. The faster the platform was moving, the fewer hits were detected. This was true for all the tests except the computer test at 176x144 resolution during which the fast speed test recorded more hits than the medium pace test. This characteristic was anticipated since a high speed exacerbates the negative effects of the differential frame capture between the left and right cameras. A higher speed also introduces other undesirable characteristics such as motion blur which has an adverse effect on hit detections. This attribute was, therefore, the limiting factor of the algorithm which could only be fully quantified and characterised once the algorithm was implemented on the boat test platform. The BeagleBone speed test at a resolution of 320x240 resulted in only one valid object hit at the fastest speed. This was unacceptable since this value was less than the maximum number of false negative hits recorded during the obstacle tests. This issue was rectified by reducing the resolution of the image frame as previously mentioned, which resulted in six valid object hits at the fastest speed. This increase in performance justified the use of a lower resolution.
10.3
Pool testing
Once the indoor control tests were complete, the algorithm was tested in a 7x4 metre, outdoor pool on the boat testing platform over the course of three days. The tests represented a more true to life collision avoidance situation than the indoor control tests, however, the speed of the vessel was limited in the relatively small body of water. 10.3.1
Pool testing day one
The first day of pool testing was used to gather general information about implementing the algorithm in an outdoor environment in order to complement the information gathered from the indoor tests. Initially, a black flag was used as the obstacle which the boat was required to avoid. It was chosen due to the fact that the indoor tests favoured darkly toned obstacles such as the hockey stick. This, however, was due to the fact that the walls of the indoor tests were lightly toned, which therefore, created a differential contrast with the obstacle. This was not the case for the outdoor tests during which the background had a dark tone. As a result, the algorithm performed poorly and a maximum of four valid hits were recorded on the flag during a test run. The image below shows the flag in a monochromatic image as the on-board cameras perceived it.
80
Figure 10-18: Image showing how the black flag is camouflaged against the dark background
As can be seen, the flag was well camouflaged against the dark background, so much so that it is difficult to identify with the naked eye. This limitation is not unique to the Pushbroom algorithm but rather a common attribute amongst most computer vision applications, and is still an activate area of research [54]. In order to mitigate this limitation, a duel modality sensing implementation is required. This notion is expanded upon in the Recommendations section. In order to rectify the contrast issue, the black flag was substituted by a white bucket as shown in the image below.
Figure 10-19: Image showing the contrast of the bucket against the dark background
81
The bucket was far more easily distinguishable from the background, and therefore, a suitable object for further testing.
Pool testing day two
10.3.2
The second day of testing was again used to gather information from which an automatic avoidance system could be designed. The BeagleBone was programmed to take a picture of the scene during testing when the number of hits rose above a specified threshold. This image was used to validate that the boat was avoiding the correct object. Three tests were performed to gather information. The first test was conducted without a valid obstacle in the way of the boat. During the second test, the object was placed on the right hand side of the pool, and lastly, the object was placed on the left hand side of the pool. Four runs were conducted for each position in order to fully characterise the algorithm’s performance. A summary of the results obtained is shown below.
Pool Testing Day Two Results Summary 12
11 10
No. of hits
10 8 6
6
6 4
3 2
2
1 0
0
0
Object Left
No Object
Maximum number of false negative hits
Object Right
Maximum number of true correspondence hits
Minimum number of true correspondence hits Figure 10-20: Graph showing a summary of the pool test results obtained on the second day of testing. The algorithm was able to identify the object in both the left and right image frames.
The pool tests generated data very similar to that obtained during the indoor tests which validated the observations and conclusions drawn from the initial tests. Similar to the indoor performance, the difference between the maximum number of false negative hits and the minimum number of valid hits for all runs recorded was three hits. The hit threshold for the outdoor tests was, therefore, three hits which was one more than the two hit threshold of the indoor tests. The algorithm also performed excellently during the no object test for which the maximum number of false negative hits was one. From this data, it was concluded that an effective collision avoidance system could be implemented whereby an avoidance threshold was utilised. If the number of hits in the left or right image plane exceeded this threshold, then a corresponding command signal was sent to the rudder in order to avoid the detected obstacle. The full collision avoidance algorithm is explained in the Development of the collision avoidance algorithm section.
82
10.3.3
Pool testing day three
The third day of testing saw the implementation of the automatic collision avoidance system on the boat test platform. The preliminary tests performed were the obstacle tests during which the speed of the boat was limited to 0.625m/s. The avoidance threshold value was also kept constant during these tests at six correspondence hits.
i.
Obstacle tests
The first test was conducted without a valid obstacle in the boat’s path in a similar fashion to the previous tests. The boat was programmed to enter into the collision avoidance state, and was manually directed toward the opposite end of the pool as shown in the image below.
Figure 10-21: Image showing the path of the boat during the no obstacle test. The boat did not divert off course in an attempt to avoid a phantom obstacle and therefore passed the no obstacle test.
The boat successfully traversed the length of the pool without incorrectly attempting to avoid a phantom obstacle. The graph below shows the hit chart for the no obstacle test.
83
10 9 8 7 6 5 4 3 2 1 0
Left Hits Right Hits Avoidance Threshold
1 12 23 34 45 56 67 78 89 100 111 122 133 144 155 166 177 188 199 210 221 232 243 254 265 276 287 298
Number of Hits
Hit Chart: Pool Test with No Obstacle
Frame Number Figure 10-22: Graph showing the hit chart for the no obstacle pool test. The maximum number of false negative hits remained below the avoidance threshold.
As expected from previous tests, the maximum number of false negative hits recorded during the test was two, which was below the avoidance threshold, and therefore, did not result in an avoidance manoeuvre. Once the no obstacle test was complete, the white bucket from the previous tests was suspended in the middle of the pool such that the obstacle tests could be performed. For the first test, the boat was aimed towards the right hand side of the obstacle. The image below shows the path of the boat during the test.
Figure 10-23: Image showing the path of the boat when aimed at the right hand side of the obstacle. The boat automatically avoided the obstacle.
84
The boat successfully avoided the obstacle with no external assistance. The graph below shows the hit chart recorded during the test.
Hit Chart: Pool Test with an Obstacle and Avoidance to the Right 16
Number of Hits
14 12 10 8 Left Hits
6
Right Hits Avoidance Threshold
4 2
1 14 27 40 53 66 79 92 105 118 131 144 157 170 183 196 209 222 235 248 261 274 287 300 313 326 339 352
0
Frame Number Figure 10-24: Graph showing the hit chart for the obstacle avoidance pool test. The object was detected in the left hand plane of the image and the boat turned right accordingly.
The boat successfully detected the bucket in the left hand image plane, and subsequently, turned right to avoid a collision. The image below was taken at the moment the algorithm detected a valid object, and therefore, corresponds to the hit chart shown above.
Figure 10-25: Image showing the frame capture at the moment the object was detected
The frame capture validates the fact that the boat was avoiding the correct obstacle as the test intended. In order to verify that the boat was able to avoid obstacles on both the left and right hand side, the above test was repeated with the boat aimed at the left hand side of the obstacle. The image below shows the path of the boat during the test.
85
Figure 10-26: Image showing the path of the boat when aimed at the left hand side of the obstacle. The boat automatically avoided the obstacle.
As before, the boat successfully avoided the obstacle with no external assistance. The graph below shows the hit chart recorded during the test.
20 18 16 14 12 10 8 6 4 2 0
Left Hits Right Hits Avoidance Threshold
1 13 25 37 49 61 73 85 97 109 121 133 145 157 169 181 193 205 217 229 241 253 265 277 289 301 313 325
Number of hits
Hit Chart: Pool Test with an Obstacle an Avoidance to the Left
Frame Number Figure 10-27: Graph showing the hit chart for the obstacle avoidance pool test. The object was detected in the right hand plane of the image and the boat turned left accordingly.
Once again a large hit spike was recorded which corresponded to a valid obstacle. Since the object was detected in the right hand image plane, the boat turned to the left. The image below was taken at the moment that the algorithm detected a valid object, and therefore, corresponds to the hit chart shown above.
86
Figure 10-28: Image showing the frame capture at the moment the object was detected
Throughout the duration of the obstacle tests, the boat completed nine runs towards the obstacle. It successfully avoided the obstacle eight times and failed once. The failed avoidance run was due to human error whereby the collision avoidance state had not been activated. The boat, therefore, avoided the obstacle with a 100% success rate at a speed of 0.635m/s and an avoidance threshold of six correspondence hits.
ii.
Speed tests
Once the obstacle tests were complete, the speed of the algorithm was tested. This was done in an attempt to characterise the limitations of the algorithm. Over the course of five individual runs, the speed of the boat was increased from 0.625m/s to 1.25m/s. The graph below shows a summary of the data collected during the speed tests.
Speed Test Data Summary 13
14
12
12
12
No. of hits
10
8 7
8 6 4
3
3
3
2 1
2 0 0.75
0.875
1
1.125
1.25
Speed (m/s) Maximum number of false negative hits
Maximum number of true correspondence hits
Figure 10-29: Graph showing a summary of the data collected during the speed tests.
The diminishing trend that was observed with the indoor tests with an increase in speed was not as evident from the speed test above. This can be attributed to the favourable lighting conditions in the outdoor environment. The trend that is suggested from the above graph is that the maximum number of valid hits was not as dependent on speed as it was during the indoor tests, but rather the object itself and the angle of approach of the boat. This theory could not be verified due to the fact that the speed of the boat was limited by the size of the pool and any further increase in speed could have resulted in the boat sustaining damage.
87
The results obtained from the pool tests verified the effectiveness of the Pushbroom algorithm at low speeds, and further testing required a larger body of water.
10.4
Open water testing
Due to the size constraint of the pool used for the preliminary tests, the boat could not be tested beyond a speed of 1.25m/s. A larger body of water was, therefore, required for further testing. This section shows the results obtained during open water testing conducted over the course of two days. 10.4.1
Open water testing day one
The UCT North stop dam was chosen as a suitable location due to its calm waters and isolated nature. The size of the dam was significantly larger than that of the pool which allowed the boat to be tested at greater speeds. The picture below shows the dam and its surrounding vegetation.
Figure 10-30: Image showing the testing location. The size of the dam allowed the boat to be tested at greater speeds. The buoy was held in place by two 20m long ski ropes anchored on either side of the dam.
The indoor tests and pool tests showed that the algorithm was very effective at detecting obstacles at slow speeds. The open water tests were conducted as a means of confirming this characteristic and to test the limits of the algorithm and determine its performance over a range of speeds. The conditions on the day of testing were as follows.
88
Table 10-2: Weather conditions on the first day of testing.
Weather Temperature Humidity Wind
i.
Partly cloudy – 40% cloud cover 20°C 72% Easterly at 3 m/s
Short detection range tests
The first tests that were conducted, utilised the same configuration settings that were used during the pool tests. This corresponded to an object detection range of 175cm. Initially, the boat was driven at slow speeds to ensure that the system was working correctly. After verifying that the algorithm had been configured correctly, the speed of the boat was steadily increased after each run in the same manner as the speed tests performed in the pool. The graph below is a summary of the speed test data obtained.
First Open Water Speed Test using a Short Detection Range
NUmber of Hits
12
11
10
10
8 7
8
7 6
6 4
2
3
2
2
3
2
2 0 0.5
0.75
1
1.25
1.5
1.75
Speed of the boat as a percentage of maximum capability Maximum number of false negative hits
Maximum number of true correspondence hits
Figure 10-31: Graph showing a summary of the dam speed test.
The figure below shows the frames captured at the moment the obstacle was detected by the boat for each corresponding speed.
89
Figure 10-32: Image showing the frame captures at the moment of detection. The frames show the variable nature of the algorithm as no hit pattern is the same as another. The speed ratings for each frame are as follows: A = 0.5m/s B = 0.75m/s C = 1m/s D = 1.25m/s E = 1.5m/s F = 1.75%
The graph above confirmed that the original trend that was observed during the indoor tests did not fully apply to the outdoor tests. The variability of the number of hits can, therefore, be attributed in part to the speed of the boat, the horizontal and vertical movement of the boat as it approaches the obstacle and the variable nature of the algorithm itself due to its thresholding characteristics. The boat was tested over the course of nine individual runs at the original range setting. Seven out of the nine runs resulted in a successful collision avoidance manoeuvre. One of the two failed attempts was due to the fact that the maximum number of true correspondence hits was below avoidance threshold. The hit chart for the failed run is shown below.
90
Hit Chart: Open Water Speed Test Failed Avoidance 7
Number of Hits
6 5 4 3
Left Hits
2
Right Hits
1 0 1 2 3 4 5 6 7 8 9 1011121314151617181920212223242526272829303132333435
Frame Number Figure 10-33: Graph showing the hit chart for the failed collision avoidance run. The maximum number of valid hits was equal to the threshold level and therefore the obstacle was not considered valid.
A pertinent observation was made upon further analysis of the data from the failed run. The maximum number of hits of the left and right image planes did not exceed the avoidance threshold in their own respective capacity. However, combining the hits from both planes in frame would have resulted in a value of eleven stereo hits in total. This value far exceeds the avoidance threshold and the boat would have successfully detected the obstacle if this had been the case. As a result of this observation, the code was adapted to accommodate this detection condition. The second failed avoidance attempt occurred not due to the fact that the boat did not detect the obstacle but rather because it hit the obstacle in its attempt to manoeuvre past it. The algorithm was initially coded such that the boat was commanded to stop the motor when an obstacle was detected. This proved to be a poor method of collision avoidance due to inertial sliding. The boat was travelling at a speed of 1.25m/s and turned from a Northerly heading to an Easterly heading which was into the wind. The combination of speed and wind resulted in the back of the boat sliding into the obstacle. Changes were subsequently made for the second open water test that enabled the boat to maintain its speed and aggressively avoid the obstacle.
ii.
Extended detection range tests
In order to mitigate the effects of inertial sliding, the detection range was increased to 220 cm and the speed tests were repeated. The graph below is a summary of the speed test data obtained with the extended detection range.
91
Dam Test 1: Speed Test using an Extended Detection Range 11
12
10
9
Number of Hits
10
9
8
6
6 4 2
1
1
2
1
1
0 0.75
1.25
1.5
1.75
2
Speed of the boat as a percentage of maximum capability Maximum number of false negative hits
Maximum number of true correspondence hits
Figure 10-34: Graph showing a summary of the dam speed test using an extended detection range. The data did not follow the expected trend that greater speed results in a reduced number of hits.
Notably, the performance of the algorithm was not hindered by the increased detection range. The maximum number of false negative hits was also lower than those observed during the initial speed tests. The figure below shows the frames captured at the moment the obstacle was detected by the boat for each corresponding speed.
Figure 10-35: Image showing the frame captures at the moment of detection. The speed ratings for each frame are as follows: A = 0.75m/s B = 1.25m/s C = 1.5m/s D = 1.75m/s E = 2m/s.
The obstacles in the figure above are visibly further away than those observed in Figure 10-32, therefore, verifying the effect of changing the detection range. As a result of the change in detection
92
range, the boat had more time to avoid the obstacle and thus the sliding effect due to inertia was nullified. The boat was tested over a course of eight individual runs during the second speed test. Of the eight runs, the boat was able to avoid the obstacle seven times. The failed attempt occurred due to a missed detection of the obstacle. Further analysis of the hit chart for the run revealed that the reason for the miss detection was the same as that observed for the first speed test, whereby the image planes were treated separately and not as one unit. As previously mentioned, the algorithm was adjusted thereafter to account for such a situation. The maximum speed of boat, for the first open water test, at which a successful avoidance manoeuvre was performed, was 2m/s. This was a significant improvement from the speed observed during the pool tests, however, the performance of the algorithm had not been fully quantified at this stage. As a result, a second open water test was conducted as discussed in the next section. 10.4.2
Open water testing day two
A second day of open water testing was conducted in order to fully characterise the performance of the boat and to determine the capabilities of the algorithm. The section below shows the results of the testing. The dam used for the first day of open water testing was not well suited to testing due to the low water level and rocky banks. As a result the dam to the South of UCT was used as the testing ground. The picture below shows the dam and the surrounding vegetation.
Figure 10-36: Image showing the test location for the second open water test. The location above was more easily accessible that the previous test location.
The conditions on the day of testing were as follows.
Table 10-3: Weather conditions on the second day of testing.
Weather Temperature Humidity Wind
Partly cloudy – 20% cloud cover 21°C 82% Easterly at 2 m/s
93
After the initial dam test, it was concluded that the system performed better when the extended detection range was utilised. As a result, it was used for all tests thereafter. The avoidance dynamics of the boat were also adapted. Instead of reducing speed in order to avoid an obstacle, the boat maintained its speed and aggressively avoided the obstacle. As a result, the boat did not slide into the obstacle as it had done previously and failed collision avoidance was not due to poor boat manoeuvrability. The boat was tested twelve times at varying speeds. The boat successfully avoided the obstacle eight times and failed to avoid the obstacle four times. The results of the tests are shown below.
No. of Hits
Second Open Water Speed Test 10 9 8 7 6 5 4 3 2 1 0
9 8
8 7 6
6 5
4
4
4
4
4
4
4
5
4
4
4 33
2 1
1
0.75
1.25
2
2
1
1.5
1.75
4 3
1
2
2.25
1
2.5
3 2
1
1
1
2.75
2.75
2.75
1
3
3.25
Speed (m/s) Maximum number of false negative hits
Maximum number of true correspondence hits
Avoidance Threshold Figure 10-37: Summary of the results obtained from the second day of open water testing. Twelve tests were conducted and eight out of the twelve tests resulted in successful avoidance of the obstacle. The failed attempts are depicted by the red avoidance threshold bars which are higher than the maximum number of true correspondence hits for the respective run.
The screenshots taken at the moment the obstacle was detected are shown below.
94
Figure 10-38: Screenshots captured at the moment the object was detected. The images are displayed in increasing speed order as per the figure above. A = 0.75m/s B = 1.25m/s C = 1.5m/s D = 1.75m/s E = 2m/s F = 2.5m/s G = 2.75m/s H = 3m/s.
The results show that the performance of the algorithm decreases with an increase in speed. Despite this, the boat managed to identify and avoid the obstacle at a speed of 3m/s. At speeds greater than 2.5m/s, the boat became difficult to control and aligning its path with the bucket became a significant challenge. Analysis of the data from the failed attempts revealed that the reasons for failure were because of the avoidance threshold being set too high on one occasion and due to a minimal number of true correspondence hits on three occasions. The reasons for why this occurred are investigated below: Speed: There was a clear correlation between the velocity of the boat and the number of successful avoidance manoeuvres. The lag effect is exacerbated by increased speeds, as explained in the algorithm anomaly section, this in turn led to a reduction in true correspondence hits. Boat pitch: As the speed of the boat increased, the boat pitched, whereby the bow of the boat rose out of the water as the stern dug into the water. Due to the fact that the pitch axis was situated toward the stern of the boat, the bow rose a significant amount above the water level. This subsequently had an effect on the field of view of the cameras. The more the boat pitched, the less it could see of the water level. The diagram below depicts this situation graphically.
95
Figure 10-39: Image showing the effect of the boat's pitch with regards to object detection. The pitch of the boat could not formally be quantified due to the lack of an on-board accelerometer or gyroscope. The image is merely an illustration of the effect.
As a result of the boat’s pitch, less of the bucket was in the FOV of the cameras which subsequently reduced the probability of successful object detection and avoidance. This pitching effect was not anticipated before testing was conducted, and was therefore, not accounted or compensated for. However, the fact that pitching only occurred at high speeds was a testament to the performance of the Pushbroom algorithm. Sun glare: Testing was performed in the afternoon and the test runs where conducted from East to West. As a result, it is possible that sun glare may have caused overexposed image frames which would have had a negative impact on obstacle detection. However, these effects could not be proven or quantified. The maximum speed that was recorded for a successful run was 3m/s. The maximum speed at which the algorithm worked without fail was 2.25m/s. In both cases, true high speed collision avoidance was achieved.
96
11. Discussion This section provides inside into the relevance of the results derived from the testing sessions.
11.1
Object detection capability
The Pushbroom algorithm surpassed expectations and performed excellently in the area of object detection. Despite the sparse detection nature of the algorithm the obstacle in question consistently appeared as a well-defined spike on the hit chart of the particular run. The reliability and consistency of the obstacle detection meant that object verification was implemented as a simple thresholding system.
11.2
Speed of the algorithm
The high frame rate of the algorithm was attributed its sparse detection method as well as the use of the algorithm accelerators, as previously mentioned. Although there was a large reduction is frame rate in comparison to that implemented by Barry et al, the algorithm still ran at a fast frame rate with respect to other implementations mentioned in the literature review. This observation is especially prevalent in terms of obstacle detection algorithms implemented on marine test platforms.
11.3
Speed of the test platform
The speeds achieved by the test platform were greater than those recorded by the previous implementation. This investigation therefore resulted in an improvement in that regard and represents development within the field of collision avoidance. 11.3.1
Analysis of the effect of speed on detection capability
Throughout the testing period the detection capability of the system was tested against the speed of the test platform. The indoor and open water tests showed a clear trend that suggested a decrease in stereo hits with an increase in test platform speed. However during the pool tests this trend was not as clear. The reason for this irregularity was because the pool tests were performed at a slow to medium paced speed. As a result, speed was not as much of a determining factor as it was during the indoor and open water tests when the system was being pushed to the limit. From these observations it can be concluded that the trend that was observed during the open water tests was true. This is based on the fact that the open water tests were most true to life and therefore represented a true characterisation of the algorithm’s performance. The significance of the test results of this investigation are discussed in the conclusion.
97
12. Conclusions An investigation into the development of a fast and reliable object detection system for collision avoidance was conducted. The questions asked at the beginning of the investigation are repeated below:
Is stereo vision the fastest and most reliable method available for the implementation of an effective collision avoidance system? Is the combination of feature detection and range estimation algorithms the fastest method of implementing an effective collision avoidance system? What frame rates are achievable with alternative algorithms? What is the maximum speed achievable by the collision avoidance system? Is the system reliable enough for remote, autonomous deployment?
A thorough examination of the available literature suggested that the best available method of fast and reliable object detection was that of stereo vision and the implementation of the Pushbroom algorithm. As a result, the cameras were calibrated using a combination of three calibration programs. The Pushbroom algorithm was then adapted for use on the BeagleBone Black with the stereo rig and the boat test platform. The boat itself was upgraded making it more efficient and lighter than the previous implementation. Tests of the algorithm were performed on land, as a means of control testing, and outdoors to simulate a real world situation. A summary of the results is given below.
12.1
Test results summary
12.1.1
Conclusions from the indoor test
Indoor tests were conducted in order to gather control data from which an algorithm could be developed to effectively avoid an obstacle. The system was able to identify the control obstacle 100% of the time but the number of hits was reduced by the speed of the test platform. 12.1.2
Conclusions from the pool test
Pool tests were conducted in order to test the algorithm at slow speeds on the test platform. Once again, the system identified the obstacle with a 100% success rate but the speed of the boat was limited by the size of the pool. 12.1.3
Conclusions from the open water test
Open water tests were performed in two bodies of water situated on UCT upper campus. The first test confirmed the results obtained during the pool tests. An observation made during the first open water test resulted in a change to the algorithm which was implemented for the second open water test. Finally, high speed collision avoidance was achieved during the second open water test and the test platform was able to successfully avoid an obstacle in its path whilst travelling at a speed of 3m/s. 12.1.4
Conclusions from the performance of the test platform
One of the objectives of this investigation was to increase the performance of the test platform constructed by de Klerk. As a result the control board was redesigned and reconstructed, and more efficient power regulators were incorporated into the design. The battery system was modified such that a single power supply powered the entire boat. As a result of the electrical upgrades made to the
98
boat, the hardware of the system performed without fail, and no problems of any nature were encountered in this regard. The battery life also surpassed expectations and outlasted even the longest testing period of four hours despite the reduction in capacity from the previous test platform. The communication between the computer and boat was seamless, enabling fast and accurate control of the boat throughout the testing period. In response to the questions asked at the beginning of the investigation:
The literature review provided evidence that stereo vision was the fastest and most reliable form of object detection and collision avoidance thereafter. Throughout the duration of the testing phase, the algorithm ran at a frequency of 14 FPS which was four times that of the previous implementation. Due to this frame rate increase, the boat was able to avoid an obstacle at 3m/s, doubling the speed reported by the previous investigation. The system eventually failed to detect and avoid the obstacle in its path at speed in exceed of 2.5m/s. This was due to unforeseen boat dynamics at high speed. Despite this, the system worked reliably at speeds equal to and less than 2.25m/s. Throughout testing, no incorrect avoidance manoeuvres were recorded which is verification of the robust nature of the algorithm. For these reasons, the system could be deployed remotely as an autonomous system.
The Pushbroom algorithm is a masterclass of algorithmic ingenuity. The principles of stereo vision were stripped down to their fundamental essentials and reconstructed into a powerful algorithm providing a fast and effective means of collision avoidance. The similarity of the components and devices used between the previous implementation and the current implementation is testament to the ability and effectiveness of the Pushbroom algorithm and worthy of the title – state-of-the-art. All credit must be given to Barry et al for the development of the algorithm.
12.2
Limitations
Despite the commendable performance of the Pushbroom algorithm, it has its respective limitations. As mentioned in the testing and results section, the Pushbroom algorithm relies on a differential contrast in order to identify an object from the background. For this reason, the algorithm cannot identify objects with the same tone as the environment they are in. Whilst this may be a common limitation of most computer vision algorithms, it is a limitation for the Pushbroom algorithm none the less. Associated to objects with low contrast, a low light level results in false positives which can lead to failed object detection and, subsequently, a collision. As mentioned in the scope, this investigation was focused on the development of a collision avoidance system which works during the day. However, the Pushbroom algorithm suffers under reduced light conditions characteristic of dusk and dawn. The BeagleBone was a hardware limitation which redirected the scope of the investigation from a recreation of the Pushbroom system implemented by Barry et al to a characterisation of the performance of the algorithm when run at a reduced frame rate.
12.3
Significance of research and success of the project
Obstacle detection and avoidance are essential building blocks that form the foundation of an autonomous system. The Pushbroom object detection algorithm has successfully been implemented on
99
both a UAV by Barry et al and a USV in this investigation. The sensing and perception of the vehicles was completely independent and performed on-board the test platforms. This, therefore, represents a contribution to the fields of computer vision and autonomous collision avoidance of unmanned vehicles. As a result, the system developed for this investigation provides a means by which a low cost and reliable system can be installed onto marine vehicles in order to give then the ability to detect and, subsequently, avoid obstacles in their path. From the analysis of the results obtained, the project can be regarded as a success. All the objectives proposed were fulfilled over the course of the investigation, and as a result, the system represented a significant improvement on the knowledge base developed by de Klerk in the previous iteration of this investigation. Future iterations will subsequently be able to build on the success of this project to enable more advanced methods of object detection and automation.
100
13. Recommendations Despite the success of the project, there are areas which can be improved and developed. This section highlights the recommendations for future investigations.
13.1
Larger test platform implementation
This investigation used a small remote controlled boat as the test platform in order to provide a proof of concept for the Pushbroom algorithm. As a means of testing the capabilities of the system, it could be tested on a larger vessel in a marine environment.
13.2
Full Pushbroom algorithm implementation
As previously mentioned, only part of the full Pushbroom algorithm was implemented for this investigation. In order to fully characterise its effectiveness, it would be beneficial to implement the complete system with state information feedback to show past detections.
13.3
Control scheme
Continuing in the same frame of mind from the above recommendation, a full state estimation and dynamic characterisation of the boat could be performed in order to create a minimum energy controller which accurately avoids obstacles with the least amount of effort required.
13.4
Hardware upgrade
1) FPGA implementation: As mentioned in the literature review, FPGA systems have been shown to operate at the same frame rate as that recorded by the Pushbroom algorithm. The difference between the two systems is the amount of information that is processed and the depth information that is subsequently extracted. Since the Pushbroom algorithm has been shown to work on an unmanned surface vehicle, a comparison could be made between the algorithm and an FPGA implementation. 2) ODROID-U2 implementation: Unhindered by the limitations of cost and available resources, the investigation could be conducted with the implementation of the hardware used by Barry et al in an attempt to achieve the same frame rate as the reported implementation, and subsequently, test its substantial capabilities in a marine environment.
13.5
Duel modality system
In order to implement a complete collision avoidance system, a duel modality sensing scheme could be implemented, enabling the system to operate at night and in situations where vision is obscured due to weather and other such factors.
101
14. List of References [1]
N. De Klerk, “Reactive Collision Avoidance and Control System for an Autonomous Sail Boat,” University of Cape Town, 2013.
[2]
Romola and R. C. Anderson, A Short History of the Sailing Ship, 1st ed. Mineola,TX: Dover Publications, 2003.
[3]
N. Nasu and S. Honjo, “Oceanographic Research and Development,” 1st ed., Tokyo: SpringerVerlag, 1993, pp. 35–36.
[4]
J. E. Manley, “Development of the autonomous surface craft ‘ACES,’” IEEE Ocean. 1997, vol. 2, pp. 827–832, 1997.
[5]
J. Manley, A. Marsh, W. Cornforth, and C. Wiseman, “Evolution of the autonomus surface craft AutoCat,” IEEE Ocean., vol. 1, pp. 403–408, 2000.
[6]
J. Curcio, J. Leonard, and A. Patrikalakis, “SCOUT - A low cost autonomous surface platform for research in cooperative autonomy,” Proc. MTS/IEEE Ocean. 2005, pp. 725–729 Vol. 1, 2005.
[7]
G. Lammons, “Naval community pursues new autonomous surface vehicle,” Sea Technol., vol. 46, no. 12, pp. 27–31, 2005.
[8]
R. Stelzer and K. Jafarmadar, “History and Recent Developments in Robotic Sailing,” in Robotic Sailing, A. Schlaefer and O. Blaurock, Eds. Berlin, Heidelberg: Springer, 2011, pp. 0–21.
[9]
R. Stelzer, “Autonomous Sailboat Navigation,” De Montfort University, Leicester, 2012.
[10]
J. Abril, J. Salom, and O. Calvo, “Fuzzy control of a sailboat,” Int. J. Approx. Reason., vol. 16, no. 3– 4, pp. 359–375, 1997.
[11]
G. Kilpin, “Modelling and Design of an Autonomous Sailboat for Ocean Observation,” University of Cape Town, 2014.
[12]
R. A. Brooks, “A Robust Layered Control System For A Mobile Robot - Rodney A Brooks.pdf,” IEEE J. Robot. Autom., vol. 2, no. 1, pp. 14–23, 1986.
[13]
M. I. Skolnik, “An Introduction and Overview of Radar,” in Radar Handbook, 3rd ed., no. 1336, New York: McGraw-Hill, 2008, pp. 1.1–1.24.
[14]
J. Byrne, M. Cosgrove, and R. Mehra, “Stereo based obstacle detection for an unmanned air vehicle,” in Proceedings of IEEE International Conference on Robotics and Automation, 2006, pp. 2830–2835.
[15]
U. Hofmann, A. Rieder, and E. D. Dickmanns, “Radar and vision data fusion for hybrid adaptive cruise control on highways,” Mach. Vis. Appl., vol. 14, pp. 42–49, 2003.
[16]
Z. Hu, F. Lamosa, and K. Uchimura, “A complete U-V-disparity study for stereovision based 3D driving environment analysis,” in Proceedings of International Conference on 3-D Digital Imaging and Modeling, 3DIM, 2005, pp. 204–211.
102
[17]
J. Liadsky, “Introduction to LIDAR,” NPS Lidar Work., pp. 1–41, 2007.
[18]
B. Peasley and S. Birchfield, “Real-time obstacle detection and avoidance in the presence of specular surfaces using an active 3D sensor,” in IEEE Workshop on Robot Vision, WORV 2013, 2013, pp. 197–202.
[19]
A. J. Barry and R. Tedrake, “Pushbroom Stereo for High-Speed Navigation in Cluttered Environments,” in 3rd Workshop on Robots in Clutter: Perception and Interaction, 2014.
[20]
M. W. Achtelik, M. C. Achtelik, S. Weiss, and R. Siegwart, “Onboard IMU and monocular vision based control for MAVs in unknown in-and outdoor environments,” in Robotics and Automation (ICRA), 2011 IEEE International Conference on, 2011, pp. 3056–3063.
[21]
O. Brock and O. Khatib, “High-speed navigation using the global dynamic window approach,” in Proceedings of1999 IEEE International Conference on Robotics and Automation (Cat. No.99CH36288C), vol. 1, pp. 341–346.
[22]
S. Soumare, A. Ohya, and S. Yuta, “Real-Time Obstacle Avoidance by an Autonomous Mobile Robot using an Active Vision Sensor and a Vertically Emitted Laser Slit,” Intell. Auton. Syst. IAS-7, pp. 301–308, 2002.
[23]
R. Hodges, “Introduction to Sonar,” in Underwater Acoustics: Analysis, Design and Performance of Sonar, 1st ed., West Sussex, England: Wiley, 2010, pp. 1–184.
[24]
I. Ulrich and J. Borenstein, “VFH+: reliable obstacle avoidance for fast mobile robots,” in Proceedings of 1998 IEEE International Conference on Robotics and Automation (Cat. No.98CH36146), 1998, vol. 2, pp. 1572–1577.
[25]
Z. Zhang, “Microsoft kinect sensor and its effect,” IEEE Multimed., vol. 19, no. 2, pp. 4–10, 2012.
[26]
A. J. Barry, H. Oleynikova, D. Honegger, M. Pollefeys, and R. Tedrake, “Fast Onboard Stereo Vision for UAVs,” 2015.
[27]
J. Han, L. Shao, D. Xu, and J. Shotton, “Enhanced computer vision with Microsoft Kinect sensor: A review,” IEEE Trans. Cybern., vol. 43, no. 5, pp. 1318–1334, 2013.
[28]
A. Harati-Mokhtari, A. Wall, P. Brooks, and J. Wang, “Automatic identification system (AIS): data reliability and human error implications,” J. Navig., vol. 60, no. 3, pp. 373–389, 2007.
[29]
A. Saxena, J. Schulte, and A. Y. Ng, “Depth Estimation using Monocular and Stereo Cues,” in Proc. 20th Int. Joint Conf. Artificial Intelligence, 2007, pp. 2197–2203.
[30]
A. Saxena, S. Chung, and A. Ng, “Learning depth from single monocular images,” in Proceedings of the 18th Annual Neural Information Processing Systems Conference (NIPS 18), 2005.
[31]
J. Michels, A. Saxena, and A. Y. Ng, “High speed obstacle avoidance using monocular vision and reinforcement learning,” in Proceedings of the 22nd International Conference on Machine Learning, 2005, vol. 3, no. 1, pp. 593–600.
[32]
B. K. Horn and B. G. Schunck, “Determining Optical Flow,” Artif. Intell., vol. 17, no. 1–3, pp. 185– 203, 1981.
[33]
R. J. Radke, Computer Vision for Visual Effects, 1st ed. Cambridge, England: Cambridge University Press, 2012.
103
[34]
J. Kim and Y. Suga, “An omnidirectional vision-based moving obstacle detection in mobile robot,” Int. J. Control. Autom. Syst., vol. 5, no. 6, pp. 663–673, 2007.
[35]
J. Zufferey, A. Beyeler, and D. Floreano, “Near-obstacle flight with small UAVs,” in UAV’ 2008: International Symposium on Unmanned Aerial Vehicles, pp. 1–16.
[36]
A. Beyeler, “Vision-Based Control of Near-Obstacle Flight,” Ecole Polytechnique Federale De Lausanne, 2009.
[37]
R. Labayrade, D. Aubert, and J.-P. Tarel, “Real time obstacle detection in stereovision on non flat road geometry through ‘v-disparity’ representation,” in Intelligent Vehicle Symposium, 2002. IEEE, 2002, vol. 2, pp. 646–651.
[38]
M. Kumano, A. Ohya, and S. Yuta, “Obstacle Avoidance of Autonomous Mobile Robot using Stereo Vision Sensor,” in Proceedings of the 2nd International Symposium on Robotics and Automation, 2000, pp. 497–502.
[39]
C.-H. Chao, B.-Y. Hsueh, M.-Y. Hsiao, S.-H. Tsai, and T.-H. S. Li, “Real-time target tracking and obstacle avoidance for mobile robots using two cameras,” in ICROS-SICE International Joint Conference, 2009, pp. 4347–4352.
[40]
N. Ayache and F. Lustman, “Trinocular stereovision for robotics,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, no. 1. pp. 73–85, 1991.
[41]
D. Honegger, H. Oleynikova, and M. Pollefeys, “Real-time and Low Latency Embedded Computer Vision Hardware Based on a Combination of FPGA and Mobile CPU,” in International Conference on Intelligent Robots and Systems, 2014.
[42]
P. P. Chu, FPGA Prototyping By Verilog Examples: Xilinx Spartan -3 Version, 1st ed. Hoboken, NJ: Wiley, 2008.
[43]
F. Hu and Y. Zhao, “Comparative Research of Matching Algorithms for Stereo Vision,” J. Comput. Inf. Syst., vol. 9, no. 13, pp. 5457–5465, 2013.
[44]
P. Viola and I. Wells, W.M., “Alignment by maximization of mutual information,” in Proceedings of IEEE International Conference on Computer Vision, 1995, vol. 24, no. 2, pp. 1–29.
[45]
R. Zabih and J. Woodfill, “Non-parametric Local Transforms for Computing Visual Correspondence,” in In Proceedings of European Conference on Computer Vision, 1994, pp. 151– 158.
[46]
G. Bradski and A. Kaehler, “Projection and 3D Vision,” in Learning OpenCV, 1st ed., M. Loukides, Ed. Sebastopol, CA 95472.: O’Reilly Media Inc., 2008, pp. 421–474.
[47]
M. Peris, “OPENCV: STEREO CAMERA CALIBRATION,” Martin Peris’ Blog, 2011. [Online]. Available: http://blog.martinperis.com/2011/01/opencv-stereo-camera-calibration.html. [Accessed: 04-Sep-2015].
[48]
A. Richardson, J. Strom, and E. Olson, “AprilCal: Assisted and repeatable camera calibration,” IEEE Int. Conf. Intell. Robot. Syst., pp. 1814–1821, 2013.
[49]
A. J. Barry, “Collision Avoidance Thesis.” [Online]. Available: Email:
[email protected]. [Accessed: 23-Jul-2015].
104
[50]
S. Shah, “Real-time Image Processing on Low Cost Embedded Computers,” University of California at Berkeley, California, 2014.
[51]
ARM The Architecture for the Digital World, “Neon,” Technologies, 2014. [Online]. Available: http://www.arm.com/products/processors/technologies/neon.php. [Accessed: 08-Oct-2015].
[52]
M. Quintero, “Install OpenCV on Ubuntu or Debian,” Manuel Ignacio Lopez Quintero blog, 2014. [Online]. Available: http://milq.github.io/install-opencv-ubuntu-debian/. [Accessed: 06-Sep2015].
[53]
H. J. Zhang, “Application Note 140: Basic Concepts of Linear Regulator and Switching Mode Power Supplies,” Linear Technology, 2013. [Online]. Available: http://cds.linear.com/docs/en/application-note/AN140fa.pdf. [Accessed: 21-Aug-2015].
[54]
C. McCarthy, J. G. Walker, P. Lieby, A. Scott, and N. Barnes, “Mobility and low contrast trip hazard avoidance using augmented depth,” J. Neural Eng., vol. 12, no. 1, p. 016003, Feb. 2015.
105
15. Appendices 15.1
Raw data
The raw unprocessed data from all testing sessions are presented in excel files on the CD attached to the back of this document.
15.2
Arduino code
The code used to control the boat via the Arduino is shown below. //******************************************************************** //*EEE4022 - Undergraduate Thesis //* Collision avoidance code for the Arduino MEGA //*==================================================================* //* WRITTEN BY: Nathan Pilbrough //* DATE CREATED: 11/09/2015 //* MODIFIED: 21/10/2015 //*==================================================================* //* PROGRAMMED IN: Arduino 1.0.5 IDE //* DEV. BOARD: Arduino Mega //*==================================================================* //* DESCRIPTION: A simple program for basic control the boat test //* test platform. The Arduino establishes communication with the //* BeagleBone Black, commands are then sent from the BeagleBone //* to the Arduino if a collision is imminent. The command tells the //* Arduino to turn the boat either left or right depending on the //* position of the obstacle. //******************************************************************** #include
//include the necessary header files
Servo myservo; // initialise program variables char inByte = 0; // incoming serial byte int servoRudBuf = 90; int servoRud = 90; int motorBuf = 0; int motor = 0; long int safety = 0; void setup() { // start serial port at 9600 bps: Serial.begin(9600); Serial2.begin(9600); while (!Serial) { ; // wait for serial port to connect. Needed for Leonardo only } myservo.attach(9); // attaches the servo on pin 9 to the servo object myservo.write(servoRud); // sets the servo position according to the scaled value
106
pinMode(13, OUTPUT); analogWrite(13, motor); }
establishContact(); // send a byte to establish contact until receiver responds
void loop() { // if a valid byte is available, read it: if (Serial2.available() > 0) { // get incoming byte: inByte = Serial2.read(); switch(inByte){ case 'q': //straighten rudder servoRudBuf = 90; servoRud = 90; myservo.write(90); break; case 'e': //kill engine motorBuf = 0; motor = 0; break; case 'a': //turn the boat left servoRudBuf+=15; break; case 'd': servoRudBuf-=15; //turn the boat right break; case 'w': motorBuf+=10; //increase speed break; case 's': motorBuf-=10; //decrease speed break; case 'c': servoRudBuf = 90; //straighten rudder break; case 'r': //hard turn right for collision avoidance servoRudBuf = 40; if(motor > 20){ motorBuf = 20; } break; case 'l': //hard turn left for collision avoidance servoRudBuf = 140; if(motor > 20){ motorBuf = 20; } break; } // this is the limit of the servo range in its current setup // write values to the servo motor if ( 40
