CLOUDSFM: 3D RECONSTRUCTION USING STRUCTURE FROM MOTION WEB SERVICE
POSTGRADUATE THESIS
By: Dany Puguh Laksono 13/353080/PTK/08949
Submitted to: GEOMATICS ENGINEERING STUDY PROGRAM POSTGRADUATE PROGRAM FACULTY OF ENGINEERING GADJAH MADA UNIVERSITY YOGYAKARTA 2016
CLOUDSFM: REKONSTRUKSI 3D DENGAN LAYANAN WEB STRUCTURE FROM MOTION
Tesis Untuk memenuhi sebagai persyaratan Mencapai derajat Sarjana S-2
Program Studi Teknik Geomatika Kelompok Bidang Ilmu Teknik
Diajukan oleh: Dany Puguh Laksono 13/353080/PTK/08949
Kepada: PROGRAM PASCASARJANA FAKULTAS TEKNIK UNIVERSITAS GADJAH MADA YOGYAKARTA 2016
ii
CLOUDSFM: 3D RECONSTRUCTION USING STRUCTURE FROM MOTION WEB SERVICE
POSTGRADUATE THESIS submitted as a partial fulfillment of the requirements for obtaining the degree of Magister of Engineering in Geomatics Engineering Study Program
By: Dany Puguh Laksono 13/353080/PTK/08949
Submitted to: GEOMATICS ENGINEERING STUDY PROGRAM POSTGRADUATE PROGRAM FACULTY OF ENGINEERING GADJAH MADA UNIVERSITY YOGYAKARTA 2016
iii
“Steter tropfen höhlt den stein”
For Allah My Parents Rindi and Syafiq
vi
ACKNOWLEDGEMENT First and above all, I would like to express my sincere gratitude to the Almighty Allah Tabaroka wa Ta’ala for His blessing and perseverance upon me, with which I could finally submit this research titled, “CloudSfM: 3D Reconstruction using Structure from Motion Web Service”. Also, I would like to send shalawaat to Rasulullah Muhammad shallallaahu ‘alaihi wa sallam for guiding me and my family to the right path, “Lailahaa kanahaari haa. Laa yaziighu ‘anhaa illa haalik”. I would like to extend my gratitude to lecturers and colleagues in Department of Geodesy, Geomatics Engineering Study Program, Faculty of Engineering, Gadjah Mada University, for the support and help during the thesis work. Specifically, I would like to express my gratefulness to: 1. Ir. Nurrohmat Widjajanti, MT., Ph.D. and Heri Sutanta, ST., M.Sc., Ph.D., the Head and Secretary of Geomatics Engineering Study Program for their continuous support towards the students 2. Ir. Djurdjani, Ph.D and Dr. Harintaka, my thesis advisors, for their willingness to take me on as a research student and to develop a thesis topic that was well-suited for my interests and background. 3. Committee members, Trias Aditya K. M., ST., M.Sc., Ph.D and Dr. Eng. Purnama B. S., ST. M.App. Sc, for their insightful comments and assistance towards my thesis. 4. All academic and supporting Staffs in Department of Geodesy, Gadjah Mada University where I spend most of the time during the thesis work. 5. Friends and Classmates in Geomatics Engineering Study Program for their company during the tough times in the pursuit of academic achievements. 6. Colleagues in Center for Spatial Data Infrastructure Developments (PPIDS-UGM) and all my friends whom I cannot mention by name in this very short list. I owe debts of gratitude for their loving supports and well-wishes during the past years in my life. This thesis would not have been possible without continuous support of my parents, the love and dedication of my wife Rindi and the smile of my little boy Syafiq. Without their love, affection and encouragement towards me, this work and most other beautiful things would not have been possible. With the ever improving technologies, especially in the field of Photogrammetry and Computer Vision, I realized that this thesis is imperfect by all means. However, I wish that this work could provide some contribution to science and humanity.
Yogyakarta, January 25, 2016
Dany Puguh Laksono
vii
ABSTRACT The needs for 3D geospatial data are growing rapidly as indicated by the emergence of new instruments and algorithms for 3D geospatial data acquisition. However, most of currently available instruments for 3D geospatial data acquisition are limited by its operational cost and requirements of technical expertise. An alternative is to utilize image-based modeling algorithm to obtain 3D data with lower operational cost and requirements. Structure from Motion (SfM) is an algorithm to obtain 3D geospatial data capable of producing point-cloud reconstruction and camera parameters from overlapping images. With the advancement of computation and internet technologies, it is possible to develop an accessible online 3D reconstruction services based on SfM algorithm. This thesis addresses the use of open source software to develop a cloud-based SfM reconstruction service, namely CloudSfM. Development of CloudSfM can be categorized into two groups: the first is the development of SfM reconstruction pipelines using open source SfM libraries, and the second is development of the web framework. Open source libraries implementing Global SfM algorithm (i.e. OpenMVG, MVE, FSSR and MVS-T) were employed in the development of SfM pipelines to produce 3D textured models, while MEAN frameworks, consist of MongoDB, ExpressJS, AngularJS and NodeJS, were used in the development of Single Page Application (SPA) using MVC (Multi-View Controller) model. Vagrant virtualization were used to produce CloudSfM box to serve cloud-based service for SfM reconstruction. The result of this thesis is a cloud-based SfM reconstruction service using Global SfM algorithm. Three datasets were compared to the result of 3D reconstruction from CloudSfM and commercial software (i.e. Agisoft Photoscan), as well as with the ground truth measurement. The test showed that the deliverables of CloudSfM does not have significant differences with the outputs from commercial software, as indicated by the RMS value for naga dataset (0.019818 m) and Tree Dataset (0.350701 m). The result of ground truth measurement resulted in high RMS value (6.9182 m) due to homogeneity of the dataset. Souce code of CloudSfM is published in a Github repository, while the Vagrant box resulted from this thesis can be distributed to other server to perform SfM reconstruction service. Utilization of open source software means that the platform can be extended and used for future researches.
Keywords 3D reconstruction, Structure from Motion, Cloud service
viii
INTISARI Kebutuhan atas data geospasial 3D tumbuh semakin pesat sebagaimana diindikasikan oleh munculnya berbagai instrumentasi dan algoritma baru untuk akuisisi data tiga dimensi. Meskipun demikian, kebanyakan instrumentasi akuisisi data yang tersedia saat ini memiliki berbagai batasan seperti biaya operasional serta persyaratan teknis yang tinggi. Pemodelan 3D berbasis foto dapat menjadi alternative untuk memperoleh data 3D dengan biaya operasional dan persyaratan operasi yang lebih rendah. Structure from Motion (SfM) merupakan algoritma untuk akuisisi data geospasial 3D yang dapat digunakan untuk memproduksi rekonstruksi point-cloud serta parameter kamera dari serangkaian foto yang bertampalan. Dengan perkembangan teknologi jaringan yang semakin pesat, adalah mungkin untuk memanfaatkan algoritma SfM untuk membangun layanan rekonstruksi 3D yang dapat diakses secara online. Tesis ini menjelaskan mengenai penggunaan perangkat lunak open source untuk membangun suatu layanan rekonstruksi SfM berbasis cloud dengan nama CloudSfM. Pembangunan CloudSfM dapat dikategorikan menjadi dua bagian besar: pembangunan alur rekonstruksi SfM dengan library SfM yang tersedia, serta pembangunan platform kerangka jaringan. Library open source yang mengimplementasikan algoritma Global SfM (yaitu OpenMVG, MVE, FSSR and MVS-T) digunakan dalam pembangunan alur SfM untuk membentuk model 3D, sedangkan MEAN framework yang terdiri dari MongoDB, ExpressJS, AngularJS dan NodeJS digunakan untuk membangun Single Page Application (SPA) dengan menggunakan model Multi-View Controller (MVC). Mesin virtual Vagrant digunakan untuk membentuk paket CloudSfM untuk layanan rekonstruksi SfM berbasis awan (cloud-based service). Hasil dari penelitian ini adalah sebuah layanan berbasis cloud yang mampu melakukan rekonstruksi SfM dengan menggunakan algoritma Global SfM. Tiga set data digunakan untuk menguji hasil dari CloudSfM dibandingkan dengan hasil dari perangkat lunak komersial (Agisoft Photoscan) serta dari ukuran lapangan. Hasil uji menunjukkan bahwa keluaran CloudSfM tidak memiliki perbedaan yang signifikan dengan keluaran yang dihasilkan oleh perangkat lunak komersial, sebagaimana ditunjukkan oleh kecilnya nilai RMS dari set data Naga (0.019818 m) dan Pohon (0.350701 m). Sebaliknya, hasil uji terhadap data lapangan menunjukkan nilai RMS yang tinggi (6.9182 m), yang disebabkan oleh adanya homogenitas set data yang digunakan. Kode CloudSfM telah dipublikasi pada repository Github, sedangkan paket box Vagrant yang dihasilkan dapat dipasang pada server lain untuk digunakan dalam rekonstruksi SfM. Penggunaan perangkat lunak open source pada penelitian ini memungkinkan platform yang dibentuk untuk dapat dimodifikasi dan digunakan dalam berbagai penelitian mendatang. Kata kunci Rekonstruksi 3D, Structure from Motion, Cloud service
ix
TABLE OF CONTENTS
Declaration ................................................................................................................... v Acknowledgement ..................................................................................................... vii Abstract ..................................................................................................................... viii Intisari ......................................................................................................................... ix Table of Contents ......................................................................................................... x List of Figures ............................................................................................................ xii List of Tables ............................................................................................................ xiv List of Acronyms ....................................................................................................... xv I. INTRODUCTION .................................................................................................... 1 I.1. Background ........................................................................................................ 1 I.2. Problems Statement ........................................................................................... 4 I.3. Research Objectives ........................................................................................... 5 I.4. Research Questions ............................................................................................ 5 I.5. Expected Outputs ............................................................................................... 5 I.6. Literature Review .............................................................................................. 6 I.7. Theoretical background ................................................................................... 10 Structure from Motion............................................................................... 10 Online Web Service Framework ............................................................... 31 II. METHODS AND IMPLEMENTATION ............................................................. 43 II.1. Method of research ......................................................................................... 43 Research Materials ................................................................................... 43 Research Tools ......................................................................................... 44 II.2. Implementation of Method ............................................................................. 47 Dataset Acquisition and Preparation ........................................................ 50 Web Framework Programming ............................................................... 55 Installation of SfM libraries ..................................................................... 75 SfM Pipeline Programming and Processing ............................................ 78 Application Deployment .......................................................................... 85 Evaluation of CloudSfM 3D Models ....................................................... 87 x
III. RESULT AND DISCUSSION ............................................................................ 91 III.1. Results ........................................................................................................... 91 SfM Pipeline Using Open Source Libraries ........................................... 91 The CloudSfM Webservice .................................................................... 95 Evaluation of SfM reconstruction ........................................................ 102 III.2. Discussion ................................................................................................... 111 IV. CONCLUSIONS AND SUGGESTIONS ......................................................... 114 IV.1. Conclusions ................................................................................................. 114 IV.2. Suggestions ................................................................................................. 115 V. REFERENCES ................................................................................................... 116 VI. APPENDICES ................................................................................................... 122 VI.1. CloudSfM Source Code .............................................................................. 122 VI.2. SfM_GlobalPipeline.py .............................................................................. 132 VI.3. OpenMVG Install Script (openmvg_install.sh) .......................................... 136 VI.4. Report of SfM Reconstruction .................................................................... 139 VI.5. Sample Terminal Output of a CloudSfM project ........................................ 160
xi
LIST OF FIGURES Figure I-1: Screenshots of some SfM webservices: CMP SfM Webservice (a) and Microsoft’s Photosynth (b) ...........................................................................................7 Figure I-2: Screenshots from Autodesk 123D Catch Reconstruction...........................8 Figure I-3: Methods of 3D Acquisition using Non-Contact Waves (Remondino and El-Hakim, 2006) .........................................................................................................12 Figure I-4: A typical setup of SfM scene (from (Westoby et al., 2012) .....................13 Figure I-5: Above: Keypoints and keypoints descriptor. Below: Keypoints matching using SIFT algorithm (from Brahmbhatt (2013)) .......................................................14 Figure I-6: Geometry of a pinhole camera (Hartley and Zisserman, 2003) ...............15 Figure I-7: Epipolar Geometry (Jebara et al., 1999), .................................................19 Figure I-8: Typical scene of Structure from Motion problem (Thormählen et al., 2010) ...........................................................................................................................20 Figure I-9: A typical Structure from Motion pipeline ................................................23 Figure I-10: Some deliverables from a SfM pipeline .................................................26 Figure I-11: An example of Visibility Graph and Pairwise Matrix (from (Moussa, 2014)) ..........................................................................................................................29 Figure I-12: Cloud-to-cloud comparison using ICP on CloudCompare (Thoeni et al., 2014) ...........................................................................................................................30 Figure I-13: Applications of Cloud Computing (Chandrasekaran, 2014) ..................32 Figure I-14: End-to-end Javascript solution for SPA .................................................34 Figure I-15: Architecture of MVC Communications (Haviv, 2014) ..........................35 Figure I-16: MEAN framework, Combinations of logos from its components (Bretz and Ihrig, 2014)...........................................................................................................36 Figure I-17: Comparison between LAMP and MEAN Architecture ..........................38 Figure I-18: Local, Centralized and Distributed VCS (Chacon, 2009) ......................41 Figure I-19: A Github repository of QGIS (https://github.com/qgis/) ........................42 Figure II-1: General Flowchart of Thesis Development .............................................50 Figure II-2: Excerpt from Naga Dataset .....................................................................52 Figure II-3: Excerpt from Tree Dataset ......................................................................52 Figure II-4: Excerpt from UGM Dataset ....................................................................53 Figure II-5: Diagrams of data acquisition at UGM Central Office Building..............53 Figure II-6: Flowchart of CloudSfM Architectural Design ........................................56 Figure II-7: The Yeoman Workflow (http://yeoman.io) .............................................59 Figure II-8: Scaffoldings of CloudSfM web application ............................................60 Figure II-9: Implementation of MVC framework for image uploading in CloudSfM....................................................................................................................63 Figure II-10: Flowchart of Manage Project Route......................................................70 Figure II-11: Structure of Server-side SfM Processing Libraries ...............................75 Figure II-12: Parameters of OpenMVG_main_ImageListing ....................................79 Figure II-13: Samsung Galaxy Grand 2 Camera Specification ..................................80 Figure II-14: Parameters of OpenMVG_main_ComputeFeatures ..............................81 Figure II-15: Parameters of OpenMVG_main_ComputeMatches..............................82 Figure II-16: Grunt serve command running in Linux Terminal................................86 Figure II-17: Registration of Naga model to reference data .......................................87
xii
Figure II-18: Iterative Closest Point Algorithm in CloudCompare ............................88 Figure II-19: Registered model and reference point cloud .........................................89 Figure II-20: Registration of point cloud to ground truth data ...................................90 Figure II-21: Pick point for ground truth evaluation ..................................................90 Figure III-1: Diagram of SfM Pipeline Constructed Using Python Script .................93 Figure III-2: CloudSfM Running SfM Pipeline in the Background ...........................94 Figure III-3: Content of zip file Containing 3D textured model from served in CloudSfM....................................................................................................................95 Figure III-4: Github Repository of CloudSfM ............................................................96 Figure III-5: CloudSfM Landing page ........................................................................97 Figure III-6: Authorization page: register and login page ..........................................98 Figure III-7: Dashboard, or main menu of CloudSfM ................................................98 Figure III-8: User Interface to create a new project ....................................................99 Figure III-9: Customized SfM settings in CloudSfM ...............................................100 Figure III-10: Options to review images before uploading ......................................100 Figure III-11: Project Manager .................................................................................101 Figure III-12: Final 3D Models of the datasets obtained from CloudSfM Projects .103 Figure III-13: Residuals histogram of the three datasets ..........................................105 Figure III-14: Camera Pose Graph of Naga Dataset .................................................106 Figure III-15: Camera Pose Graph of Tree Dataset ..................................................107 Figure III-16: Camera Pose Graph of UGM Building Dataset .................................107 Figure III-17: Cloud-to-cloud evaluation on Naga Dataset ......................................108 Figure III-18: Cloud-to-cloud evaluation on Tree Dataset .......................................109 Figure III-19: Possible error due to inaccuracies of the 3D models in point #19 and #21 ............................................................................................................................112
xiii
LIST OF TABLES
Table I-1: Overview of SfM software...........................................................................9 Table II-1: Datasets and their respective testing purpose ...........................................44 Table II-2: Technical Specification of Samsung Galaxy Grand 2 Duos (www.samsung.com) ..................................................................................................45 Table II-3: Measured sample points in UGM dataset obtained using TS* .................54 Table II-4: Description of server Request-Response in CloudSfM ............................72 Table III-1: RMS Error of Global SfM reconstruction on each dataset ...................104 Table III-2: Result of Cloud-to-Cloud Comparison .................................................110 Table III-3: Ground Truth Evaluation (units are meter, scaled, not georeferenced) 110
xiv
LIST OF ACRONYMS
FSSR
Floating-scale Surface Reconstruction
ICP
Iterative Closest Point
MEAN
MongoDB, ExpressJS, AngularJS, NodeJS
MVC
Multi-View Controller
MVE
Multiview Environment
MVS
Multiview Stereo
MVS-T
Multiview-Stereo Texturing
OpenMVG
Open Multiview Geometry
RMS
Root Mean Square
SaaS
Software as Service
SfM
Structure from Motion
SIFT
Scale-invariant Feature Transform
SPA
Single-page Application
VCS
Version Control System
xv
I. INTRODUCTION
I.1. Background Demands on three dimensional models as representation of real world are growing rapidly for decades. As human need more space for their activities, a detailed and accurate three dimensional representation of their surroundings is required more than ever. As outlined by Stoter (2000), as well as Elwannas and Emirates (2011), the needs for 3D reconstruction of real world are an important aspect in 3D GIS, cadaster, urban planning, assets inventories, disaster management, and many other fields related to geospatial engineering. Even more, the lack of 3D spatial data could lead to conflict on space utilization (Hassan et al., (2006); Shoshani et al., (2004)). All these supporting factors lead to progresses in 3D data acquisition, processing pipeline and visualization. Some methods are employed to develop three dimensional representation of real world. For example, in order to obtain a 3D representation of a building users have options to acquire the data using various methods, such as traditional land surveying using Total Station (TS), Airborne Laser Scanner (ALS) and Terrestrial Laser Scanner (TLS). While providing users with various options to generate 3D representation according to their needs, each of the method have limitations at some degrees, such as costly instrument and limited operations (Grussenmeyer et al. (2008); Klette and Reulke (2005)). In order to generate 3D representation of a building, traditional land surveying using Total Station is one of the available options. By using Total Station, users will be able to measure the accurate dimension of a building, as well as generate 3D representation from surveyed data. This method is great to obtain accurate, geo-referenced data, but failed to provide detailed picture of building’s façade and its texture. In order to obtain detailed façade of buildings, the users should be able to obtain more points and detailed measurement. Using Total station
1
for non-topographic purposes is quite difficult to obtain complete details of regularly-shaped object such as buildings (Fröhlich and Mettenleiter, 2004). Another data acquisition method to generate 3D model of buildings is Airborne Laser Scanner (ALS). Some notable researches (such as Schwalbe et al. (2005), Sohn and Dowman (2007), and Dorninger and Pfeifer (2008)) suggest that 3D model of buildings can be reconstructed from Airborne Laser Scanner, combined with additional data from ground measurements. However, this method is severely limited by its high operational cost and complicated processing pipelines, and thus is only suitable to be implemented at a large scale projects (Dorninger and Pfeifer, 2008). A more efficient method is the use of Terrestrial Laser Scanner (TLS) to generate detailed, accurate 3D representation of a building (Arayici (2007), Fröhlich and Mettenleiter (2004)). Points cloud data from TLS survey can be extracted to obtain basic structure of buildings (Vosselman et al., 2004). Haala and Kada (2010) mentioned that TLS survey could be used to obtain building facades and to develop 3D model of a building. However, this method possess some limitations. For example, the placement of its instrument is a critical point in obtaining detailed result of surveyed objects. In this context, it is hard to obtain point cloud from TLS data in obstructed environment, especially when users cannot find ideal place for TLS instrument (Grussenmeyer et al., 2008). Therefore, surveying and processing 3D model from TLS for a single building is inefficient, costly and laborious workflow, as demonstrated in Andrews et al. (2013) and Golparvar-Fard et al. (2011). The lack of efficient, low-cost and accurate 3D data acquisition method is a bottleneck in development of 3D building models (Klette and Reulke, 2005). Structure from Motion (SfM) is an effective, albeit low-cost method to fill the gap in 3D data acquisition (Westoby et al., 2012). SfM has been identified as an emerging method to obtain 3D data from multiple images, and was born as branch of both computer vision and photogrammetry (Barazzetti and Scaioni (2009) and 2
Förstner (2002). By understanding geometric relationship between camera from photogrammetric point of view, as well as image analysis techniques provided by stereo vision analysis, SfM pipeline can be used to obtain metric, point-cloud data from real world objects. Compared to other means of 3D data acquisition methods, SfM is a promising method with relatively low-cost operational and processing pipelines (Westoby et al., 2012). SfM is one of the most popular and actively researched problem in Computer Vision due to its flexible implementation, academically challenging algorithm and commercial prospects (Aanaes, 2003). Technological advances of hardware and software in the last decades pave the way to a more robust, rapid implementation of various algorithm to solve SfM problems. Commercial software, as well as its open source counterpart, have been developed to implement various method and algorithm in SfM. A comparison on some of these software (i.e. Bundler, Bundler+ PMVS, Project Photofly and Arc3D Webservice) is shown in Wang (2011). Free or open source desktop software such as Bundler, VisualSfM, PPT (Python Photogrammetry Toolbox), and OpenMVG provide lowcost solution to obtain Structure and Motion from images. Overview of some of these open source software can be found in Bartoš et al. (2014). SfM also found its way to internet and mobile technologies. Although still in early stage of research, the implementation of mobile and online SfM have found to be very promising. A mobile application for SfM have been developed by Tanskanen et al. (2013). Online implementation of SfM such as Microsoft Photosynth (Pomaska, 2009) have been around for some time now. Another online implementation of SfM is CMP-SfM online Web service (Jancosek and Pajdla, 2011). However, the currently available online SfM services have lots of limitations, such as commercial license, limited service, black-box and outdated algorithm, and so on (Remondino et al., 2012). For example, the widely used algorithm for SfM reconstruction is sequential algorithm which reconstruct the camera pose sequentially. Using state-of-the-art algorithm such as Global SfM
3
reconstruction as shown in Moulon et al. (2013a), it is possible to perform SfM reconstruction with more robust, faster calculation and with better result. While open source desktop SfM software is growing in number, an online SfM service which employs open source software is yet to be researched. Moreover, the utilization of open source software usually needs some degree of programming expertise, which is not suitable for practical purposes. This thesis addresses the use of open source software to develop an online platform for SfMbased image processing to produce 3D reconstruction of real-world objects such as buildings or monuments. The whole system is built as a virtualization machine, which make it reproducible and can be continuously developed and deployed to various servers. An evaluation of the result is provided to demonstrate the SfM web service developed in this thesis compared to other commercial software and ground truth data.
I.2. Problems Statement With various options to generate 3D reconstruction of real world objects, users are still faced with technical as well as economical drawback of the existing data acquisition methods, including expensive cost of instrumentation and software, steep learning curve of operation and data processing as well as limited flexibility. On the other hand, recent advances in computer science opens up new possibilities for extraction of 3D structures from images, i.e. by using method called Structure from Motion (SfM). However, the pipeline for performing SfM reconstruction requires some degree of programming knowledge on state-of-the-art algorithm, while commercially available software often use outdated SfM algorithm and are not suitable for research purposes. Therefore, a 3D reconstruction system which is low-cost, easy-to-use and implementing state-of-the-art algorithms needs to be developed in order to fill the gaps as stated above.
4
I.3. Research Objectives Main objective of this research is to implement Structure from Motion approach of obtaining 3D structure from images as an online web service using open source software. This objective can be broken down into specific objectives as follow: a. To perform 3D reconstruction of different datasets using the SfM pipeline implemented from various SfM open source libraries. b. To develop an accessible, easy-to-use and inexpensive online SfM web service using cloud-computing technologies. c. To evaluate the geometrical quality of point-cloud resulted from the SfM pipeline developed in this thesis against commercial software and other ground truth measurement.
I.4. Research Questions This research was conducted to answer the following questions: a. How to combine various open source libraries to develop a pipeline capable of producing 3D models using SfM algorithm? b. How to implement SfM algorithm as web service using open source software? c. How are the results of the proposed system compared to other software and ground truth data, specifically in geometric quality against commercial software and ground-truth measurement?
I.5. Expected Outputs The expected output is an online SfM web service which is capable of solving SfM problem and produce 3D reconstruction of real world objects on-the-fly. The system is also reproducible, which means that it was built using open source software and can be deployed to different server machine when necessary. The 5
open source nature of the system provide the possibilities for further research and development. The utilization of Global Structure from Motion algorithm were expected to produce robust 3D reconstruction compared to other (commercial) software implementing SfM algorithm, as well as ground truth measurement using Total Station. I.6. Literature Review Researcher as early as Ullman (1979) suggested an algorithm that can be used to interpret three dimensional object from a single moving camera. Further researches such as Pollefeys et al., (1998) and Jebara et al. (1999) elaborate the applicability of SfM algorithm for three dimensional reconstruction of real world object from sequence of images. The efficiency of SfM algorithm is discussed in Oliensis (2000) and Szeliski and Kang (1997), while optimization of feature detection algorithm and the use of non-metric camera to produce metric 3D representation can be found in Pollefeys et al. (1999). SfM, as well as another algorithm in Machine Vision are discussed in depth in Hartley and Zisserman, (2003) and Szeliski (2010). Recent developments in Structure from Motion revolve around the utilization of technological advances to produce 3D representation Wei et al. (2013). Another development is concerned on improvement of SfM algorithm and processing pipeline to develop a more accurate 3D model (Quan and Wu (2013); Klingner et al. (2013)). Free to commercial software which implement SfM algorithm are in active development. Some notable software that produce 3D representation using SfM principle are Bundler, VisualSFM, Autodesk 123D Catch, Agisoft Photoscan, etc. Research on comparison of the aforementioned softwares can be found in Koutsoudis et al. (2014) and Remondino et al. (2012). The rising popularity of handheld device enables more possibilities on its use for developing three dimensional models. Tanskanen et al. (2013) propose a working pipeline to obtain three dimensional reconstructions of real world objects using an Android device.
6
Aside from commercial software, open source software packages which implement SfM algorithm are also available (Bartoš et al., 2014). Open source software are often used in a SfM research for its customization, various algorithm and low-cost implementation. For this reason, open source software have more updated options, and implement more recent algorithm. An example of such open source software is OpenMVG (Moulon et al., 2012) and MVE (Fuhrmann et al., 2014). Utilization of open source software to implement SfM algorithm have another advantages, which is to ensure the continuous development and research with compromising license. Thus, an open source software can be implemented with more customization, while using state-of-the-art algorithm which have not been implemented in commercial software yet (Weber, 2004). Another means of SfM implementation is a SFM web service. In such software, users need to upload images, which then processed in the server to produce 3D point cloud. Examples of SfM web service are Microsoft’s Photosynth (Pomaska, 2009), CMP SfM webservice (Jancosek and Pajdla, 2011) and Autodesk 123D Catch (Chandler and Fryer, 2013). Screenshots of these web services are shown in Figure I-1.
b)
a) Figure I-1: Screenshots of some SfM webservices: CMP SfM Webservice (a) and Microsoft’s Photosynth (b)
7
Notable example of SfM Web service is Autodesk 123D Catch. Beside the web service, this software also implemented as Android and iOS application, as well as desktop software. It offers free services for limited numbers of images (as much as 70 images are allowed for each reconstruction project). Also, advanced options are available as commercial services in premium edition. In addition to 3D reconstruction service, the free membership also offer 3D model sharing and 3D printing. Screenshots from Autodesk 123D Catch are shown in Figure I-2.
Figure I-2: Screenshots from Autodesk 123D Catch Reconstruction
SfM algorithm makes it possible to develop a fully automatic system to reconstruct 3D structures and camera motion from multiple images. However, current implementation of SfM have limitations. Commercial software offers simple, easy to use interface to deal with non-technical user. Aside from its commercial license, these software usually have outdated SfM algorithm (Moulon et al., 2012) and limited customization on 3D reconstruction parameters (Remondino et al., 2012). Open source software offers highly customizable environment and low-cost solution. However, such software have steep learning
8
curve, and the user is required to have photogrammetry and computer vision background, as well as some programming knowledge. On the other hand, SfM web services provided simple interface and easy to operate. Minto and Remondino (2014), as well as Vergauwen and Van Gool (2006) reviewed some aspects on presenting a platform for 3D sharing of SfM models. Unfortunately, most SfM web services were developed as black-box or closed source software (hidden or unknown source code and algorithm), which limit the usability of such software for research purposes (Remondino et al., 2012). Another notable example of web cloud technologies for 3D reconstruction online are shown in Chen et al. (2014), which use cloud processing for 3D reconstruction of indoor environment using crowdsourced data. An overview regarding some SfM software available on the market is shown in Table I-1. Table I-1: Overview of SfM software No 1
Name Agisoft Photoscan
2
Photomodeler Scaner
3
4
Pix4D Mapper Bentley Acute3D
5
Platform
License
Blackbox
Reference
Status
Desktop
Commercial
Yes
www.agisoft.com
Active
Desktop
Commercial
Yes
http://www.photomodeler.com/
Active
Commercial
Yes
https://pix4d.com
Active
Commercial
Yes
http://www.acute3d.com/
Active
Desktop, webservice Desktop
Bundler
Desktop
6
VisualSfM
Desktop
7
PPT (Python Photogrammetry
Free and Opensource Free
No Yes
Open
No
Source
Toolbox) 8 Desktop
Open
OpenMVG
Desktop
Free and opensource
Unknown
http://184.106.205.13/arcteam/ppt.p
(last
hp
update
Active Yes
Source 9
ccwu.me/vsfm/
Active
2012)
Free and Insight3D
er
Active
Free and Desktop
www.cs.cornell.edu/~snavely/bundl
http://sourceforge.net/projects/insig
(last
ht3d/
update 2013)
Yes
https://github.com/openMVG/open MVG
Active
9
10
Microsoft Photosynth
12
Autodesk 123D Catch
13
CMP SfM webservice
Free, Webservice
Limited
No
https://photosynth.net/
Active
No
http://www.123dapp.com/catch
Active
access Free, Webservice
Limited access
Webservice
Free
No
http://ptak.felk.cvut.cz/sfmservice/
Inactive
http://www.hypr3d.com/
Inactive
14
Hypr3D
Webservice
Free
No
15
Arc3D
Webservice
Free
No
http://homes.esat.kuleuven.be/~visit 3d/webservice/v2/
Active
There are many options available to perform Structure from Motion 3D reconstruction, whether using desktop software or as an online web service. However, this thesis focus on some aspects which differs it from the aforementioned works, which are: 1. Utilization of open source software 2. Using Global Structure from Motion algorithm (Moulon et al., 2013a) 3. Online web service using MEAN frameworks The output of thesis is a web based, online SfM web service implemented using open source software. The system, called CloudSfM, was built to cope the limitation of SfM software as mentioned above. The result of this thesis could be used for further research on cloud-based photogrammetry using state-of-the-art 3D reconstruction algorithms.
I.7. Theoretical background Structure from Motion I.7.1.1. The Structure from Motion (SfM) problem. The Structure from Motion problems is stated as follow: given corresponding view of images, how to reconstruct the 3D structures of objects and the camera 10
motion? (Hartley and Zisserman, 2003). In other words, the SfM algorithm aims to solve two kinds of unknowns: object’s structure and relative camera motion (position and orientation) under the lack of data such as initial camera position and their respective orientation. Thus, the SfM algorithm consist of methods for feature identification and matching, homography estimation, dense reconstruction and other methods to produce 3D coordinates of objects in question using limited initial knowledge of the scene (Sonka et al., 2014) Literatures on Structure from Motion (SfM) can be traced back to two fields: photogrammetry and computer vision. Although initially developed in computer vision community, modern definition of SfM is closely related to photogrammetry (Jebara et al., 1999). With the rapid development in sensor and processing technology, the knowledge exchange between photogrammetry and computer vision are inevitable. Jebara et al., (1999), Förstner (2002), Klette and Reulke (2005) and Wang et al. (2012) are some that have emphasized the relationship of photogrammetry and computer vision, including on Structure from Motion. Structure from Motion shared the same principle with classic photogrammetry in that both methods reconstruct 3D structure from a sequence of overlapping images (Westoby et al. (2012) and Micheletti et al. (2015)). As shown in Ma et al. (2012), the so-called Kruppa’s Equation (according to Kruppa (1913)) stated that from two overlapping images, five points are sufficient to determine the relative transformation between camera view and 3D position of the points using projective geometry. The projective geometry, derived from perspective projection, leads to the mathematical formulation of homography or projective transformation (Sonka et al., 2014). Another principal work on SfM was conducted by Marr and Poggio (1979), which showed the computational relationship between human stereo vision, image matching and 3D geometry of a scene. Using iterative algorithm, unique matches between points in corresponding views can be used to retrieve the points in a 3D space. Ullman (1979) expanded the previous works on scene geometry by proposing the use of camera motion for 3D structure reconstruction. His work proved that nonlinear algorithm could be 11
used to provide unique solution of four corresponding points in three views. Modern approach of SfM algorithm are provided by the works of Nister (2004), which improvise the Kruppa’s five points algorithm to be used for computational purposes. In line with this improvement, Longuet-Higgins (1987) and Hartley et. al. (1997) focus on eight-point algorithm to reconstruct camera pose and 3D structure. The aforementioned works are the basis of the computer implementation on SfM (see for example on Szeliski (2010).
Figure I-3: Methods of 3D Acquisition using Non-Contact Waves (Remondino and El-Hakim, 2006)
In computer vision community, Structure from Motion is also known with other terms, such as Structure and Motion (Triggs, 1996), Depth from Motion or Shape from Motion (Sonka et al., 2014). The latter terms coined to explain the approach of SfM method as part of depth reconstruction methods called Shape from X, with X being various techniques such as Depth from Defocus, Shape from Texture, Shape from Shading, etc (Sonka et al., 2014). However, in photogrammetry, the workflow for generating 3D models from images is collectively referred to as close-range photogrammetry (Remondino and El12
Hakim, 2006). Figure I-3 shows terminologies used in 3D data acquisition using non-contact waves or remotely-sensed data. The SfM method in Figure I-3 is referred to as ‘photogrammetry’, while other method such as shape from shading and focus/defocus are mentioned explicitly. A typical setup for a SfM reconstruction scene is shown in Figure I-4. A successful reconstruction of a SfM scene requires the camera to be ‘displaced’ relative to the object in question. This is not necessarily means that the camera moves against a static object, but rather to keep base distances from the camera in order to avoid local minima in non-linear solution (Soatto and Brockett, 1998). Since SfM relies on camera-object geometry, providing more images generally means more successful reconstruction, as shown in Jebara et al. (1999).
Figure I-4: A typical setup of SfM scene (from (Westoby et al., 2012)
Prior to solving the unknown parameters (camera pose and matrix), a set of keypoints from corresponding images are detected using feature detection algorithm. Matching keypoints in corresponding images can be done using so13
called keypoint descriptor, which is a unique identifier for every single keypoint (Hartley and Zisserman, 2003). It is important to note that in Structure from Motion, the keypoints are not merely extracted, but also identified with one of the keypoint extraction algorithms so as to identify not only movement of the keypoint (hence the name Structure from Motion), but also the change in their orientation and scale. Some notable keypoint extraction algorithm are SIFT (Scale-Invariant Feature Transform), SURF (Speeded-up Robust Features), BRIEF (Binary Robust Invariant Scalable Keypoints) and ORB (Oriented FAST and Rotated BRIEF). Figure I-5 shows keypoints, keypoint descriptor and keypoint matching using SIFT algorithm.
Figure I-5: Above: Keypoints and keypoints descriptor. Below: Keypoints matching using SIFT algorithm (from Brahmbhatt (2013))
The Structure from Motion Algorithm solves the problem of generating 3D reconstruction of the object’s structure as well as its estimation of the camera motion (position and orientation) in a particular scene (Jebara et al., 1999). Thus, 14
SfM differs from conventional photogrammetry in that SfM requires no prior knowledge on the scene, whether on the camera parameters (internal or external) or the structure of the object (Jebara et al., 1999). SfM uses computer vision algorithms to perform automatic recognition of features and feature matching, then reconstruct the geometry of scene based on these information. The algorithm estimates camera pose and iteratively solve the 3D structure of the object, hence the name (Ullman, 1979). I.7.1.2. Algorithms of Structure from Motion A. Single-view geometry To solve the SfM problem, a SfM reconstruction scene consist of several overlapping images can be explained using simplified geometry of the scene. A single-view geometry defines the relationship between a single images 3D structures (or ground coordinate) of points in an object. The single-view geometry establish the mathematical properties of image formation in camera and the transformations needed to obtain projected coordinates of 3D points into the 2D camera frame (Ma et al., 2012).
Figure I-6: Geometry of a pinhole camera (Hartley and Zisserman, 2003)
Basic equations for SfM are similar to conventional photogrammetry. (Hartley and Zisserman, 2003) uses pinhole camera (Figure I-6) to model the relationship between three dimensional object of the scene and two dimensional frame of the camera. A pinhole camera is a simplified model of perfect camera 15
where each point is projected to the camera center C through the principal point p. The pinhole camera model assumes same scale for both X and Y axis, and that no distortion occurred to the camera lenses. Modern camera with CCD sensor should employ distortion model K to eliminate the effect of distortion (Hartley and Zisserman, 2003). These relationship of images and 3D structure of points in each images can be elaborated as three principal transformation as follow (Robertson and Cipolla, 2008): 1. Transformation of points in ground coordinate 𝑋̃ = [𝑋 𝑌 𝑍 1]𝑇 to camera ̃𝐶 = [𝑋𝐶 𝑌𝐶 𝑍𝐶 1]. In matrix form, this transformation is coordinates 𝑋 expressed as: 𝑋𝐶 𝑌 𝑅 [ 𝐶] = [ 0 𝑍𝐶 1
𝑋 𝑇 𝑌 ][ ] 1 𝑍 1
(1)
With R and T is rotation and translation matrix to the origin of real world coordinate system, respectively. Both of these parameters define camera pose (position and orientation) or external orientation of the camera. 2. 2D to 3D coordinate transformation from the camera coordinate system ̃𝐶 = [𝑋𝐶 𝑌𝐶 𝑍𝐶 1] to plane coordinate of the camera 𝑥̃ = [𝑥 𝑦 1]𝑇 . Using 𝑋 the pinhole camera estimation of focus where f = 1, the relationship can be expressed as: 𝑥 1 [𝑦] = [0 1 0
0 0 1 0 0 1
𝑋𝐶 0 𝑌 0] [ 𝐶 ] 𝑍𝐶 0 1
(2)
3. The third transformation is from the camera plane coordinate system 𝑥̃ to ̃ = [𝑢 𝑣 1]𝑇 . Given K as matrix of the image’s frame coordinate system 𝒖 camera calibration (Hartley and Zisserman, 2003): 𝛼𝑢 𝑠 𝑢0 𝐾 = [ 0 𝛼𝑣 𝑣0 ] 0 0 1
(3)
16
With αu and αk are scale factors, s defines skewness and principal point 𝒖𝟎 = [𝑢0 𝑣0 ]𝑇 , the relationship of these three parameters is:
̃ = 𝑲𝒙 ̃ 𝒖
(4)
The scale factors, skewness and coordinates of principal points are known as internal parameters of the camera. The equations above could be expressed by simplifying K, R and T into projection matrix P with 3P4 ~ K [R T] into: ̃ ̃ = 𝑷𝑿 𝒖
(5)
Given the intrinsic and extrinsic parameters of the camera, 3D coordinates of the points in the object could be retrieved using the equations above (Robertson and Cipolla, 2008). If 𝒖𝑖 = [𝑢𝑖 𝑣𝑖 ]𝑇 is the measured image position of 3D point [𝑋𝑖 𝑌𝑖 𝑍𝑖 ]𝑇 , the projection matrix P (Eq. 5) could be expressed as (Robertson and Cipolla, 2008):
(6)
(7)
The parameters 𝑝11, 𝑝12,… 𝑝34, in the above equation represent unknowns in the projection matrix P which needs to be solved in order to obtain camera position and orientation 𝒖𝑖 and object structure [𝑋𝑖 𝑌𝑖 𝑍𝑖 ]𝑇 . For n number of initial points, the equation below are valid:
17
(8)
Thus, in order to obtain scene structure and camera position, at least six points needs to be available in the initial view of the two overlapped photos to solve the twelve parameters above. This relationship is valid for a simple, pinholelike camera in a single-view geometry. However, Structure from Motion works on multiple camera, thus resulting in a more complex geometry of the scene. The geometry of multiple view camera can be elaborated using two view-geometry and N-view geometry which models camera movements in a SfM scene (Hartley and Zisserman, 2003).
B. Multiple-view Geometry The two-view geometry of a SfM scene is also referenced as stereo-vision geometry, which is well known in field of photogrammetry (Jebara et al., 1999). The two-view geometry is also known as Epipolar Geometry. The Epipolar Geometry defines the intrinsic projective geometry between two views, and is independent of the global scene structure, as it depends entirely on internal parameters of camera and their relative pose, respectively (Hartley and Zisserman, 2003).
18
Figure I-7: Epipolar Geometry (Jebara et al., 1999),
Figure I-7 depict epipolar geometry of a scene. Given P as a corresponding points viewed through two images, COP1 and COP2 is the Center of Projection in each camera, respectively. The plane formed by COP1, COP2 and P is called the epipolar plane, while epipolar lines defined as the intersection of this plane with each of camera frame plane. Epipoles is the line connecting the two COP, and intersect with the image frame in e1 and e2, respectively (Hartley and Zisserman, 2003). According to Eq. (5), the point P is projected into the image frames as p1 and p2, respectively. Thus, the coordinate of p can be expressed in camera frame coordinate as p1 (u1, v1, 1) and p2 (u2, v2, 1). The relationship between p1 and p2 relative to camera pose can be expressed as (Jebara et al., 1999): 𝑝1 𝑡 ∗ 𝐹 ∗ 𝑝2 = 0
(9)
Matrix F in above equation is known as the Fundamental Matrix, which represents the algebraic relation of an epipolar geometry (Hartley and Zisserman, 2003). (Hartley and others, 1997) demonstrated that as much as eight corresponding points (i.e. tie points) between the two images are sufficient to solve F, and thus 19
recover the intrinsic and extrinsic parameters of the scene. Thus, if corresponding points in two images could be recovered, reconstruction of the scene can be performed using algorithm shown in Hartley and Zisserman (2003): i.
Using known point correspondences, compute the fundamental matrix
ii.
Using the fundamental matrix, compute camera (intrinsic) matrices
iii.
For each correspondences xi and xi’, compute the 3D coordinates of the point projected to the two points.
If the intrinsic parameters of the camera is known, the fundamental matrix could be further simplified into another matrix called Essential Matrix (Ma et al., 2012). Given the intrinsic parameters of the camera K (Eq. (3)), the essential matrix E could be expressed as (Hartley and Zisserman, 2003): 𝐸 = 𝐾 𝑇 𝐹𝐾
(10)
The single and two-view geometry is applied to the whole SfM scene recursively. The rigid-body motion or Euclidean motion is the term used to express camera’s position in a SfM scene. Such term coined since the camera (or cameras, in case of more than one camera is used) moved in Euclidean space relative to the object in question (Hartley and Zisserman, 2003).
Figure I-8: Typical scene of Structure from Motion problem (Thormählen et al., 2010)
20
A SfM scene with three views is depicted in Figure I-8. In the case of a monocular camera, the three views are produced by moving the camera in three different position (k-1), k, and (k+1). If the point P in object projected as Pj,k-1, Pj,k, and Pj,k+1 in each image frame respectively, the geometry of the scene can be reconstructed using the relationship of Essential matrix and Fundamental matrix as shown in Eq. (9). When more cameras are used, the equation could be expanded and solved as linear problem as shown in Thomas and Simoncelli (1994) or nonlinear problem as stated in Szeliski and Kang (1994).
I.7.1.3. Methods of Scene Estimation In conventional photogrammetry, various methods are employed to obtain both of the parameters, whether by carefully measuring camera position and orientation to determine the external parameters and performing camera calibration to obtain internal parameters. SfM methods are able to solve the structure and motion problem using various algorithm, e.g. by using estimation of known parameters and perform iterations to refine the parameters (Tomasi and Kanade, 1992). SfM employs mathematical model to find camera pose and iteratively solve the structure of the objects until all of camera pose and tie points are solved with minimum errors (Jebara et al., 1999). There are various SfM algorithm to solve a scene of SfM problems. (Jiang et al., 2013) underlines three main method used to estimate position and orientation of the camera prior to solving SfM problems: a. Sequential/incremental method. This method is the conventional uses of SfM algorithm, and the most widely implemented for its simplicity (Kim et al., 2013). This method estimates the relative pose of two first images of the scene and gradually iterates all the structure and camera pose of the scene based on the next images until all images’ position and orientation are defined. This algorithm works well on the scene with monocular camera capturing images along a straight line. A variation of this method known as 21
Hierarchical SfM (proposed by Bergen et al., (1992)) is conducted by assessing the hierarchical relationship among the images instead of sequentially solve the scene. b. Factorization Method. This SfM method are introduced by Tomasi and Kanade (1992), thus also known as Tomasi and Kanade Factorization method. This estimation method assumes that in all the image sequence, there exist some set of features captured by most of the images. This method employs bilinear factorization to find the relationship between such features and the shape and motion of images. Given the projection equation of a view (pair of images) and estimated coordinates of features known in all view, the whole geometry of the scene can be reconstructed. c. Global Method. The previous methods are prone to drifting errors as a result of gradually solving each view. The global method solve all the camera poses and 3D structure of points at once, thus minimizing the possibilities for local maxima or minima from other methods (Sweeney et al., 2015). Global method works by solving all of the images’ rotational parameters, and afterward solve the translations (Jiang et al., 2013). This method provides better accuracy and computational efficiency compared to other methods, especially with faster computing capabilities available (Cui et al. (2015) and Sweeney et al. (2015)). The global method were used in this thesis for its faster solution and less processing resource needed (Moulon et al., 2013a), thus is suitable for solving SfM problems on-the-fly. Global Method in this thesis refers to Moulon et al. (2013a), and was implemented using open source SfM library called OpenMVG. The OpenMVG’s pipeline for solving SfM using Global method are (Moulon et al., 2013a): 1) Compute relative pairwise rotation from pairs of images 2) Detect and remove errors in pairwise rotation 3) Calculate global rotation (R in Eq. (1)) using dense least square and rotation estimates 22
4) Calculate relative (pairwise) translation from the general model 5) Calculate the global translation (T in Eq. (1)) 6) Finalize Structure from Motion using known rotation and translation parameters 7) Reiterate and refine rotation and translation parameters to minimize error 8) Recalculate Structure from Motion I.7.1.4. The Structure from Motion pipeline The solution of camera pose (position and orientation) are done by iteratively estimating unknown parameters through multiple view geometry equation (Moons et al. (2009); Hartley and Zisserman (2003), while the intrinsic parameter (i.e. camera matrix) can be solved using self-calibration (Pollefeys et al., 1999). In case of Incremental SfM, the calculated parameters are then refined iteratively so that precise 3D position of surveyed object can be obtained (Pollefeys et al., 1999). Image acquisition Calibration
Keypoints and keypoints descriptor
Camera Matrix
Keypoint Matching or Optical flow detection
Enhanced Pose Estimation
3D Reconstruction (point cloud)
Texture Mapping
Bundle Adjustment and geometric reconstruction
3D Model
Figure I-9: A typical Structure from Motion pipeline
23
A common Structure from Motion pipeline for producing 3D models of realworld project are explained in Figure I-9. According to Rossi et al. (2012), the high-level SfM reconstruction pipeline of scene structure and camera motion consists of these steps as follow: a. Feature extraction In this step, feature detection algorithm is employed to detect features (‘keypoints’) and their description (‘keypoints’ descriptor’) in each image. The most common feature detection algorithm used in SfM is SIFT (Scaleinvariant Feature Transform), a patented method by Lowe (1999). As the name suggest, SIFT are able to detect features in spite of changes in image scale, contrast, translation or rotation (Brahmbhatt, 2013). Other methods for feature detection including SURF, ORB and the more recent KAZE (Alcantarilla et al., 2012). b. Feature matching This step consist of two main algorithm: 1) feature tracking and 2) feature matching (Ma et al., 2012). In feature tracking, an algorithm is employed to trace detected features from the previous step and over impose the features in the next images, thus creating a track for each particular features. Afterwards, feature matching algorithm is employed to find definitive correspondence between the features. c. Bundle Adjustment When features and their corresponding matches are found, the next step is to reconstruct the projective two-view geometry. Eq. 6 and Eq. 7 are then used for estimating the relationship of corresponding features. To solve the non-linear problem of estimating pose and structure, a bundle adjustment method is used. The bundle adjustment algorithm commonly used for pose and structure estimation is Levenberg-Marquardt (Sonka et al., 2014). d. Point-Cloud generation and processing This step consist of two main process: sparse and dense point-cloud reconstruction. Sparse reconstruction produces 3D structure of points 24
obtained from bundle adjustment. Dense point-cloud used 3D structure from sparse point-cloud to refine camera pose, then repeat the reconstruction process to obtain denser point cloud (Westoby et al., 2012). e. Visualization The last step for 3D modeling using SfM pipeline is visualization. Two process falls into this category: meshing and texturing. Meshing is the process to interpolate the surface of 3D models based on dense point cloud. Algorithm such as Poisson Surface Reconstruction (Kazhdan et al., 2006) could be used to perform meshing. There are various other algorithms which can be used in this step, for example Floating Scale Surface Reconstruction (Fuhrmann and Goesele, 2014) used depth map as intermediate product to develop more robust surface mesh. Texturing or texture draping is the next and final step in the pipeline. In this step, the images are ‘draped’ or mapped into the surface mesh, creating more realistic visualization (Rossi et al., 2012). Figure I-10 shows some typical deliverables of a high-level SfM pipeline. The sparse point-cloud (Fig. I-10 a) is the first product, which is a 3D coordinates of reconstructed structure using initial feature matches (or tie points). Refined SfM sparse reconstruction resulted in dense point cloud (Fig. I-10 b and c), which in some case is comparable to point clouds resulted from Terrestrial Laser Scanning (Westoby et al., 2012). From the dense point cloud, a meshing algorithm is used to produce surface or mesh (Fig. I-10 d), and finally the texture or images from camera is used to cover the surface into a textured model (Fig. I-10 e). This is the common product of a SfM pipeline as shown in Rossi et al., (2012), but some particular cases might only need dense point cloud as the final product instead of a textured model.
25
a) Sparse point-cloud
d) Mesh surface
c) Dense point-cloud (colorized)
b) Dense point-cloud (shaded)
e) Textured surface
Figure I-10: Some deliverables from a SfM pipeline
I.7.1.5. Implementations of SfM The advancement in computational science and technology led to various number of SfM implementation. SfM relies on the computer’s ability to automatically identify features, perform feature matching, and reconstruct SfM scenes. Thus, implementations or high-level abstraction of SfM grows along with the growth of computer technologies. In Computer Vision, SfM influences many fields, such as visual odometry (Kitt et al., 2010) and robotic navigation and localization (Zaman, 2007). In 3D reconstruction of real world object, SfM has been implemented to develop a 3D model of the City of Rome using images acquired from tourists (Agarwal et al., 2011). Various researches showcase the 26
agility of SfM in modeling architectural objects (Sinha et al., 2008), as well as archeological site (Westoby et al., 2012), topographic mapping (Fonstad et al., 2013) and street view reconstruction (Klingner et al., 2013). With developments in internet technologies, SfM reachs its way to be implemented in this field as well, as shown in Chapter I.6. Among the implementations of SfM are software and libraries that provide SfM reconstruction. According to its license of use, there are four categories of SfM software/library (Remondino et al., 2012): a. Commercial software, which obliged user to pay for the software in order to use the SfM reconstruction features. Examples of software in this category are Agisoft Photoscan, Photomodeller Scanner, Pix4D Mapper, and so on. b. Sharewares, where most of the features are available for use, but at some point user are obliged to pay (for additional features or time). An example of this software is Autodesk 123D catch c. Free software. Users of this software are not obliged to pay any fees for using the SfM reconstructions. However, the user will not be able to modify the algorithm behind the software. Examples of these software are VisualSfM and CMPMVS d. Open Source software. Of all the software above, this category is the least strictly licensed software. The user are free to use the software and modify the source code as they want. Examples of this software are OpenMVG, MVE and Theia. Open source software usually developed as a library which serves particular purposes from SfM pipeline. Users need to combine the libraries to perform complete SfM pipeline, as shown in Rossi et al. (2012). The developed software architecture chain for SfM reconstruction used free and/or open source software, i.e. Bundler, CMVS/PMVS, Poisson Surface Reconstruction, and Meshlab.
27
I.7.1.6. Assessment of SfM products A. Internal and external evaluation Numbers of researches have been conducted to investigate the quality of SfM result. Since products of SfM reconstruction depends on a lot of variables (such as camera sensor size, distortion model, illumination of scene, homogeneity of the surface, texture of the model, etc), the research of SfM product evaluation also highly varied in terms of control variable and object of studies. The internal evaluation and external evaluation are often used to evaluate the robustness of SfM products. Internal evaluation assess the consistency of SfM reconstruction. For example, to check for missing correspondences in each image pair or to validate the consistency of Epipolar Geometry. Jiang et al. (2012) propose the formula for evaluating correctness of epipolar geometry EG: 1
1
1
𝑁 ̂ 𝑀 ̂ 𝐸𝑅 = 𝑀 ∑𝑀 𝑝=1 𝑃𝑚𝑖𝑠𝑠𝑖𝑛𝑔 (𝑝) = 𝑀 ∑𝑝=1 𝑁 ∑𝑖=1 𝑃𝑚𝑖𝑠𝑠𝑖𝑛𝑔 (𝑝, 𝑖)
(11)
With N being the total number of images and M is the total number of 3D points reconstructed from the view. 𝑃𝑚𝑖𝑠𝑠𝑖𝑛𝑔 (𝑝, 𝑖) is the probability that the feature descriptor (e.g. SIFT) does not match its image projection in 𝑖, with 𝑃̂𝑚𝑖𝑠𝑠𝑖𝑛𝑔 (𝑝) being its average in all views. Since SfM depends on the correctness of feature detection and matching, evaluating the erroneous detected features against the whole scene would give estimated robustness of a scene. The internal evaluation can be expressed as a connected visibility graph or pairwise matrix which shows correspondences between each camera in a scene.
28
Figure I-11: An example of Visibility Graph and Pairwise Matrix (from (Moussa, 2014))
The external evaluation is performed by comparing result of SfM reconstruction (whether camera calibration matrix, dense point cloud or mesh) with ground truth data. Strecha et al. (2008) provide some early method of comparing SfM result with ground truth data (using Lidar data). Seitz et al. (2006) provides platform to evaluate state-of-the-art multi-view stereo algorithm with predefined ground truth model. With increasing popularity of SfM, more research were conducted to evaluate its result against other data acquisition method, for example by Grussenmeyer et al. (2008), Wang (2011), Westoby et al. (2012), Rasztovits and Dorninger (2013), Andrews et al. (2013) and Thoeni et al. (2014). Method used in these papers to compare result of SfM reconstruction against other data is the Iterative Closest Point (ICP).
B. Iterative Closest Point (ICP) Iterative Closest Point (ICP) is the method used to register one point cloud (known as ‘source’) to another (‘target’ or ‘reference’). The so-called Iterative Closest Point algorithm performs transformation of ‘source’ point cloud to ‘target’ and iteratively refines the transformation until minimal error is achieved (Besl and 29
McKay, 1992). Variations of ICP have been presented, such as Zhang (1992) and Chetverikov et al. (2002). The algorithm of ICP is as follow (Besl and McKay, 1992): 1) For every single point in ‘source’ point cloud, find the corresponding closest point in the ‘target’ point cloud. 2) Calculate cost function of mean-squared translation and rotation error from the previous step 3) Calculate transformation matrix and perform the transformation based on the previously calculated cost function 4) Repeat each step until the cost function is minimized
Figure I-12: Cloud-to-cloud comparison using ICP on CloudCompare (Thoeni et al., 2014)
ICP can be used to register two point clouds from different sources, e.g. from SfM reconstruction and terrestrial laser scanner (Westoby et al., 2012). By registering the two point cloud, it is possible to calculate the differences between both of the point cloud using mean distance. The method to employ cloud-to-cloud method for comparison of SfM reconstruction against ground truth data are found in researches such as Thoeni et al. (2014) as shown in Figure I-12. Implementation 30
of ICP for cloud-to-cloud comparison can be implemented using software such as CloudCompare (Girardeau-Montaut, 2011).
Online Web Service Framework I.7.2.1. Cloud Computing According to NIST (National Institute of Standard and Technology), the definition of Cloud Computing is: “A model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction” (Mell and Grance, 2009). According to the definition, a cloud computing system possesses the following characteristics: a. On demand self-service. The user of cloud computing service could provision computing capabilities without assistance of human service provider b. Broad network access. The cloud service can be accessed through standard thick or thin client platforms c. Resource pooling. The cloud service computing resource are able to serve multiple consumers d. Rapid elasticity. The server’s capabilities can be dynamically provisioned and released according to demand on those service e. Measured service. The service given with metered service, which means that the cloud server’s resource are monitored and measured A cloud service as the above characteristics can be implemented using various models. The NIST definition of cloud computing provide three models of cloud computing services (Mell and Grance, 2009): a. Software as a Service (SaaS). This model give the user capabilities of software stored in cloud server. The software’s service can be accessed from any 31
thick or thin client such as web browser or a program interface. The user in this model will not be able to manage or control how the software works or its underlying infrastructure. An example of this model is Google apps (e.g. Google Spreadsheet, Google Doc) and the aforementioned CMP SfM web service (See Chapter I.6). b. Platform as a Service (PaaS). The user create applications with some kind of programming language and deploy the software into cloud server. In this case, the consumer does not control the underlying infrastructure (such as available memory or processor capabilities), but are able to modify the way the software flows. Notable examples are Heroku and Amazon AWS c. Infrastructure as a Service (IaaS). Users are capable to provision storage, memory or other underlying platform of the software, while being able to modify the flow of their software on the cloud. Notable examples are Amazon EC2 and Windows Azure
Figure I-13: Applications of Cloud Computing (Chandrasekaran, 2014)
32
SaaS cloud computing services are commonly targeted for thin client users. Chandrasekaran (2014) underlines the best use-cases characteristics for a SaaS services as follow: a. Applications where the end user did not require a full-term service. Instead, the user only needs on-demand service which available when needed. b. For users or companies who are unable to spend money on buying licensed software c. For applications which target thin-client users, such as availability to be served on a mobile devices d. Applications with dynamic, unpredictable resource load. Another definition of Cloud Computing is based on the deployment models (Mell and Grance, 2009). A cloud computing service is either private (exclusively used in an organization), public (open use by practically everyone), community cloud (shared by several organizations to support common needs in the community) or a hybrid (possess two or more deployment characteristics). The use of cloud computing are increasing with the growth of server capabilities to handle large number of processing at once. Employment of a cloud computing service means that the user only need to focus on data acquisition and provision, while the processing and its infrastructure are performed on a cloud server. The utilization of cloud service in many fields reduce the cost for infrastructure and optimize the productivity (Miller, 2008). Armbrust et al. (2010) listed several reasons for the opportunities of Cloud Computing, including 1) mobile interactivity, 2) parallel processing, 3) the growing needs for analytics, and 4) cost-benefit of applications. Siegele (2008) quoted the applicability of live applications as: “The future belongs to services that respond in real time to information provided either by their users or by nonhuman sensors”. In the field of geospatial science and engineering, cloud computing and on-the-fly applications are proven to have many advantages 33
and is growing into a pivotal aspect for the growth of spatial technologies. Yang et al., (2011) provides some example of geospatial applications of cloud computing are shown. For example, the application of cloud computing for spatially-enabled resource and energy management, emergency response, climate change and sustainable development. Another example is shown in Bhat et al. (2011), which proposed a cloud-based web interface and server for a GIS infrastructure. In particular, the utilization of cloud computing technology for the field of photogrammetry has been discussed in Chapter I.6.
I.7.2.2. Single Page Applications Single Page Applications (SPA) is defined as a web application which update or load new content to client (e.g. browser) without the need to reload the page (Mikowski and Powell, 2013). A SPA gives user experience similar to a desktop software instead of ordinary website. The implementation of SPA often use Javascript as end-to-end language (frontend-backend). Mikowski and Powell (2013) elaborate the reason behind the use of Javascript as the main language of SPA, which are its flexibility in server, client and database applications.
Figure I-14: End-to-end Javascript solution for SPA
A Single Page Application can be of any forms. A common form is an application with CRUD (Create, Read, Update and Delete) database operations (Monteiro, 2014). This kind of application put the user interface to dynamically 34
interact with the database and update the view asynchronously. Such application would give the user fluid experiences to manage and interact with the database. Further derivative of such system can be applied to lots of scenario. For example, the CRUD operations can be used in an authentication system to manage user accounts and their password (Haviv, 2014). A design-pattern known as MVC (Model-View-Controller) are often used to implement Single Page Application with database operations. MVC uses three tiers architectural system to communicate between user in frontend and the database in server backend. Each of the three tiers, namely Model, View and Controller (hence the name), have their roles in communication between frontend and backend, respectively. The principal idea behind MVC is Separation of Concerns between data management (model), application logic (controller) and data presentation (view). This pattern helps web developer to build a robust system and focus on a single task, rather than combining all the functionalities into a single requestresponse unit (Seshadri and Green, 2014).
Figure I-15: Architecture of MVC Communications (Haviv, 2014)
35
The three components of MVC pattern define how client and server communicate. When a user send request through a web page frontend (‘view’), the request were accepted by the web application logic (‘controller’) which send the request to a backend database management system (‘model’). The ‘model’ then returns responses to the ‘controller’, which update user ‘view’ based on the response. This design-pattern guarantee that each unit is independent to each other, thus developing modular structure of the web application (Seshadri and Green, 2014). I.7.2.3. The MEAN Framework MVC can be implemented into a Single Page Application using so-called MEAN framework. MEAN is a set of applications based on Javascript language consist of a server application environment (NodeJS), database management system (MongoDB), web framework (ExpressJS) and a frontend framework (AngularJS). MEAN is a full-stack Javascript framework which serve similar purpose to the LAMP framework (Linux, Apache, MySQL and PHP) which are widely used prior to Web 2.0 (Bretz and Ihrig, 2014).
Figure I-16: MEAN framework, Combinations of logos from its components (Bretz and Ihrig, 2014)
MEAN framework consist of four components as follow (Haviv, 2014): a. NodeJS NodeJS is the core components of MEAN framework. NodeJS is the Javascript implementation of server-side application framework which support event driven, non-blocking I/O model for realtime communications between client and server (Haviv, 2014). The evented nature of NodeJS is useful to build highly scalable and fast application 36
on the web, and thus is a perfect match for building Single Page web Application (Gackenheimer, 2013). b. MongoDB MongoDB is a no-SQL database system which based its structure and query on Javascript (Hows et al., 2014). The No-SQL concept means that MongoDB does not have tables, relations, schemas, or any other characteristics of relational databases. MongoDB uses JSON-like format (namely BSON format – Binary JSON) to save data into various collections. Given the Javascript nature of MongoDB, the CRUD operation can be performed faster although large dataset were used (Hows et al., 2014) c. ExpressJS ExpressJS is a NodeJS middleware in form of a set of commonly used web application features (Haviv, 2014). Examples of ExpressJS utilization is to handles request and response route. ExpressJS also serve as middleware, which able to extend standard http request and response into various data format. Thus, ExpressJS can handle file upload/download in NodeJS, as well as other critical function in a web application. Another function of ExpressJS is to maintain MVC framework in a web application by managing application structures and routes (Haviv, 2014). d. AngularJS The last part of MEAN framewok is AngularJS. AngularJS is a frontend Javascript framework for development of Single Page Application implementing MVC framework (Haviv, 2014). AngularJS enrich standard HTML elements into customized element tags.
It also
supports two-way data binding and client-side templating to build rich features SPA. AngularJS is highly supportive on developing RESTful web application with its native support for REST parameterization. As for CRUD (Create, Read, Update, Delete) database operation, AngularJS also enriched with native form validation and routing 37
capabilities, thus make it works seamlessly with other MEAN components (Panda, 2014).
LAMP Framework Backend (Server)
Frontend (Client)
HTML, XML, etc
Javascript
Database
SQL
Apache with PHP
MySQL
Backend (Server)
Database
MEAN Framework Frontend (Client)
Javascript (BSON)
Javascript
AngularJS
NodeJS and ExpressJS
MongoDB
Figure I-17: Comparison between LAMP and MEAN Architecture
Combination of the four components makes MEAN stack become a viable alternative to LAMP framework. One of the most prominent advantages of using MEAN framework over LAMP is that it use a single Javascript language for the whole purpose of database, server and client-side applications, thus increasing cost-efficiency and unifies the whole ecosystem (Haviv, 2014). Figure I-17 shows 38
the analogies of LAMP framework and MEAN framework components in client, server and database applications. I.7.2.4. Open-sourcing application with Github Software development is an ongoing process which involves different stages at which the software is being developed. The lifetime of software development involves continuous improvement such as feature upgrades and bug fixes. The different stages of software development are expressed as software version, and the act of staging multiple version of a software is called versioning. When discussing software versioning, an important tool which manage and tracks the different version of the software needs to be taken into account. Such a tool is called a Version Control System (VCS), which also known as Revision Control System (RCS) or Source Code Manager (SCM) (Loeliger and McCullough, 2012). A Version Control System is a system which records any changes made to a software or file historically (Chacon, 2009). A Version Control System captures every state of the software development, and thus providing the ability to revert into previous state whenever needed. Also, using VCS make possible of comparing and monitoring the progress of the development (Loeliger and McCullough, 2012). Another important function for software developer is the ability to collaboratively work on the same project while recording every contribution to the software. Below are some of the advantages of using a VCS for software development (Bell and Beer, 2014): a. Ability to undo changes. Since VCS record every state of the software, any erroneous stage of the software can be easily reverted into the previously non-erroneous state. In VCS, each state is called commit. b. Provides complete history of all the changes. Every addition or subtraction on the code is recorded. If an error occurred, the software developer could trace back which steps resulted in the error and fix it accordingly.
39
c. Documentation of why changes is made. Every changes recorded in the software history contains remark of what that change is about. This remark is called commit message. d. Confidence for changing anything. Since everything is recorded and are able to be reverted to any previous state, software developer would be more confidence to change any part of the code. e. Multiple streams of history. A single project could be duplicated into multiple projects with different modification (called repository). Each of this project would have independent records of history and can be reverted to their respective states. f. Ability to resolve conflicts. Software development, especially on open source software, is often a result of collaborative works between numbers of software developers. In VCS, each of the developers could work on a particular part of the software while being able to merge their work with the main software (or, in VCS term, master repository). g. Independent streams of history. If some developers work on the same project, each of them would have their own record of software commits. Thus it is possible for each developers to work with their own projects, revert to previous states, etc. before merging their work with the master repository. According to the way files and their history is recorded, there are three different setup of Version Control System: Local VCS, Central VCS and Distributed VCS (Chacon, 2009). Local VCS stores histories of a revision in the same machine as the software being developed. Central VCS connects a local machine to centralized machine which record histories of the software, separated from local drives. Distributed VCS used centralized server machine with several connected local computers which save their own histories of software modifications. Figure I-18 shows the three different systems of VCS available.
40
Local Version Control System
Centralized Version Control System
Distributed Version Control System
Figure I-18: Local, Centralized and Distributed VCS (Chacon, 2009)
One of the well-known VCS used by software developers is Git. Git is a Distributed Version Control System which save different copies of software history locally, from a repository stored in a central server (Bell and Beer, 2014). Thus Git is a system which made possible of software collaboration among several software developers. In order to manage Git system, a Git tool is needed. A Git 41
tool hosts repositories of software and provide platform for developers to contribute to the software development (Loeliger and McCullough, 2012).
Figure I-19: A Github repository of QGIS (https://github.com/qgis/)
One of Git tools available for software developer is Github. Github is a Git tool which provides services for developer to publish their work and to contribute to other developer’s work. Github records each modification performed by the developer, and also record other developer’s contribution to the software. Github is growing into de facto application used in development of open source software around the world (Dabbish et al., 2012).
42
II. METHODS AND IMPLEMENTATION
II.1. Method of research This thesis use open source web framework and structure from motion libraries to build an online structure from motion web service which able to do SfM reconstruction on-the-fly. Datasets are used to test the robustness (i.e. point cloud comparison) of this online service. Research Materials These datasets below are used to test the result of online structure from motion service: a. Tree dataset, consist of 91 images of a tree in front of Department of Geodetic Engineering’s building b. Naga dataset, consist of 81 photos of a Javanese Dragon statue. c. UGM Central building dataset, consist of 35 photos of the southern part of UGM Central office building. Each of the dataset were obtained using mobile handset, i.e. Samsung Galaxy Grand 2 Duos. Each of the photo has image dimensions of 3264 x 2448 pixels with 72 dot per inch image resolution and 24 bit sRGB color depth. Different ISO speed and Exposure Time were obtained for all images in the three datasets due to the automated settings of the handheld device’s camera, although the focal length remains the same for each image (i.e. 3 mm). Of the three datasets, two datasets were used to perform cloud-to-cloud evaluation, while one dataset were employed for ground-truth evaluation. The evaluation conducted to each of the dataset are shown in Table II-1.
43
Table II-1: Datasets and their respective testing purpose
Dataset Name
Acq. Device
Tree
obtained using
# of Photos 91 images
Cloud-to-cloud
mobile handset’s Naga
camera (Samsung
Testing Purpose
Comparison 81 Images
Galaxy Grand 2 UGM Main
Duos)
Building
35 Images
Ground truth evaluation
Each of the dataset were uploaded to CloudSfM service as a single reconstruction project. The result of evaluation for each dataset were provided in Chapter III.1.
Research Tools Research tools for this thesis can be divided into two categories, which are hardware and software. Hardware used in this thesis were as follows: a. Samsung Galaxy Grand 2, with Samsung 8 MP (megapixel) rear camera. Used to obtain the datasets. Specifications of this handheld device are shown in Table II-2. b. Leica Reflectorless Total Station TS02-5 (serial number: 1338740), used to obtain ground truth data on UGM dataset c. HP Pavillion g4 as server machine, with specifications as follow: a. Processor: Intel® Core i5, 2.5 Ghz CPU b. 1st Graphic Card: Intel® HD Graphics 3000 c. 2nd Graphic Card: AMD Radeon HD 7400M d. RAM: 4GB The specification above shows that the server is a mid-range class laptop which able to do simple graphical processing, thus affect time consumption during SfM reconstruction. 44
Table II-2: Technical Specification of Samsung Galaxy Grand 2 Duos (www.samsung.com) Device Model
Galaxy Grand 2 Duos (SM-G7102)
Dimension
Operating System
Cyanogenmod 12.1 (Android 5.1.1 Lollipop)
Chipset
Quadcore 1.2GHz Cortex-A7 (CPU) Adreno 305 (GPU)
Memory
RAM 1.5GB, Storage 8GB
Camera
8 megapixel (rear), 1.9 megapixel (front) with Autofocus Enabled
Software for this thesis can be categorized into three categories, namely software for SfM processing; software for web app development; and miscellaneous software. The details are as follow: 1). Software for SfM processing Software used for SfM processing are various open source SfM libraries which serves different purposes for SfM reconstruction pipeline, which are: a. OpenMVG v0.9 ‘Paracheirodon simulans’ (Moulon et al., 2013b). This libraries are used for Global SfM reconstruction. Deliverables are structure (sparse and dense, colorized point-cloud) and motion (camera path).
45
b. MVE - Multiview Environment (Fuhrmann et al., 2014). This software consist of libraries for conducting sparse and dense SfM reconstruction, generate depth maps and producing mesh from sparse point cloud. c. FSSR - Floating-Scale Surface Reconstruction (Fuhrmann and Goesele, 2014). This software consist of libraries used for noise reduction and surface reconstruction from dense point cloud. As of October 2015, the libraries in this software were merged with MVE. d. MVS-Texturing (Waechter et al., 2014). This library used state-of-the-art techniques for texturing reconstructed surface. Deliverables are 3D model of textured object in OBJ format. Python script were used to parameterize and bind user input to the web service, so that all the libraries run in the server with minimum end-user intervention.
2). Software for Web development The online web service were developed using MEAN framework, which consist of components as follow: a. MongoDB. MongoDB is a no-SQL database management system which use javascript as its basic structure. MongoDB has advantages for its utilization in a web environment, i.e. its speed and simple structure. b. ExpressJS. Express is a javascript library which serve as middleware for NodeJS. ExpressJS deals with routing request and response between backend and frontend framework. c. AngularJS. Angular is a Javascript library for building web interface d. NodeJS, is the server behind MEAN framework.
MEAN-MVC framework as above were implemented using Daftmonk Yeoman Angular-Fullstack Generator (Henkel, 2014), which create a basic template for MEAN framework. Angular-File-Upload (Farid, 2015) is used to provide basic service for file upload pipeline. Other than the aforementioned software, some other web development libraries were also employed in order to
46
develop the online SfM web service. A full list of web development libraries utilized in this thesis were listed in Appendix VI.1 (package.json and bower.json).
3). Other software Other software are used in the development of this thesis, which are: a. Linux Mint 17.1 Rebecca. The operating system used to develop CloudSfM and act as server Operating System in this thesis b. Atom Editor. Code editor for editing Python and Javascript implementation of the SfM and web service c. Github and Git, for code management and publishing open source software d. Meshlab, used to display camera motion, sparse point cloud, dense point cloud and textured model of SfM reconstruction e. Cloud Compare. This open source software was used for assessment of SfM result. f. Phone Tester (Android App). Used for obtaining camera parameters according to the hardware specification of the mobile phone used in this thesis. g. Microsoft Excel. This software was used for statistical testing of SfM result against ground truth data. h. Agisoft Photoscan. This commercial software was used to provide point cloud data to be compared with the result of CloudSfM. All of the software above except Microsoft Excel and Agisoft Photoscan are open source software, which means each of them can be used without any charge.
II.2. Implementation of Method Based on related works and literature review, as well as the principle of CloudSfM in the previous chapters, there are three conditions that can be inferred for SfM processing of images: 47
a. Scenario 1: Full Desktop processing. This is the most common implementation of SfM processing. Images are obtained during field survey, and then processed back in the laboratory. b. Scenario 2: Mobile processing. All of the application will be fully operated in and use processing power of handheld device. Example implementation of this scenario can be found in Tanskanen et al. (2013). c. Scenario 3: Cloud Processing. Camera or handheld mobile devices are used for data acquisition. The images are then sent to a server for SfM processing on-the-fly, and the reconstructed model can be downloaded afterwards.
This research aims to implement Structure from Motion method of obtaining 3D structure from images as an online environment using open source software. In order to achieve the goal, a robust, accurate yet easy-to-use workflow needs to be employed. In this case, the last scenario will be implemented in this thesis based on these reasons: a. Using a cloud processing workflow will eliminate the weakness of postprocessing workflow as outlined in Koutsoudis et al. (2014). By utilizing cloud-based processing, the inabilities to evaluate data acquisition on the spot could be minimized. b. Implementation of SfM workflow usually needs high computation and processing ability. Thus, using desktop computer as server in cloud processing pipeline has advantages of both processing power and ease of use. c. Applicability for the use of the system for non-technical users. Cloud processing could be used to develop a user friendly web application frontend. This can be used for less-advanced target user who wants to develop 3D models with ease.
A pipeline for development of cloud processed SfM has been conducted in the following manner:
48
Start
Literature Review
Server Preparation
Dataset Acquisition
Total Station Sample Points
Web frontend-backend Programming
Images from camera Dataset Preparation (image selection)
A
B
Agisoft Photoscan
CloudSfM webservice
C
D
SfM pipeline Programming
Successful
Web Framework Run Testing
Failed
49
A
C
D
Agisoft Processing
CloudSfM Processing
Failed
B
Successful reconstruction ?
Successful
Agisoft 3D Models
CloudSfM 3D Models
Cloud-to-cloud comparison
Ground truth comparison
Analysis of results
Recording and reporting
End
Figure II-1: General Flowchart of Thesis Development
Dataset Acquisition and Preparation Prior to the development of the CloudSfM, a literature review was conducted to build understanding on the two major components of CloudSfM, which is Global Structure from Motion and MEAN web framework development. Activities in this step including an literature study of the previous research, state-of-the-art methods for Structure from Motion algorithm, pros and cons of currently available SfM software and web service, as well as survey on the web framework which could be used for the CloudSfM system. The result of this step is elaborated in Chapter I. 50
Based on the literature review, some open source libraries were chosen for SfM pipeline used in this thesis. Each of the software has different functionalities in the SfM pipeline to generate 3D textured model from a set of images. The SfM pipeline were developed similar to those of Rossi et al. (2012). The SfM pipeline used in this thesis were open source and represent state-of-the-art progress in Structure from Motion algorithm with Global method. The next step was to acquire dataset which would be used for this thesis. Three datasets with different characteristics were obtained to be used with the CloudSfM. The datasets were obtained using mobile handheld device and consist of different number of images (See II.1.1). Afterwards, each of the dataset are filtered to remove blurry or out of focus images from dataset collection. This filtering process was done to ensure that the images which would be used for SfM reconstruction using CloudSfM should provide as much identified points as possible. However, the CloudSfM were built with user interactivity to remove unwanted images before processing (See II.2.4 for more details). Below are overview of each dataset, respectively: a. Naga Dataset The Naga dataset is series of images obtained using the mobile handheld camera (8 Megapixels) on a small Javanese dragon statue (~ 30cm x 42cm x 10cm). Using the mobile device, 81 images are acquired and used to assess the quality of SfM reconstruction on handheld camera.
51
Figure II-2: Excerpt from Naga Dataset
b. Tree Dataset The tree dataset, situated in front of Department of Geodetic Engineering building, consist of 91 initial images obtained using mobile handheld camera. Targets were applied to the objects to aid in 3D reconstruction process. Figure II-3 shows excerpt from Tree Dataset used in cloud-to-cloud comparison.
Figure II-3: Excerpt from Tree Dataset
52
c. UGM Main Building Dataset
Figure II-4: Excerpt from UGM Dataset
The southern part of Central Office Building of Gadjah Mada University was used as testing dataset for ground truth measurement. The dataset consists of 66 overlapping images of the southern part of the building obtained using the handheld device. Figure II-5 depicts configuration of data acquisition at UGM Central Building, with each photo were taken at approximately 50 meters to the object.
Directions of photo
Side of building to reconstruct
Figure II-5: Diagrams of data acquisition at UGM Central Office Building
The ground truth measurement in UGM Central Building were conducted using Leica Reflectorless Total Station, sampled 37 points 53
from the building, of which 5 points were used as control point for evaluation purposes. Table II-3 shows coordinates of each control points obtained by means of Total Station measurement.
Table II-3: Measured sample points in UGM dataset obtained using TS*
Pt. No
Xts
Yts
Zts
Pt. No
Xts
Yts
Zts
1
-40.7139
-16.4868
10.4644
20
27.3583
-24.9429
5.34614
2
-37.0597
-16.9348
10.4639
21
-8.49458
-20.475
5.34423
3
-33.4895
-17.3774
10.4598
22
-12.0713
-20.0241
5.35994
4
-29.9264
-17.8158
10.4583
23
-15.6422
-19.5898
5.33818
5
-26.3568
-18.2602
10.4595
24
-19.2133
-19.1368
5.34162
6
-22.7851
-18.7033
10.4588
25
-22.7761
-18.693
5.34566
7
-19.2204
-19.1517
10.452
26
-26.3345
-18.247
5.35106
8
-15.6592
-19.5987
10.4392
27
-29.9007
-17.8123
5.36542
9
-12.0759
-20.0522
10.4516
29
-37.0272
-16.92
5.3483
10
-8.51626
-20.4847
10.4457
30
-41.4942
-12.0199
8.91267
11
-2.64643
-21.4168
13.069
31
-41.0489
-8.47431
8.93858
12
-1.03974
-21.4514
12.964
32
-40.6062
-4.90847
8.94401
13
-2.61082
-21.4417
9.56334
33
-40.2041
-4.85332
3.85764
14
-0.02546
-21.7618
9.5624
34
-2.60275
-21.4227
1.16148
15
-1.04113
-21.5288
6.41252
35
-2.60473
-21.4355
2.63317
16
-0.01315
-21.7465
6.44151
36
-0.03813
-21.7506
1.16851
17
2.11675
-16.2981
7.85729
37
-0.03157
-21.762
4.24468
18
23.7888
-24.5266
10.4396
19
14.6117
-15.0431
3.26487
* Point #28 was measured, but was excluded from the dataset due to obstruction
54
Web Framework Programming The web framework is the main environment for frontend web interface and backend SfM workflow. This thesis use MEAN (MongoDB, ExpressJS, AngularJS and NodeJS) for the web framework as elaborated in Chapter II. This stage of the research can be explained as follow: II.2.2.1. Design of CloudSfM application flowchart The first step of developing CloudSfM web application was to design the application workflow. The software’s workflow is shown in Figure III-6.
Start Landing page
Login page
N o
Logged in?
Y o Upload Page
Documentation
Project definition
Display Help
B
55
B
Image selection
Images
SfM Setup
Project Manager Page
Global SfM Finished? Dense Reconstruction
N o
Y o Mesh, textured OBJ
Mesh reconstruction
Textured surface Download page
End
Figure II-6: Flowchart of CloudSfM Architectural Design
The next step is to implement the flowchart into software. For the User Interface (UI), Bootstrap and AngularJS framework were used in order to implement Model-View-Controller (MVC) framework used in the flowchart. ExpressJS were employed as middleware to handle file upload to a MongoDB database collection, while NodeJS was used to handle the server operation.
56
II.2.2.2. Server preparation After the flowchart and its corresponding software listed and prepared, the next step was to construct the basic environment for the server machine. The server preparation consists of the following step: a) Setup operating system and dependencies LinuxMint were chosen as server machine for its open source nature and sustainable support. LinuxMint are based on Ubuntu, and have wide support for various libraries. Some additional setup were done in order to prepare Linux Mint for developing a web framework. This include basic updating/upgrading and the installation of code editor software. The command line as shown below were run through Linux Mint’s Terminal, which expressed as a dollar symbol ($): $ sudo apt-get update $ sudo apt-get upgrade
At this stage, the server are ready for next step, which is to setup MEAN framework.
b) Setting up NodeJS NodeJS installation can be done using several different ways. This thesis use the latest version of NodeJS and npm (Node Package Manager) from the official release, i.e. in Ubuntu repository. Below were the commands executed in order to install NodeJS and npm into LinuxMint: $ curl -sL https://deb.nodesource.com/setup | sudo bash $ sudo apt-get install -y nodejs $ sudo apt-get install -y build-essential
57
Commands above installs NodeJS and npm along with its dependencies. In order to be able to use Yeoman seamlessly with the current npm, the ownership of ~/.npm folder needs to be changed to enable writing operations for installing new package. Also, a new system variable, NODE_PATH needs to be created so that it can be used with Grunt and Bower inside Yeoman package. $ sudo chown -R $(whoami) ~/.npm $ export NODE_PATH=$HOME/.node/lib/node_modules/
Next step of the flowchart is to install MongoDB as the core database management system.
c) Installing MongoDB MongoDB installation were executed using commands as follow: $
sudo
apt-key
adv
--recv
7F0CEB10
--keyserver
hkp://keyserver.ubuntu.com:80
The above command was to import public keyring of MongoDB into the LinuxMint’s package management system $
echo
"deb
http://repo.mongodb.org/apt/ubuntu
"$(lsb_release -sc)"/mongodb-org/3.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list
Next was to create mongodb list using echo command as above. The list were used by the system to identify if a libraries exist within the system $ sudo apt-get update
58
As the local list were added, the system needs to be updated in order to apply the changes. After the update, MongoDB can finally be installed using $ sudo apt-get install -y mongodb-org
which will install the latest stable version of MongoDB into the system. The database service, namely ‘mongod’ needs to be started to run in the background, allowing CRUD (Create, Read, Update, Delete) operation to be executed, for example by the CloudSfM app later.
d) Installing Yeoman and Yeoman Angular-Fullstack Yeoman is a web app scaffolding tool which simplifies the development of a web application through ‘Yeoman workflow’. Yeoman consist of build system (such as Grunt or Gulp) and a package manager (such as Bower), both of which is an important tool to create a web application for development and production step.
Figure II-7: The Yeoman Workflow (http://yeoman.io)
Yeoman and its dependencies were installed using this command: $ npm install -g yo grunt-cli bower
The installation were set to use Grunt as the build system and Bower as the package manager. With these setup finished, any new package dependencies of the CloudSfM will be managed by Bower. 59
Once Yeoman installed, the generation of MEAN framework scaffolding itself will be handled by Yeoman Generator called AngularFullstack. Angular-Fullstack is a Yeoman Generator to scaffold MEAN web framework. This generator were installed using: $ npm install -g generator-angular-fullstack
Next, the CloudSfM scaffoldings could be created using $ mkdir cloudsfM && cd $_ $ yo angular-fullstack CloudSfM
Commands above will scaffold basic MEAN framework with predefined folder structure as follow:
Figure II-8: Scaffoldings of CloudSfM web application
The folder structure and files were automatically generated by Angularfullstack. Among them are: -
Server : this folder is used to save all backend api configuration. Including Authorization and MongoDB setup
-
Client : this folder store all configuration of frontend application, including Angular User Interface and controller.
-
e2e
: this folder contains script file used for Karma Testing
60
-
Gruntfile.js: this is the file for building the production and development version of the application using Grunt
-
package.json and bower.json: these files list all the installed packages and dependencies in CloudSfM. Full listing of package.json and bower.json can be found in Appendix VI.1.
At
this
point,
a
local
server
was
made
available
at
http://localhost:9000/, which is the default port for Grunt. The output of Angular-fullstack is a basic template for MEAN framework, including settings for Bootstrap-AngularJS and ExpressJS, Passport libraries for Authentification, and SocketIO for realtime, bidirectional communication between client and server. This is a basic template which needs to be modified for the purpose of developing CloudSfM by means of setting frontend and backend service. Additional dependencies were needed to serve specific purposes, such as middleware for handling file uploading, generating uuid, handling received file, and so on. These additional dependencies should be installed beforehand.
e) Installing Angular-File-Upload and other dependencies Angular-File-Upload is an Angular directive for handling file upload using HTML5. This libraries were needed to handle image upload prior to SfM processing. The installation were conducted using Bower, and executed using the command, $ bower install ng-file-upload --save $ bower install ng-file-upload-shim --save
The install parameter --save was used to record the installation command
into
bower.json
and
list
Angular-file-upload
as
dependencies for CloudSfM.
61
Other dependencies, such as Multer, were needed for the SfM webservice. Multer is an ExpressJS middleware to handle ‘multipart/form-data’ sent by upload controller. Multer was installed using the command: $ npm install multer --save
The same command was used to install Express-zip (middleware for generating zip files); shelljs (to execute bash command inside NodeJS); and present (to calculate script’s running time). All the dependencies were included either in package.json file or referenced directly inside each controller. As the preparation for dependencies were finished, the next step is to develop the web framework for CloudSfM, which consist of frontend and backend component.
II.2.2.3. Frontend-backend programming As the server environment ready, the next step was to setup the CloudSfM single page application to provide user interface as well as to conduct cloud processing of SfM reconstruction. The frontend serve as user interface (or ‘View’ in MVC framework), and can be generated using ‘route’ command in Angularfullstack generator. On the other hand, the backend is the ‘Model’ in terms of MVC framework, and can be generated using ‘endpoint’ command in Angular-fullstack generator. Both frontend and backend are communicated using a ‘Controller’ which redirect response and request to each other, respectively.
62
Upload Image
Request
Upload
(Controller)
Response
(Controller)
Angular-UI
MongoDB
(View)
(Model)
Backend
Frontend
Figure II-9: Implementation of MVC framework for image uploading in CloudSfM
The CloudSfM web framework programming can be elaborated as follow: a) ‘Main’ route and controller A ‘route’ in Angular-fullstack terms is the components that form a web frontend. ‘Main’ route is the landing page of the web application and the main user interface of CloudSfM. Command to generate ‘main’ route is as follow: $ yo angular-fullstack:route main [?] Where would you like to create this route? client/app/ [?] What will the url of your route be? /main
As
it
finished,
‘Main’
route
can
be
accessed
from
http://localhost:9000/main. The command generates files needed for a page implementing MVC framework, which are: client/app/main/main.js client/app/main/main.controller.js client/app/main/main.controller.spec.js client/app/main/main.html client/app/main/main.scss 63
main.js is the javascript file to provide controller and templating setup for ‘main’ route. main.controller.js is the controller of main route, which connect the view main.html and main.scss to the database. The main.controller.spec.js serves the same function as main.controller.js, only that it is run for testing purposes. The User Interface (UI) were developed using Bootstrap and AngularJS to create a responsive web application. The landing page UI consists of user login menu and some other menus to redirect user request to other page. The most important is the upload menu, which constructed by the code below: Excerpt from main.html
...
Upload Images ...
The form action defines user request to be redirected into ‘/upload’ route using http get method, which elaborated in II.2.2.3.b). Other button which have similar functionalities is Help button, which redirect user request to the help page. As for the login button which redirect user to login page, the HTML code were injected using AngularJS’ directive ng-include, as Angular-fullstack already provide the authorization route in ‘/signup’ and ‘/login’. The complete HTML page of main route can be referred in Appendix VI.1. 64
b) ‘Upload’ route and controller The ‘/upload’ page provide UI for selecting images prior to uploading them into server. The Angular-File-Upload directive were employed to handle file upload after user finished selecting images. This page consist of three part: a. Project definition and description The HTML form were used to obtain information on the project name and description. This information were in turn would be used for generating SfM reconstruction based on project name. b. Images review Images review were provided using AngularJS’ ngrepeat so that the user would be able to remove bad images (e.g. blurry or out-of-focus). When all the user has satisfied with the result they can proceed to upload the images into server for further processing. c. Images upload This part consist of a single button which has two functionalities: 1) upload the images to the server; 2) redirect user to ‘/manage’ route for processing SfM reconstruction based on the images. All of the UI were coded into the upload.html file, while the actions (such as Angular-File-Upload and http get redirection) were defined in the controller upload.controller.js file. An excerpt of upload.controller.js which shows upload image directive using Angular-File-Upload are as follow:
65
Excerpt from upload.controller.js ... $scope.upload = function(files) { $scope.numUploaded = 0; $scope.uploaded = true; if (files && files.length) { for (var i = 0; i < files.length; i++) { var file = files[i]; Upload.upload({ url: 'api/uploadImages', method: 'POST', fields: { projectName: $scope.projectName, projectDescription: $scope.projectDescription }, file: file, fileFormDataName: 'cloudsfm' }) ...
}).success(function (data, status, headers, config) { $scope.numUploaded += 1; if ($scope.numUploaded == files.length) { $window.location.href='/manage'; }}); }} ...
The request were sent using http post to server endpoint ‘/api/uploadImages’, which then save the images into the server. c) ‘uploadImages’ endpoint The ‘/api/uploadImages’ is an endpoint which function is to handle file upload request and save the project into a MongoDB database collection. This endpoint were created using Yeoman generator: $ yo angular-fullstack:endpoint uploadImage [?]
What
will
the
url
of
your
endpoint
be?
/api/uploadImages 66
This command generates files related to uploadImages endpoint: server/api/uploadImage/index.js server/api/uploadImage/uploadImage.spec.js server/api/uploadImage/uploadImage.controller.js server/api/uploadImage/uploadImage.model.js server/api/uploadImage/uploadImage.socket.js
This endpoint receive user’s POST request, and proceed to save the images to NodeJS server. Multer middleware were employed to redirect the user upload destination to folders based on its username and projectname. Below are some excerpt from index.js to handle routing from user request:
Excerpt from index.js (/api/uploadImage) ... var multer = require('multer'); var destination = multer({ dest:"./uploaded/", changeDest: function(dest, req, res) { var newDestination = dest + req.user.name + "/" + req.body.projectName + "/images/"; fs.mkdirsSync(newDestination, function(err) { console.log('Error creating directory ',err); }); return newDestination } }); router.post('/', auth.isAuthenticated(), destination, controller.postUpload); ...
Therefore,
when
the
user
requested
http://localhost:9000/api/uploadImages through Upload frontend, the request are accepted and redirected to Multer middleware and 67
controller.postUpload. For example, if the username is ‘user’ and projectname is ‘project’, the server’s response will be to save uploaded images in ‘/uploaded///images’. Thus, each user have unique storage for each of their project. The postUpload function is as follow: Excerpt from uploadImage.controller.js (/api/uploadImage) ... exports.postUpload = function(req, res) { var file = req.files; exports.projectname = req.body.projectName; console.log(file); //console.log(projectname); console.log(req.body) console.log("Images successfully retrieved"); delete req.body.date; var project = new UploadImage(_.merge({author:req.user._id}, req.body)); project.save(function(err, project) { if(err) { return handleError(res, err); } return res.json(201, project); console.log('project saved successfully'); }); } ...
The postUpload controller above save the project properties into MongoDB database. The database’s collection were configured using predefined MongoDB schema in uploadImage.model.js. The function loadRecent(cb) in uploadImage.model.js were used for loading project information to populate the in ‘/manage’ route.
68
Excerpt from uploadImage.model.js (/api/uploadImage) ... var UploadImageSchema = new Schema({ projectName: String, projectDescription: String, date: { type: Date, default: Date.now }, author: { type: Schema.Types.ObjectId, ref: 'User' } });
UploadImageSchema.statics = { loadRecent: function(cb) { this.find({}) .populate({path:'author', select: 'name'}) .sort('date') .exec(cb); } }; ...
d) ‘Manage’ route and controller The Manage page would be used to display information of the last active project by the user, thus limiting the ability to perform reconstrunction to one project for each user. The user cannot start other project before the current project finished running and producing SfM reconstruction. This frontend is where SfM reconstruction request sent to the user through http get request to ‘/api/startsfm’. The flowchart for this route is as follow:
69
request
response
‘Manage’ page
Check if SfM output exist
Exist?
Download point cloud
yes
Display download link
no
Display processing link
SfM reconstruction
Output of SfM reconstruction
Figure II-10: Flowchart of Manage Project Route
The manage.controller.js file handles the request-response as shown above. The http get request was coded as shown below:
70
Excerpt from manage.controller.js ... $scope.startsfm = function() { $scope.message = 'Processing Request...'; console.log('requesting pipelines'); var request = { method: 'POST', url: '/api/startsfms', data: { username: $scope.currentUsername, projectname: $scope.currentProject.projectName } }; $http(request).success(function(msg){ if (msg == 'Completed') { $scope.message = msg; $scope.downloadReady = true; } }).error(function(err){ console.log('Error occured!', err) }); }; ...
$scope.downloadReady is the angular scope variable which control if the SfM reconstruction process has finished or the download file is ready. To generate download link, manage.controller.js was coded with get request to the finished SfM reconstruction (dense point-cloud or textured model). The code is shown below:
71
Excerpt from manage.controller.js ... $scope.changedVal = function(theparam) { $scope.downloadUrl = '/api/startsfms/' + Auth.getCurrentUser()._id + '?download=' + theparam; $http({method:'GET', url:$scope.downloadUrl}).success(function(data,status,headers ,config){ console.log(status); }); }; $scope.getfile = function() { $window.open($scope.downloadUrl, '_blank'); } ...
The frontend-backend communication to serve SfM reconstruction result were provided using REST (REpresentational State Transfer) implemented using ExpressJS. The HTTP GET request, as shown in the above code, were implemented using user’s ID assigned by Passport middleware to download result of SfM reconstruction (Auth.getCurrentUser()). The request and response of each parameter are shown in Table II-4.
Table II-4: Description of server Request-Response in CloudSfM Request
Parameter
Response
Description Report of SfM reconstruction, including reported
/id:/download
'sfmreport'
SfMReconstruction_Report.html
accuracy
and
robustness from each
SfM
reconstruction view
(image
pairs)
72
Colorized dense point-cloud 'colorizedcloud'
robust_colorized.ply
from
draping
image texture to points in point cloud Mesh
and
textured surface. Zip
file
was
generated
by
Express 'textured'
out_textured.obj (zipped)
Zip
library
to
combine multiple texture and OBJ files for download purposes.
The REST communication were done between ‘manage’ route and ‘startsfm’ endpoint. This startsfm endpoint accept parameterized request from ‘manage’ route and return the result as downloadable content to be served in ‘manage’ route view. e) ‘StartSfM’ endpoint This endpoint’s functionalities is to receive project parameters and process SfM reconstruction accordingly. This endpoint use ShellJS dependencies to execute python script using bash script, which in turn run Structure from Motion (SfM) reconstruction libraries. Some of the most important part of this script is: a. Function to check the availability of SfM output in user dedicated folder. If available, this function return a message saying that the file exist.
73
b. Function to run SfM reconstruction. This function return SfM output and report and message saying that the reconstruction process finished and download is ready. c. Function to return download links for each SfM output (i.e. dense point cloud and textured model) based on user’s ID when the file is available.
A function were employed to check whether the SfM reconstruction run successfully and producing point-cloud or not. The function asynchronously check the availability of reconstructed point cloud in server. Once the SfM reconstruction finished, the function returns a message to ‘manage’ route view that the files are ready to be downloaded. Excerpt from startsfm.controller.js ... exports.check = function(files) { // Check to see if the process has finished var message = ''; try { var stats = fs.lstatSync(files); if (stats.isFile()) { console.log('File found!'); message='Completed'; } } catch(err) { console.log(err); console.log('File Not Found'); message='Ready to Start'; } return message; } ...
This endpoint use Express-zip dependencies to create zip file of textured model resulted from SfM reconstruction. The detail for SfM reconstruction pipeline will be explained in the ensuing subchapter.
74
f) ‘Feedback’ and ‘Help’ route The Feedback UI is used to obtain user input for CloudSfM, while the Help UI is to provide documentation and useful resources on CloudSfM. Feedback frontend obtain user comments and save the comments to a MongoDB collection. Only user with administrator rights would be able to retrieve the user comments.
Installation of SfM libraries The web framework as explained in previous step was used to provide User Interface (UI) for conducting SfM reconstruction as a web application. The core process in CloudSfM is the SfM reconstruction using various libraries to conduct sparse reconstruction, dense reconstruction, mesh generation, colorization and texturing, respectively. The libraries are part of OpenMVG (Moulon et al., 2013b), MVE (Fuhrmann et al., 2014), FSSR (Fuhrmann and Goesele, 2014) and MVSTexturing (Waechter et al., 2014).
Final Output
MVS-Texturing’s Libraries FSSR’s deliverables
FSSR’s Libraries MVE’s deliverables
MVE’s Libraries OpenMVG’s deliverables
OpenMVG’s Libraries Python Script (underlying framework)
Figure II-11: Structure of Server-side SfM Processing Libraries
75
Given that each of the software is an independent libraries which serves different aspects of Structure from Motion algorithm and have different input and output format, and underlying framework needs to be used to assure that each libraries works seamlessly. A script written in Python were developed to handle the task of integrating all the libraries. The libraries of openMVG, MVE, FSSR and MVS-Texturing sit on top of this script, which perform operation on the deliverables of each libraries (See Figure II-11). Prior to developing the Structure from Motion pipelines, these script and all the libraries should be installed to the server. The installation steps might differs when implemented in each server. Thus, CloudSfM is packaged as vagrantbox under Ubuntu Trusty 64 bit as its guest machine, which allows the user installs CloudSfM without the need to repeat these installation process. a. Installation of OpenMVG and its dependencies. OpenMVG depends on various libraries to perform SfM sparse and dense reconstruction. The installation were conducted using CMAKE on
OpenMVG
source
code
available
in
Github
(https://github.com/openMVG/openMVG.git). A bash shell script (openMVG_install.sh) were modified to perform this installation on the server machine (see Appendix VI.3). The install script worked with these steps as follow: a) Setting up working directories in Linux b) Installing dependencies c) Cloning OpenMVG Github repository d) Compiling OpenMVG using CMAKE e) Copying binary executables to system environment f) Perform testing g) Clean installation files
76
b. Installation of MVE, FSSR and MVS-T Installation of libraries from MVE, FSSR and MVS-T were done by cloning each of the libraries’ Github repository. The command to clone and install MVE using CMAKE were as follow: $ git clone https://github.com/simonfuhrmann/mve.git $ make -j8 -C mve $ make -C mve/apps/makescene $ make -C mve/apps/sfmrecon $ make -C mve/apps/dmrecon $ make -C mve/apps/scene2pset
Four libraries were created as result of the installation. Each of the library have their own roles in SfM pipeline. Next, FSSR installation were also conducted using Git and CMAKE. The installation of FSSR libraries were conducted as follow: $ git clone https://github.com/simonfuhrmann/fssr.git $ make -j8 -C fssr
FSSR installation produced two libraries for mesh generation and cleaning. The last step is to install MVS-T for the final step of SfM reconstruction. The installation of MVS-T in the server were done using Git and CMAKE as follow: $
git
clone
https://github.com/nmoehrle/mvs-
texturing.git texrecon $ cd texrecon && mkdir build && cd build $ cmake -DRESEARCH=ON .. $ make -j8
77
This installation produces libraries for texture mapping of a meshed surface. The use of all this libraries, as well as their role in SfM reconstruction, will be elaborated in Subchapter II.2.4.
SfM Pipeline Programming and Processing Python scripts were employed to provide a working pipeline for SfM reconstruction. The python scripts develop an integration of Global structure from motion pipeline, point-cloud densification, multi-view meshing and texture draping in order to seamlessly integrate various SfM libraries into the desired output. The python script run in the server-side of web application, so that the user will not deal directly with reconstruction pipeline. The script are available in Appendix VI.2.
II.2.4.1. Global SfM pipeline OpenMVG used in this thesis consist of several libraries to perform basic SfM reconstruction algorithm using Global or Sequential algorithm. The general workflow and command of OpenMVG’s implementation on SfM reconstruction are as follow (Moulon et al., 2012): a. Create list of images and camera parameters (if available). The openMVG_main_SfMInit_ImageListing library were used as initial process in OpenMVG reconstruction by listing all images and their respective intrinsic parameters. This command needs some parameter to produce a JSON file, sfm_data.json, which would be used for further processing in OpenMVG. The parameters needed were shown in Figure II-12.
78
Figure II-12: Parameters of OpenMVG_main_ImageListing
Figure
II-12
above
shows
that
the
openMVG_main_SfMInit_ImageListing command handles all the necessity for further SfM reconstruction, including list of images, camera sensor database, output directory, camera focus, calibration matrix (K matrix), camera model for SfM algorithm, and configuration for group camera model (whether all the images use the same camera or not). For its default settings, CloudSfM uses estimated focal camera based on the following formula (Moulon et al., 2013a):
Focal𝑝𝑖𝑥 =
𝑚𝑎𝑥(𝑤pix ,ℎpix )∗focalmm ccdwmm
(12)
where: focal𝑝𝑖𝑥
: Focal length from EXIF data in pixel
focal𝑚𝑚
: Focal length in mm
𝑤pix , ℎpix
: Image size in pixel (width*height)
ccdwmm
: Sensor width size (CCD) of the camera
In this thesis, the handheld devices used is Samsung Galaxy Grand 2 duos with 8 Megapixel camera. In order to retrieve the properties of the camera, an Android application, namely Phone Tester, were employed. 79
Figure II-13: Samsung Galaxy Grand 2 Camera Specification
According to the camera properties (See Figure II-13), the focal length of Samsung Galaxy Grand 2 camera is 2.93 mm, the image size (width and length) is 3264x2448 and the sensor size is approximately 3x2.3 mm. Thus, according to the formula above, the focal length of Samsung Galaxy Grand 2 used in this thesis can be calculated as:
Focal in pixel = (2.93 (Focal in mm) * 3264 (image width))/ 3 (sensor width) = 3187.84 pixel
As for camera with unknown internal, the focal length were iteratively recomputed into new approximate value based on known image size and/or existing EXIF data. Also, for advanced users, all of the parameter were available to be modified in CloudSfM’s source code.
80
b. Detect features in each images using preselected feature detection algorithm. The library used for this step is openMVG_main_ComputeFeatures. Its main purpose was to compute features in each images using feature detection algorithm. The feature detection algorithm of CloudSfM uses DG Lowe’s SIFT (Lowe, 1999) as the default. However, advanced options are available in case the user needs to modify the settings. Also, the user can choose the feature detection level, whether it would be Normal (the default setting), High or Ultra. The higher the setting, the more points can be detected by openMVG. Default CloudSfM settings uses ‘High’ for features detection, since Global SfM algorithm need denser point cloud to conduct SfM reconstruction (Moulon et al., 2013a).
Figure II-14: Parameters of OpenMVG_main_ComputeFeatures
c. Find best matches between pairs of corresponding images. Based on detected features, the next step is to perform automatic images matching. This step defines the estimated camera motion to be used for SfM (sparse) reconstruction. Users are able to choose geometrical model for feature matching (i.e. using fundamental matrix, essential matrix or 81
homography matrix), or define number of matching images if the data is sequential in nature (such as extracted video frame). The deliverables of this command is a folder containing a pair of matched images.
Figure II-15: Parameters of OpenMVG_main_ComputeMatches
d. Solve SfM reconstruction OpenMVG can perform SfM reconstruction using two algorithms: Global and Sequential SfM method. CloudSfM adapted Global SfM reconstruction algorithm for its faster implementation which shows better capabilities to handle images compared to the classic Sequential/Incremental method (Moulon et al., 2013a). However, the user could modify the source code of CloudSfM if they prefer using sequential method over Global method. The openMVG_main_GlobalSfM command uses matched pairs of images from the previous steps. However, in order to run the Global SfM reconstruction properly, the openMVG_main_ComputeFeatures should use
at
least
High
setting
for
feature
detection.
Also,
the
openMVG_main_ComputeMatches needs to be run with parameter ‘-r 0.8’ which set the Nearest Neighbor distance ratio to use more features for the matching.
82
The output from this Global SfM reconstruction are 1) sparse point cloud, 2) dense point cloud, and 3) camera pose. These deliverables are presented as *.ply file, which can be opened using software such as Meshlab. e. Colorize point cloud, i.e. draping colors to each point in point cloud. After successfully creating reconstructed 3D structure and camera motion (i.e. camera pose), the next step is to colorize each point in the point cloud with texture from images. The result is a colorized point cloud, which shows estimated objects with their texture. OpenMVG’s command to run this feature is openMVG_main_ComputeSfM_DataColor. f. Recompute structure from known motion (camera position).
This
command is originally used when camera motion is known prior to SfM reconstruction. However, this step can also means evaluating the previous SfM result by calculating Standard Deviation of the produced camera pose against camera pose from the result of next iteration. The command used for recomputing structure from known camera pose in CloudSfM is openMVG_main_ComputeStructureFromKnownPoses. command
produces
another
point
As
cloud,
this the
openMVG_main_ComputeSfM_DataColor needs to be called again to produce colorized point cloud, ‘colorized.ply’.
All of this command were programmed into the previously mentioned Python script to be run in the server. The script translates default and advanced parameters from the client and setup the SfM reconstruction accordingly. The script were called by CloudSfM’s server endpoint using bash script as an interpreter so that the command runs in the background of the web interface.
II.2.4.2. Multiview Stereo Texturing pipeline The previous step using OpenMVG only produces colorized, dense point cloud. Some other libraries need to be employed to conduct mesh generation and
83
texture draping over the surface. This part of CloudSfM can be elaborated as follow: a. Import scene and camera parameters, reconstruct SfM in MVE format ‘makescene’ is the command used for converting openMVG structure and camera parameter from the previous step into MVE scene. This command is part of MVE library which produces a folder, called ‘/scene’ to store MVE’s view (the name given in MVE for corresponding pairs of images). Next, MVE also needs to perform its own SfM reconstruction against the converted data using the ‘sfmrecon’ library. b. Depth map generation and conversion to point set Before developing surface reconstruction of the scene, a depth map needs to be created using MVE view from previous step. The library ‘dmrecon’ is used for generating depthmap raster images for surface draping. The next library, ‘scene2pset’ is used to convert the depth map into a set of point attributed with the depth information as a PLY file. The algorithm were derived from (Fuhrmann et al., 2014), and are provided by MVE. c. Floating-scale surface reconstruction and mesh cleaning The previous step only produced a depth map and point sets associated with the depth map. The surface reconstruction itself were handled by other libraries available in FSSR (Fuhrmann and Goesele, 2014). The command to perform surface reconstruction using point set and depth map information is ‘fssrecon’. The output of this command is a PLY surface interpolated from point sets of the scene using Floating Scale Surface Reconstruction algorithm. Afterwards, a mesh cleaning algorithm needs to be employed to clean the surface produced from any noises from depth map or point set generation. The command used were ‘meshclean’, which produced PLY files, surfaceclean.ply, as a cleaned surface from the FSSR surface generation. d. Texturing The last step of CloudSfM reconstruction pipeline was to perform texture mapping to the cleaned surface. The library from MVS-Texturing 84
(Waechter et al., 2014) were used to drape texture derived from input images into the surface. The command used is ‘texrecon’, which produces the final product of CloudSfM, i.e. textured 3D model in OBJ format and its texture map.
Same as the SfM reconstruction before, this multiview and texturing workflow were implemented using Python script which will be called by server endpoint. The Python script were called using bash script so that it will run in the server machine. All of the scripts are available in Appendix VI.1.
Application Deployment With Angular-fullstack, the deployment of the software was done using Grunt. Grunt is a Javascript runner which automatize some of javascript task. The command to serve CloudSfM into the local server (i.e. localhost) were: $ grunt serve
The Grunt command will run several task of serving javascript files in the server. Also, the command will test the application for error and shows which part of the script contains error, whether the erroneous script is in server side or client side of CloudSfM. When all the script has been successfully loaded, the CloudSfM web application will be shown in the default browser pointed to http://localhost:9000.
85
Figure II-16: Grunt serve command running in Linux Terminal
CloudSfM is deployed to Github, so that all of its source code are openly available to the developers or researchers. After setting up repositories and SSH keys, the deployment to Github were conducted using the basic steps of Github deployment: /cloudsfm $ git pull /cloudsfm $ git add --all /cloudsfm $ git commit –am “commit message” /cloudsfm $ git push
The steps were repeated each time the code of source SfM were changed. The code can be accessed through http://github.com/danylaksono/cloudsfm. Users who
86
want to install CloudSfM into their own machine will be able to clone the repositories, install dependencies and run CloudSfM in their server machine.
Evaluation of CloudSfM 3D Models II.2.6.1. Cloud-to-cloud Comparison Cloud-to-cloud comparison of 3D models resulted from CloudSfM were performed using CloudCompare, an open source software for comparing pointclouds. The datasets used for the comparison were Naga and Tree dataset. For the comparison, the same images from both the dataset were run through commercial software Agisoft Photoscan. The CloudSfM 3D models would be used as the ‘model’, while the Agisoft Photoscan’s 3D models would be used as the ‘reference’.
Figure II-17: Registration of Naga model to reference data
In order to obtain the cloud-to-cloud distance, these steps below were conducted:
87
a. Performed a cloud-to-cloud registration of both point-cloud. The model dataset obtained from CloudSfM were registered using similar points to the reference dataset. Figure II-17 shows the aligned model to the reference dataset using some tie points b. Performed Iterative Closest Point (ICP) algorithm to further align both of the dataset. The algorithm iterates the position of a point cloud so as to obtain minimum distances between both set of points and establish the position of the model data to the reference data (Figure II-18).
Figure II-18: Iterative Closest Point Algorithm in CloudCompare
c. As both the point-cloud were situated in its optimum position using ICP (i.e. minimum distances between the two point cloud), the function to calculate the distance were called. The function would calculate the distance of each single point in the point cloud to the nearest corresponding point in the reference dataset. Figure II-19 shows the final point cloud and statistics calculated from the differences between the datasets.
88
Figure II-19: Registered model and reference point cloud
II.2.6.2. Ground truth evaluation CloudCompare was used in order to register and obtain coordinates of sample points from 3D models of UGM Main building dataset run through CloudSfM. After the 3D models of UGM building and coordinates of sample point obtained from Total Station Measurement, these steps were taken to perform the ground truth evaluation: a. Register the 3D models of UGM building using CloudCompare. Five points were selected as reference points based on their distribution in the 3D models to register the model to (local) ground truth data.
89
Figure II-20: Registration of point cloud to ground truth data
b. After the point cloud registered to ground truth data, using five tie points as reference, coordinates of the other sample points in registered model were obtained using Pick point tool in CloudCompare.
Figure II-21: Pick point for ground truth evaluation
c. As two set of point coordinates were obtained, the last step was to conduct
statistical analysis of the data. Root Mean Square (RMS) Error were calculated using Microsoft Excel. The result of this test were provided in Chapter III.
90
III. RESULT AND DISCUSSION
III.1. Results The results of this thesis are a SaaS (Software as a Service) web-service which are capable of performing 3D reconstruction using Structure from Motion algorithm from series of images. Subchapters below discuss the results of this thesis, which answer the questions in Chapter I.4.
SfM Pipeline Using Open Source Libraries The SfM pipeline was constructed following the steps elaborated in Chapter II.2.4. The pipeline was used to perform backend SfM processing in CloudSfM architecture. A python script were employed to combine deliverables of OpenMVG, MVE, FSSR and MVS-Texturing (see Figure II-11) into a single pipeline to produce 3D textured models from images as input. Figure III-1 presented the constructed SfM pipeline consists of libraries from the open source software. The complete result of this script is presented in Appendix VI.2 (SfM_GlobalPipeline.py).
91
Initial Setup (Checking Project Directory)
Read Parameters (from CloudSfM Interface)
OpenMVG’s Libraries
openMVG_main_SfMInit_ImageListing
Image match directory
sfm_data.json
openMVG_main_ComputeFeatures
Image match directory (Update)
sfm_data.json (update)
openMVG_main_ComputeMatches
Image match directory (Update)
sfm_data.json (update)
openMVG_main_GlobalSfM
Point-cloud
sfm_data.bin
openMVG_main_ComputeStructureFromKnownPoses
Colorized Point-cloud
openMVG_main_ComputeSfM_DataColor
A
92
A
Format Conversion Sfm_data.bin
Colorized Point-cloud
MVE Scene Dir.
openMVG_main_openMVG2MVE2
MVE+FSSR
MVE Scene Dir.
dmrecon
Point-set
Depth Map
fssrecon
Scene2pset
Mesh Surface
meshclean
Cleaned Mesh Surface
MVS-Texturing Cleaned Mesh Surface
texrecon
Image Dir.
Texture+3D model
Update project setting (settings.json)
Finish and Serve Results Figure III-1: Diagram of SfM Pipeline Constructed Using Python Script
93
The SfM pipeline run in the background (i.e. backend processing) when the corresponding command from CloudSfM is called. User will not be able to see the whole data processing activities, as the process happends in the server side (see Figure III-2 and Appendix VI.5). However, user would be able to download a log file consist of the whole background processing activities (Appendix VI.4).
Figure III-2: CloudSfM Running SfM Pipeline in the Background
The backend processing resulted in some files that would be presented to the user through CloudSfM interface. The deliverables of the pipeline are as follow: a. Colorized point-cloud and reconstructed camera position (PLY format) b. Textured 3D Model (OBJ format, delivered as a Zip file. See Figure III-3) c. SfM Reconstruction report (HTML format) d. SfM pipeline log file (TXT format)
94
Figure III-3: Content of zip file containing 3D textured model from served in CloudSfM
The CloudSfM Webservice The CloudSfM application consist of web User Interface (UI) which present a Single Page Application (SPA) needed to collect user input as client side application and a server side processing application for SfM reconstruction. The thin-client architecture were developed to ease the use of Global Structure from Motion algorithm to be used as an online services, especially using the combination of OpenMVG, MVE, FSSR and MVS-Texturing. The deliverables of this thesis are CloudSfM, an online SfM webservice and its source code which available as an opensource application with GPL license. This subchapter discuss about the result of CloudSfM: deployment of the system and the user interface of CloudSfM.
III.1.2.1. Deployment and Installation of CloudSfM CloudSfM is deployed as cloud service using Vagrant, a virtualization machine wrapper which work
on VirtualBox. The vagrant-wrapped (so-called
‘vagrantbox’) CloudSfM were built and can be used as a ready-to-use package containing all functionalition of CloudSfM. In order to start a virtualization environment and install CloudSfM Vagrantbox in local machine, the users need to follow the procedure below: 1. Clone the CloudSfM repository (http://github.com/danylaksono/cloudsfm) 2. Start the virtualization machine using command: vagrant up 95
3. Log in to the machine using command: vagrant ssh Afterwards, the machine will boot and perform installation of CloudSfM dependencies in the local machine. The CloudSfM webservice can be accessed from host machine at http://localhost:9000. CloudSfM were also developed as open source software which can be built manually
from
its
source
code.
The
CloudSfM
Github
repository
(http://github.com/danylaksono/cloudsfm) contains all the source code needed to build CloudSfM into a local machine (Figure III-4). However, the vagrantbox version of CloudSfM provides user with ready to use virtual machine running CloudSfM, while this installation from source was intended for developers who want to customize their own version of CloudSfM.
Figure III-4: Github Repository of CloudSfM
III.1.2.2. CloudSfM User Interface Below some screenshot of the CloudSfM Web Interface are presented. The user interface follows the architectural design of CloudSfM shown in Figure II-6. Some of the user interface shown below is the Landing page, User Authentication page (login and signup), Dashboard, New Project and Project Manager page. 96
a. CloudSfM Landing Page The landing page is the first page that shows up when a user load the URL of CloudSfM (i.e. http://localhost:9000 or a user’s respective public domain). The Landing Page is a simple welcoming page consisting only small information regarding CloudSfM and a button to proceed to the Dashboard.
Figure III-5: CloudSfM Landing page
b. Dashboard and User Authorization Interface CloudSfM uses PassportJS middleware for request authentication. As the user proceed to Dashboard without identified login authentication, the Passport middleware would redirect the request to a User Authentication Page. The user needs to login or register (if the user have not registered previously) to use the service. Social media account (i.e. Google, Facebook or Twitter account) can also be used to login to the service.
97
Figure III-6: Authorization page: register and login page
PassportJS assigns unique ID for each user, thus every user have their private account for SfM reconstruction. Only registered user would have access to the Dashboard (Figure III-7), which allows the user to manage their existing projects, create new SfM projects or read the documentation of CloudSfM. The user ID, username, and their projects were stored in a MongoDB database collection, so that the user could retrieve their last project (only one project can be run by the same user at a time).
Figure III-7: Dashboard, or main menu of CloudSfM
98
Each user in CloudSfM have specific privilege. Only user with administrative privilege are allowed to modify other user’s account, including to remove a user or read all comments by other user. At this time, CloudSfM uses Passport local authentication, which means that users is not required to use real email when using CloudSfM services. c. ‘New Project’ Page
Figure III-8: User Interface to create a new project
When a registered user clicks on New Project button in Dashboard, it will redirect the user to frontend which allows the definition of a project. This page has three main functionalities as follow: i. Define a new project The user interface (Figure III-8) consist of forms to provide general settings for the project. Aside from defining project name and definition, users have options whether to use default settings (by leaving the ‘use advanced settings’ checkmark unchecked) or to use customized settings as shown in Figure III-9.
99
Figure III-9: Customized SfM settings in CloudSfM
ii. Review the images Before finally uploads images to the server, the user have option to review each of the image. For example, if an image contains out of focus object or blur, the user could remove the images and excluded it from SfM processing in the server (See Figure III-10).
Figure III-10: Options to review images before uploading
100
iii. Upload the image to server The last menu of the page is a confirmation to upload the selected images to server. When the user have set up all the parameters and images used for SfM reconstruction, the button in this page redirects a command to save all the settings into the MongoDB database along with the user information, as well as the images used for SfM reconstruction server-side.
d. ‘Manage Project’ page The Manage Project page is where the SfM reconstruction being processed. This page retrieve list of projects the user have been processed previously from MongoDB Database and display the project info is a project is selected. The Add Project button redirect the user to New Project Page, which allows user to create a new project.
Figure III-11: Project Manager
The Project info consist of three tabs, which display information about each project related to a user in MongoDB database. Basic info displays information whether the data can be processed in the server, or has been processed previously. 101
Also, Delete Project button is available when a user wants to delete the project and all of its attributes (including images saved in the server). The Start Processing button starts SfM reconstruction to a previously unprocessed images saved in the database. Details tab provides information on the SfM reconstruction parameters used for the SfM process. During the SfM processing, the user could close the page, and only need to login to display the Manage Project page with the updated status. As the SfM processing finished, the manage project page display download links for the output of SfM reconstruction. The user could choose to download the deliverables of CloudSfM, which is a 1) colorized point cloud and camera pose, and 2) Textured 3D model. An option to display SfM accuracy report is also available.
Evaluation of SfM reconstruction III.1.3.1. Result of CloudSfM reconstruction The result of CloudSfM reconstruction was a set of textured 3D models for each dataset used in this thesis, i.e. Naga dataset, Tree Dataset and UGM Dataset. The 3D models resulted from CloudSfM project was essentially the deliverables of OpenMVG, MVE, FSSR and MVS-Texturing pipeline used in the server side of CloudSfM. Meshlab was used to perform visual evaluation of each model. Figure III-12 shows the 3D models resulted from three different CloudSfM project. Figure III-12.a and Figure III-12.b shows the result of 3D reconstruction using Naga and Tree dataset, respectively. Both the result were visually identical to the object in question, and possessed little to no noise in the images. A different result can be observed in Figure III-12.c which shows the result of 3D reconstruction on UGM Main Office Building dataset. Visual inspection of the 3D model showed that the reconstruction failed to identify both wings of the building. Only western part of the building have been reconstructed from the images, although the shape of the building is still recognizable.
102
a. 3D Model of Naga dataset
b. 3D Model of Tree dataset
c. 3D Model of UGM dataset
Figure III-12: Final 3D Models of the datasets obtained from CloudSfM Projects
The cause of the error in UGM dataset was the similarity between both wings of the building, which resulted in ambiguity error during feature recognition step of CloudSfM reconstruction. Details on this error will be reviewed in Chapter III.1.3.2.
III.1.3.2. Result of Evaluation The result of CloudSfM reconstruction were evaluated using three kinds of evaluation: Internal evaluation, which evaluates the internal reliability of each 103
reconstruction project; and the external comparison, consist of cloud-to-cloud comparison, which employs Iterative Closest-Point (ICP) algorithm to assess two point clouds from commercial software and CloudSfM and Ground-truth evaluation, which was performed by comparing coordinates of sample points obtained from Total Station measurement and coordinates extracted from the result of CloudSfM reconstruction. The internal evaluation were performed on the three datasets, while cloud-tocloud evaluation were performed on the Naga and Tree dataset using 3D models from CloudSfM and 3D models obtained from Agisoft Photoscan. The cloud-tocloud test wasn’t performed on UGM building dataset since the reconstruction process conducted on the dataset using Agisoft Photoscan caused the program to crash and stop running. The last test, ground truth measurement, was conducted on UGM building by extraction point coordinates from the 3D model to be compared with Total Station measurement of the same point. a. Internal Evaluation The internal evaluation were performed in order to assess the geometric configuration of Global Structure from Motion Algorithm on the dataset (Moulon et al., 2013a). The internal evaluation consist of observation on the residuals and outlier test and the observation of RMSE (Root Mean Square Error) value in the dataset. Another means of SfM geometric configuration analysis is to observe the visibility graph, which analyze the connectivity between the images. Table III-1: RMS Error of Global SfM reconstruction on each dataset
Dataset Naga
No. of reconstructed camera pose 81 from 81
RMSE (meter, unscaled) 0.90439
Tree
91 from 91
0.607263
UGM Building
67 from 68
0.845978
The small value of RMS Error indicates the geometrical closeness between the observed coordinates of points in all the iterations of views (i.e. each of the estimated camera pose), and thus indicates consistency of the 104
observation. However, the value did not necessarily means that the reconstructed structure were accurate. An evaluation of residual value need to be conducted in order to give better understanding of the scene.
Observations
a. Naga Dataset
Residuals
Observations
b. Tree Dataset
Residuals
Observations
c. UGM Building dataset
Residuals
Figure III-13: Residuals histogram of the three datasets
Figure III-13 shows the histogram plot of residuals in each dataset using SfM reconstruction. It can be observed that while the RMS Error (in relative point-cloud unit) yields relatively small number (i.e. 0.845978), the UGM dataset have large numbers of maximum residual value (16.0597) compared to that of Naga dataset (5.44819) and Tree dataset (3.99226).
105
This indicates the existence of outliers and erroneous computation of camera pose in the scene. The Appendix VI.4 shows detail of each dataset’s Global SfM reconstruction result, as well as value of residuals in each observation.
Figure III-14: Camera Pose Graph of Naga Dataset
The erroneous observation can also be detected from the camera pose and view graph. The graph expressed how an images is connected or sharing views with other images, and is the result of SfM reconstruction aside from the 3D models (i.e. the camera ‘motion’). Figure III-14, Figure III-15 and Figure III-16 shows camera pose or visibility graph of Naga dataset, Tree dataset and UGM Building dataset, respectively. The visibility graph of Naga and Tree dataset shows the connected views as expected. The connected images (shown in the graph as nodes) were consistent with the relative position of the camera during data acquisition.
106
Figure III-15: Camera Pose Graph of Tree Dataset
Group of images capturing eastern wing of UGM building
Group of images capturing western wing of UGM building
Figure III-16: Camera Pose Graph of UGM Building Dataset
107
Meanwhile, the visibility graph for UGM Building datasets in Figure III-16 shows inconsistencies with the camera position during data acquisition. The graph shows two groups connected to each other, while in reality each of the group is collection of images capturing different wings of the building. It is possible that the images were connected during feature detection due to the similarity of the feature. However, according to Moulon et al. (2013a), the utilization of Global SfM should be able to handle reconstruction of objects with similar façade. Thus, a more detailed test on CloudSfM’s algorithm needs to be conducted in future researches.
b. Cloud-to-cloud evaluation Cloud-to-cloud evaluation were conducted to both the Naga and Tree dataset. The evaluation were conducted using CloudCompare on the two pairs of datasets: 3D models from Agisoft Photoscan and 3D models resulted from CloudSfM.
Figure III-17: Cloud-to-cloud evaluation on Naga Dataset
The result of cloud-to-cloud comparison of Naga dataset are shown in Figure III-17. The color represent the distance of each point in ‘model’ point cloud (Naga 3D model from CloudSfM) to the closest point in 108
‘reference’ point cloud (Naga 3D model from Agisoft Photoscan). The color space used to represent the point cloud distance is blue-green-yellowred, where blue color indicates the closest distance (close to zero or identical point cloud’s position), and vice versa. The same result for Tree dataset is shown in Figure III-18. It can be concluded that both the tested datasets (Naga and Tree dataset from CloudSfM projects) and the reference datasets (Naga and Tree dataset from Agisoft projects) were geometrically similar.
Figure III-18: Cloud-to-cloud evaluation on Tree Dataset
The result of cloud-to-cloud comparison are shown in Table III-2. The cloud-to-cloud comparison conducted using CloudCompare calculates distances between the tested dataset and the reference dataset. Among the value obtained from the test are Mean Distance and Standard Deviation, which express the closeness of both model and reference dataset. The value for mean distances (in relative point cloud unit) of Naga dataset and Tree dataset are 0.028808 and 0.176421 in relative point cloud unit, respectively. This result indicates that the 3D models (from CloudSfM and Agisoft Photoscan) are close to each other. Another indication that the results from CloudSfM perform on par with the result from Agisoft Photoscan is the
109
value of their standard deviations, which yield 0.019818 and 0.350701 relative point cloud unit for Naga dataset and Tree dataset, respectively. Table III-2: Result of Cloud-to-Cloud Comparison
Mean Distance
Standard Deviation
(meter, unscaled)
(meter, unscaled)
Naga Dataset
0.028808
0.019818
Tree Dataset
0.176421
0.350701
c. Ground-truth evaluation Ground-truth evaluation were conducted by comparing set of coordinates measured using Total Station and coordinates retrieved from the 3D models of a CloudSfM project using CloudCompare. Table III-3 shows both coordinates of groud measurement data and coordinates retrieved from the same point in CloudSfM. Points in bold were used as control point.
Table III-3: Ground Truth Evaluation (units are meter, scaled, not georeferenced) No
XTS
YTS
ZTS
XCloudSfM
YCloudSfM
1
-40.7139
-16.4868
10.46441
-40.7139
-16.4868
2
-37.0597
-16.9348
10.46386
-36.5453
3
-33.4895
-17.3774
10.45978
-33.115
4
-29.9264
-17.8158
10.45828
5
-26.3568
-18.2602
6
-22.7851
7
ZCloudSfM
RMSi
10.46441
0.00
-17.4467
10.7306
0.77
-17.8399
10.65884
0.63
-29.5962
-18.2895
10.42208
0.58
10.45945
-26.0941
-18.7216
10.28729
0.56
-18.7033
10.45877
-22.6077
-19.0821
9.994242
0.63
-19.2204
-19.1517
10.452
-19.2434
-19.3221
9.864934
0.61
8
-15.6592
-19.5987
10.43919
-15.716
-19.6849
9.635551
0.81
9
-12.0759
-20.0522
10.45163
-12.2442
-20.0084
9.511401
0.96
10
-8.51626
-20.4847
10.4457
-8.82484
-20.4065
9.287446
1.20
11
-2.64643
-21.4168
13.06902
-2.76517
-20.9244
11.41929
1.73
12
-1.03974
-21.4514
12.96405
-1.20931
-21.1066
11.60797
1.41
13
-2.61082
-21.4417
9.563336
-2.61082
-21.4417
9.563336
0.00
14
-0.02546
-21.7618
9.562398
-1.37014
-21.1549
7.944175
2.19
15
-1.04113
-21.5288
6.412518
-3.10734
-21.1827
4.961673
2.55
16
-0.01315
-21.7465
6.441513
-1.56416
-21.0627
4.854207
2.32
110
17
2.116753
-16.2981
7.857287
2.116753
-16.2981
7.857287
0.00
18
23.78883
-24.5266
10.43957
23.38862
-24.2561
7.610996
2.87
19
14.61174
-15.0431
3.264869
22.91587
-24.1584
2.462347
12.36
20
27.35834
-24.9429
5.346136
27.35834
-24.9429
5.346136
0.00
21
-8.49458
-20.475
5.344231
27.65028
-24.9218
2.454023
36.53
22
-12.0713
-20.0241
5.359937
-8.73816
-20.6074
4.441019
3.51
23
-15.6422
-19.5898
5.338176
-12.4661
-19.931
4.800025
3.24
24
-19.2133
-19.1368
5.341622
-15.9989
-19.5681
4.834555
3.28
25
-22.7761
-18.693
5.345658
-19.4977
-19.276
4.82535
3.37
26
-26.3345
-18.247
5.351059
-22.8565
-19.0222
5.064403
3.57
27
-29.9007
-17.8123
5.365424
-26.3992
-18.5732
5.207166
3.59
29
-37.0272
-16.92
5.348298
-29.8908
-18.2255
5.514187
7.26
30
-41.4942
-12.0199
8.912665
-40.6215
-16.6834
5.990805
5.57
31
-41.0489
-8.47431
8.938582
-41.1898
-12.3132
9.449731
3.88
32
-40.6062
-4.90847
8.944011
-40.7666
-8.98207
9.567879
4.12
33
-40.2041
-4.85332
3.85764
-40.2041
-4.85332
3.85764
0.00
34
-2.60275
-21.4227
1.161481
-3.13789
-20.9181
2.830894
1.82
35
-2.60473
-21.4355
2.633172
-3.19733
-21.0484
1.093124
1.69
36
-0.03813
-21.7506
1.168508
-1.58199
-21.022
1.049733
1.71
37
-0.03157
-21.762
4.244675
-1.54258
-21.1046
2.736701
2.23
The result of this comparison showed quite large value of Root Mean Square (RMS) Error. The calculated value are 6.4733 m for RMSX; 2.1322 m for RMSY; and 1.1879 m for RMSZ. Thus, the accumulated error or RMSTOTAL are 6.9182 m. The large error were obtained due to inaccuracies of the 3D models as discussed in III.2. The largest RMS Error were observed in point #19 and #21, as shown in Table III-3.
III.2. Discussion This thesis result is a web service, named CloudSfM, which is capable of solving Structure from Motion problems based on some overlapped images. The workflow of the CloudSfM web service are similar to that of other image-based online 3D reconstruction system such as Autodesk 123D Catch, which is consist of merely three main steps (Image uploads/selection > Processing > Downloads). The user interface of CloudSfM itself were built with the purpose of simplifying 111
the SfM pipeline which naturally needs Linux environment or customized settings which are not familiar to basic users. Future improvement and modification to the UI might be needed to optimize user experience with CloudSfM. The result of CloudSfM projects shows that the system was capable of producing detailed 3D models. The comparison of two datasets, i.e. Naga dataset and Tree dataset, processed using CloudSfM and a commercial software Agisoft Photoscan showed that both the models were similar to each other. However, the result ground truth measurement showed that the 3D model of UGM dataset possess large errors which were caused by similarities of the object in question. A more detailed test of the system and the algorithms used in CloudSfM needs to be conducted in future researches.
Figure III-19: Possible error due to inaccuracies of the 3D models in point #19 and #21
According to Moulon et al., (2013a) and Sweeney et al. (2015), the utilization of GlobalSfM should be able to overcome the problem of homogeneity in the dataset. As shown in Fonstad et al. (2013) and also in Andrews et al. (2013), the result of SfM reconstruction performed well compared to other means of 3D data 112
acquisition (i.e. aerial and topographic laser scanner). Other researches such as Green et al. (2014) demonstrated that the quality of SfM reconstruction is quite acceptable in terms of its geometric accuracy, as elaborated previously in Chapter I.6. However, the UGM dataset in this thesis has resulted in poor condition. The large errors observed in the UGM dataset were probably caused by the inaccuracies of point picking due to the erroneous 3D model or ground truth error during total station measurement. Figure III-19 shows the location of point #19 and #21 in the 3D model. More detailed test needs to be performed using coded targets instead of building features. Also, an investigation of the SfM performance using CloudSfM needs to be conducted, for example by running CloudSfM in a distributed environment. All of the CloudSfM projects of the three datasets were conducted using default settings. The open source nature of CloudSfM makes it possible to customize settings such as algorithm of feature detection, feature matching, bundle adjustment, etc. For example, CloudSfM use David Lowe’s SIFT as the default feature detection algorithm. While the user would be able to customize the settings using the advanced parameters in CloudSfM interface, further customization could be performed to employ another algorithm. One of the most important aspect of this thesis is that CloudSfM was built using open source software implementing the state-of-the-art algorithm in Structure from Motion (i.e. Global SfM). CloudSfM’s open source nature would allow researchers to perform modification of the algorithm and/or the user interface based on the new researches. Thus, instead of reinventing the wheel, researchers would only need to utilize some part of CloudSfM (either the web framework part or the SfM pipeline) to fit their particular purpose and modify the source code accordingly.
113
IV. CONCLUSIONS AND SUGGESTIONS
IV.1. Conclusions a. In this thesis, various open source libraries with different purposes has been successfully combined to produce a working Global SfM pipeline. The python script which combine the libraries were able to perform 3D reconstruction using images as input data, producing 3D models and point clouds of a real world objects. b. This thesis showed that open source libraries used as web framework (i.e. MEAN framework) can be employed to develop a Software as Service (SaaS) cloud platform. The web framework used to host SfM libraries to perform 3D reconstruction on-the-fly. The system, namely CloudSfM, has been successfully developed in this thesis. c. The assessment conducted to test the result of this thesis shown that 3D models from a CloudSfM project performs on par with the results obtained from commercial software. However, future research needs to focus on the performance of CloudSfM’s accuracy using more robust test on ground truth data. The current test dataset failed to show that the result of CloudSfM are comparable to ground truth method of obtaining 3D models. d. CloudSfM developed in this thesis provides simplified user interface for nontechnical user, as well as advanced options for users who wants more customization for SfM reconstruction. e. The utilization of open source software means that CloudSfM can be replicated into another server machine with ease. Therefore it can be customized and developed into other application. This also means that CloudSfM can be used for further researches on Structure from Motion by modifying its source code, which are available in CloudSfM’s Github repository.
114
IV.2. Suggestions a. The availability of open source software are increasing with the needs to provide open environment for researchers and developers in various scientific and engineering works. Given the open, less strict nature of this software, it is preferable to build applications in a research activity to use open source environment, as shown in this thesis. b. Implementation of Structure from Motion algorithm needs huge processing power of a CPU (Central Processing Unit). Therefore, development of a web service which able to conduct SfM reconstruction on-the-fly is a challenging task. This thesis provide a first version of a cloud-based SfM which able to do so using open source software. However, more research needs to be conducted in more powerful computers with higher processing capabilities c. Future researches need to be conducted to assess the accuracy of the 3D models from CloudSfM projects compared to other 3D data acquisition methods, such as Terrestrial Laser Scanner or LiDAR measurement. Also, there needs more researches on the performance of CloudSfM’s algorithm in 3D reconstruction of images with similar features, as this thesis shown that in such condition the algorithm is still prone to errors. d. Further tests could also be conducted on objects with different texture and size, as well as utilizing different camera with different specification for data input.
115
V. REFERENCES
Aanaes, H., 2003. Methods for Structure from Motion. Informatics and Mathematical Modelling, Technical University of Denmark, Denmark. Agarwal, S., Furukawa, Y., Snavely, N., Simon, I., Curless, B., Seitz, S.M., Szeliski, R., 2011. Building rome in a day. Communications of the ACM 54, 105–112. Alcantarilla, P.F., Bartoli, A., Davison, A.J., 2012. KAZE features, in: Computer Vision–ECCV 2012. Springer, pp. 214–227. Andrews, D.P., Bedford, J., Bryan, P.G., 2013. A Comparison of Laser Scanning and Structure from Motion as Applied to the Great Barn at Harmondsworth, UK. ISPRS-International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences 1, 31–36. Arayici, Y., 2007. An approach for real world data modelling with the 3D terrestrial laser scanner for built environment. Automation in Construction 16, 816–829. Armbrust, M., Fox, A., Griffith, R., Joseph, A.D., Katz, R., Konwinski, A., Lee, G., Patterson, D., Rabkin, A., Stoica, I., others, 2010. A view of cloud computing. Communications of the ACM 53, 50–58. Barazzetti, L., Scaioni, M., 2009. Automatic orientation of image sequences for 3-D object reconstruction: first results of a method integrating photogrammetric and computer vision algorithms. Proceedings of 3D-ARCH 25–28. Bartoš, K., Pukanská, K., Sabová, J., 2014. Overview of available open-source photogrammetric software, its use and analysis. Int. J. Innov. Educ. Res 2, 62– 70. Bell, P., Beer, B., 2014. Introducing GitHub: A Non-technical Guide. O’Reilly Media, Inc., California. Bergen, J.R., Anandan, P., Hanna, K.J., Hingorani, R., 1992. Hierarchical model-based motion estimation, in: Computer Vision—ECCV’92. Springer, pp. 237–252. Besl, P.J., McKay, N.D., 1992. Method for registration of 3-D shapes, in: Robotics-DL Tentative. International Society for Optics and Photonics, pp. 586–606. Bhat, M.A., Shah, R.M., Ahmad, B., 2011. Cloud Computing: A solution to Geographical Information Systems(GIS). International Journal on Computer Science and Engineering 3, 594–600. Brahmbhatt, S., 2013. Practical OpenCV. Apress, New York. Bretz, A., Ihrig, C.J., 2014. Full stack Javascript development with MEAN. Sitepoint Press, Australia Chacon, S., 2009. Pro git. Apress. Springer Science and Business Media, New York, US. Available at: http:progit.org/book Chandler, J., Fryer, J., 2013. Autodesk 123D catch: how accurate is it. Geomatics World 2, 28–30. Chandrasekaran, K., 2014. Essentials of Cloud Computing. CRC Press, Boca Raton, Florida Chen, S., Li, M., Ren, K., 2014. The power of indoor crowd: Indoor 3D maps from the crowd, in: 2014 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS). Presented at the 2014 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), pp. 217–218. doi:10.1109/INFCOMW.2014.6849233
116
Chetverikov, D., Svirko, D., Stepanov, D., Krsek, P., 2002. The trimmed iterative closest point algorithm, in: Pattern Recognition, 2002. Proceedings. 16th International Conference on. IEEE, pp. 545–548. Cui, Z., Jiang, N., Tan, P., 2015. Linear Global Translation Estimation from Feature Tracks. arXiv preprint arXiv:1503.01832. Dabbish, L., Stuart, C., Tsay, J., Herbsleb, J., 2012. Social coding in GitHub: transparency and collaboration in an open software repository, in: Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work. ACM, pp. 1277–1286. Dorninger, P., Pfeifer, N., 2008. A comprehensive automated 3D approach for building extraction, reconstruction, and regularization from airborne laser scanning point clouds. Sensors 8, 7323–7343. Elwannas, R., Emirates, U.A., 2011. 3D GIS: It’sa Brave New World, In: FIG Working Week 2011 - Bridging Gap between Cultures, Marrakech, Morocco Farid, D., 2015. Angular File Upload. Available at: http://github.com/danialfarid/ng-fileupload Fonstad, M.A., Dietrich, J.T., Courville, B.C., Jensen, J.L., Carbonneau, P.E., 2013. Topographic structure from motion: a new development in photogrammetric measurement. Earth Surface Processes and Landforms 38, 421–430. Förstner, W., 2002. Computer vision and photogrammetry–mutual questions: geometry, statistics and cognition. Bildteknik/lmage Science, Swedish Society for Photogrammetry and Remote Sensing 151–164. Fröhlich, C., Mettenleiter, M., 2004. Terrestrial laser scanning–new perspectives in 3D surveying. International archives of photogrammetry, remote sensing and spatial information sciences 36, W2. Fuhrmann, S., Goesele, M., 2014. Floating scale surface reconstruction. ACM Transactions on Graphics (TOG) 33, 46. Fuhrmann, S., Langguth, F., Goesele, M., 2014. MVE-A Multi-View Reconstruction Environment, in: Eurographics Workshop on Graphics and Cultural Heritage. The Eurographics Association, pp. 11–18. Gackenheimer, C., 2013. Node. js Recipes: A Problem-Solution Approach. Apress. Springer Science and Business Media, New York, Girardeau-Montaut, D., 2011. CloudCompare-Open Source project. OpenSource Project. Golparvar-Fard, M., Bohn, J., Teizer, J., Savarese, S., Peña-Mora, F., 2011. Evaluation of image-based modeling and laser scanning accuracy for emerging automated performance monitoring techniques. Automation in Construction 20, 1143– 1155. Green, S., Bevan, A., Shapland, M., 2014. A comparative assessment of structure from motion methods for archaeological research. Journal of Archaeological Science 46, 173–181. doi:10.1016/j.jas.2014.02.030 Grussenmeyer, P., Landes, T., Voegtle, T., Ringle, K., 2008. Comparison methods of terrestrial laser scanning, photogrammetry and tacheometry data for recording of cultural heritage buildings, in: ISPRS Congress Proceedings, Beijing. pp. 213–18. Haala, N., Kada, M., 2010. An update on automatic 3D building reconstruction. ISPRS Journal of Photogrammetry and Remote Sensing, ISPRS Centenary Celebration Issue 65, 570–580. doi:10.1016/j.isprsjprs.2010.09.006 Hartley, R., others, 1997. In defense of the eight-point algorithm. Pattern Analysis and Machine Intelligence, IEEE Transactions on 19, 580–593.
117
Hartley, R., Zisserman, A., 2003. Multiple view geometry in computer vision. Cambridge university press, Cambridge, UK Hassan, M.I., Rahman, A.A., Stoter, J., 2006. Developing Malaysian 3D Cadastre System—preliminary findings, in: Innovations in 3D Geo Information Systems. Springer, pp. 519–533. Haviv, A.Q., 2014. MEAN Web Development. Packt Publishing Ltd, Birmingham, United Kingdom Henkel, T., 2014. Angular Fullstack. Available at: http://github.com/DaftMonk/generator-angular-fullstack Hows, D., Membrey, P., Plugge, E., 2014. MongoDB Basics, Apress, Springer Science and Business Media, New York, US Jancosek, M., Pajdla, T., 2011. Removing hallucinations from 3D reconstructions. Technical Report CMP CTU, 2011. Jebara, T., Azarbayejani, A., Pentland, A., 1999. 3D structure from 2D motion. Signal Processing Magazine, IEEE 16, 66–84. Jiang, N., Cui, Z., Tan, P., 2013. A global linear method for camera pose registration, in: Computer Vision (ICCV), 2013 IEEE International Conference on. IEEE, pp. 481–488. Jiang, N., Tan, P., Cheong, L.-F., 2012. Seeing double without confusion: Structurefrom-motion in highly ambiguous scenes, in: Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on. IEEE, pp. 1458–1465. Kazhdan, M., Bolitho, M., Hoppe, H., 2006. Poisson surface reconstruction, in: Proceedings of the Fourth Eurographics Symposium on Geometry Processing. Kim, K., Sugiura, T., Torii, A., Sugimoto, S., Okutomi, M., 2013. Instant Surface Reconstruction for Incremental SfM, in: Proceedings of the IAPR Conference on Machine Vision Applications (MVA2013). pp. 371–374. Kitt, B., Geiger, A., Lategahn, H., 2010. Visual odometry based on stereo image sequences with ransac-based outlier rejection scheme, in: Intelligent Vehicles Symposium (IV), 2010 IEEE. IEEE, pp. 486–492. Klette, R., Reulke, R., 2005. Modeling 3D scenes: Paradigm shifts in photogrammetry, remote sensing and computer vision. CITR, The University of Auckland, New Zealand. Klingner, B., Martin, D., Roseborough, J., 2013. Street View Motion-from-Structurefrom-Motion, in: Computer Vision (ICCV), 2013 IEEE International Conference on. IEEE, pp. 953–960. Koutsoudis, A., Vidmar, B., Ioannakis, G., Arnaoutoglou, F., Pavlidis, G., Chamzas, C., 2014. Multi-image 3D reconstruction data evaluation. Journal of Cultural Heritage 15, 73–79. doi:10.1016/j.culher.2012.12.003 Kruppa, E., 1913. Zur Ermittlung eines Objektes aus zwei Perspektiven mit innerer Orientierung. Hölder. Loeliger, J., McCullough, M., 2012. Version Control with Git: Powerful tools and techniques for collaborative software development. O’Reilly Media, Inc. Longuet-Higgins, H.C., 1987. A computer algorithm for reconstructing a scene from two projections. Readings in Computer Vision: Issues, Problems, Principles, and Paradigms, MA Fischler and O. Firschein, eds 61–62. Lowe, D.G., 1999. Object recognition from local scale-invariant features, in: The Proceedings of the Seventh IEEE International Conference on Computer Vision, 1999. Presented at the The Proceedings of the Seventh IEEE International Conference on Computer Vision, 1999, pp. 1150–1157 vol.2. doi:10.1109/ICCV.1999.790410 118
Marr, D., Poggio, T., 1979. A computational theory of human stereo vision. Proceedings of the Royal Society of London B: Biological Sciences 204, 301–328. Ma, Y., Soatto, S., Kosecka, J., Sastry, S.S., 2012. An invitation to 3-d vision: from images to geometric models. Springer Science & Business Media. Mell, P., Grance, T., 2009. The NIST definition of cloud computing. National Institute of Standards and Technology 53, 50. Micheletti, N., Chandler, J.H., Lane, S.N., 2015. Structure from Motion (SfM) Photogrammetry. Geomorphological Techniques, Chap. 2, Sec 2.2. British Society for Geomorphology, UK Mikowski, M.S., Powell, J.C., 2013. Single Page Web Applications. Manning Publications Co. New York, US Miller, M., 2008. Cloud computing: Web-based applications that change the way you work and collaborate online. Que publishing. Pearson Education. New Jersey. USA Minto, S., Remondino, F., 2014. Online access and sharing of reality-based 3d models. SCIRES-IT 4, 17–28. Monteiro, F., 2014. Learning Single-page Web Application Development.Packt Publishing Ltd. Birmingham, UK Moons, T., Van Gool, L., Vergauwen, M., 2009. 3D Reconstruction from Multiple Images. Now Publishers Inc. Breda, Netherlands Moulon, P., Monasse, P., Marlet, R., 2013a. Global fusion of relative motions for robust, accurate and scalable structure from motion, in: Computer Vision (ICCV), 2013 IEEE International Conference on. IEEE, pp. 3248–3255. Moulon, P., Monasse, P., Marlet, R., 2013b. Adaptive Structure from Motion with a contrario model estimation, in: Computer Vision–ACCV 2012. Springer, pp. 257–270. Moulon, P., Monasse, P., Marlet, R., 2012. OpenMVG (open Multiple View Geometry). Available at: http://github.com/openMVG/openMVG Moussa, W., 2014. Integration of digital photogrammetry and terrestrial laser scanning for cultural heritage data recording. University of Stuttgart, Netherlands Nister, D., 2004. An efficient solution to the five-point relative pose problem. IEEE Transactions on Pattern Analysis and Machine Intelligence 26, 756–770. doi:10.1109/TPAMI.2004.17 Oliensis, J., 2000. A Critique of Structure-from-Motion Algorithms. Computer Vision and Image Understanding 80, 172–214. doi:10.1006/cviu.2000.0869 Panda, S., 2014. AngularJS: Novice to Ninja. SitePoint. Melbourne, Australia Pollefeys, M., Koch, R., Van Gool, L., 1999. Self-calibration and metric reconstruction inspite of varying and unknown intrinsic camera parameters. International Journal of Computer Vision 32, 7–25. Pollefeys, M., Koch, R., Vergauwen, M., Van Gool, L., 1998. Metric 3D surface reconstruction from uncalibrated image sequences, in: 3D Structure from Multiple Images of Large-Scale Environments. Springer, pp. 139–154. Pomaska, G., 2009. Utilization of photosynth point clouds for 3D object reconstruction, in: Proceedings of the 22nd CIPA Symposium, Kyoto, Japan. Quan, H., Wu, M., 2013. A Real-Time SFM Method in Augmented Reality, in: Proceedings of the 2012 International Conference on Information Technology and Software Engineering. Springer, pp. 841–848. Rasztovits, S., Dorninger, P., 2013. Comparison of 3D reconstruction services and terrestrial laser scanning for cultural heritage documentation. ISPRS-
119
International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences 1, 513–518. Remondino, F., Del Pizzo, S., Kersten, T.P., Troisi, S., 2012. Low-cost and open-source solutions for automated image orientation–A critical overview, in: Progress in Cultural Heritage Preservation. Springer, pp. 40–54. Remondino, F., El-Hakim, S., 2006. Image-based 3D Modelling: A Review. The Photogrammetric Record 21, 269–291. Robertson, D.P., Cipolla, R., 2008. Structure from motion, in: Practical Image Processing and Computer Vision. Wiley, Blackwell. Rossi, A.J., Rhody, H., Salvaggio, C., Walvoord, D.J., 2012. Abstracted workflow framework with a structure from motion application, in: Image Processing Workshop (WNYIPW), 2012 Western New York. IEEE, pp. 9–12. Schwalbe, E., Maas, H.-G., Seidel, F., 2005. 3D building model generation from airborne laser scanner data using 2D GIS data and orthogonal point cloud projections. Proceedings of ISPRS WG III/3, III/4 3, 12–14. Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R., 2006. A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms, in: Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on. IEEE, pp. 519–528. doi:10.1109/CVPR.2006.19 Seshadri, S., Green, B., 2014. AngularJS: Up and Running: Enhanced Productivity with Structured Web Apps. O’Reilly Media, Inc. Siegele, L., 2008. Let it rise: A special report on corporate IT. Economist Newspaper. Sinha, S.N., Steedly, D., Szeliski, R., Agrawala, M., Pollefeys, M., 2008. Interactive 3D architectural modeling from unordered photo collections, in: ACM Transactions on Graphics (TOG). ACM, p. 159. Soatto, S., Brockett, R., 1998. Optimal structure from motion: Local ambiguities and global estimates, in: Computer Vision and Pattern Recognition, 1998. Proceedings. 1998 IEEE Computer Society Conference on. IEEE, pp. 282–288. Sohn, G., Dowman, I., 2007. Data fusion of high-resolution satellite imagery and LiDAR data for automatic building extraction. ISPRS Journal of Photogrammetry and Remote Sensing 62, 43–63. doi:10.1016/j.isprsjprs.2007.01.001 Sonka, M., Hlavac, V., Boyle, R., 2014. Image processing, analysis, and machine vision. Cengage Learning. Boston, Massachusetts, US Stoter, J., 2000. Considerations for a 3D Cadastre. TU Delft, The Netherlands 1–18. Strecha, C., von Hansen, W., Gool, L.V., Fua, P., Thoennessen, U., 2008. On benchmarking camera calibration and multi-view stereo for high resolution imagery, in: Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on. IEEE, pp. 1–8. Sweeney, C., Hollerer, T., Turk, M., 2015. Theia: A Fast and Scalable Structure-fromMotion Library, in: Proceedings of the 23rd Annual ACM Conference on Multimedia Conference. ACM, pp. 693–696. Szeliski, R., 2010. Computer vision: algorithms and applications. Springer, London. Szeliski, R., Kang, S.B., 1997. Shape ambiguities in structure from motion. Pattern Analysis and Machine Intelligence, IEEE Transactions on 19, 506–512. Szeliski, R., Kang, S.B., 1994. Recovering 3D shape and motion from image streams using nonlinear least squares. Journal of Visual Communication and Image Representation 5, 10–28. Tanskanen, P., Kolev, K., Meier, L., Camposeco, F., Saurer, O., Pollefeys, M., 2013. Live metric 3D reconstruction on mobile phones, in: Computer Vision (ICCV), 2013 IEEE International Conference on. IEEE, pp. 65–72. 120
Thoeni, K., Giacomini, A., Murtagh, R., Kniest, E., 2014. A comparison of multi-view 3D reconstruction of a rock wall using several cameras and a laser scanner, in: Proceedings of ISPRS Technical Commission V Symposium, Riva Del Garda, Italy. pp. 23–25. Thomas, I., Simoncelli, E., 1994. Linear structure from motion. NSF Science and Technology Center for Research in Cognitive Science. Thormählen, T., Hasler, N., Wand, M., Seidel, H.-P., 2010. Registration of sub-sequence and multi-camera reconstructions for camera motion estimation. Journal of Virtual Reality and Broadcasting 7, 1–10. Tomasi, C., Kanade, T., 1992. Shape and motion from image streams under orthography: a factorization method. International Journal of Computer Vision 9, 137–154. Triggs, B., 1996. Factorization methods for projective structure and motion, in: Computer Vision and Pattern Recognition, 1996. Proceedings CVPR’96, 1996 IEEE Computer Society Conference on. IEEE, pp. 845–851. Ullman, S., 1979. The Interpretation of Structure from Motion. Proc. R. Soc. Lond. B 203, 405–426. doi:10.1098/rspb.1979.0006 Vergauwen, M., Van Gool, L., 2006. Web-based 3D reconstruction service. Machine vision and applications 17, 411–426. Vosselman, G., Gorte, B.G., Sithole, G., Rabbani, T., 2004. Recognising structure in laser scanner point clouds. International archives of photogrammetry, remote sensing and spatial information sciences 46, 33–38. Waechter, M., Moehrle, N., Goesele, M., 2014. Let There Be Color! Large-Scale Texturing of 3D Reconstructions, in: Computer Vision–ECCV 2014. Springer, pp. 836–850. Wang, H., Shen, S., Lu, X., 2012. Comparison of the camera calibration between photogrammetry and computer vision, in: 2012 International Conference on System Science and Engineering (ICSSE). Presented at the 2012 International Conference on System Science and Engineering (ICSSE), pp. 358–362. doi:10.1109/ICSSE.2012.6257207 Wang, Y.-F., 2011. A comparison study of five 3D modelling systems based on the sfm principles. Technical Report 2011-01. Visualize Inc., Goleta, USA. Weber, S., 2004. The success of open source. Cambridge Univ Press. UK Wei, Y., Kang, L., Yang, B., others, 2013. Applications of structure from motion: a survey. Journal of Zhejiang University SCIENCE C 14, 486–494. Westoby, M.J., Brasington, J., Glasser, N.F., Hambrey, M.J., Reynolds, J.M., 2012. “Structure-from-Motion” photogrammetry: A low-cost, effective tool for geoscience applications. Geomorphology 179, 300–314. doi:10.1016/j.geomorph.2012.08.021 Yang, C., Goodchild, M., Huang, Q., Nebert, D., Raskin, R., Xu, Y., Bambacus, M., Fay, D., 2011. Spatial cloud computing: how can the geospatial sciences use and help shape cloud computing? International Journal of Digital Earth 4, 305–329. Zaman, M., 2007. High precision relative localization using a single camera, in: Robotics and Automation, 2007 IEEE International Conference on. IEEE, pp. 3908–3914. Zhang, Z., 1992. Iterative point matching for registration of free-form curves.International Journal of Computer Vision, 13:2, 119-152. Kluwer Academic Publishers, Boston
121
VI. APPENDICES
VI.1. CloudSfM Source Code Below are some of the files used in CloudSfM. The complete code is available at http://github.com/danylaksono/cloudsfm.
Bower.json { "name": "sfm", "version": "0.0.0", "dependencies": { "angular": ">=1.2.*", "json3": "~3.3.1", "es5-shim": "~3.0.1", "jquery": "~1.11.0", "bootstrap-sass-official": "~3.1.1", "bootstrap": "~3.1.1", "angular-resource": ">=1.2.*", "angular-cookies": ">=1.2.*", "angular-sanitize": ">=1.2.*", "angular-bootstrap": "~0.11.0", "font-awesome": ">=4.1.0", "lodash": "~2.4.1", "angular-socket-io": "~0.6.0", "angular-ui-router": "~0.2.10", "ng-file-upload": "~4.1.3", "ng-file-upload-shim": "~4.1.3" }, "devDependencies": { "angular-mocks": ">=1.2.*", "angular-scenario": ">=1.2.*" } }
122
Package.json { "name": "sfm", "version": "0.0.0", "main": "server/app.js", "dependencies": { "body-parser": "~1.5.0", "composable-middleware": "^0.3.0", "compression": "~1.0.1", "connect-mongo": "*", "cookie-parser": "~1.0.1", "ejs": "~0.8.4", "errorhandler": "~1.0.0", "express": "*", "express-jwt": "*", "express-session": "*", "express-zip": "^2.0.0", "jsonwebtoken": "^0.3.0", "lodash": "~2.4.1", "method-override": "~1.0.0", "mongoose": "^4.0.3", "morgan": "~1.0.0", "multer": "*", "node-static": "^0.7.6", "passport": "*", "passport-local": "*", "serve-favicon": "*", "socket.io": "*", "socket.io-client": "*", "socketio-jwt": "*" }, "devDependencies": { "grunt": "~0.4.4", "grunt-autoprefixer": "~0.7.2", "grunt-wiredep": "~1.8.0", "grunt-concurrent": "~0.5.0", "grunt-contrib-clean": "~0.5.0", "grunt-contrib-concat": "~0.4.0", "grunt-contrib-copy": "~0.5.0", "grunt-contrib-cssmin": "~0.9.0", "grunt-contrib-htmlmin": "~0.2.0", "grunt-contrib-imagemin": "~0.7.1", "grunt-contrib-jshint": "~0.10.0", "grunt-contrib-uglify": "~0.4.0", "grunt-contrib-watch": "~0.6.1", "grunt-google-cdn": "~0.4.0", "grunt-newer": "~0.7.0", "grunt-ng-annotate": "^0.2.3", "grunt-rev": "~0.1.0", "grunt-svgmin": "~0.4.0", "grunt-usemin": "~2.1.1",
123
"grunt-env": "~0.4.1", "grunt-node-inspector": "~0.1.5", "grunt-nodemon": "~0.2.0", "grunt-angular-templates": "^0.5.4", "grunt-dom-munger": "^3.4.0", "grunt-protractor-runner": "^1.1.0", "grunt-asset-injector": "^0.1.0", "grunt-karma": "~0.8.2", "grunt-build-control": "DaftMonk/grunt-build-control", "grunt-mocha-test": "~0.10.2", "grunt-contrib-sass": "^0.7.3", "jit-grunt": "^0.5.0", "time-grunt": "~0.3.1", "grunt-express-server": "~0.4.17", "grunt-open": "~0.2.3", "open": "~0.0.4", "jshint-stylish": "~0.1.5", "connect-livereload": "~0.4.0", "karma-ng-scenario": "~0.1.0", "karma-firefox-launcher": "~0.1.3", "karma-script-launcher": "~0.1.0", "karma-html2js-preprocessor": "~0.1.0", "karma-ng-jade2js-preprocessor": "^0.1.2", "karma-jasmine": "~0.1.5", "karma-chrome-launcher": "~0.1.3", "requirejs": "~2.1.11", "karma-requirejs": "~0.2.1", "karma-coffee-preprocessor": "~0.2.1", "karma-jade-preprocessor": "0.0.11", "karma-phantomjs-launcher": "~0.1.4", "karma": "~0.12.9", "karma-ng-html2js-preprocessor": "~0.1.0", "supertest": "~0.11.0", "should": "~3.3.1" }, "engines": { "node": ">=0.10.0" }, "scripts": { "start": "node server/app.js", "test": "grunt test", "update-webdriver": "node node_modules/grunt-protractorrunner/node_modules/protractor/bin/webdriver-manager update" }, "private": true }
124
Manage.controller.js 'use strict'; angular.module('sfmApp') .controller('ManageCtrl', function ($scope, $http, $window, Auth) { $scope.currentUsername = Auth.getCurrentUser().name; $scope.currentProject =''; $scope.downloadReady = false; $scope.message = ''; $http.get('/api/uploadImages').success(function(projects) { for (var i=0; i < projects.length; i++) { var username = projects[i].author.name; if ($scope.currentUsername == username) { $scope.currentProject = projects[i]; } else { console.log('project not found') } }; checkAvailability(); }); var checkAvailability = function(){ var url = '/api/startsfms/' + '?name=' + $scope.currentUsername + '&project=' + $scope.currentProject.projectName; console.log(url); $http.get(url).success(function(msg){ $scope.message = msg; if ($scope.message == 'Completed') { $scope.downloadReady = true; } else { $scope.downloadReady = false; } }); }; //for list $scope.param = 'none'; $scope.paramsopt = [ {value:'colorized-cloud', name:'Colorized Point-cloud'}, {value:'sfmreport', name:'HTML Report'}, {value:'textured', name:'Textured OBJ'} ];
$scope.changedVal = function(theparam) { $scope.downloadUrl = '/api/startsfms/' + Auth.getCurrentUser()._id + '?download=' + theparam; $http({method:'GET', url:$scope.downloadUrl}).success(function(data,status,headers,config){ console.log(status); });
125
}; $scope.getfile = function() { $window.open($scope.downloadUrl, '_blank'); }
$scope.startsfm = function() { $scope.message = 'Processing Request...'; console.log('requesting pipelines'); var request = { method: 'POST', url: '/api/startsfms', data: { username: $scope.currentUsername, projectname: $scope.currentProject.projectName } }; $http(request).success(function(msg){ if (msg == 'Completed') { $scope.message = msg; $scope.downloadReady = true; } }).error(function(err){ console.log('Error occured!', err) });
};
});
126
startsfm.controller.js 'use strict'; var var var var var var var
_ = require('lodash'); Startsfm = require('./startsfm.model'); shell = require('shelljs'); fs = require('fs'); path = require('path'); zip = require('express-zip'); present = require('present');
exports.check = function(files) { // Check to see if the process has finished var message = ''; try { var stats = fs.lstatSync(files); if (stats.isFile()) { console.log('File found!'); message='Completed'; } } catch(err) { console.log(err); console.log('File Not Found'); message='Ready to Start'; } return message; } // Whole processing route exports.startProcess = function(req, res, next) { // identifying working directories var currentDir = shell.pwd(); var usernameDir = req.body.username; var projectnameDir= req.body.projectname; var workingDir = currentDir + '/uploaded' + '/' + usernameDir + '/' + projectnameDir; console.log(exports.usernameDir); // Defining SfM processing commands // (initial sfm parameter are set to default now. future release will use parameter based on user input) //var GlobalSfM = 'python SfM_GlobalPipeline.py -i ' + workingDir + '/images/ -o '+ workingDir + '/output/ -f 2000 > '+ workingDir +'/report_global.txt'; var GlobalSfM = ''; var SequentialSfM = 'python SfM_GlobalPipeline.py -i ' + workingDir + '/images/ -o '+ workingDir + '/output/ -f 2000 > '+ workingDir +'/report_sequential.txt'; var MVS = "printf \\r\\n | python MVE_FSSR_MVS.py -i " + workingDir + '/images/ -o '+ workingDir + '/scene/ > '+ workingDir +'/report_mvs.txt';
127
//Select SfM Method based on user input. Default is Global var SfMmethod = 'Global'; exports.global_sfm_dir = workingDir + '/output/reconstruction_global/'; exports.sequential_sfm_dir = workingDir + '/output/reconstruction_sequential/'; exports.mvs_scene_dir = workingDir + '/scene/'; var start = present(); var HTMLreport = ''; if(SfMmethod == 'Global') { HTMLreport = exports.global_sfm_dir + 'SfMReconstruction_Report.html'; shell.exec(GlobalSfM); } else { HTMLreport = exports.sequential_sfm_dir + 'SfMReconstruction_Report.html'; shell.exec(SequentialSfM); } //executing MVS shell.exec(MVS); var end = present(); console.log("Process took " + (start - end).tofixed(3) + " milliseconds.")
var fileExist = exports.check(HTMLReport); return res.json(fileExist); };
// Download exports.download = function(req, res, next) { //console.log(exports.usernameDir); var out_colorizedSfM = exports.global_sfm_dir + 'robust_colorized.ply'; var out_SfMReport = exports.global_sfm_dir + 'SfMReconstruction_Report.html'; var out_textured = exports.mvs_scene_dir + 'out_textured.obj'; if (req.query.download == 'colorized-cloud'){ console.log('Requesting colorized cloud..'); return res.download(out_colorizedSfM); } else if (req.query.download == 'sfmreport') { console.log('Requesting SfM Report...'); return res.download(out_SfMReport); } else if (req.query.download == 'textured') { console.log('Requesting textured model...'); //return res.download(out_textured); return res.zip([ {path: exports.mvs_scene_dir + 'out_textured.obj', name: 'out_textured.obj'},
128
name: 'out_textured.obj'}, {path: exports.mvs_scene_dir + 'out_textured.mtl', name: 'out_textured.mtl'}, {path: exports.mvs_scene_dir + 'out_textured_data_costs.spt', name: 'out_textured_data_costs.spt'}, {path: exports.mvs_scene_dir + 'out_textured_labeling.vec', name: 'out_textured_labeling.vec'}, {path: exports.mvs_scene_dir + 'out_textured_material0000_map_Kd.png', name: 'out_textured_material0000_map_Kd.png'} ]);
} else { return res.json('Unknown query parameter') } };
// Check if output already exist exports.index = function(req, res) { var pwd = shell.pwd(); var name = req.query.name; var project = req.query.project; console.log(name); console.log(project); var mvsobj = pwd + '/uploaded' + '/' + name + '/' + project + '/scene/out_textured.obj'; var checkExist = exports.check(mvsobj); console.log(checkExist); return res.json(checkExist); }; function handleError(res, err) { return res.send(500, err); }
129
Manage.html
Project Manager
Project Properties
Your last (active) project is:
| Owner | : | {{currentProject.author.name}} |
| Project Name | : | {{currentProject.projectName}} |
| Project Description | : | {{currentProject.projectDescription}} |
| Status | : | {{message}} |
This project is recorded on {{currentProject.date}}
You can close this page as we process your data. You can always return to this page by signin in and select 'Manage' menu on the toolbar. Upon completion, download links will be available below. Choose which type of output that you want, then click on 'Download'
130
{{downloadReady? 'Download is Ready' : message}}
Download Output
Reconstructed Structure and Motion can be downloaded here. Please refer the Documentation for explanation on each options
Download this
131
VI.2. SfM_GlobalPipeline.py #!/usr/bin/python #! -*- encoding: utf-8 -*# # Created by Dany Laksono # Thanks to Romuald Perrot, @vins31 and Pierre Moulon # # This script performs background SfM processing of CloudSfM Webservice # Input is username and project name based on CloudSfM web interface # # openMVG parameters are based on version 0.9 (Paracheirodons Simulans)
# Indicate the openMVG binary directory BIN_DIR = "/usr/local/bin" # Indicate the openMVG camera sensor width directory CAMERA_SENSOR_WIDTH_DIRECTORY = "/home/dany/github/openMVG/src/software/SfM" + "/cameraSensorWidth" import import import import import import
commands os, stat subprocess sys, getopt atexit json
# define arguments of this python script if len(sys.argv) < 2: print ("Usage %s username projectname" % sys.argv[0]) sys.exit(1) try: opts, args = getopt.getopt(sys.argv[1:],"hw:",["workingdir="]) except getopt.GetoptError: print 'sfmglobal.py -u -p ' sys.exit(2) for opt, arg in opts: if opt == '-h': print 'sfmglobal.py -u -p ' sys.exit() elif opt in ("-w", "--workingdir"): project_dir = arg
132
# INITIAL PARAMETERS project_path = project_dir #read the parameters from settings.json with open(project_path+'/settings.json') as params: sfmparams = json.load(params) print(sfmparams['projectStatus']) input_dir = os.path.join(project_path, "images") output_dir = os.path.join(project_path, "output") matches_dir = os.path.join(output_dir, "matches") reconstruction_dir = os.path.join(output_dir, "global") mve_dir = os.path.join(output_dir, "mve") camera_file_params = os.path.join(CAMERA_SENSOR_WIDTH_DIRECTORY, "cameraGenerated.txt")
# ========================================================= # STRUCTURE FROM MOTION # ========================================================= # Create these folders if not present if not os.path.exists(output_dir): os.mkdir(output_dir, 0755) if not os.path.exists(matches_dir): os.mkdir(matches_dir, 0755) if not os.path.exists(reconstruction_dir): os.mkdir(reconstruction_dir, 0755) if not os.path.exists(mve_dir): os.mkdir(mve_dir, 0755)
# OpenMVG Libraries print ("1. Intrisics analysis") if sfmparams["intrinsic"] == "focal": print ("Using camera focus") pIntrisics = subprocess.Popen([os.path.join(BIN_DIR, "openMVG_main_SfMInit_ImageListing"), "-i", input_dir, "-o", matches_dir, "-f", sfmparams["focal"]]) pIntrisics.wait() else: print ("Using Known Calibration Matrix") pIntrisics = subprocess.Popen([os.path.join(BIN_DIR, "openMVG_main_SfMInit_ImageListing"), "-i", input_dir, "-o", matches_dir, "-k", sfmparams["kmatrix"]]) pIntrisics.wait() print ("2. Compute features") pFeatures = subprocess.Popen( [os.path.join(BIN_DIR, "openMVG_main_ComputeFeatures"), "-i", matches_dir+"/sfm_data.json", "o", matches_dir, "-m", sfmparams["featDetector"], "-u", sfmparams["isUpright"], "-p", sfmparams["detPreset"]]) pFeatures.wait()
133
print ("3. Compute matches") pMatches = subprocess.Popen( [os.path.join(BIN_DIR, "openMVG_main_ComputeMatches"), "-i", matches_dir+"/sfm_data.json", "-o", matches_dir, "-r", sfmparams["annRatio"], "-n", sfmparams["nearMethod"], "-g", sfmparams["geomModel"]] ) pMatches.wait() print ("4. Do Global reconstruction") pRecons = subprocess.Popen( [os.path.join(BIN_DIR, "openMVG_main_GlobalSfM"), "-i", matches_dir+"/sfm_data.json", "-m", matches_dir, "-o", reconstruction_dir, "-r", "2", "-t", "1"] ) pRecons.wait() print ("5. Colorize Structure") pRecons = subprocess.Popen( [os.path.join(BIN_DIR, "openMVG_main_ComputeSfM_DataColor"), "-i", reconstruction_dir+"/sfm_data.bin", "-o", os.path.join(reconstruction_dir,"colorized.ply")] ) pRecons.wait() # optional, compute final valid structure from the known camera poses print ("6. Structure from Known Poses (robust triangulation)") pRecons = subprocess.Popen( [os.path.join(BIN_DIR, "openMVG_main_ComputeStructureFromKnownPoses"), "-i", reconstruction_dir+"/sfm_data.bin", "-m", matches_dir, "-f", os.path.join(matches_dir, "matches.e.bin"), "-o", os.path.join(reconstruction_dir,"robust.json")] ) pRecons.wait() pRecons = subprocess.Popen( [os.path.join(BIN_DIR, "openMVG_main_ComputeSfM_DataColor"), "-i", reconstruction_dir+"/robust.json", "-o", os.path.join(reconstruction_dir,"robust_colorized.ply")] ) pRecons.wait()
# MULTIVIEW RECONSTRUCTION print ("7. Conversion to MVE Format") pRecons = subprocess.Popen( [os.path.join(BIN_DIR, "openMVG_main_openMVG2MVE2"), "-i", reconstruction_dir+"/sfm_data.bin", "-o", mve_dir]) pRecons.wait() print (" ") print ("Finished SfM Reconstruction. Continuing with MVS...") print (" ") scene_dir = mve_dir+ "/MVE"
134
print ("8. Depth Map reconstruction (images + camera parameters --> depth maps)") # Depth-map reconstruction pDMrecon = subprocess.Popen( [os.path.join(BIN_DIR, "dmrecon"), "-s3", "p", scene_dir]) pDMrecon.wait() # Depth-map to point set pPointset = subprocess.Popen( [os.path.join(BIN_DIR, "scene2pset"), "-F3", scene_dir, scene_dir+"/point-set.ply"]) pPointset.wait()
print ("9. Surface reconstruction (depth maps + camera parameters --> 3D model)") # Floating-Scale Surface Reconstruction pFSSR = subprocess.Popen( [os.path.join(BIN_DIR, "fssrecon"), scene_dir+"/point-set.ply", scene_dir+"/surface.ply"]) pFSSR.wait() # Clean Reconstruction Mesh pMeshclean = subprocess.Popen( [os.path.join(BIN_DIR, "meshclean"), "t10", "-c10000", scene_dir+"/surface.ply", scene_dir+"/surfaceclean.ply"]) pMeshclean.wait()
# TEXTURING print (" ") print ("Texturing Mesh...") print (" ")
print ("10. MVS-Texturing (3D model + images + camera parameters --> textured 3D model)") # MVS-Texturing pTexrecon = subprocess.Popen( [os.path.join(BIN_DIR, "texrecon"), scene_dir+"::undistorted", scene_dir+"/surface-clean.ply", scene_dir +"/textured"]) pTexrecon.wait()
print(' ') print('Updating settings.json...') print(' ') sfmparams['projectStatus'] = 'Finished' with open(project_path+'/settings.json', 'w') as params: json.dump(sfmparams, params, sort_keys=True, indent=4, separators=(',', ': ')) print ("===================================================") print(' ') print("Finished all process!")
135
VI.3. OpenMVG Install Script (openmvg_install.sh) #!/bin/bash # # v1 2014,08,30 # v0 from Natanael1234 # # This script is available for Debian distros and maybe for other ones using the aptitude mecanism (tested on Debian only) # # Its purpose is to automate openMVG compilation and installation under a Debian like distro. # # realased as this and free to use, modify, copy and so on ... # # binaries are installed in /usr/local/bin and camera databse in /usr/local/etc # # ToDo # add CMVS/PMVS automation # # Adjustement in OpenGL might be necessary regarding your graphic card. Add or modify it in installDeps (e.g. Nvdia) # # get number of cpu cores to use NBcore=$([[ $(uname) = 'Darwin' ]] && sysctl -n hw.logicalcpu_max || lscpu -p | egrep -v '^#' | wc -l) # MainWorkDir=/usr/local/src openMVGSrcDir=$MainWorkDir/openMVG/src openMVG_BuildDir=$openMVGSrcDir/build if which sudo >/dev/null; then echo sudo installed, ok! else echo sudo not there! Installing it! apt-get install -qq -y sudo fi if [ -d $MainWorkDir ]; then echo working directory exists, ok! else echo working directory does not, creating it! mkdir -p $openMVG_BuildDir fi
136
installDeps(){ echo echo "Installing necessary dependencies..." sudo apt-get install -qq -y apt-listchanges build-essential cmake git libpng12-dev libjpeg8-dev libxxf86vm1 libxxf86vm-dev libxi-dev libxrandrdev sudo apt-get install -qq -y python-sphinx freeglut3-dev zlib1g-dev libncurses5-dev glew-utils libdevil-dev libboost-all-dev libatlas-cpp-0.6dev sudo apt-get install -qq -y libatlas-dev libgsl0-dev liblapack-dev liblapack3 libpthread-workqueue-dev }
cloneGitHub(){ echo echo "Cloning OpenMVG from Github..." cd $MainWorkDir git clone https://github.com/openMVG/openMVG.git #git clone -b develop_newIO --single-branch https://github.com/openMVG/openMVG.git cd $MainWorkDir/openMVG #git checkout develop git submodule init git submodule update cd .. } compileAndInstallOpenMVG(){ echo echo "Compiling OpenMVG..." echo mkdir $openMVG_BuildDir cd $openMVG_BuildDir cmake -DCMAKE_BUILD_TYPE=RELEASE -DOpenMVG_BUILD_TESTS=ON DOpenMVG_BUILD_EXAMPLES=ON . $openMVGSrcDir make -j $NBcore
# Install main binaries in /usr/local/bin find $openMVG_BuildDir/software/SfM -type f -executable -exec cp {} /usr/local/bin \; find $openMVG_BuildDir/software/globalSfM -type f -executable -exec cp {} /usr/local/bin \; find $openMVG_BuildDir/software/colorHarmonize -type f -executable -exec cp {} /usr/local/bin \; find $openMVG_BuildDir/software/SfMViewer -type f -executable -exec cp {} /usr/local/bin \; #/usr/local/src/openMVG/src/build/software/globalSfM
137
# Install camera database in /usr/local/etc cp $MainWorkDir/openMVG/src/software/SfM/cameraSensorWidth/cameraGenerated.tx t /usr/local/etc # Install samples binaries in /usr/local/bin find $openMVG_BuildDir/openMVG_Samples -type f -executable -exec cp {} /usr/local/bin \; } testOpenMVG(){ echo echo "Testing..." cd $openMVG_BuildDir make test } clearInstall(){ if [ -d $MainWorkDir/openMVG ] ; then echo echo "Removing previous files ..." rm -fr $MainWorkDir/openMVG fi } installDeps clearInstall cloneGitHub compileAndInstallOpenMVG testOpenMVG echo echo "Ready to go. "
138
VI.4. Report of SfM Reconstruction
Naga Dataset
SFM report.
Dataset info: #views: 81 #poses: 81 #intrinsics: 1 #tracks: 53579 #residuals: 307065
IdVie w
Basename
Residu Residu Residu #Observati Residuals als als als media ons min mean max n
0
01d911e461fb719426056bea3 2211 9600033
0.000167 0.6476 0.8393 4.3165 171 82 7 2
1
07bb3c2e5d37aedd9daf569e0 4944 1793be1
3.08557e- 0.5657 0.7302 4.3346 06 03 84 5
2
0871350c4c31993699e9fa072 3055 8a50fa0
0.000291 0.4056 0.5819 3.6859 176 47 98
3
097a5b9d1c80243b159c81524 2855 7b92593
0.000336 0.4940 0.6635 4.0918 445 9 73 1
4
0c36b3e98c545d9e28084a806 4352 29a99a9
9.29081e- 0.4863 0.6343 3.9729 05 05 7 7
139
5
106c71327eeaf28c61b0916e5 2447 a2dfc59
0.000138 0.6478 0.8586 4.2854 68 46 34 1
6
109a229d7bbfdda2267697735 4027 d02efc7
6.22278e- 0.6862 0.8616 4.4588 05 48 21 9
7
1a4e3f3c5bdf0c3014c1fcab52 5752 625851
9.00837e- 0.5483 0.7073 4.3685 05 33 89 9
8
1ac9ba24d259455b71e2a7f4fe 6653 ad6c6f
0.000207 0.5852 0.7311 4.6351 041 91 35 8
9
1b14b9a802b58f4338e79f3bcc 4490 522c7b
0.000180 0.4840 0.6387 4.4335 235 03 47 7
10
1c97e2fc79bdc474d2ff8895dfc 1096 2e783
0.001100 0.5421 0.7263 4.0263 99 26 57 4
11
1de3d0a1770af09ba70c1dcea 4786 22f07a5
7.90503e- 0.4707 0.6241 4.9466 05 75 46 9
12
1e62e7cea8e01fc0f781cd2225 3956 37774e
3.09209e- 0.4710 0.6119 4.0644 05 23 11 1
13
23a7a19e25db58e68097b7f72 6566 a6e4708
1.59268e- 0.4788 0.6291 4.217 05 06 47
14
24322ff75486d8eb84b7f671c4 1113 7549dc
0.000335 0.4568 0.6360 4.0654 651 82 94
15
2b2bc4e1f30ae9b3302ef8a2bd 4211 f1f827
4.97605e- 0.4811 0.6354 3.7887 05 55 78
16
34cd2f60b525ad48404acf1b12 5030 95e9f5
0.000188 0.6666 0.8314 4.1113 162 26 13 2
17
34e15f93fbb6352a1bd38a66a 6221 037a02b
5.22684e- 0.4571 0.6157 4.4186 05 2 33 2
18
3648a519f08e526d3490cc5a6 4400 5ce2c82
3.26227e- 0.5528 0.6987 4.4910 05 02 26 7
140
19
3cde95d3fb5ba21e044a759fe2 3989 32f56b
4.55134e- 0.4756 0.6202 5.4118 05 89 01 2
20
3dc1d783a817f742bf6bc9e32d 2978 3af86f
0.000119 0.4943 0.6410 4.2889 972 86 78 6
21
3dd86378fcbaf985af3b9b6858 5550 bff70d
4.77016e- 0.4016 0.5517 4.0799 05 88 28 4
22
40b43b8b39701481db74207e 3693 d4f4b4b6
9.84996e- 0.5747 0.7591 4.9892 07 03 95 7
23
41a761feddfb3b6f454aad1710 1691 826e3b
0.000100 0.4273 0.5940 3.9469 604 68 77 9
24
41b6e44c8ec7048df0cb82deb 6248 dc0cf82
2.72908e- 0.6557 0.7999 4.1391 05 44 39 9
25
4409f2a8c681dd20b44dfdc40a 4987 8ff18e
0.000116 0.4491 0.5878 4.2791 938 72 63 3
26
450e339a2d41b9773d3221e76 5010 25a6070
9.66767e- 0.4333 0.5789 4.1543 06 15 87 5
27
463eedb8689cb8209f3b89be4 832 81a2c8a
0.000158 0.4309 0.5859 3.9883 812 86 26 1
28
4695b33aea0386d30e85cb433 847 9a30d4b
0.000220 0.4718 0.6488 4.0062 893 87 79 3
29
4b5ff4a9f0b5d4788e1c8c93b6 4927 184c9d
0.000120 0.7114 0.8780 4.8459 377 77 59 5
30
4ef548a04d9b05110d17cae73 4923 db4a04b
2.28625e- 0.5022 0.6640 4.5620 05 09 4 7
31
5a00d059919e1a69e0c7b6985 4801 85fcbaf
6.19326e- 0.4336 0.5724 3.8469 05 72 68 7
32
5a10c1b1781cc452af8ff4fdacf 5151 93f06
2.06648e- 0.4334 0.5670 4.2794 05 9 84 3
141
33
5e5b15f5545bf80a07e76238bf 4999 9184ef
1.47475e- 0.5300 0.6921 4.4371 05 41 36 8
34
5efea191a925de207db172625 871 8cc0460
0.000223 0.6014 3.7941 0.4472 118 36 9
35
621ba6567e3921af425e61cc6c 5590 741668
9.90216e- 0.4827 0.6226 4.173 05 77 57
36
64e9c90210396b7d8f66c2a6d 1637 064ee60
8.34515e- 0.6246 0.8220 4.3068 05 34 12 7
37
6625a00bba2d3c660973b243c 6027 f2017a5
2.80737e- 0.4582 0.6056 4.0547 05 66 14 6
38
6b3369a8248b1697ea7a15b7a 3946 1afd535
9.89833e- 0.5776 0.7274 4.3194 06 39 39 3
39
6da3ccf53a70200b0519acb115 5624 4c0b93
2.54189e0.568 05
40
74aae46ebec9957c38803d39f 4411 6102a66
0.000138 0.4994 0.6477 4.5080 241 2 53 1
41
7a2c3cec6113603ff0831fc0232 730 72eb0
0.000805 0.5631 0.7429 4.2805 935 36 42 2
42
7a869f593a510884256d2b97a 3587 ec59f68
0.000288 0.3995 0.5299 3.7524 573 23 74 7
43
82049001d99250c7acdd452fff 2963 0f523b
3.17001e- 0.5393 0.7198 4.0247 05 77 12 1
44
822260e3921445710e058e2c7 446 6021eef
0.000307 0.4007 0.5598 3.6595 927 44 69 8
45
875bc8625163634e9cfe9f2e23 1071 e7bb9d
0.000140 0.4814 0.6655 4.0006 767 05 27 8
46
887de0ca28ea2f4c5ef8e8c3be 1132 63b7ef
9.76943e- 0.7439 0.9717 5.1884 05 38 75 7
0.7227 4.3389 99 5
142
47
8ba080187a5581f1a8f4cf508b 4707 505264
5.69794e- 0.6912 0.8377 4.5595 05 46 01 7
48
8d2a565b68a99683a61e5b925 3421 453409e
9.29107e- 0.4682 0.6374 4.8171 05 33 01 7
49
966584b5258f73364449ae1a0 6071 599517a
5.85752e- 0.6233 0.7758 3.8424 05 47 75 7
50
96d52764792551f8c659e3e5a 4595 41ee8b8
8.51603e- 0.5328 0.6867 4.0720 05 11 02 7
51
974af4163b29d9d0dbeb88b6b 4612 ead32c0
6.06834e- 0.4294 0.5728 3.9399 06 27 33
52
9b7c3ea2be74e841f0691e658 4668 935fc31
5.07153e- 0.4953 0.6468 4.6446 05 09 9 8
53
a12eab48142013cf6279856e5 3330 0efa529
4.58571e- 0.6184 0.7977 5.4481 05 04 37 9
54
a4097956b90755d08b01386e5 2956 f7e0bd6
0.000166 0.4519 0.6095 4.1375 718 3 66 3
55
a643dbf40550e0f5a9ae2ee834 2952 2936b2
8.06466e- 0.4336 0.5901 4.0707 06 46 9 5
56
a8d5c28eb5d2b98e57d5535d4 4987 d3c8a11
0.000185 0.4591 0.6095 4.4255 532 58 63 4
57
aba6aced5fb10d8de828b4a6cc 4975 a009e6
3.1923e- 0.4518 0.5969 4.0125 05 23 41 6
58
ae99c684c86943449eb696cb3 2036 5b712de
0.000333 0.5447 0.7236 5.2107 43 66 62 8
59
b642ce0f024c6ad14b09956f07 5398 ec6084
2.51213e- 0.5390 0.7003 4.5268 06 2 34 9
60
b88c4e2d2e40f94135041e76a 689 a133ac5
0.002266 0.8589 1.0309 4.2392 2 31 6
143
61
ba9e1a481848c365bda9ae4c0 3678 0089144
4.77249e- 0.5580 0.7144 4.0115 05 27 86 7
62
bfa2a6c711246a57417c188a1c 3797 a17af5
7.85633e- 0.5068 0.6610 4.8614 05 44 33 9
63
c2ee89dfa0e99d0b2f59bf74a5 3402 7466c7
2.46828e- 0.6131 0.7681 4.1550 05 09 59 9
64
c42b6e92a53240e30b813d663 2487 01afe21
9.55266e- 0.4843 0.6453 3.9569 05 22 43
65
c4a8685ec17ca7f96794ac1eb2 2958 5294e5
4.51247e- 0.4915 0.6682 4.3026 06 84 05 1
66
cb3164dec711615ca5bff54675 4885 ce00bf
7.88036e- 0.4389 0.5820 3.6801 06 24 5 5
67
cbd656c28d5a639a44eddbd28 4555 02bf365
0.000185 0.5608 0.7264 5.0346 447 65 94 3
68
d3131acb416df12bc29aea608 3461 d9e710b
3.61045e- 0.4954 0.6395 3.9365 05 88 89 9
69
de0324eabc1ba1f3f032d2b5e3 5171 6dafb9
8.49249e- 0.5045 0.6347 4.3670 05 17 76 1
70
e1a589c49740b2ac889320775 1884 b246232
4.60779e- 0.7937 0.9928 4.2675 06 06 99
71
eed484bdb68b9a8ae7b21f6d0 3668 42a76ca
7.62051e- 0.5046 0.6658 3.8639 05 8 31 3
72
eef505187c0824478893a994d 1888 ae9bea9
9.72852e- 0.5456 0.7052 3.7226 05 01 11 8
73
ef57aeecef4258a08acfe0ce4c7 1837 e4d12
0.000118 0.7491 0.9767 5.1234 537 74 69 3
74
ef829ad50e43771068300bea9 4977 548ed92
0.000169 0.4297 0.5802 3.8568 935 49 46 2
144
75
ef84ecaa346dc452bf7a431185 5205 a9d225
4.97727e- 0.4853 0.6364 4.0816 06 53 93 2
76
f3d06a52c6e9dca3b6ef9e10b5 4540 8e8340
1.73024e- 0.4969 0.6412 4.2486 05 6 24 7
77
f4582b758635cff8a7b5ddcc65 2494 7b490e
7.6887e- 0.4035 0.5528 4.1431 05 82 1 4
78
f668ae48977eb6a483a01c279 3094 2b0a177
5.11656e- 0.5624 0.7406 4.4901 05 54 22 3
79
f8604c4b73d884a7376fec48f5 4627 39e02b
2.74161e- 0.5394 0.6998 4.3264 05 79 86 8
80
fd4ba6ecab4fcd9a54b0f48bbe 6234 4c40f4
1.79873e- 0.4691 0.6135 4.3860 05 29 4 3
SfM Scene RMSE: 0.90439
145
Tree Dataset
SFM report.
Dataset info: #views: 91 #poses: 91 #intrinsics: 1 #tracks: 55853 #residuals: 268711
IdVie w
Basename
Residu Residu Residu #Observati Residuals als als als media ons min mean max n
0
026091638c04915e5f172579e 4272 6219ee0
2.39272e- 0.2791 0.4023 3.7598 05 46 39
1
05f3b43c8a6c84f7f355fd54352 3098 d017e
0.000175 0.4534 0.5811 3.7295 322 21 53 6
2
066f04f46467740fccffc010b7d 3196 f88d1
5.94455e- 0.3049 0.4243 3.9472 06 88 01 3
3
0802804a16283630819c059b9 4034 29c0656
9.64134e- 0.3233 0.4475 3.9398 05 31 54
4
0d1f76b443090b46e224f0784 4255 6a52801
5.1061e- 0.2639 0.3809 3.5125 05 65 8 2
5
0e563477d37dab9b9e59a3fe9 2073 ba0d983
0.000241 0.3203 0.4593 3.8617 291 32 73 4
6
10952e4bdae09cd22faf24c4b4 1551 d5d033
4.19678e- 0.3028 0.4283 3.6730 05 17 78 4
146
7
11a46b067c90951e9b63f32d5 4446 84f3603
2.80778e- 0.2659 0.3955 3.9373 05 6 17 2
8
135bce1d183c72d69375b68f6 6881 b87f559
1.32792e- 0.2545 0.3604 3.3974 05 99 12 9
9
14a1fa4eb7af30ad5e7440636f 2638 cbd63a
6.56267e- 0.2703 0.3928 3.8792 05 52 3 1
10
160ae5c94f58dd6b001517425 3244 8fc48bc
0.000122 0.3047 0.4224 3.8870 15 73 18 5
11
1ce4aa16f696f26a9dc76d49ba 3529 ab9b4d
3.31047e- 0.3318 0.4239 3.8836 05 51 38 1
12
1e16aafbdbdd64c693d298d3d 2094 1081f17
7.29779e- 0.3312 0.4431 3.3664 05 44 06 3
13
1e65f82ce480962377a09a12e 4965 a6bf33a
6.76805e- 0.3207 0.4052 3.9316 06 67 71 8
14
2047a71d02b1970cffbc00259f 532 a2086b
0.002875 0.3677 0.5161 3.2513 41 81 61 7
15
24c08846130322aee31817a9c 2438 7077cd4
0.000130 0.2898 0.4247 3.5230 895 03 26 1
16
274ed46f2d98ee3a7d880fda6 2011 01b5990
3.11825e- 0.1817 0.3218 3.2403 05 34 21
17
2bfdba9d0ac7fa83fa7750c9eef 2349 88343
3.41404e- 0.3445 0.5047 3.9501 05 94 62 2
18
368f3bcf869661bd46eef2d716 1046 b4a9c8
3.34142e- 0.3034 0.4424 3.3622 05 23 88 8
19
3752ea95deaf511f70f91f781af 1070 7a027
0.000315 0.2709 0.4012 3.3237 504 9 11 4
20
399707a8a3dfc05140e8751db 2860 efdf71c
5.00209e- 0.3458 0.4634 3.8612 06 79 26 7
147
21
3be74278f883e4a040324b956 2214 eacaebd
0.000176 0.5311 0.6587 3.7562 473 02 73 4
22
4882d440d9c652a41722b6a4e 6391 1a1cdc8
2.83278e- 0.2921 0.3844 3.4381 06 19 92 2
23
4c2cf8fa2b2465ae3fe1fc2819e 3441 a9d41
5.13251e- 0.3456 0.4636 3.8129 05 44 28 4
24
4ce56b4d46dd7e2b92969b6ca 3970 6e9b8ce
2.49177e- 0.3069 0.4406 3.9302 05 22 03 7
25
56353dae1d9319c24a46fde10 2517 e019195
0.000117 0.5095 3.7393 0.6525 188 4 4
26
5887a8889bf12fb79c040ab815 2212 3eb5d8
0.000409 0.3364 0.4281 3.7516 767 24 1 8
27
5ae41a27a64b67d103445ed7b 6531 f3e5efc
2.28091e- 0.3303 0.4347 3.5822 06 15 55 5
28
5cdcbb66ebdeafda5bc56c7420 1682 74276e
5.83115e- 0.2933 0.4422 3.9922 05 1 49 6
29
5d90359df644f1de00b0149ba 3330 9e0c525
0.000143 0.3612 0.4800 3.8268 256 39 26 8
30
5e94959acee9eeafbc8a60fff95 2252 c015a
3.94035e- 0.3166 0.4402 3.4938 07 19 06 8
31
632dfd8288a54e4ef305152f70 2931 06c1f9
1.06042e- 0.2899 0.4125 3.3306 05 14 08 3
32
67b5df1d2bc131357349e6403 6667 4e6cbd5
1.18051e- 0.2631 0.3832 3.6804 06 08 9 3
33
69bc0b08c1e6d19efad39ad4c 5668 5da1043
8.25619e- 0.3404 0.4426 3.6786 05 33 03 9
34
6d7c3ac3bc80073c32ce3c6af8 3149 66cbd5
0.000147 0.3193 0.4343 3.6424 524 65 98 6
148
35
7f677e0706921a774db8ca9a1 2860 05f1523
0.000257 0.3389 0.4462 3.9130 116 17 44 9
36
80961aee880c3fea328226415 1779 d374d95
2.66943e- 0.3375 0.4756 3.6831 05 63 72
37
81077fb61c4ce5a7bccfbc5deb 2128 4355b6
1.71926e- 0.3130 0.4748 3.9098 05 23 52 4
38
841edef46bbfc71c8fb81c8259 2683 98c274
4.83095e- 0.3149 0.4399 3.9186 05 44 33 8
39
8d06e0a4e18595baf79ef756d 2760 3ee8361
5.30141e- 0.3712 0.4965 3.9258 06 48 09
40
8d4020d4435f7a4e942e979a2 7397 56e213b
1.98778e- 0.2748 0.3666 3.7962 05 3 35 6
41
8f40e8a195cf1dbfab9049f6ed 1860 101dae
5.20617e- 0.2910 0.3923 3.5975 05 95 07
42
8f9a3d466ebe5b66ed11ce42c 675 60721d9
0.000167 0.2856 0.3998 3.4676 155 14 92
43
8ff8cdf1b0b720fbbe5dfbc2259 687 14ab4
0.000349 0.2837 0.4031 3.5881 354 88 64 8
44
901edfef048bb4dfcccd1288bd 1697 45f92b
0.000111 0.2652 0.3812 3.6512 68 2 48
45
917b19aa1052996146a55dba1 915 6a69117
6.99466e- 0.2974 0.4443 3.2623 05 25 55 7
46
93c693b0bd437642cbb234142 1433 a9249d5
0.000328 0.4049 0.5374 3.8729 541 79 82 5
47
94a12f1e56d53a0c4c236f479d 1897 e5e20f
1.12578e- 0.2723 0.4079 3.9811 05 82 43 8
48
952ad1bd4ec8f66e108cb2299 4585 909e1d6
0.000158 0.3597 0.4844 3.9775 172 37 99 5
149
49
997818ecf4a57def4c249fb5b3 2759 bd2ffe
0.000118 0.3568 0.4854 3.7629 181 94 14 3
50
99ed79f870e6f210e8c157cb86 2225 9c842e
5.62073e- 0.2719 0.3860 3.7080 05 73 19 5
51
9a12c7115f557d8721496b26cf 2622 592c90
4.0867e- 0.2389 0.3756 3.9570 05 37 16 8
52
9a6558294246cb4038f528e9a 1253 0178e8b
4.94973e- 0.2861 0.4076 3.5583 05 67 09 3
53
9de14a7a7071156a8e83a1024 1762 3a6bd1e
6.91256e- 0.2948 0.3912 3.6833 05 5 21 9
54
9e0e00cce62de9c75ac6ae5db 5196 31be6d0
8.6286e- 0.2690 0.3931 3.7762 06 55 69 7
55
9e90be2b97ec9af6eb151c7c13 2248 44e40d
0.000109 0.3081 0.4106 3.4635 927 5 76 9
56
a0c05fb0398b885433ab6e9ea 1252 be221ed
0.000306 0.2936 0.4069 3.6714 123 52 84 4
57
b01609a4880b15a704b082161 1670 3d687e8
0.000152 0.3018 0.4071 3.2528 314 45 05 8
58
b0f1aff3b70f9740c4991d122d 2038 93f4f0
2.21015e- 0.2759 0.4332 3.6769 05 35 36 3
59
b286ea58635135261fe9b888c 997 c12c4c6
0.000460 0.2791 0.3889 3.2379 748 08 29 4
60
b2ad60f705e224a3f837f3b768 2558 0e4745
0.000237 0.4075 0.5118 3.7677 886 75 78 5
61
b497aa31fbb85d5f8f1c8cd8d4 7164 cfdb28
7.29837e- 0.2705 0.3541 3.7860 05 68 11 2
62
b8c00541735d82c2cc25f4f029 1636 8b0b2c
0.000129 0.2401 0.3434 3.6952 562 36 25 1
150
63
bbfde2f923ca2c665a66b197aa 7021 ec9e46
5.15089e- 0.2916 0.3731 3.6271 05 1 19 2
64
bf1db3e2e682a3a1c79487cdf2 1287 8f9b39
0.000683 0.3787 0.4956 3.4546 174 59 46 4
65
c5de39455c1040509f2d2d9e5 5463 77f6a80
2.7857e- 0.2830 0.3774 3.7463 06 41 82 7
66
c72830fd73ef2ef5f4e42b1d17 2443 50ec5e
6.22313e- 0.2847 0.3931 3.8972 05 89 22 2
67
c7ddf4d3c513fdcebb2372a15f 2126 b520f1
4.63906e- 0.3811 0.5162 3.8860 05 08 43 5
68
c8da058dfaf12f5e1035660d9d 7974 841873
6.3388e- 0.4159 0.5207 3.9798 05 94 35
69
cb70abb007a2f7f1f6e7ffbde4e 1658 827cf
0.000131 0.3718 0.5170 3.5947 084 16 08 6
70
cc0e7babd3ff66914e2bac0810 3003 07b9cd
4.6028e- 0.3634 0.4600 3.6495 05 55 06 9
71
cd17f7c991a1ebd8cc475fc56e 3283 9e72f5
0.000137 0.3281 0.4400 3.7972 108 58 18 2
72
cd46e4d8c6da36ac0ae08e16b 2531 dd06890
0.000300 0.3863 0.4997 3.2810 899 4 28 6
73
cfbd27d0c466df76014f3f5a45 1463 8a640f
0.000185 0.3445 0.4711 3.7650 388 27 38 6
74
d1dec98875009fae9b9c18c844 2194 a0e534
7.22952e- 0.3742 0.5040 3.7695 05 53 03 5
75
d3581a582ce36425f355944bd 2285 18144b0
0.000191 0.3347 0.4816 3.8288 943 66 24 5
76
d523feed7e00d46532ed9f4a4 2803 455946a
5.88147e- 0.3067 0.4213 3.7282 05 57 95 8
151
77
d593665c1a06d29081c582f41 1297 0ae8747
0.000224 0.3534 0.4622 3.5168 878 07 94 3
78
d8df6b76006e9e9d1ec64b0a5 3423 45ed989
2.82945e- 0.3421 0.4675 3.6809 05 85 99 6
79
df80e28e8810d6b5953ea16c1 1136 4c5f77a
0.000177 0.2613 0.4306 3.8811 448 06 18
80
e17511242cbde4c8f874bf854d 4782 ae76fa
1.19149e- 0.3295 0.4110 3.7326 06 21 4
81
e3518dcda56bcabfba9e20845 2473 34419c3
0.000117 0.3216 0.4614 3.9670 696 72 98 3
82
e439302636af65b536bdb128b 1632 964736f
5.49849e- 0.3458 0.4523 3.2964 05 96 02 1
83
e4c3470a3a808ffb0397f7c368 1080 311a1b
7.61259e- 0.3289 0.4363 3.8096 05 5 85
84
e6a8d3d93dc0ba90dd8bf4ea8 3500 ee9cf1e
8.16626e- 0.3540 0.4749 3.8276 05 58 72 9
85
ecc69b6ad746729294b669c89 2008 00777dc
0.000113 0.2969 3.3028 0.4099 843 91 1
86
f03fa130dcdb326d61427a718b 4656 3a8a4e
2.09339e- 0.2381 0.4047 3.9328 06 43 46 4
87
f1513a1e248a27d88e977d34d 2429 9e64d29
2.50748e- 0.3531 0.4807 3.8196 05 64 26 5
88
f674d4102cf37915a1fe2cfb189 2609 3c7b3
4.17473e- 0.3395 0.4516 3.5157 05 41 06 4
89
fc0e1c103098e21cdc0c427e2a 6634 dbfb79
5.633e-05
90
fc93f5b91cc2964d0023bf29a9 1245 6568df
0.000634 0.4016 0.5321 3.9084 012 71 87 7
0.3869 0.4932 3.9027 77 62 8
152
SfM Scene RMSE: 0.607263
153
UGM Building Dataset
SFM report.
Dataset info: #views: 67 #poses: 67 #intrinsics: 1 #tracks: 13832 #residuals: 55664
IdVie Basename w
Residu Residu Residu #Observati Residuals als als als media ons min mean max n
0
00c1f7b8ca530f3e7c60818a4e 1012 bb1044
1.61859e- 0.3907 0.5538 5.8165 05 75 73 3
1
09217ad491c645b6c07210608 786 4de92e7
0.000127 0.6054 0.7541 8.9573 858 26 97 1
2
0edce68587fd672a7dc85ec04 479 de116a2
0.000116 0.3477 0.5398 5.6200 463 52 56 8
3
0f833531d004c832781f4d844f 1454 e26bce
0.000280 0.4716 0.6090 4.0441 196 38 11 5
4
0f9899a09c5b38a1d2a83bc9cd 523 428cae
0.000281 0.4104 0.5804 9.8539 186 09 99 9
5
11705d74aa0b05e74764dea4b 177 892dace
0.003747 0.4691 0.6570 3.5971 78 67 04 3
6
1371a412d9e05e04678acd133 574 67ab7df
0.000556 0.4056 0.5845 4.1306 453 05 61 3
154
7
13a2d84fe8eb91cc05e16e799 627 5fea277
0.000100 0.4220 0.5822 4.0764 585 37 73 8
8
17c3947a54c0d5a5867e04282 1230 4759ce4
7.3752e- 0.3080 0.4767 6.6157 05 23 52 7
9
1c9cc5f8c9ed167d46c2150d36 2227 52b66f
2.62449e- 0.4119 0.5436 9.2516 05 84 63 8
10
1d8221294f9dc484d238665df 439 251f7f1
6.23162e- 0.3098 0.5172 8.7598 06 48 47 4
11
1d85b49cb4de3d92e7b3deb76 835 77f6a3a
0.000509 0.3779 0.5503 4.3229 858 23 79 1
12
25d98489a5c4bd1fcbfa9c3282 580 d405ca
6.02424e- 0.4291 0.6088 5.4115 05 27 36 6
13
2debab297507afaaec9bd3f65c 88 bbf22d
0.001405 0.4957 0.6458 3.2020 7 28 25 9
14
2f0e0a33deb78a27ca829e7ea 431 20e5602
9.26197e- 0.4348 0.6383 6.7610 05 13 81 5
15
2f0ea7336794900a8fe83977cf 1419 c60621
5.19307e- 0.3720 0.5248 5.8675 05 27 15 4
16
313933d530aeef53764b26570 1232 f75dafa
0.000106 0.3404 0.4924 5.1310 887 9 02 1
17
343d8f95417da0eaedf1171a3 926 d2ff8d1
8.54853e- 0.3660 0.5312 8.7006 05 28 32 8
18
3d775f239c7c65c859e814371c 2038 dc8b00
4.15055e- 0.3923 0.5558 4.4489 06 55 93 2
19
3f6b88461d6a4666bfe412eafb 1104 81d1d5
1.22297e- 0.3286 0.4978 3.868 06 81 47
20
41d5341514c9327716ab29bc0 747 0dbac6f
2.15238e- 0.4101 0.5948 3.7328 05 11 64
155
21
52c04681b64750ebac58c49d1 723 63a6f4b
0.000681 0.6011 0.7576 6.3186 746 96 59 8
22
5f1eeb80166b487ada6d92fad 407 5918b1f
1.15987e- 0.4277 0.6139 13.156 05 11 99 9
23
68376016a63f70896aa1ce5c6a 1156 f67f5d
1.72678e- 0.4166 0.5718 5.4999 05 56 71
24
684d8baac3f530c8a0be0b5c75 562 01319f
1.79643e- 0.3192 0.4535 3.7581 05 68 18 2
25
6a19cbfc162afac6dbe8748259 557 3e94c2
0.000190 0.3548 0.5681 5.3448 372 78 48
26
6adc4141da53577ead66478d0 616 2210c88
8.04654e- 0.4115 0.5833 5.9686 05 46 63 5
27
6eb694b8857b2a96bab5234a2 245 c0e6cee
0.000334 0.7377 0.9125 14.109 889 75 59 9
28
705e147a6546706c3b1740429 1059 a792a40
0.000147 0.3825 0.5373 5.7847 455 56 06 1
29
74a980a72da538e8602a3a314 519 d54df5c
0.000424 0.4045 0.6073 11.203 41 54 45 4
30
7519b8d8c5ef4fdba9e569b48 617 86d9402
2.49968e- 0.2281 0.4181 4.0228 05 1 75 2
31
75c3e81d54fb8dd8ed2a59274 1587 3b49952
2.56085e- 0.3658 0.5119 6.3656 05 09 68 9
32
7bd710dd79b4e08945d3c46c8 1155 c556376
3.06227e- 0.3527 0.5108 4.4979 05 36 84 4
33
81c0089b030e532121ef6c1c3 458 9fa5cce
6.38605e- 0.4508 0.6028 4.7054 05 39 08 7
34
83796151c4aeeab833c03a025 1213 dbb299a
0.000300 0.4507 0.5929 5.1621 531 8 62 8
156
35
858d4c65e3faa6b370d817846 346 e2fe3a1
0.000357 0.5986 0.7887 14.092 909 7 23 3
36
867d062516e71e0cae5f548c9 172 d053519
0.003902 0.5985 0.7935 6.0179 92 12 69 8
37
8907d198229ea05dd7be43409 500 c7181d5
0.000166 0.4668 0.6249 6.6422 3 91 91 1
38
89c37eadba7db2ad092afe335 132 9f67da4
2.94904e- 0.3211 0.5418 3.9354 05 91 79
39
8cb5c78f6f1b56485ffbd62bb4 1789 88635e
2.73922e- 0.3791 0.5151 5.0496 06 6 82 9
40
97a49360627068c217c82a069 811 02526c5
6.859e-05
41
98c2fdb6a99c0004dbcc2273ba 746 37f892
6.97441e- 0.3633 0.5432 4.0537 06 67 23 1
42
9e330a16d9c80f608112060d5 506 732b963
0.000512 0.4837 0.6524 8.2416 697 72 57 1
43
9fff754fdae9a50e7cdf285632c 2006 06e1e
0.000102 0.7763 0.9406 5.5867 604 63 38 5
44
a344ef081e185341d7543c7e2 474 11dd853
0.000212 0.5075 0.7041 9.0406 737 45 55 4
45
a5a0f53a888243f16eceea945f 1851 16ad70
4.54918e- 0.4315 0.5800 4.3741 05 11 43 2
46
a6eab25e0899923f0ad7953ae 532 39090cd
4.12629e- 0.4357 0.5892 4.4333 05 57 89 2
47
ac3fd8cd7ddfd42502eaa92a96 560 f5cf56
0.000225 0.4416 0.6326 6.9475 178 78 34
48
aca0c1598b026629b8c2e2ab8 1166 9f445ac
5.26851e- 0.4023 0.5535 11.017 06 89 17 4
0.3697 0.5657 4.9047 68 48 6
157
49
b78ed74f0a619407bbc94c870 226 343fbab
0.000281 0.4779 0.6241 3.8983 874 62 7 6
50
b90269d303729b9201ee72a2e 917 a512a6f
7.97687e- 0.4004 0.5873 16.059 05 91 57 7
51
b9a490c5910e2a8ef4846a3d9 1071 b8504cc
9.67099e- 0.3674 0.5280 3.8270 05 57 76 7
52
bd3b719d161acd5530595cde3 625 b192b73
0.000141 0.3906 0.5609 4.5064 941 04 11 2
53
bd99eadbb5c6730879715e641 529 9e067d1
8.7421e- 0.4693 0.6314 8.1751 05 67 6 7
54
c52c1070d2b7f714143ec50dd 623 50408a5
0.000183 0.5252 0.7101 4.8810 305 82 44 1
55
ca9b9d0a4960ac02edc280d29 655 77edd77
1.65878e- 0.4070 0.5843 9.7567 05 14 37 8
56
cc2f152b5863fd66a0d1aae80b 1055 c986ee
0.000166 0.4416 0.6159 9.3434 446 23 16 1
57
d0d35d20065710d69888860b 2273 89e95eec
0.000250 0.4476 0.5975 4.4378 816 01 79 4
58
d16b9c4cfabb44208d70afc576 794 61936c
8.88406e- 0.4488 0.6447 9.6688 06 47 41 5
59
dd38019ff80b7a6f16c0b48174 1228 ca2d5f
2.83695e- 0.3675 0.5211 6.6001 05 57 59 4
60
e1883ca15ca44473770344f5c9 514 857555
0.000136 0.3276 0.4659 4.1345 435 02 12 7
61
e74f6476c0c2b7f840de5683de 495 8b1fa7
6.93125e- 0.4349 0.6702 9.9447 05 39 58 6
62
e81401f9f70c2d8e1c8b38fc6c 976 4765ca
0.000105 0.3920 0.5486 4.0756 058 75 38
158
63
fc3702efe7e9e192576ced8293 741 6a2d7f
7.72381e- 0.4363 0.6481 10.474 06 88 22 4
64
fc5e8a4e62dc14c1b244b7758 426 808ae8c
9.50642e- 0.4584 0.6237 4.4554 05 25 33 5
65
fe804f8e190942dd6df434fbe7 402 c81bfa
0.000890 0.4597 0.6107 4.5124 138 95 48 9
66
febb3fe70d7263fdbbd510ea63 721 292874
0.000116 0.3777 0.4909 4.3387 665 09 8 4
SfM Scene RMSE: 0.845978
159
VI.5. Sample Terminal Output of a CloudSfM project Sample terminal output produced in the server side processing of a CloudSfM project (UGM Dataset) You called : openMVG_main_SfMInit_ImageListing --imageDirectory images/ --sensorWidthDatabase --outputDirectory ./output --focal 3916.8 --intrinsics --camera_model 3 --group_camera_model 1 - Image listing 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| *************************************************** SfMInit_ImageListing report: listed #File(s): 67 usable #File(s) listed in sfm_data: 67 You called : openMVG_main_ComputeFeatures --input_file ./output/sfm_data.json --outdir ./output/matches --describerMethod SIFT --upright 1 --describerPreset HIGH --force 0 - EXTRACT FEATURES 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| *************************************************** Task done in (s): 358.102 You called : openMVG_main_ComputeMatches --input_file ./output/sfm_data.json --out_dir ./output/matches Optional parameters: --force 0 --ratio 0.8 --geometric_model e --video_mode_matching -1 --pair_list --nearest_matching_method AUTO --guided_matching 0 - Regions Loading 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| *************************************************** - PUTATIVE MATCHES Use: exhaustive pairwise matching Using FAST_CASCADE_HASHING_L2 matcher Using the OPENMP thread interface
160
0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| *************************************************** Task (Regions Matching) done in (s): 359.531 - Geometric filtering 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| *************************************************** Task done in (s): 248.667 Export Adjacency Matrix of the pairwise's geometric matches ----------------------------------------------------------Global Structure from Motion: ----------------------------------------------------------Open Source implementation of the paper: "Global Fusion of Relative Motions for Robust, Accurate and Scalable Structure from Motion." Pierre Moulon, Pascal Monasse and Renaud Marlet. ICCV 2013. ------------------------------------------------------------ Features Loading 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| *************************************************** CleanGraph_KeepLargestBiEdge_Nodes():: => connected Component: 1 Connected component of size: 67 Cardinal of nodes: 67 Cardinal of edges: 654
- Relative pose computation 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| *************************************************** ------------------------------Global rotations computation: #relative rotations: 654 #global rotations: 67 Statistics about rotation triplets: min: 0.0418159 mean: 1.98372 median: 1.38579 max: 21.2826
0 1.06 2.13 3.19 4.26 5.32 6.38
| | | | | | |
1189 877 426 186 131 79 42
161
7.45 8.51 9.58 10.6 11.7 12.8 13.8 14.9 16 17 18.1 19.2 20.2 21.3
| | | | | | | | | | | | |
16 26 5 6 7 4 3 1 1 2 1 0 1
Triplets filtering based on composition error on unit cycles #Triplets before: 3003 #Triplets after: 2786 #Edges removed by triplet inference: 14 CleanGraph_KeepLargestBiEdge_Nodes():: => connected Component: 1 Connected component of size: 67 Cardinal of nodes: 67 Cardinal of edges: 640 Found #global_rotations: 67 ------------------------------Relative translations computation: ------------------------------#Triplets: 3003 Relative translations computation (edge coverage algorithm) 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| *************************************************** TRIPLET COVERAGE TIMING: 32.021 seconds -------------------------------- #Relative translations estimates: 400 computed from 3003 triplets. -- resulting in 1200 translations estimation. -- timing to obtain the relative translations: 32.021 seconds. ------------------------------CleanGraph_KeepLargestBiEdge_Nodes():: => connected Component: 1 Connected component of size: 67 Cardinal of nodes: 67 Cardinal of edges: 649
------------------------------Global translations computation: - Ready to compute 67 global translations. from #relative translations: 1200 Feasibility 1 -------------------------------- #relative estimates: 1200 converge with gamma: 9.56536.
162
timing (s): 6.779. ------------------------------Found solution: 0 0 0 -1.4748 -4.00167 5.38707 2.46067 4.76456 -1.25723 -27.4755 6.03062 19.9156 -2.13829 5.1591 -4.03799 17.8428 5.69521 9.26064 -13.7053 -4.3301 7.98424 0.46562 3.95093 4.32829 -12.0511 -0.597329 4.4976 -9.14331 -0.249056 7.95946 13.6383 -4.36645 8.52983 9.29309 9.18562 1.42154 -7.00767 -4.12708 8.23209 9.5444 -4.48025 3.65593 15.4335 -3.51977 6.24958 -1.36957 9.34496 7.71844 -5.61906 9.49831 -1.8261 0.378647 0.0492944 6.61987 -17.0885 -0.793224 13.2123 0.113568 5.06165 6.26658 -13.8423 -4.24257 -1.8678 -1.46928 4.56935 5.69403 2.60725 5.31733 2.10548 -4.97049 9.10703 7.12155 28.6719 14.4583 20.8848 15.9132 -3.50352 -0.6558 1.24784 9.31506 7.10464 -9.14413 0.00709534 0.194085 -3.30905 9.26059 8.02132 8.54417 -4.44164 5.43128 11.2522 9.01228 9.72592 -2.53228 -0.330979 6.61935 -17.8894 4.80702 16.2759 -0.327387 9.01394 5.95439 -27.3994 -3.82973 25.2892 0.425196 5.20121 -0.0711218 -14.467 -4.39674 10.1051 -10.0845 5.03568 0.233062 25.8923 5.09491 15.5976 -5.59274 -0.216572 5.49835 -23.5605 4.63776 7.88443 -7.30857 9.16304 7.07899 -5.08293 4.99613 6.02994 -12.802 -0.0694083 7.73652 0.715677 4.93563 -0.426937 -8.18578 9.0806 -1.88523 7.60168 -4.09336 2.35922 -0.0824946 -4.2572 -2.45648 -1.96354 -0.243104 7.15843 4.52466 5.32864 7.61255 -9.57881 -0.40366 3.59584 -2.85965 -0.0458017 -2.19944 5.49227 0.35054 4.63785 -36.5147 4.3673 17.4053 0.318426 -1.57309 6.07294 4.38905 5.2274 -1.62884 -1.43419 -0.174192 9.68839 -9.70446 8.93844 15.3484 -10.6266 -4.18779 -0.0494333 -1.03268 1.32744 -0.334215 28.3171 5.46415 21.7514 -14.3099 -4.02214 0.0647728 5.8462 9.22708 -1.89891 -1.62196 5.24439 -2.23925 -8.9644 -0.162471 0.193447 6.03952 8.28743 -1.33593 3.26359 -0.644265 4.12689 26.8891 13.9176 1 52.8863 56.8717 78.8244 1 79.084 76.4035 17.8078 37.672 76.8673 1 1 9.50969 28.5836 74.1381 15.0438 51.3701 79.4262 26.9236 33.5857 44.6059 1 20.7148 113.028 70.4558 1 111.336 16.5831 1 1 1.61809 80.4181 1 39.542 1 39.5743 35.3966 1 28.8855 30.2939 34.2134 91.427 30.2464 144.999 26.2503 16.962 36.6898 41.22 18.4698 145.013 1 70.589 20.2449 87.183 25.0229 68.0094 79.278 108.886 24.8423 41.8183 2.69747 22.1713 42.7895 10.8502 1 3.8824 82.4145 23.4764 33.3488 35.201 29.128 32.8696 24.2371 25.5525 11.2021 8.81119 69.4078 10.9335 17.8383 47.1218 11.636 1 84.5685 26.8674 28.3472 14.0589 1 102.655 53.2943 28.2544 82.2622 45.8618 69.176 55.1743 1 55.5483 102.547 1 53.8557 61.9698 13.662 11.73 36.9432 1 52.6232 10.1524 28.2479 1 18.4052 7.57864 47.7494 23.5682 67.0428 1 32.1791 90.0058 60.9561 61.1135 97.055 44.0919 2.34468 58.7481 75.4812 63.4913 79.1677 10.2454 70.9274 77.9862 8.848 1 97.0976 29.2414 15.7121 40.58 37.6601 1 42.2365 61.6671 79.2851 2.74019 1 80.5082 18.5295 33.9126 58.9154 13.6653 111.072 76.1709 57.2591 39.659 70.9096 31.3695 56.0977 74.2071 46.7732 44.7143 36.4185 66.3312 15.465 43.0154 39.6655 81.7335 26.0095 53.1546 12.8874 34.526 36.4265 61.8513 22.767 33.7536 53.1475 58.5207 91.2025 1 17.8569 3.97436 9.33786 1 88.4136 26.6393 1 1 13.3923 47.009 1 54.0862 56.7008 115.909 1 12.9283 62.696 57.5749 1 92.5107 64.6519 1 1 46.7199 62.7172 90.3891 1 1 4.50042 1 115.692 25.8104 27.876 1 86.5677 144.767 26.2181 26.7801 84.4404 31.8483 11.6387 13.6741 57.2367 31.9808 1 1 2.85522 33.3133 26.8581 24.9452 90.663 1 52.4808 1 15.5603 32.4123 36.3483 80.8457 46.1805 49.1006 49.6377 5.50777 13.321 1 16.9555 73.2659 44.3414 46.6965 108.338 18.136 1 31.1638 1 1 1 56.3556 1 44.4967 21.2613 39.3617 44.0444 1 116.312 123.201 33.3105 7.64435 7.50351 44.0151 21.807 1 3.00978 11.3577 1 1 1.64422 45.8218 1 1 77.4158 50.6852 96.4844 24.0175 30.604 53.804 15.8702 1 1 48.3453 18.2027 33.6737 56.0344 4.97321 20.6356 14.1571 26.251 1 25.2479 10.6809 20.476 62.0415 11.379 10.7736 31.1768 72.2478 1.78637 86.6095 23.9755 82.428 64.3448 52.9479 37.6352 1.26406 69.9158 202.608 86.5235 40.1316 33.5802 1 12.6767 34.5071 22.1715 33.7528 1 32.4463 8.9494 4.39434 67.0931 1 12.2749 12.0898 38.2334 17.0801 75.5034 21.4404 49.247 7.79245 37.397 1 308.923 10.3884 6.38611 34.0366 81.9719 22.1272 32.3983 3.38106 87.4245 11.0641 1 9.3711 1 1 31.706 1 52.2661 1.32505 1 1.22255 48.1186 8.80568 86.5779 8.91391 16.53 73.0257 27.8897 65.0178 46.1915 1 66.3723 7.49828 33.8492 32.4534 55.2425 81.968 55.7955 39.9335 4.8856 117.951 84.0281 1.59003 45.1351 20.9834 10.6509 26.6789 1 1 26.6144 16.8441 79.8244 4.81261 11.7788 16.8421 52.4063 118.641 32.5485 19.358 97.4213 28.6157 4.00226 7.68215 42.7041 23.7484 2.03361 1 9.56536
163
cam position: 0 0 0 -1.4748 -4.00167 5.38707 2.46067 4.76456 -1.25723 -27.4755 6.03062 19.9156 -2.13829 5.1591 -4.03799 17.8428 5.69521 9.26064 -13.7053 -4.3301 7.98424 0.46562 3.95093 4.32829 -12.0511 -0.597329 4.4976 -9.14331 -0.249056 7.95946 13.6383 -4.36645 8.52983 9.29309 9.18562 1.42154 -7.00767 -4.12708 8.23209 9.5444 -4.48025 3.65593 15.4335 -3.51977 6.24958 -1.36957 9.34496 7.71844 -5.61906 9.49831 -1.8261 0.378647 0.0492944 6.61987 -17.0885 -0.793224 13.2123 0.113568 5.06165 6.26658 -13.8423 -4.24257 -1.8678 -1.46928 4.56935 5.69403 2.60725 5.31733 2.10548 -4.97049 9.10703 7.12155 28.6719 14.4583 20.8848 15.9132 -3.50352 -0.6558 1.24784 9.31506 7.10464 -9.14413 0.00709534 0.194085 -3.30905 9.26059 8.02132 8.54417 -4.44164 5.43128 11.2522 9.01228 9.72592 -2.53228 -0.330979 6.61935 -17.8894 4.80702 16.2759 -0.327387 9.01394 5.95439 -27.3994 -3.82973 25.2892 0.425196 5.20121 -0.0711218 -14.467 -4.39674 10.1051 -10.0845 5.03568 0.233062 25.8923 5.09491 15.5976 -5.59274 -0.216572 5.49835 -23.5605 4.63776 7.88443 -7.30857 9.16304 7.07899 -5.08293 4.99613 6.02994 -12.802 -0.0694083 7.73652 0.715677 4.93563 -0.426937 -8.18578 9.0806 -1.88523 7.60168 -4.09336 2.35922 -0.0824946 -4.2572 -2.45648 -1.96354 -0.243104 7.15843 4.52466 5.32864 7.61255 -9.57881 -0.40366 3.59584 -2.85965 -0.0458017 -2.19944 5.49227 0.35054 4.63785 -36.5147 4.3673 17.4053 0.318426 -1.57309 6.07294 4.38905 5.2274 -1.62884 -1.43419 -0.174192 9.68839 -9.70446 8.93844 15.3484 -10.6266 -4.18779 -0.0494333 -1.03268 1.32744 -0.334215 28.3171 5.46415 21.7514 -14.3099 -4.02214 0.0647728 5.8462 9.22708 -1.89891 -1.62196 5.24439 -2.23925 -8.9644 -0.162471 0.193447 6.03952 8.28743 -1.33593 3.26359 -0.644265 4.12689 cam Lambdas: 26.8891 13.9176 1 52.8863 56.8717 78.8244 1 79.084 76.4035 17.8078 37.672 76.8673 1 1 9.50969 28.5836 74.1381 15.0438 51.3701 79.4262 26.9236 33.5857 44.6059 1 20.7148 113.028 70.4558 1 111.336 16.5831 1 1 1.61809 80.4181 1 39.542 1 39.5743 35.3966 1 28.8855 30.2939 34.2134 91.427 30.2464 144.999 26.2503 16.962 36.6898 41.22 18.4698 145.013 1 70.589 20.2449 87.183 25.0229 68.0094 79.278 108.886 24.8423 41.8183 2.69747 22.1713 42.7895 10.8502 1 3.8824 82.4145 23.4764 33.3488 35.201 29.128 32.8696 24.2371 25.5525 11.2021 8.81119 69.4078 10.9335 17.8383 47.1218 11.636 1 84.5685 26.8674 28.3472 14.0589 1 102.655 53.2943 28.2544 82.2622 45.8618 69.176 55.1743 1 55.5483 102.547 1 53.8557 61.9698 13.662 11.73 36.9432 1 52.6232 10.1524 28.2479 1 18.4052 7.57864 47.7494 23.5682 67.0428 1 32.1791 90.0058 60.9561 61.1135 97.055 44.0919 2.34468 58.7481 75.4812 63.4913 79.1677 10.2454 70.9274 77.9862 8.848 1 97.0976 29.2414 15.7121 40.58 37.6601 1 42.2365 61.6671 79.2851 2.74019 1 80.5082 18.5295 33.9126 58.9154 13.6653 111.072 76.1709 57.2591 39.659 70.9096 31.3695 56.0977 74.2071 46.7732 44.7143 36.4185 66.3312 15.465 43.0154 39.6655 81.7335 26.0095 53.1546 12.8874 34.526 36.4265 61.8513 22.767 33.7536 53.1475 58.5207 91.2025 1 17.8569 3.97436 9.33786 1 88.4136 26.6393 1 1 13.3923 47.009 1 54.0862 56.7008 115.909 1 12.9283 62.696 57.5749 1 92.5107 64.6519 1 1 46.7199 62.7172 90.3891 1 1 4.50042 1 115.692 25.8104 27.876 1 86.5677 144.767 26.2181 26.7801 84.4404 31.8483 11.6387 13.6741 57.2367 31.9808 1 1 2.85522 33.3133 26.8581 24.9452 90.663 1 52.4808 1 15.5603 32.4123 36.3483 80.8457 46.1805 49.1006 49.6377 5.50777 13.321 1 16.9555 73.2659 44.3414 46.6965 108.338 18.136 1 31.1638 1 1 1 56.3556 1 44.4967 21.2613 39.3617 44.0444 1 116.312 123.201 33.3105 7.64435 7.50351 44.0151 21.807 1 3.00978 11.3577 1 1 1.64422 45.8218 1 1 77.4158 50.6852 96.4844 24.0175 30.604 53.804 15.8702 1 1 48.3453 18.2027 33.6737 56.0344 4.97321 20.6356 14.1571 26.251 1 25.2479 10.6809 20.476 62.0415 11.379 10.7736 31.1768 72.2478 1.78637 86.6095 23.9755 82.428 64.3448 52.9479 37.6352 1.26406 69.9158 202.608 86.5235 40.1316 33.5802 1 12.6767 34.5071 22.1715 33.7528 1 32.4463 8.9494 4.39434 67.0931 1 12.2749 12.0898 38.2334 17.0801 75.5034 21.4404 49.247 7.79245 37.397 1 308.923 10.3884 6.38611 34.0366 81.9719 22.1272 32.3983 3.38106 87.4245 11.0641 1 9.3711 1 1 31.706 1 52.2661 1.32505 1 1.22255 48.1186 8.80568 86.5779 8.91391 16.53 73.0257 27.8897 65.0178 46.1915 1 66.3723 7.49828 33.8492 32.4534 55.2425 81.968 55.7955 39.9335 4.8856 117.951 84.0281 1.59003 45.1351 20.9834 10.6509 26.6789 1 1 26.6144 16.8441 79.8244 4.81261 11.7788 16.8421 52.4063 118.641 32.5485 19.358 97.4213 28.6157 4.00226 7.68215 42.7041 23.7484 2.03361 1 Track stats
164
------------------- Tracks Stats -Tracks number: 28270 Images Id: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, -----------------TrackLength, Occurrence 3 15603 4 3633 5 2514 6 1439 7 1081 8 786 9 625 10 499 11 372 12 365 13 249 14 211 15 151 16 138 17 120 18 92 19 71 20 75 21 46 22 50 23 28 24 19 25 18 26 15 27 15 28 18 29 9 30 4 31 5 32 4 33 5 34 1 35 1 36 1 37 1 39 2 40 1 41 1 46 1 47 1 Blind triangulation progress: 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| *************************************************** #removed tracks (invalid triangulation): 3985 Triangulation took (s): 0.04 Bundle Adjustment statistics (approximated RMSE):
165
#views: 67 #poses: 67 #intrinsics: 1 #tracks: 24285 #residuals: 251222 Initial RMSE: 76.5901 Final RMSE: 15.7652 Time (s): 13.6206
Bundle Adjustment statistics (approximated RMSE): #views: 67 #poses: 67 #intrinsics: 1 #tracks: 24285 #residuals: 251222 Initial RMSE: 15.7652 Final RMSE: 13.4023 Time (s): 14.6749
Bundle Adjustment statistics (approximated RMSE): #views: 67 #poses: 67 #intrinsics: 1 #tracks: 24285 #residuals: 251222 Initial RMSE: 13.4023 Final RMSE: 12.6062 Time (s): 20.0047 Outlier removal (remaining #points): initial structure size #3DPoints: 24285 pixel residual filter #3DPoints: 17455 angular filter #3DPoints: 13832 Bundle Adjustment statistics (approximated RMSE): #views: 67 #poses: 67 #intrinsics: 1 #tracks: 13832 #residuals: 111328 Initial RMSE: 1.08586 Final RMSE: 0.598196 Time (s): 7.57729
Total Ac-Global-Sfm took (s): 168.95 ...Generating SfM_Report.html ...Export SfM_Data to disk. Compute Structure from the provided poses - Regions Loading 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| *************************************************** Compute frustum intersection 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| ***************************************************
166
Compute pairwise fundamental guided matching: 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| *************************************************** Per triplet tracks validation (discard spurious correspondences): 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| *************************************************** Robust triangulation progress: 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| *************************************************** Structure estimation took (s): 648.801. #landmark found: 19483 Note: this program writes output in MVE file format. Note: this program writes output in MVE file format. [1;31mCreating directory: ./output/mve/MVE[0m [1;31mCreating directory: ./output/mve/MVE/views[0m Writing bundle (67 cameras, 13832 features): to synth_0.out... 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| *************************************************** MVE Scene to Pointset (built on Oct 6 2015, 18:38:44) Using depthmap "depth-L3" and color image "undist-L3" Initializing scene with 67 views... Initialized 67 views (max ID is 66), took 651ms. Processing view "images//0edce68587fd672a7dc85ec04de116a2.JPG" Processing view "images//0f833531d004c832781f4d844fe26bce.JPG" Processing view "images//09217ad491c645b6c072106084de92e7.JPG" Processing view "images//00c1f7b8ca530f3e7c60818a4ebb1044.JPG" Processing view "images//0f9899a09c5b38a1d2a83bc9cd428cae.JPG" Processing view "images//11705d74aa0b05e74764dea4b892dace.JPG" Processing view "images//1371a412d9e05e04678acd13367ab7df.JPG" Processing view "images//13a2d84fe8eb91cc05e16e7995fea277.JPG" Processing view "images//17c3947a54c0d5a5867e042824759ce4.JPG" Processing view "images//1c9cc5f8c9ed167d46c2150d3652b66f.JPG" Processing view "images//1d8221294f9dc484d238665df251f7f1.JPG" Processing view "images//1d85b49cb4de3d92e7b3deb7677f6a3a.JPG" Processing view "images//25d98489a5c4bd1fcbfa9c3282d405ca.JPG" Processing view "images//2debab297507afaaec9bd3f65cbbf22d.JPG" Processing view "images//2f0e0a33deb78a27ca829e7ea20e5602.JPG" Processing view "images//2f0ea7336794900a8fe83977cfc60621.JPG" Processing view "images//313933d530aeef53764b26570f75dafa.JPG" Processing view "images//343d8f95417da0eaedf1171a3d2ff8d1.JPG" Processing view "images//3d775f239c7c65c859e814371cdc8b00.JPG" Processing view "images//3f6b88461d6a4666bfe412eafb81d1d5.JPG" Processing view "images//41d5341514c9327716ab29bc00dbac6f.JPG" Processing view "images//52c04681b64750ebac58c49d163a6f4b.JPG" Processing view "images//5f1eeb80166b487ada6d92fad5918b1f.JPG" Processing view "images//68376016a63f70896aa1ce5c6af67f5d.JPG" Processing view "images//684d8baac3f530c8a0be0b5c7501319f.JPG" Processing view "images//6a19cbfc162afac6dbe87482593e94c2.JPG" Processing view "images//6adc4141da53577ead66478d02210c88.JPG" Processing view "images//6eb694b8857b2a96bab5234a2c0e6cee.JPG" Processing view "images//705e147a6546706c3b1740429a792a40.JPG" Processing view "images//74a980a72da538e8602a3a314d54df5c.JPG" Processing view "images//7519b8d8c5ef4fdba9e569b4886d9402.JPG" Processing view "images//75c3e81d54fb8dd8ed2a592743b49952.JPG" Processing view "images//7bd710dd79b4e08945d3c46c8c556376.JPG"
(with (with (with (with (with (with (with (with (with (with (with (with (with (with (with (with (with (with (with (with (with (with (with (with (with (with (with (with (with (with (with (with (with
colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)... colors)...
167
Processing view "images//81c0089b030e532121ef6c1c39fa5cce.JPG" (with colors)... Processing view "images//83796151c4aeeab833c03a025dbb299a.JPG" (with colors)... Processing view "images//858d4c65e3faa6b370d817846e2fe3a1.JPG" (with colors)... Processing view "images//867d062516e71e0cae5f548c9d053519.JPG" (with colors)... Processing view "images//8907d198229ea05dd7be43409c7181d5.JPG" (with colors)... Processing view "images//89c37eadba7db2ad092afe3359f67da4.JPG" (with colors)... Processing view "images//8cb5c78f6f1b56485ffbd62bb488635e.JPG" (with colors)... Processing view "images//97a49360627068c217c82a06902526c5.JPG" (with colors)... Processing view "images//98c2fdb6a99c0004dbcc2273ba37f892.JPG" (with colors)... Processing view "images//9e330a16d9c80f608112060d5732b963.JPG" (with colors)... Processing view "images//9fff754fdae9a50e7cdf285632c06e1e.JPG" (with colors)... Processing view "images//a344ef081e185341d7543c7e211dd853.JPG" (with colors)... Processing view "images//a5a0f53a888243f16eceea945f16ad70.JPG" (with colors)... Processing view "images//a6eab25e0899923f0ad7953ae39090cd.JPG" (with colors)... Processing view "images//ac3fd8cd7ddfd42502eaa92a96f5cf56.JPG" (with colors)... Processing view "images//aca0c1598b026629b8c2e2ab89f445ac.JPG" (with colors)... Processing view "images//b78ed74f0a619407bbc94c870343fbab.JPG" (with colors)... Processing view "images//b90269d303729b9201ee72a2ea512a6f.JPG" (with colors)... Processing view "images//b9a490c5910e2a8ef4846a3d9b8504cc.JPG" (with colors)... Processing view "images//bd3b719d161acd5530595cde3b192b73.JPG" (with colors)... Processing view "images//bd99eadbb5c6730879715e6419e067d1.JPG" (with colors)... Processing view "images//c52c1070d2b7f714143ec50dd50408a5.JPG" (with colors)... Processing view "images//ca9b9d0a4960ac02edc280d2977edd77.JPG" (with colors)... Processing view "images//cc2f152b5863fd66a0d1aae80bc986ee.JPG" (with colors)... Processing view "images//d0d35d20065710d69888860b89e95eec.JPG" (with colors)... Processing view "images//d16b9c4cfabb44208d70afc57661936c.JPG" (with colors)... Processing view "images//dd38019ff80b7a6f16c0b48174ca2d5f.JPG" (with colors)... Processing view "images//e74f6476c0c2b7f840de5683de8b1fa7.JPG" (with colors)... Processing view "images//e1883ca15ca44473770344f5c9857555.JPG" (with colors)... Processing view "images//e81401f9f70c2d8e1c8b38fc6c4765ca.JPG" (with colors)... Processing view "images//fc3702efe7e9e192576ced82936a2d7f.JPG" (with colors)... Processing view "images//fc5e8a4e62dc14c1b244b7758808ae8c.JPG" (with colors)... Processing view "images//fe804f8e190942dd6df434fbe7c81bfa.JPG" (with colors)... Processing view "images//febb3fe70d7263fdbbd510ea63292874.JPG" (with colors)... Writing final point set (3865837 points)... Writing PLY file (3865837 verts, with colors, with normals, with confidences, with values, 0 faces, with colors)... done.
168
