Visualization of building models and sensor data ...

Visualization of building models and sensor data using open 3D platforms Juho-Pekka Virtanen, School of Engineering, Aalto University (email: [email protected]) Matti Kurkela, School of Engineering, Aalto University (email: [email protected]) Hannu Hyyppä, School of Engineering, Aalto University, Built Environment Hubic, Helsinki Metropolia University of Applied Sciences (email: [email protected]) Sami Niemi, Vahanen Building Physics Ltd (email: [email protected]) Sami Kalliokoski, Electria, Helsinki Metropolia University of Applied Sciences (email: [email protected]) Seppo Vanhatalo, Electria, Helsinki Metropolia University of Applied Sciences (email: [email protected]) Juha Hyyppä, The Finnish Geospatial Research Institute, National Land Survey of Finland (email: [email protected]) Henrik Haggrén, School of Engineering, Aalto University (email: [email protected])

Abstract Modern facilities management in digital systems requires user-friendly tools for visualization and decision making, and improved data. To help achieve this, 3D models can operate as a platform for integrating sensor data and other documentation for a building. If 3D models are not readily available for a specific building, measuring methods can be applied. In this article, a prototype of a building model with a sensor data visualization system is presented using an open source 3D platform. The prototype is utilized to visualize 3D models produced by terrestrial laser scanning, thermal camera images, and sensor data. Several development directions are identified for such systems. Keywords: indoor air quality, building model, laser scanning, maintenance, visualization

1

1. Introduction In building information modeling (BIM), the information representing different disciplines in the field of construction (architecture, engineering, etc.) are integrated into a three dimensional (3D) digital model. With the help of BIM, it is possible to detect collisions and conflicts in an early stage of design (Azhar, 2011) and, for example, improve scheduling in the construction phase (Tulke et al., 2008). However, the construction and planning only represent a small fragment of the total life cycle of a building. Contemporary buildings contain a multitude of different technical systems that need to be managed during the operational life of the building. The increasing complexity of the buildings is putting pressure on the traditional methods of facilities management, as stated by Ahn and Cha (2014). Several authors have suggested that a BIM-based solution has to be applied for facilities management (Becerik-Gerber et al., 2012; Ahn and Cha, 2014). By aggregating facilities management data into a BIM model, the data management issues caused by missing or nonsynchronous documentation can be solved (Ahn and Cha, 2014). Sick building syndrome exists in Finnish public buildings (Audit Committee of the Parliament of Finland, 2013), as in so many other countries globally. Complicated building structures and growing needs for better indoor air quality increase the need to monitor, adjust, and supervise building parameters properly and illustrate data visually in a compact form (Hietsalo et al., 2014). More sensors and wireless networks are needed to fulfill those requirements. In particular, there is a demand for better control of relative humidity (RH), dew point calculations, CO2 sensors and total volatile organic compound (TVOC) sensors to assure healthy indoor air. The absolute amount of water vapor in indoor air can be determined by combining data on RH and temperature. The difference between the amount of water vapor in indoor and outdoor air is a good indicator of the efficiency of building ventilation. In a similar manner, the pressure difference between the building interior and exterior is a good indicator of a properly operating ventilation system: If the ventilation system is correctly adjusted, the pressure difference should be low to prevent any unintentional air flows, and so forth. For monitoring the building envelope, continuous measuring is important, as the amount of moisture can, for example, vary significantly during a single day. As a result, one time measurements become unreliable. Sensors can also be integrated into building structures. By measuring the moisture of building structures, it is possible to monitor, for instance, the drying of a concrete structure after repair work. By combining the data from indoor condition measurements and structural measurements, there is the potential to estimate the risk of mold growth. If BIM-based facilities management methods are to be applied in these cases, measuring and modeling becomes essential. On this point, Tang et al. (2010) provide an overview of the automated as-built BIM reconstruction problem and some of the partial solutions. This reconstruction can be performed manually, by viewing a point cloud in a computer-aided design (CAD) suite and building the model (Mill et al., 2013). A similar approach can be found in building modeling using terrestrial laser scanning (TLS), as presented by Arayici et al. (2007) for

2

example. In most cases, however, the modeling is too laborious. While the project may be carried out without the comprehensive model, many of the benefits of BIM are not attained. For both manual and semi-automatic BIM reconstruction, laser scanning can be used as a measuring method. In measuring the built environment, it is usually applied as TLS, with the instrument being mounted on a tripod. In laser scanning, the laser pulse travel time or phase difference is used to calculate distances relative to the scanner position. By combining data from the scanning angle and impact of atmospheric reflection caused to the speed of light on the time data, 3-dimensional locations of the scanned points can be identified. If the scanner positions are known, several scans can be combined to create a larger point cloud of the built and natural environment (Virtanen et al., 2014c; Lehtola et al., 2014; Kankare et al., 2013). The high point density and fast acquisition speed make laser scanning an efficient method for documenting complex building surfaces (Virtanen et al., 2015), rapid prototyping (Virtanen et al., 2014a), and detecting structural deformations in buildings with great accuracy (Virtanen et al., 2014b). On this point, Chee Wei et al. (2010) present the workflow for utilizing terrestrial laser scanning from a cultural heritage documentation project. The documentation of the chosen artifact is conducted with a laser scanner. Scanning targets are used for both registering the scans and geo-referencing the project. In addition, digital images are taken with the integrated camera system of the scanner to obtain RGB values for the scanned points (Chee Wei et al., 2010). Mobile laser scanning (MLS) has generated a large amount of interest for measuring and modeling urban environments. In particular, it is suited for measuring in outdoor environments, producing models of e.g. building facades. Typically, MLS systems are mounted on regular cars, but there have been implementations of MLS on a cart or an all-terrain vehicle (ATV). One such example of a MLS system is the ROAMER system, developed at the Finnish Geodetic Institute (Kukko et al., 2007). The essential components of a MLS system are a laser scanner, GPS receiver, inertial measuring unit (IMU), and data recording system. The GPS and IMU are jointly used for the accurate localization of the MLS systems (Kukko et al., 2007). In addition to laser scanning, image-based measuring methods (digital photogrammetry) can be applied for precise 3D documentation (Virtanen et al., 2012; Kurkela et al., 2012). Different types of images (panoramic image mosaics, image databases, sensor images, and multispectral photographs) are employed to produce information from the environment. Another significant source of data concerning buildings are sensors. Several types of sensors can be installed in a building to measure a variety of parameters (such as CO2, temperature, and amount of light). In this field, one of the research directions is wireless sensor networks (WNS) that can be applied in construction and building maintenance. As the “health” of a building is dependent on a large group of factors, different sensors are required to follow a group of parameters. To produce comprehensive data, these sensors have to be scattered across all parts of the building and also installed inside the structures. WNS can help reduce the amount of cabling needed for such installations and thus help reduce costs. When using sensor data for decision making in construction, data visualization becomes a central task. A large amount of data must be presented, preferably dynamically, so that it can be seen in near real time (Hammoudeh et al.,

3

2015). If this data is to be utilized in decision making with the 3D building models, they have to be visualized together. This both simplifies the interpretation of data and enables the detection of correlations (Motamedi et al., 2014). It is possible to identify temporal correlations from the sensor data, e.g. the impact of microbiological activity on the CO2 concentration. It is also possible to search for long term trends and thus try to identify potential problems before they emerge and become acute. In current planning and maintenance systems, buildings are mostly studied as individual entities. This reduces the overall efficiency that could be reached by looking at interactions not only within a single building but between buildings. Networking the buildings and analyzing their data, energy, and resource flows is the key to unlocking the full potential of these networked, smart buildings. The resulting Internet of Buildings can share resources and thus optimize the use of energy and services, to maximize the efficiency when using renewable natural resources. For buildings in this networked ecosystem, new business models can be created, thus improving the conditions of the building owner as both a consumer and provider of services (Lukin, 2015). The four mentioned aspects are combined in the future digital models used for building upkeep and maintenance (Figure 1). Firstly, a digital 3D model operates as a platform for information and forms the starting point for all building maintenance activities. Secondly, all data concerning the building is aggregated on top of this model, the individual data sets being bound to respective model components. Thirdly, the sensor data from the building is integrated into the same model and visualized. And finally, the model has to operate in a networked system, enabling the integration of several buildings or other online resources. In addition, if the proposed solution is to be applied to the current building stock, for which no BIMs exist, the starting data for the model has to be generated using the existing 3D measuring techniques.

Figure 1: The aspects of future digital models used for building upkeep and maintenance

4

Thus, this paper presents a 3D visualization method as a tool for analyzing and viewing 3D building models with additional sensor data. 3D measuring techniques are utilized to produce models that depict the building as it is in use. The visualization results focus on indoor air quality measurements. This solution will provide information for monitoring the status of the indoor environment of the building. The goal of the system is to improve indoor air quality and reduce the cost of unnecessary renovations. To proof the system, various sensor data will be combined with a 3D model. Data from imaging sensors, such as a thermal camera, and individual sensors, such as temperature, moisture, and carbon dioxide, is used as the test data set. Moreover, an open 3D application development platform is utilized to visualize the building models. A user interface is then developed to facilitate the study of time series. The resulting model can then be used to study the state of the building and its indoor air quality. Based on this prototype, the aim is to outline future development directions. To take the whole operational life of the building into account, models have to include, as much as possible, all information concerning the use and upkeep of a building.

2. Materials and methods 2.1 Measuring methods for indoor environments The open source 3D Internet application platform was used to host the building models created and operate the developed applications. The resulting virtual scene is defined in a TXML file, following the XML syntax. Each object in the scene is defined, with the object geometry being stored in an external file, accessible over an http connection (realXtend, 2015; Alatalo, 2011). To obtain indoor measurement data from one of the test sites, the test utilized the Faro Focus 3D terrestrial laser scanner and Matterport Pro 3D Camera. Faro Focus 3D is a conventional TLS instrument with an integrated camera designed to obtain texture images that can be used to colorize the point cloud in later processing (Faro, 2015). The distance measurement accuracy of the instrument is ±2 mm at a distance of 25 m (Chow et al., 2012). Faro Scene software was utilized to process and co-register the TLS scans. Matterport is a commercial solution for measuring indoor environments based on depth camera technology (Matterport, 2015). The device is used to acquire a set of panoramic depth images from the environment, with each image covering a full 360° field of view. The 3D reconstruction process is performed automatically with cloud computing. After the user has uploaded the measured data, an optimized alignment of the 3D captures is solved, and a textured mesh model created (Bell et al., 2013). Four different model reconstruction methods were employed: three dimensional models were created from the TLS point cloud using AutoCAD 2016 and Geomagic Design X. In addition, the automated generation of a mesh model was performed from the TLS point cloud by using Geomagic Studio 11. For the Matterport data, the automatic online service of Matterport was used to obtain the mesh models. After the modeling, game engine compatible models were created in Blender for use as game engine equivalents for the visualization.

5

2.2 Sensor networks Data from the 720 indoor air quality measuring system (720, 2015) was employed in the experiment. The sensors were installed in two rooms of the test building. The data consisted of the hourly averages of the CO2 concentration, relative humidity, temperature, and TVOC concentration. In total, the time series contained some 1030 readings, spanning a time period of 43 days. A small application was also written on the platform to visualize the data.

2.3 Online 3D building maintenance model To assemble an online 3D representation of all data available from the building, a data model was utilized that divided the data into both individual spaces and data types. In the resulting model, each single entity of data (e.g. a Flir image or a 3D indoor model) was given two identifiers, a room code and a data type. The data types are provided in Table 1. The full set of metadata defined for objects is given in Table 2. Table 1: Data types Name

Description

2D CAD

Two dimensional CAD drawing

3D CAD

Three dimensional CAD model

Mesh

Triangulated 3D mesh

FLIR

Thermal image

Picture

Image

Point cloud

Three dimensional point cloud

Document

Text document

Other

Other than above types

Table 2: Object metadata Attribute

Description

Data type

Data type (listed in Table 1.)

Data source

Description of source (e.g. measuring instrument used)

Room identifier

Unique identifier of room in building (e.g. room number)

Building identifier

Identifier of building (e.g. address or name)

URL

Path of the full data (e.g. on intranet server)

Additional information

Any additional descriptions of data set

As the chosen game engine-based system was unable to represent all object types, game engine equivalents were created for all objects. These equivalents were created by producing mesh models of the reduced polygon counts for all 3D objects and textured planes for the 2D objects.

6

After this, the location of the actual data was input to the metadata of the object in TXML. Thereafter, an application was developed that allows the user to select individual spaces of the building, to choose the displayed objects according to data types, and to study the values of the sensor time series.

3. Results By using the methods described, a virtual scene depicting two rooms of the test building was created. The orientation of these data sets was resolved manually, and the metadata was input for the objects. The resulting scene was hosted using a realXtend server. In the tests, a local machine was used to run the server and access the scene. In the scene, different measuring data sets are displayed in the same virtual 3D space. In the scene, a user can filter the displayed object according to the data structure described, fetching data sets of a certain class from each room individually. The counts of objects available on the server for each type are displayed. In addition, there are tools to hide all data sets or isolate an individual data set for study. The values obtained by the 720 sensors are displayed next to each room. To enable functionality, a user interface is provided for jumping backwards and forwards in the time series. Accordingly, Figure 2 illustrates the system displaying Matterport mesh models for both rooms, the sensor data for both rooms, and a set of thermal images for the other room. As shown, the user can move freely in the virtual 3D space. Figure 3 also demonstrates a preview of a thermal image being shown inside a mesh model.

Figure 2: System in use, displaying some data for both rooms

7

Figure 3: Studying an individual thermal image in the data set

4. Discussion The presented setup is able to display objects of different types in a virtual 3D space and also sort them according to the metadata. There are a number of advantages that can be found in the proposed system. Firstly, the concept of using game engine equivalents is essential for the development of computationally light applications that operate over the Internet. As there are already browser-based systems that utilize the same architecture (Meshmoon, 2015), the possibility of developing fully browser-based tools remains. A simplified, game engine-optimized model can, in these systems, be used to depict a larger, more complex model. In this way, the user can preview the model relatively quickly and then download the full model later, using the existing tools of the industry (e.g. point cloud processing software or full-featured CAD) to study and process the actual model. Secondly, the usage of game engine equivalents enables the utilization of existing game engines for application development. This is already being conducted in several cases where, for example, Unity (Unity, 2015) is used to display building models. Thirdly, by leveraging the typical features of game engines, the system is able to represent both 2D and 3D data along with dynamic and static data in the same environment. However, as the system is still a very early prototype, it has a number of apparent shortcomings. Firstly, the amount of automation in the content production is very low. Content building methods more familiar to the game development industry than construction have to be applied for making game engine equivalents of the 3D models. As such, the system is unable to directly support model formats commonly used in construction (e.g. DWG or IFC). Secondly, the orientation of the data sets with respect to each other has to be solved manually, if the data sets are in a different coordinate system. For example, Matterport mesh models had to be manually moved to match the coordinate system of the TLS campaign. Finally, there are no automated tools for adding new

8

objects to the scene; the user must use the tools offered by the platform. These shortcomings render the current system unfeasible for immediate wider adaption. Comparing the presented prototype to the aspects of future digital models employed for building upkeep and maintenance as presented in the introduction, it is possible to identify several development directions for the system. The future development roadmap is summarized in Table 3. The main development directions are the increase in the applicability of the system to large models, the inclusion of the entire building life cycle, and finally the support for sensor networks. With these development directions, the integration of geometric, sensor, and maintenance data becomes an important research focus. Table 3: Future development directions

Application

Large, detailed models

Building Life Cycle

Network of smart buildings

Renovation

Life cycle simulation

Multi-building systems

Existing building stock

Building maintenance

Modeling of large complexes

Comparison with asbuilt data, identification of deviations

Prediction models, alerts

Data integration

Integration of geometric, sensor, and planning data

Platform development

Support for large data sets Segmentation, Optimization

Data sources

IFC models

Interfaces

Integration to planning systems

Data visualization

Model change detection & updating

Big Data

Facilities management

Internet of Buildings

Open Data

Building planning

Dense sensor networks

Measuring methods

5. Conclusions Smart management of existing buildings requires new tools that integrate data from several sources and enable the study of networks of buildings. Based on the literature, the four features of future models used for building upkeep and maintenance were identified: the use of a virtual model as a starting point for operations, the aggregation of data on top of the model, the integration of sensor data to the model, and the building of an Internet technology-based networked system for maintaining and accessing the model. In the presented experiment, data from 3D measuring techniques and imaging was used to build a prototype of such a model, consisting of two rooms in a larger facility. To achieve this, virtual world technology was used as a platform for the prototype. A simple application was then developed to facilitate the study of the models and other data. As such, the system was able to

9

visualize the data. However, the low degree of automation in constructing the model prevents the immediate adoption of the proposed system. Future development directions include the increase of automation in the content building of the presented system, the utilization of browser-based technologies, and the pilot use of the system to better identify development needs. With measuring techniques such as MLS, a 3D time series can be created for identifying geometric changes in the environment. 3D application platforms based on client-server architecture are well-suited for prototyping these models and visualizing data from the various sensors together with the 3D models.

Acknowledgments The authors wish to thank 720 Degrees Oy for sample data. This work is funded by the Academy of Finland with its support in the projects “Centre of Excellence in Laser Scanning Research (CoE-LaSR)” (No. 272195) and Strategic Research Council project “COMBAT” (No. 293389); the Tekes project "Healthy Building" and partly Tekes funded RYM “EUE” program”; the European Union; the European Regional Development Fund "Leverage from the EU 2014–2020” projects "AKAI" (301130) and "Soludus" (301192); the Aalto Energy Efficiency Research Programme ("Light Energy—Efficient and Safe Traffic Environments"); and the Aalto University doctoral program.

References 720 (2015) Your answer to indoor air quality, (available online http://www.720.fi/en/ [accessed on 30/11/2015]) Ahn D and Cha H (2014) “Integration of Building Maintenance Data in Application of Building Information Modeling (BIM)” Journal of Building Construction and Planning Research 2(2). Alatalo T (2011) “An entity-component model for extensible virtual worlds”, Internet Computing, 15(5): 30-37. Arayici Y (2007) “An approach for real world data modelling with the 3D terrestrial laser scanner for built environment” Automation in Construction 16(6): 816-829. Audit Committee of the Parliament of Finland (2013) Rakennusten kosteus- ja homeongelmat, TrVM 1/2013 vp - M 5/2013. Azhar S (2011) “Building information modeling (BIM): Trends, benefits, risks, and challenges for the AEC industry” Leadership and Management in Engineering 3(11): 241-252. Becerik-Gerber B, Jazizadeh F, Li N and Calis G (2011) “Application areas and data requirements for BIM-enabled facilities management” Journal of construction engineering and management 138(3): 431-442.

10

Bell M, Gausebech D and Beebe M (2013) U.S. Patent Application 13/776,688. Chee Wei O, Siew Chin C, Majid Z and Setan H (2010) “3D documentation and preservation of historical monument using terrestrial laser scanning” Geoinformation Science Journal 10(1): 7390. Chow J, Lichti D and Teskey W (2012) “Accuracy assessment of the FARO Focus3D and Leica HDS6100 panoramic-type terrestrial laser scanners through point-based and plane-based user self-calibration”, FIG Working Week 2012: Knowing to manage the territory, protect the environment, evaluate the cultural heritage, 6-10 May, Rome, Italy. FARO (2014) 3D Surveying (Available online www.faro.com/products/3d-surveying [Accessed 30/11/2015]). Hammoudeh M, Newman R, Dennett C, Mount S and Aldabbas O (2015) “Map as a Service: A Framework for Visualising and Maximising Information Return from Multi-ModalWireless Sensor Networks” Sensors 15(9): 22970-23003. Hietsalo P, Lindholm M, Ahlavuo M and Hyyppä H (2014) ”Näkökulmia terveeseen taloon”, Yhteistä tulevaisuutta rakentamassa ja kartoittamassa, Metropolia Ammattikorkeakoulu, Helsinki: 152-162. Kankare V, Holopainen M, Vastaranta M, Puttonen E, Yu X, Hyyppä J, Vaaja M, Hyyppä H and Alho P (2013) ”Individual tree biomass estimation using terrestrial laser scanning” ISPRS Journal of Photogrammetry and Remote Sensing 75: 64-75. Kukko A, Andrei CO, Salminen VM, Kaartinen H, Chen Y, Rönnholm P Hyyppä H, Hyyppä J, Chen R, Haggren H, Kosonen I and Čapek K (2007) “Road environment mapping system of the Finnish Geodetic Institute—FGI Roamer”, Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci, 36: 241-247. Kurkela M, Hyyppä H, Virtanen JP, Zhu L, Ahlavuo M, Alho P, Haggrén H and Rönnholm P (2012) "Social and Interactive 3D Imaging and Computational Photography Techniques", Helsinki Photomedia, 28-30 March, 2012, Helsinki, Finland. Lehtola VV, Kurkela M and Hyyppä H (2014) “Automated image-based reconstruction of building interiors–a case study” Photogrammetric Journal of Finland 24(1). Lukin E (2015) Internet of Buildings – Kiinteistöt tuottamaan kiertotaloudella, (available online https://www.tekes.fi/nyt/uutiset-2015/fiksu-kaupunki-uutiset/internet-of-buildings--kiinteistottuottamaan-kiertotaloudella/ [accessed on 30/11/2015]). Matterport (2015) 3D For the real world, (available online http://matterport.com/ [accessed on 30/11/2015])

11

Mill T, Alt A and Liias R (2013) “Combined 3D building surveying techniques–terrestrial laser scanning (TLS) and total station surveying for BIM data management purposes” Journal of Civil Engineering and Management 19: 23-32. Motamedi A, Hammad A and Asen Y (2014) “Knowledge-assisted BIM-based visual analytics for failure root cause detection in facilities management” Automation in Construction 43: 73-83. realXtend (2015) Open source platform for the 3D internet, (available online http://realxtend.org/ [accessed on 30/11/2015]) Tang P, Huber D, Akinci B, Lipman R and Lytle A (2010) “Automatic reconstruction of as-built building information models from laser-scanned point clouds: A review of related techniques” Automation in construction 19(7): 829-843. Tulke J, Nour M and Beucke K (2008) “A Dynamic Framework for Construction Scheduling based on BIM using IFC”, Proceedings of 17th Congress of IABSE: 17-19. Virtanen JP, Hyyppä H, Ahlavuo M, Markkula M, Miikki L, Hyyppä J, Kurkela M, Launonen P and Hollström T (2014a) ”Monialaisesti ratkaisuja kaupungistumiseen - Energizing Urban Ecosystems”, Yhteistä tulevaisuutta rakentamassa ja kartoittamassa, Metropolia Ammattikorkeakoulu, Helsinki: 73-81. Virtanen JP, Puustinen T, Pennanen K, Vaaja MT, Kurkela M, Viitanen K, Hyyppä H and Rönnholm P (2015) ”Customized visualizations of urban infill development scenarios for local stakeholders” Journal of Building Construction and Planning Research 3(02). Virtanen JP, Hyyppä H, Kurkela M, Vaaja M, Alho P and Hyyppä J (2014b) ”Rapid Prototyping—A Tool for Presenting 3-Dimensional Digital Models Produced by Terrestrial Laser Scanning” ISPRS International Journal of Geo-Information 3(3): 871-890. Virtanen JP, Kurkela M, Hyyppä H (2014c) “Using 3D in Design–An Overview of Measuring Methods and Experiences”, Proceedings of NordDesign 2014, 27-29 August, Espoo, Finland. Virtanen JP, Hyyppä H, Kurkela M, Manner J, Alho P, Jaakkola A and Vaaja M (2012) "Emerging technologies for capturing spaces – review of selected research-oriented demo cases", Helsinki Photomedia, 28-30 March, 2012, Helsinki, Finland.

12