Making Visible the Invisible. Augmented Reality Visualization for 3D ...

Making visible the invisible. Augmented Reality visualization for 3D reconstructions of archaeological sites Roberto Pierdicca1? , Emanuele Frontoni1 , Primo Zingaretti1 , Eva Savina Malinverni1 , Francesca Colosi2 , and Roberto Orazi2 1

Universit´ a Politecnica delle Marche, Via Brecce Bianche 12, 60131 Ancona (Italy) {r.pierdicca,e.frontoni,p.zingaretti,e.s.malinverni}@univpm.it http://www.univpm.it 2 CNR-ITABC Via Salaria km. 29,300, 00016, Monterotondo Street, Roma, Italy {francesca.colosi,roberto.orazi}@itabc.cnr.it http://www.itabc.cnr.it/

Abstract. In archaeology, findings are completely freed from the earth which had been covering for centuries but in most cases they must be covered again in order to protect them. In this paper we present a augmented reality (AR) experience for the visualization of 3D models in-situ, giving the possibility to see conceived findings. The experience was carried out for archaeological purposes in Chan Chan, the America’s greatest pre-Columbian town. From 2001, our mission is operating at Chan Chan carrying on a wide action of documentation, conservation and exploitation. We propose an interesting workflow: image acquisition, 3D photogrammetric reconstruction, 3D simplification, AR visualization. Also a knowledge base applied to archaeological sites is here presented. Keywords: Augmented Reality, Archaeology, Knowledge base, 3D Reconstruction, Survey, Visualization

1

Introduction

Augmented Reality (AR) is a growing technology in the field of Computer Vision. AR has the capability of adding virtual objects in the physical reality, allowing users a direct communication with the exhibitions. AR can be summarized as an enhancement of sensory experience using digital or computer-generated contents, with the aim to increase knowledge about everything that surrounds us. Thanks to the cinematography, the 1960 is considered the year of birth of the AR, when Morton Heilig [14] proposed Sensorama, a system able to increase the sensory perception of the reality. Only thanks to the work of Krueger et al. [13], the first system of Virtual Reality (VR) that allowed the users to interact with ?

Corresponding author: tel. +39 3283248263

2

Pierdicca et al.

virtual object has been created, known as Videoplace. The passing from the VR to the AR has only occurred in the 1990 [12], when the information came out from the device, due to the overlapping of digital contents. AR has reached a high degree of development during the year 2000 when it has involved different fields of application [2], [1]: medicine, education, navigation, urban planning and architecture, military, entertainment and cultural heritage. Considering the cultural heritage aspect, only in the recent years a large number of works has been proposed [8], [18], [23]. An important guide for archaeological sites is represented by the work of Vlahakis et al. [21]. Archeoguide provides a personalized augmented tour of archaeological sites, using augmented reality methods to reconstruct ruined sites, simulating also the ancient life. The system allows to recreate the archaeological site, maintaining the user in the real world. Another AR system is Lifeplus applied to historical and archaeological sites that uses handled devices on site displays [19]. Another most recent work [3] combines historical and archaeological details to improve the experience of the visitors by imaging the relation archaeological ruins with the ancient landscape. ARAC Maps [10] is a very recent AR application for archaeological sites, with the aim to enhance archaeological maps using 3D models together with other interactive approaches. Nevertheless, the use of AR for archaeological applications is still broadly missing; although, as we have previously said, there are several examples of projects with this aim, there is the necessity to introduce such a technology to drive people towards a new concept of archaeological tourism. For this purpose, the aim of our project is to cope with the lack of widespread interactive solutions to serve archaeologist and tourists. Augmented reality allows to discover, in an alternative way, monuments or ruins by simply scanning the surrounding environment, loading contents from a remote repository and visualizing them as virtual objects. There would be many advantages using AR solutions for both visitors and scientists. From one hand, additional information can be given at any point of the tour, visitor interests can be closely matched, user interaction supports and increase learning process, Edutaiment (Educational Enterteinment) fascinates children as well as adults. From the other side, scientists and insiders can raise their awareness of cultural heritage, have a useful tool to better conduct their research, verify their archaeological interpretation of findings, improve their documentation activity. The most important feature is the possibility to visualize findings, also when they are covered. It is well known that, due to conservation reasons, once the rests, monuments or graves are discovered, if there is not the possibility to store and save them, they are covered again. AR allows the visualization of rests, also where they are invisible. To exploit AR experience the main components are wearable or handled devices with a series of embedded sensors for tracking systems (e.g. camera, compass, GPS receiver and so on), Network Connectivity card and input devices for user interaction (e.g. display or vocal commands). For the development, many free and open-source software solutions are growing, which allow to integrate sensors with the contents, in order to make the device

Augmented Reality for Archaeology

3

able to analyze and know the surrounding reality. Every device able to geo-localize the user and to overlap contents could be used, as well as to inform, also to drive him with a sort of augmented route guidance. This technology is just beginning but combining GIS data, localization and a simple tablet, it is already possible to walk out onto the site and augment it with sights, sounds, 3D reconstructions and virtual models of past people. Linking them is a good solution for way finding and opens up new perspectives and possibilities of exploring the results of spatial modeling, changing chronologies, and different types of building reconstructions. The paper is organized as follow: next session introduces Chan Chan archaeological site, with particular focus over those findings used for our AR experience Section 3 describes the acquisition campaign and the techniques of survey, section 4 is an explanation to achieve a good solution of AR visualization, 4 and 5 are focused on the methodology related to, respectively.

2

Case Study: Chan Chan Archaeological Site

The archaeological complex of Chan Chan is located in the northern part of Peru no far from the town of Trujillo and about 550 km. from Lima. The town, that represents the largest mud brick pre-Columbian settlement covers an area of 14 km2 and is placed on a sedimentary terrace a few hundred meters from the Pacific Ocean. Chan Chan was the capital of the Chimu culture, one of the great civilizations that arose along the coast of northern Peru during the early centuries of our age. The most important architectural typology of the urban area is represented by nine palaces or ciudadelas. Each of them is made of a large enclosure four to seven hundred meters long and two to five hundred meters wide. The nine ciudadelas are characterized by some recurrent architectural features: the main entrance is placed on the north side, everyone of the palaces is divided in three main sectors the first of which is characterized by the presence of a large square for public ceremonies and meeting with the king. The second and the third sectors are reserved for the king and his court and present a great quantity of small yards, warehouses, houses and audiencias (small U shaped rooms with ritual function). Moreover the second sector is characterized by the “plataforma de entierro” or royal grave where the king was going to be buried at the moment of his death ([4], [6]). The entire palace was usually decorated with geometrical or natural bass-relieves along the walls of the corridors or of the audiencias representing scenes referred to the activity of fishing and to the marine world. From a strictly geographic point of view, Chan Chan is located in the tropical zone between the Equator and the Tropic of Capricorn, but due to a cold water stream that runs along the coast of Peru, the temperature of the region is quite mild and the rainfall extremely scarce. The climate is therefore ideal for the conservation of earthen architecture, though some heavy factors of decay are present. These last are mainly caused by illegal excavations that have been carried on since the time of the Conquistadores, by marine salt transported by the wind or absorbed by the walls through capillarity and, finally by

4

Pierdicca et al.

the sometime devastating phenomenon of the Ni˜ no. Unfortunately, due to the precarious state of conservation, many of the friezes have been lost. The few which are still on the site have been documented by simple photographs and protected from the atmospheric agents by covering the original decoration with a new mud brick wall. Due to this situation, to the large dimension of the site and to the fact that the touristic visit is limited to Palacio Tschudi, the only one restored, it is impossible to have a look to the precious decorative aspects of the Chimu civilization if not by mean of old and bi-dimensional pictures. Up to now it was difficult to plan an exposition of the friezes of the town, but thank to new technologies for the diffusion of information this action is not anymore impossible. An occasional opportunity arose last year while the PECACH (Proyecto Especial Complejo Arqueologico Chan Chan), that is the local operative branch of the Ministero de Cultura, was working at the restoration of some parts of Palacio Rivero. The main door introducing into the large square of the first sector is the architectural element that we exploited for the test presented in this paper. The door was completely freed from the earth which had been covering the structure for centuries; at this stage we had the possibility, in just few hours, to take photogrammetric shoots of the original entrance before being covered again in order to protect it. Figure 1 shows the original entrance door before and after its coverage.

Fig. 1. The main door introducing into the first sector. Upper image: before the excavation the door covered by earth. Lower image: the door freed from the earth

The Italian Mission in Peru, which is operating on the site since 2002 ([7]), in accordance with a PECACHs program of documentation and monitoring of all the hidden friezes of Chan Chan, is going to realize the 3D models of the wall decorations in order to show them in a specific immersive room of the


5

new Museum and arrange, in this way, a kind of pre-visit to the wide and still unknown Chimu decorative world.

3

Acquisition and Photogrammetric reconstruction

Besides the extreme importance of keeping safe such an important monument, the work carried out over the entrance door pioneered several issues in the field of archaeological conservation and dissemination, which for years are object of discussion in the scientific community. It is well known that excavations are generally conducted in emergency conditions, surveys are often unprogrammed and the documentation of findings is a hard task during campaigns. The workflow presented in this paper has the purpose to face these issues and to outline best-practices in the field of archaeology, from the excavation phase until the dissemination and fruition through AR. The introduction of new photogrammetric techniques acquisition and cutting edge solution for the visualization of rests are mandatory for both conservation/documentation and promotion of archaeology. With our approach we provide: – Free and low cost solutions allowing unplanned data acquisition; – Rapid and smart techniques to facilitate documentation during excavation work; – High level of detail for the representation of complex 3D objects; – Contextualized visualization of archaeological findings.

3.1

Acquisition Campaign

During our campaign in Chan Chan in the past years, to perform the 3D survey we used different types of technologies such as GPS, laser scanner or spherical photogrammetry [5]. However, these techniques require an accurate a priori planning of the survey campaign, as well as a costly and cumbersome equipment. In this case we faced with a condition of emergency, besides the fact that the documentation and acquisition was totally unprogrammed; furthermore, the only available equipment was a calibrated SONY SLT-A77V camera, hence we performed the survey via terrestrial photogrammetry technique. The camera features CMOS Exmor APS-C with 24,3 MP resolution, coupled with an 24-70 mm objective. Approximately 440 images were acquired with the same survey, carried out in November 2013. The images were acquired keeping the camera at the focal length of 40 mm, with the highest resolution (6000 x 4000 pixels), with an overlap of about 30”%”. Pictures were taken from 5 points of view (POV) in order to have a full coverage of the architecture; each set of pictures for each POV has been used to create a panorama (see Figure 2), using a stitching software. This allows to speed up the process of photogrammetric reconstruction and to simplify the texturing procedure, as explained in the following section.

6

Pierdicca et al.

Fig. 2. Panoramas output from the stitching software. Left and right part of the archaeological finding .

3.2

Photogrammetric Reconstruction

For the 3D model reconstruction of the findings we adopted Structure from Motion (SfM) techniques. Even if there are many different packages available [11] for this kind of process, the steps to achieve a reality-based model of the object are quite similar. In particular in our case we have: – – – –

A set of pictures of the door, with a very high resolution; Images alignment to fix the camera position; Point cloud generation; Mesh generation.

For our work we used Agisoft Photoscan, a commercial software able to create 3D content from still images. The final output of this procedure is summarized in Figure 3, where are visible the images POVs, as well as detailed information over the pictures and the 3D model characteristics. At this stage, the 3D model is ready to be used for different purposes; for archaeologist the maximum level of detail (LOD) is needful for documentation or restoration aims. However, such a high LOD requires very high computational expenses, and the 3D model would be unusable for touristic or public visualization purposes. This is true especially for an Augmented Reality experience; 3D models are visualized into commercially available smart devices, where real time rendering performances are yet low; besides, due to portability issues, applications require strong optimization. 3D models, developed with the aforementioned techniques, are visible in Figure 4; the high accuracy of reconstruction, as well as the resolution of the textures, would not allow the visualization in AR. A hard task is the model simplification with the higher lossless compression possible. This problem has been addressed and the AR was made possible, as explained in the next section.


7

Fig. 3. Images during the SFM process. After the orientation phase, the relative position of the photos is fixed. The final result is a detailed 3D mesh of the door (in this case a single portion) .

Fig. 4. 3D models with maximum LOD. .

4

Augmented Reality

The application developed for this case study performs location-based AR to visualize in-context the entrance door rests. As the archaeological findings are conceived, a classical image-based target detection is not possible. Furthermore, a location-based AR allow to simply retrieve contents from the database. There are several key aspects that must be taken into account for this kind of AR experience; terrestrial coordinates are mandatory for a correct positioning of 3D models. This is possible thanks to GPS receiver built-in into smart devices, while the coordinates of the model have been calculated from the images exif file; in fact, the camera is equipped with GPS receiver that register the position of

8

Pierdicca et al.

each single shot. Transformation parameters have been stored into the database, while the device orientation is expressed as a 3-D vector thanks to the gyroscope. Managing all these information it is possible to project the 3D model on the screen from the perspective of the camera’s view, based on the orientation of the device. The application was developed for iOS devices, and in particular was tested on iPad Air due to its highest performances (i.e. hardware components and display resolution). The development has been conducted using XCode 5 and Metaio SDK framework an Augmented Reality framework which provides advanced development tools 3 . The user experience visualization is strictly related to the ability of displaying accurate and detailed 3D models quickly. As said before, the problem is to provide simplified model, without loosing quality. However, polygon counting, textures dimensions and file formats for mobile applications reveal several limitations4 . To overcome these issues, a tidy process of simplification was carried out in order to achieve the results visible in Figure 5, while data about 3D model are shown in Table 1.

Fig. 5. On site augmented reality visualization .

5

Discussion: towards a Cultural Object Standardization

In this part we present our proposal of creating a knowledge base applied to archaeological sites, that can be easily extended to cultural objects in general, always in term of augmented reality. In our paper we refer to an archaeological site, in this context our idea is to create a standard for the augmented reality 3 4

http://www.metaio.com https://dev.metaio.com/content-creation/3d-animation/polygon-count/generalguidelines/


9

Table 1. List of parameters and features about 3D models. Max LOD is referred to the photogrammetric process output, while AR LOD is referred to the 3D model after the simplification. Features Max LOD AR LOD Polygon counting 84697 14029 Faces 199999 20058 Vertexes 99805 10173 3D model dimension 134,33M b 1,953M b 3D model file format .ply .obj Texture dimension 54M b 4,5M b Texture file format .tif f .png

applied to archaeological sites with the aim of adding to the real world, a 3D model, together with multimedia objects (such as image, text, audio, video) and/or a description, based on this standard. For the aim of this paper, the idea is to dictate general guidelines valid to build an AR environment for archaeological sites. Where most of the monuments become ruins have the necessity to be restored or rebuilt. From the standardization point of view, ARCO project [16], [17], [22] also proposed a novel metadata element set useful to describe cultural objects and their representation for building virtual exhibition. However, this application is for museum as other referring works [15], [20], [9].

Fig. 6. Graphical representation of XML Schema archaeological site.

Figure 6 shows the graphical representation of XML schema of an archaeological site that is also valid for any cultural object. The scheme shows the attributes characterizing the cultural object class. The cultural object is described both regarding physical, localization, conservation aspects and also from

10

Pierdicca et al.

the AR point of view. This last refers to the 3D model of the virtual object and to the multimedia objects that can be superimposed to the real environment.

Fig. 7. Graphical representation of Class Description.

According to the specifications based on ICCD (Istituto Centrale per il Catalogo e la Documentazione)5 we have defined the Description class, as Figure 7 shows. In this class are listed physical, localization, conservation and chronological aspects describing the archaeological site, important for its correct identification and geolocalization. The attributes of the 3D model and of the multimedia objects overlapping to the site are described together with their features in figures 8 and 9. The 3D model of the virtual object superimposed on the real environment is created on the base of the attributes of Model3D. Observing Figure 8, we listed the attributes of the 3D model, highlighting the key characteristic of being a scalable 5

http://www.iccd.beniculturali.it/index.php?it/473/standard-catalografici


11

Fig. 8. Graphical representation of Class 3D Model.

object. In other words, the 3D model can be adapted to build any different object compatible with whichever resolution or file format, and so on. The purpose is to create a 3D model class more scalable possible in view of the standardization. A scalable structure means making data multipurpose; as explained in section 3.2, the LOD needed by archaeologists or scientist is different from tourist’s one.

Fig. 9. Graphical representation of Class Multimedia Object.

12

6

Pierdicca et al.

Conclusions

In this paper we outlined best practices for the applicability of Augmented Reality in the archaeological domain, starting from unplanned survey campaign. Chan Chan, as well as many other archaeological sites in the world, hide their priceless rests under earth due to conservation reasons; although this practice allows to preserve cultural heritage, prevents the wider public to see these rests. AR allows the visualization of rests, also where they are invisible. 3D models have been tested and handled to be suitable for AR visualization, for a quick and responsive user experience. Our solution pursue the aim to outline a workflow for archaeologists during their campaigns. The 3D visualization and the distribution of additional information is needful for both insiders or non-expert users. In the last paragraph, we have also introduced the idea to create a knowledge base applied to cultural object in general and then to archaeological sites, always in term of augmented reality. Even if the approach here presented is still experimental, we retain that it is important to further develop this aspect in order to construct a knowledge base for Cultural Heritage using augmented reality. This knowledge base would become a reference standard to develop cultural object project that exploit augmented reality.

References 1. Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., MacIntyre, B.: Recent advances in augmented reality. Computer Graphics and Applications, IEEE 21(6), 34–47 (2001) 2. Azuma, R.T., et al.: A survey of augmented reality. Presence 6(4), 355–385 (1997) 3. Bernardini, A., Delogu, C., Pallotti, E., Costantini, L.: Living the past: Augmented reality and archeology. In: 0002, J.Z., Schonfeld, D., Feng, D.D. (eds.) ICME Workshops. pp. 354–357. IEEE (2012) 4. Campana Delgrado, C.: Estudio de la ciudad de adobe ms grande de amrica latina. Editorial Orus (2006) 5. Colosi, F., Gabrielli, R., Malinverni, E., Orazi, R.: Strategies and technologies for the knowledge, conservation and enhancement of a great historical settlement: Chan chan, peru. In: Proceedings of Conference Built Heritage 2013 Monitoring Conservation and Management. pp. 56–64. Editor in chief M. Boriani, edited by R. Gabaglio, D. Gulotta (2013) 6. Colosi, F., Orazi, R.: Studi e progetti per la creazione del parco archeologico di chan chan. Rivista italiana di studi americanistici 30/31, 139–172 (2011) 7. Colosi, F., Fangi, G., Gabrielli, R., Orazi, R., Angelini, A., Bozzi, C.A.: Planning the archaeological park of chan chan (peru) by means of satellite images, gis and photogrammetry. Journal of Cultural Heritage 10, e27–e34 (2009) 8. Damala, A., Marchal, I., Houlier, P.: Merging augmented reality based features in mobile multimedia museum guides. In: Anticipating the Future of the Cultural Past, CIPA Conference 2007, 1-6 October 2007,. pp. 259–264 (2007) 9. Doerr, M.: The cidoc conceptual reference module: an ontological approach to semantic interoperability of metadata. AI magazine 24(3), 75 (2003) 10. Eggert, D., H¨ ucker, D., Paelke, V.: Augmented reality visualization of archeological data. In: Cartography from Pole to Pole, pp. 203–216. Springer (2014)


13

11. Herrmann, H., Pastorelli, E.: Virtual reality visualization for photogrammetric 3d reconstructions of cultural heritage. In: Augmented and Virtual Reality, pp. 283– 295. Springer (2014) 12. Kato, H., Billinghurst, M.: Marker tracking and hmd calibration for a video-based augmented reality conferencing system. In: Augmented Reality, 1999.(IWAR’99) Proceedings. 2nd IEEE and ACM International Workshop on. pp. 85–94. IEEE (1999) 13. Krueger, M.W., Gionfriddo, T., Hinrichsen, K.: Videoplacean artificial reality. In: ACM SIGCHI Bulletin. vol. 16, pp. 35–40. ACM (1985) 14. L, H.M.: Sensorama simulator (Aug 28 1962), http://www.google.com/patents/ US3050870, uS Patent 3,050,870 15. McKenna, G., Association, M.D., et al.: SPECTRUM: The UK museum documentation standard. Museum Documentation Association (2005) 16. Mourkoussis, N., Liarokapis, F., Darcy, J., Pettersson, M., Petridis, P., Lister, P., White, M.: Virtual and augmented reality applied to educational and cultural heritage domains. In: In proceedings of Business Applications of Virtual Reality, Workshop (2002) 17. Mourkoussis, N., White, M., Patel, M., Chmielewski, J., Walczak, K.: Ams– metadata for cultural exhibitions using virtual reality. In: International Conference on Dublin Core and Metadata Applications. pp. 193–202 (2003) 18. Noh, Z., Sunar, M.S., Pan, Z.: A review on augmented reality for virtual heritage system. In: Learning by Playing. Game-based Education System Design and Development, pp. 50–61. Springer (2009) 19. Papagiannakis, G., Ponder, M., Molet, T., Kshirsagar, S., Cordier, F., Magnenatthalmann, N., Thalmann, D.: Lifeplus: Revival of life in ancient pompeii. In: Proceedings of the 8th International Conference on Virtual Systems and Multimedia (VSMM 02) Gyeongju, Korea. pp. 25–27 (2002) 20. Trant, J., Richmond, K., Bearman, D.: Open concepts: museum digital documentation for education through the amico library. Art Libraries Journal 27(3), 30–42 (2002) 21. Vlahakis, V., Ioannidis, N., Karigiannis, J., Tsotros, M., Gounaris, M., Stricker, D., Gleue, T., Daehne, P., Almeida, L.: Archeoguide: an augmented reality guide for archaeological sites. IEEE Computer Graphics and Applications 22(5), 52–60 (2002) 22. White, M., Mourkoussis, N., Darcy, J., Petridis, P., Liarokapis, F., Lister, P., Walczak, K., Wojciechowski, K., Cellary, W., Chmielewski, J., et al.: Arco-an architecture for digitization, management and presentation of virtual exhibitions. In: Computer Graphics International, 2004. Proceedings. pp. 622–625. IEEE (2004) 23. Zollner, M., Keil, J., Wust, H., Pletinckx, D.: An augmented reality presentation system for remote cultural heritage sites. In: Proceedings of the 10th International Symposium on Virtual Reality, Archaeology and Cultural Heritage VAST. pp. 112–116. Citeseer (2009)