3D Augmented Reality Mobile Navigation System ...

Download PDF
4 downloads 148579 Views 4MB Size Report
Comment
information via earphones. This system is the pioneer of the .... platforms of Window, iOS, and Android in order to beautify the scene, add special effects, and ...
3D Augmented Reality Mobile Navigation System Supporting Indoor Positioning Function Ching-Sheng Wang

Ding-Jung Chiang

Yi-Yun Ho

Dept. of Computer Science and Information Engineering Aletheia University Taipei, Taiwan [email protected]

Dept. of Digital Multimedia Design Taipei Chengshih University of Science & Technology Taipei, Taiwan [email protected]

Dept. of Computer Science and Information Engineering Aletheia University Taipei, Taiwan [email protected]

The National Palace Museum in Taiwan developed a passive RFID digital navigation system [1, 2], which added an RFID Reader to a mobile device and affixed RFID tags to the exhibition cabinets of exhibits. The RFID Reader could sense RFID Tag to further read the number and data of exhibits. Moreover, such a system enabled visitors to listen to relevant information via earphones. This system is the pioneer of the digital navigation system of museums. However, because this system used old-fashioned PDA and passive RFID device, visitors could not read relevant data of exhibits until they walked to and stood in front of exhibits to detect them. Moreover, this system could neither actively provide the information on the location of visitors; therefore, its application scope was limited.

Abstract—“Oxford College,” well-known as “the earliest edification institution in northern Taiwan,” was planned by Rev. George Leslie Mackay. It is a typical Chinese and Western style architecture with rich historical and cultural content, now is a Class 2 national monument. This paper took “Oxford College” as an example to develop a 3D augmented reality mobile navigation system that supports the function of indoor positioning. This system collected the historical data to develop the 3D models according to the ratio of actual objectives, and constructed the 3D external and internal structures of Oxford College of the past and present. Moreover, this system combined RFID positioning function with the technology of markerless augmented reality to actively detect the location of visitors and to further instantaneously present 3D and multimedia navigation information on mobile devices.

In recent years, the improved effectiveness of smart mobile devices and the popularity of wireless internet have made various industries aggressively engage in the development of applications (App) applicable to smart phones. In individualized smart navigation service, visitors can use their personal smart phones or mobile devices to download a specific App or access specific navigation webpages, via wireless internet, to rapidly acquire more abundant and diversified navigation information. Smart navigation service not only provides detailed illustrations and explanations of exhibits, but also the information of audio navigation, maps of exhibition facilities, schedules, etc. Diversified navigation information is provided in such service, and the effect of navigation is far superior to that of traditional textual or verbal explanations.

Keywords-3D; Augmented Reality; mobile nvigation; indoor; positioning; RFID; Localization ; Oxford College; Mackay

I.

INTRODUCTION

Traditional exhibition facilities mainly display simple exhibition information or historical pictures for visitors. However, because each piece of exhibits has its own background and story, visitors can neither presume the history of exhibits nor fully understand their historical background and stories simply by observing the appearance of exhibits or their brief explanations. Moreover, some of exhibition facilities provide guides or audio navigation service to increase visitors’ understanding of background of exhibits. In terms of guides, exhibition facilities usually arrange professional guides to lead visitors through exhibition sites and systemically introduce the theme and important exhibits. However, because guides can only introduce exhibits verbally, the navigation form is restricted, and because guides are usually outnumbered by visitors, such service is mainly provided for visitor groups. Therefore, visiting hours are restricted and individual needs cannot be met. On the contrary, audio navigation system can provide individualized service, which enables visitors to freely choose to listen to explanations according to their personal interest, visiting speed, and length of stay. Moreover, visitors can manually enter the corresponding audio navigation number of exhibits to listen to various introductions. However, as audio navigation system can only verbally describe the historical background of exhibits the navigation form remains restricted.

978-1-4673-0890-8/12/$31.00 ©2012 IEEE

At present, some famous museums have commenced the promotion of smart navigation services [3, 4, 5, 6]. In the smart navigation system of the National Museum of Taiwan History, visitors can listen to navigation explanations, and look up information, such as maps of exhibition facilities and current exhibitions [3]. In addition to providing textual introductions of exhibits, the American Museum of Natural History App of the American Museum of Natural History can also reflect the location of visitors, via Wi-Fi positioning introduced into the indoor AGPS, which can provide a guide in the flow of exhibition facility [4]. The Brighton Museums App, as promoted by the British, combines the museum information of five museums. In addition to providing museum maps, historical pictures, and the latest information of activities and museums, it also assists visitors in planning one-day trips [5].

64

COMNETSAT 2012

The Street Museum App of the Museum of London collects historical pictures of the museum and overlaps them with the current street scenery to reflect the difference between the ancient times and modern time [6].

to the exhibition layout of Oxford College, while the RFID Tag was placed on visitors. When visitors carried mobile devices and an active RFID Tag and walked into the exhibition area, the Reader at the exhibition area would detect the signal of Tag within the sensing area and send it to the positioning server for analysis. In the end, the mobile devices of visitors would present positioning results and the information of exhibits is sent back from the navigation information server system.

Moreover, the Taipei International Flora Expo in Taiwan in 2011 also developed a flora expo navigation system [7]. This system was integrated with the technologies of GPS, NFC, and RFID, which enabled visitors to use their smart phones to sense QR codes or Tag, and rapidly acquire the navigation information of the exhibition facilities, and obtain more complete navigation services. Tatung University also developed a mobile navigation system using Longshan Temple in Taipei as the background, which was integrated with the technology of augmented reality [8]. This system would recognize QR codes, and when visitors walked to a target scenic spot with a code or tag, and use their mobile devices to scan the QR code, they could obtain relevant texts and pictures. However, the additional QR code or Tag may affect the appearance of the exhibition area.

The navigation information system stored the 3D architectural appearance of Oxford College at each major period, its indoor 3D virtual scene and layout, and the multimedia information of various exhibits, such as texts, pictures, audio service, and 3D models. The augmented reality exhibition system included the functions of image recognition and object mapping and imaging of markerless augmented reality. When visitors used the camera of a mobile device to shoot exhibits, the system would instantaneously analyze, recognize, and present the augmented reality of the corresponding object to display various diversified exhibit information (e.g. virtual 3D external architectural model of Oxford College 100 years ago)

g Ta

The aforementioned systems showed that, simple texts/pictures or audio navigation service can no longer meet the needs of modern visitors, and only diversified navigation services can win the recognition of visitors. On the other hand, the aforementioned systems usually only use GPS outdoor positioning or simple Wi-Fi indoor positioning; therefore, their accuracy of positioning is restricted. Consequently, this paper took the Oxford College as an example to combine active RFID positioning with the technologies of augmented reality and mobile navigation to develop a 3D augmented reality mobile navigation system that supports the function of positioning.

MM in of form exh at ibi ion ts

id

This paper proposed a more precise active RFID positioning system to precisely identify the location of visitors. Moreover, this system could determine the exhibits and exhibition facility information around visitors according to their location. Furthermore, a markerless augmented reality exhibition system could effectively improve identification effects without changing the layout of the exhibition area. In addition, it could present a diversified 3D model and multimedia information to further improve the overall effectiveness of 3D mobile navigation system.

Figure 1. Framework of the System

Each Tag of the RFID positioning system has a specific ID number. When visitors walked into the Oxford College, the staff would guide them in downloading the required App into their mobile device and distributed the Tag specifically for use in the RFID positioning system to them to enable the system to acquire the information on individual visitor’s location. Once the prerequisite procedures were completed, visitors could carry the Tag while walking through the indoor space where the Reader had been installed in advance. The Reader at each exhibition area would receive the ID signal transmitted by Tag and instantaneously sent it to the positioning server for analysis and comparison to further identify the location of visitors.

The remainder of this paper is organized as follows. Section II describes the framework and functions of the system; Section III explains the details of system implementation; Section IV shows the result of the system; and Section V presents the conclusions of this paper. II. FRAMEWORK AND FUNCTIONS OF THE SYSTEM

After the information of individual visitor’s location was obtained, the navigation information server system would send the navigation information at the exhibition area of visitors to their mobile devices to enable them to acquire the exhibit information of the location and specific exhibition area of their location. Moreover, this system would divide the exhibit information into several files according to exhibition areas in advance for visitors to download in batches. In addition to reducing the burden on hardware devices, this method could narrow down the scope of the augmented reality image

The framework of the system is as shown in Figure 1. This system was mainly divided into three major parts: (1) RFID indoor positioning system of the server; (2) navigation information server system; and (3) augmented reality exhibition system for visitors. The RFID indoor positioning system was mainly used to analyze the location of visitors and provide them with guidance on location and the selection of an adequate server to convey information. The hardware devices of the RFID indoor positioning system include an active RFID Reader and Tag. The RFID Reader was established according

65

recognition system to significantly improve recognition effectiveness and accuracy. Moreover, this system supported the information pre-loaded mechanisms, which could predict the exhibition area where visitors were going to visit according to their location and navigation route. This system would use spare time to download the exhibit information at the exhibition areas in advance to effectively reduce the time spent on waiting for information downloads.

download procedure would be initiated to download exhibit information to their mobile devices. If they had downloaded it, the pre-loaded mechanism would be initiated to download the navigation information of the next exhibition area. D.

Information presentation After the navigation information was downloaded, multimedia exhibit information, such as texts, pictures, audio service, and 3D models, would be presented on mobile devices. Moreover, the procedures of instantaneous positioning and information download would be continuously conducted. Furthermore, the augmented reality exhibition system of visitors could also instantaneously analyze and identify the exhibits that were photographed to present the augmented reality of corresponding exhibits, as well as to present the overlapped virtual and actual objects.

The operating procedures of the system are shown in Figure 2. The main procedures are in the order, as follows: (A) prerequisite procedures; (B) instantaneous positioning; (C) information download; (D) information presentation. The detailed descriptions of each procedure are given as follows: A.

Prerequisite procedures Visitors that firstly visited Oxford College had to download and install the navigation App applicable to their mobile devices. In their first login of the system, they had to manually enter the Tag ID to facilitate establishing a connection between mobile devices and Tag. After completing the procedures above, they could download basic navigation inflation of the exhibition area from the navigation information server and their mobile devices would present the initial navigation screen.

III. SYSTEM IMPLEMENTATION The procedures of implementation of the system mainly include three parts: construction of 3D scenes, augmented reality, and RFID positioning. This system constructs the actual 3D image of the current Oxford College and the virtual scene of external appearance and internal layout of Oxford College in early days based on the historical data of the Mackay exhibits. This system used Sketch Up to construct various 3D models of the external appearance and internal layout of the building. The models proportional to real objects were constructed, and the images of original materials are attached to the models to render the virtual scenes more real. In terms of 3D interactive technology, this system used Unity software in support of the platforms of Window, iOS, and Android in order to beautify the scene, add special effects, and include interactive functions and scene integration. For augmented reality, it used the App developed by Eclipse for mobile devices and the actual pictures of the scene as the bases for markerless image recognition. Therefore, its identification effect remained accurate, and without changing the layout of exhibition areas. Regarding indoor positioning, an active RFID positioning mechanism was used to complete accurate positioning of visitors.

B.

Instantaneous positioning With the Reader pre-installed at individual exhibition areas received the signal of the Tag that visitors carried, the positioning system would instantaneously compare the signal strength value with the positioning information in the server to further analyze the location of visitors. In addition, the system would send back the location information to visitors’ mobile devices for guidance through exhibition areas. C.

Information download The navigation information server system would send specific navigation information to the exhibition area where the visitors were located. If visitors did not download the navigation information of the exhibition area, the automatic

Figure 2. Flow chart of navigation system

66

with information of augmented reality, the recognition program would convert it into a black-and-white image, and then detect its feature points and distribution. Moreover, the program would analyze the angle and distance between image and camera to reflect the 3D coordinates of the image. The coordinates would be rotated and zoomed in/out to present the 3D objects. In the end, the program would search for corresponding virtual objects according to the distribution of feature points and present the virtual objects on mobile devices.

A.

Construction of 3D scenes To enable visitors to fully experience the sense of reality, in addition to using AutoCAD to precisely depict floor plans proportional to actual floor plans, according to the original floor plans of buildings, this paper also conducted on-site inspections to take photographs of real objects as the materials to design virtual images. In terms of outdoor 3D scenes, this system first input the floor plans developed by AutoCAD into SketchUp, and then developed the 3D building according to the actual appearance and ratio of the building (as shown in Figure 3). In terms of indoor 3D scenes, this system used actual scene pictures as the images to be attached to the model in order to enhance the sense of reality. After the models of all of buildings were completed, they were converted into FBX files and integrated into Unity. In Unity, after it was certain that the materials of images to be attached onto the buildings were correctly connected; more scenes and objects would be added into the system. In addition, natural materials (e.g. sky, tree, etc.) would be timely added as well. In the end, light effects would be added to beautify the models and scenes (as shown in Figure 4). Furthermore, after the self-compiled interactive program was added to the models, a 3D mobile navigation system, with the effect of realistic scenes and interactive functions, was completed.

C.

RFID Positioning

Figure 6. Layout of RFID in the Oxford College

In terms of the practical operations of RFID positioning [9], this system installed a total of 7 RFID readers in the Oxford College, located at the corner of the entrance of each exhibition hall (as shown in Figure 6), in order to avoid, as much as possible, the possibility that one Tag was read by several readers at the same time. The reader at each exhibition area would detect the signal strengthen value of the Tag and send it to the positioning server for comparison of positioning information in order to determine the location of the exhibition area of visitors. When visitors walked into a vague area, namely, the Tag was sensed by several readers at the same time, the positioning system would determine their location based on the corresponding signal strength to facilitate the subsequent information download procedure. In addition, the system would send the exhibit information of the exhibition area nearest to visitors according to their mobile devices.

Figure 3. 3D building in SketchUp Figure 4. Integrated scene in Unity

B.

Augmented Reality

IV.

Figure 5. Flow chart of processing procedures for augmented reality

DEMONSTRATION

Figures 7(a)(b)(c)(d) are the navigation interfaces of the system provided for visitors. As shown in Figure 7 (a), there are three options on the main screen: (1) 3D model; (2) exhibit information; and (3) a MAP for visitors to choose. Moreover, the small map in the lower left corner of the main screen could reflect the location and exhibition area of visitors based on the returned signal from the RFID positioning system. The name of the exhibition area and the exhibit information are presented on the bottom of the navigation screen. In terms of the navigation

Figure 5 shows the processing procedures of the augmented reality system in this paper, which includes five major procedures: (1) tracking of image; (2) calculation of 3D image coordinate; (3) identification of image marker distribution; (4) searching of virtual objects of corresponding images; and (5) presentation of virtual objects. In the practical operations of recognition of augmented reality, this paper used real procures at the scenes as the basis. When a camera tracked an image

67

of 3D models, visitors could use the camera of mobile devices to scan the pictures of exhibits to browse the augmented reality 3D models. Moreover, they could also switch the models into a panoramic model (as shown in Figure (b)). The function interface of the system could be hidden to facilitate visitors’ adjustment of camera angles, which enabled them to observe the 3D models various angles. To exit the panoramic model and return to the standard screen, they simply had to click the exit button in the upper right corner.

V. CONCLUSIONS This paper took “Oxford College,” a Class 2 national monument in Taiwan, as an example to combine RFID positioning with the technologies of augmented reality and mobile navigation to develop a 3D augmented reality mobile navigation system that supports the function of indoor positioning. The proposed active RFID position system could precisely identify the location of visitors. Moreover, the markerless augmented reality technology of this system could be integrated with the positioning and pre-download mechanisms to effectively improve the performance of image recognition, and to further present more abundant and diversified navigation information without changing the layout of exhibition areas. This system was not only applicable to “Oxford College,” but also various exhibition facilities. It could improve the effectiveness of navigation and increase understanding and interest of visitors significantly.

Figure 7 (c) shows the navigation screen of exhibit information. After the visitors clicked the button of “exhibit information,” the screen would present the basic navigation information in texts and images. In addition, visitors could click the “speaker” button to listen to the corresponding audio service to increase their understanding of exhibits. (Figure (c) shows the description of the architectural appearance of Oxford College). Moreover, when a visitor clicked the “Map” button, the floor plan of Oxford Collage could be presented in full screen mode, including the exhibition theme of each exhibition area and the location/exhibition area of the visitor (red box in Figure 10). Such information enabled visitors to plan their navigation routes. Figure 8 shows a picture of the system taken at Oxford College.

ACKNOWLEDGMENT The authors would like to thank the National Science Council of the Republic of China, Taiwan for financially supporting this research under Contract No. NSC-100-2221-E156-008 and NSC-99-2632-H-156-001-MY3. Besides, the authors would like to acknowledge Kai-Bin Wang for his assistance in making this system possible. [1]

(a)Main screen (standard mode) (b) Panoramic navigation mode [2]

[3] (c) Exhibit information

(d) Map of exhibition areas and guidance on positioning

[4]

Figure 7. Screen for operation of mobile navigation system

[5]

[6]

[7]

[8]

[9]

Figure 8. On-site testing results

68

REFERENCES Y. C. Chuang, S. C. Yu, C. C. Liang, “A Study of Applying Digital Mobile Museum Guide,” Journal of Library and Information Studies, vol. 1, No. 2, pp. 1–24, September 2003. National Palace Museum Virtual Museum, http://www.npm.gov.tw/vrmuseum/index.html [Accessed: April 2012]. National Museum of Taiwan History, http://www.npm.gov.tw/en/home.htm [Accessed: April 2012]. American Museum of Natural History, http://www.amnh.org/apps/explorer.php [Accessed: April 2012]. Brighton Museums, http://www.brighton-hoverpml.org.uk/Museums/social-media/Pages/Smartphoneapp.aspx [Accessed: April 2012]. Museum of London Street Museum, http://www.museumoflondon.org.uk/Resources/app/youare-here-app/home.html [Accessed: April 2012]. Taipei International Flora Exposition, http://www.2010taipeiexpo.tw/MP_3.html [Accessed: November 2011]. K. H. Wang, L. C. Chen, P. Y. Chu, and Y. M. Cheng, “A Study on the Design of Augmented Reality User Interfaces for Mobile Learning Systems in Heritage Temples,” Proceedings of the 3rd International Conference on Virtual and Mixed Reality, pp. 282-290, July 2009. C. S. Wang, “LOVINA: Location-aware Virtual Navigation System Based on Active RFID,” International Journal of Ad Hoc and Ubiquitous Computing, Vol. 8, No. 4, pp. 211-218, November 2011.