applications on a laptop computer connected to the INTER-. NET over a wireless ... instance calculate the fastest way to the next available phar- macy, doctor, or ...
Mobile, on Demand Access of Service-Annotated 3D Maps Michael Przybilski, Stefano Campadello and Titos Saridakis NOKIA Research Center PO Box 407, FIN-00045 NOKIA GROUP, FINLAND ABSTRACT The ever-increasing capabilities (processing power, memory, connectivity, display, etc.) of personal devices such as PDAs and mobile phones, as well as their convergence, have enabled the development of 3D map applications for mobile users. In addition to 3D map layouts, textures and route finding features, such applications can also provide access to location-specific services (e.g. restaurants and their ”menu-of-the-day”). The challenge that those mobile 3D applications have to face is how to efficiently provide up-to-date location-specific data and services. Based on the TellMaris and Robocop projects this paper addresses these issues and provides a solution that is based on dynamically obtaining the location-specific data and services on user’s demand.
applications on a laptop computer connected to the INTERNET over a wireless LAN network connection, thus providing a first mobile 3D solution. However, one of the first projects, implementing a 3D map application directly for a mobile device, is the TellMaris project. Figure 1 shows the TellMaris guide application on a Nokia Communicator terminal.
KEY WORDS Context Awareness, Service-Annotated Maps, Mobile Services, Secure Access Figure 1. Mobile 3D guide application
1 Introduction Within the last years, the increasing capabilities of mobile devices have led to the development of 3D application for the mobile user. While initially 3D was considered to be hardly applicable on mobile devices, more and more applications have been developed directly for wireless terminals, or have been ported from the PC world. Starting initially with concept applications, and games, also other applications have started to appear on mobile terminals. One category of those applications is that of 3D maps and guidance application for mobile terminals. While those applications started from 2D, computer version of ordinary paper maps, they quickly evolved into more and more advanced applications. Initially, those applications were primarily deployed as a route finding application in cars, while more advanced applications are now also available for other users, such as pedestrians and cyclists, and provide additional functionality besides route finding. First such application were for instance ”Cyberguide” project, develop at the Georgia Institute of Technology in the late 1990’s [1], or the ”GUIDE” project, developed at the Lancaster University [2, 3]. One of the first 3D map application was the ”Tre-D / 3D city info” project of the Tampere University of Technology [4, 5]. While these projects dealt with 3D primarily on desktop computers, it was certainly also possible to deploy the
In this paper we present a solution for obtaining ondemand 3D map data annotated with location-specific service information on a mobile client, based on the implementations from the TellMaris and Robocop projects.
2 Background 2.1 Service-Annotated 3D Maps Especially for pedestrians, 3D maps can provide an easier way of orientation, as they are more comprehensible, and require less abstraction than 2D maps [6, 7]. Furthermore, it is not necessary for the user to understand the abstractions and symbols that a 2D map imposes [8]. The use of such an application provides also the ability to disable the display of information that is currently not relevant for the user [9], thus avoiding an overload of information. This feature is perhaps most useful for people who are not familiar with their current surroundings, such as tourists, visiting a new location. It allows them to orient themselves and can, in combination with a route-finding mechanism, provide them the ability to find a specific location. However, the use of 3D is not limited to the provision of maps, for orientation, and guidance, and the provision of route finding information. The use of 3D on mobile devices enables also other important applications. The extension,
or combination with additional services, such as the overlay of additional information of the 3D view of ones surroundings, or the sharing of personal location information, can be extremely useful also for people who are familiar with the environment. Examples include the provision of a restaurants menu or pricing information, the opening times of a shop, or similar information, in an overlay, over the 3D view of ones surroundings. The idea of these services can be taken also indoors, where it would for instance be possible to display additional information of a painting, the consumer currently views. Furthermore, it is possible to provide additional information also outside of the context of a city, and provide information about ones surroundings, that is not apparently visible, and that can even help minimize risks. An example hereof is the provision of a weather forecast, or the overlay of avalanche probabilities on a specific mountain area, one intends to climb. Even in a familiar environment, a mobile guide application can be extremely useful, as it can take into account the latest up-to-date information, and for instance calculate the fastest way to the next available pharmacy, doctor, or hospital, in the case of an emergency. In addition to these services, also a possibility to search for other services should be offered.
2.2 Context Awareness In the current implementation, it is possible for the user to manipulate the location in the 3D model by himself, or obtain the current location from a GPS device, which can be either integrated or connected via Bluetooth to the mobile terminal. Future extensions can include the possibility of obtaining the position based on the location, and signal strength or signaling times of cellular base stations, such as in the E-OTD approach [10], or based on the relative position of other users, and their Bluetooth, or wireless LAN enabled devices, or a combination of these possibilities. The possibility of automatically obtaining the users position not only makes it easier for the user to orientated himself, find a way to s specific location, from his current position, or identify visible landmarks, and get additional information, as described before. Other services include the possibility of automatically locating other persons relative to ones own position or the possibility of overlaying personal information, and the sharing thereof, in the virtual environment. Also, an automated guidance system, which informs the user of interesting landmarks, or guides him through the historic parts of a city, and supplies additional information, can be provided. Besides the users location, the mobile terminal can also take into account other information that is available, such as the users phonebook, to inform him about the presence of another person in his phonebook, or preferences, to inform him about potential points of interests in the vicinity, or insert booking information for the current location into his calendar. The application should also integrate with, and use other applications on the mobile device,
whenever possible, for instance using a devices browser application to provide information that is outside of the context of the 3D map application.
2.3 3D Map Data In order to provide above services, the required information needs to be available on the mobile device. The basic information that needs to be available is that of the 3D model. This includes primarily the vertices and the textures that are laid over the planes. While this depicts the very basic layer, additional services require further layers of information. This includes for example layers that contain the route finding information, relative to the target, and the users current location, the additional information of points of interest, that should be displayed to the user, the location of other users, or a layer, or combination thereof, which allow the user to enter personal information. In order to structure and combine these different layers, the virtual model has been divided into cells. While it is possible to have cells of varying size, it was for simplicity decided to divide the model into cells of equal size. To provide the basic 3D information, a single cell contains • a unique cell ID • a list of all adjacent cells • the coordinates of the NW corner of the cell • the points and vertices of the 3D model, of the cell • the textures, which are mapped on the planes of the 3D model of the cell While this depicts a minimum of required information for a 3D model, different approaches can be taken for the efficient computation and storage of such basic 3D information, as well as more elaborate 3D models [11, 12, 13].
3 TellMaris 3D Model While the TellMaris project primarily concentrated on the 3D aspect mobile devices, specifically, with the restricted resources and the limited user interface mechanisms, the communications mechanisms were only marginally explored. Although extensive development has been done, and is still ongoing, in bringing 3D to mobile devices, those devices still have the limitation to be unable to store all desired information locally. It is usually anticipated that also the mobile technology evolves and mobile devices become more powerful, providing more computing power, memory, and storage, thus providing the possibility to develop more and more advanced applications. Also wireless network connections will be able to provide a higher bandwidth, become faster and more reliable. Still, one of the fundamental challenges in mobile computing is that, even though mobile devices will continue to evolve, they will always have less available resources than static devices [14].
Thus, the challenge of bringing up-to-date information to the mobile terminal, as well as using the available resources efficiently,remains. For most applications it is nowadays necessary, to store the information of the location that one wishes to visit beforehand, by synchronizing the mobile device, with an information base, using a high speed, and high bandwith cable connection. Storing all information on the mobile device will however in many cases not even be desirable, since it is difficult to keep the information up-to-date. Also certain services, mentioned above, require the connection to other devices, via some kind of network. Thus, it is necessary, to transfer the updated information over the air, which is in more and more terminals these days possible, as they are, or can be integrated or combined with, mobile phones. Since however the bandwidth for these devices is limited, it is necessary to efficiently choose, which information is relevant at a particular point in time, for the specific user. It is furthermore necessary to efficiently distribute the data, before it can be transferred to the mobile device, again in the most efficient way. In order to structure the data that needs to be transferred to the mobile device, the data has in this case been divided into cells, as explained earlier. Due to the limited storage on the mobile device, only a limited number of cells are available on the device, at a given point in time, while additional information needs to be transferred as required. Assuming and initial number of 9 cells, which are stored in the mobile devices memory, there are different ways to decide, which cells should be transferred next. In this example, it is assumed that the cells are available in memory and new cells have to be transferred, while in reality it will usually be implemented in such a way, that a number of cells is stored in a cache mechanism of the device, and loaded into memory on demand, instead of requesting them from the network. The exact caching strategy is not of interest here.
G
H
L Q
S
A
B
C
D
G
H
I
E
Figure 3. Rearranged Cells This can be enhanced even further, by proving a lesser resolution for the textures of more distant objects, and thus allowing even more cells to be displayed, however the amount of vertices in a given cell remains the same, and thus this approach is not shown here. Alternatively, increasing the size of a more distant cell, to contain more information can be used as another, and perhaps more flexible approach. The result would however be similar, and thus this approach is not shown here.
A
B
C
D
F
G
H
I
L
I N
R
H, I and M) are visible on the display. In order to optimize the situation, and allow the user to see a further distance, it is possible to choose cells differently, as the example in Figure 3 shows. Here, all 9 cells, which are available in the device memory, are visible to the user.
E
Newly loaded Cells
Figure 4. Turning User Cells in Memory Visible Cells Person Viewing Angle
Figure 2. Initial Situation An initial situation is shown in Figure 2. In this example it can be seen, that 9 different cells, containing the information of the area around the users, are stored in memory. At the given viewing angle, only 4 of these cells (G,
Now the problem that needs to be resolved is the selection of new cells, which need to be loaded, as the user either moves in a certain direction, or as the user turns. The example in Figure 4 shows that a number of additional cells, in this case 2, needs to be displayed if the user turns only slightly, thus changing the viewing angle, and thus those cells must first be fetched from the network, or the caching mechanisms, before they can be loaded into memory, and displayed. As the user turns further however, it is possible to discard a number of cells from the memory, as they leave the
A
B
C
D
F
G
H
I
L
E
B
C
D
G
H
I
L
Newly loaded Cells Discarded Cells
N
Q
R
S
Figure 5. User turning further user field of view (see Figure 5). This again reduces the memory requirements. Upon a further turn, to almost 90 degrees, additional cells can be discarded, while other cells need to be loaded into memory, thus resulting again in a number of 9 cells (see Figure 6).
A
B
C
F
G
H
K
L
D
Figure 7. User moving cussed, this solution does however have the disadvantage of not showing more distant objects, and their associated information.
G
H
L Q
R
I
G
N
L
S
Q
H
I N
R
S
Figure 8. Fast turning Figure 6. User turned 45 degrees A similar decision mechanism can be applied to the situation where the user moves from one cell to another. As the user moves forward, crossing the boundary between two cells, it is necessary to load the additional cells, that should now appear in the distance. Yet, at the same time it is possible to discard the cells that are further behind the users position. An example of this approach can be seen in Figure 7. One problem that has been found with this approach however is that the user can turn much faster, then (s)he is able to move forward to the next cell. Since it takes time to load a new cell into memory, it was found to be much better to provide the user with a faster ability to turn, and instead deprive him of the ability to see more distant objects. Thus, in this implementation, it was decided to keep all the cells, adjacent to the users current position in memory, instead of providing all currently visible cells. In such a case, the user has the advantage of getting a very fluent impression when (s)he turns around, into another direction. This is shown in Figure 8, and depicts the current implementation. As dis-
In the current implementation the data size of a 100m by 100m cell, as described earlier, including only the 3D model, and the textures is in average less then 6 kB ( 5 kB texture, 600 bytes 3D model). Thus, the transfer of the basic cell data takes about 7 seconds per cell, assuming a basic GMS data connection (9600 bps). Other transfer bearers provide a higher bandwidth, and thus can provide a better user experience. The communication mechanism also needs to be able to recover from a degrading, or even lost connection, in a way that is transparent to the user. One possibility is the provide a different caching mechanism, storing more data on the device itself, and proving the user with the available data, while at the same time indicating the connection status [15, 16]. Furthermore, in order to use the available bandwidth more efficiently, it is also possible to introduce an intelligent scheduling mechanism for transmitting data packets. However, a loss of packets is in this application not acceptable. For availability, efficiency, as well as administrative and ownership reasons, the 3D data, as well as the data required to provide the different services, must
exist on multiple databases and computer-systems.
4 Obtaining Location-specific Data Location-specific data such as 3D map segments, the associated texture and the services available in the corresponding geographic areas, are distributed in a number of repositories. Each repository belongs to a distinct administrative domain and contains data specific to a geographic location (e.g. the repository owned by the City of Helsinki contains maps and routing information regarding the city of Helsinki). These location-specific data include references to services available in the corresponding map segments, which may be provided by individual service providers and be available for some fee. A TellMaris user will first have to obtain the 3D map data of a given area of interest (presumable the area where he is currently located). Then, the user can access the services available in that area using the references to those services with which the 3D maps are annotated. To obtain the 3D map data, the TellMaris client software that executes in mobile devices has to contact the correct repository that contains those data. A centralized directory service with a ”well-known” address contains all the mappings of 3D map cell IDs (i.e. the geographical coordinates of a cell) to the addresses of repositories, which contain the 3D map data that correspond to the given location. The registration and location schemes are straightforward. Every repository that provides 3D map data for TellMaris clients contacts the directory service at its ”well-known” address and registers its address and the coordinated of the geographic area it covers. The address of the directory service is also placed in every TellMaris client when it is first configured. When a TellMaris client needs to contact the appropriate repository in order to obtain 3D map data for a specific geographic area (presumably the area where the TellMaris client is currently located), it sends the coordinates of that area. Any cell ID of the city of Helsinki suffices for the directory service to identify the repository owned by the City of Helsinki as the appropriate repository for a TellMaris client that needs 3D map data regarding Helsinki. When the TellMaris client receives the address of the repository owned by the City of Helsinki, it connects to it and requests the 3D map data that correspond to a given cell ID (i.e. the 3D map of the given cell plus the 3D maps of its 8 neighbor cells). The registration and location schemes are graphically illustrated in Figure 9.
5 Secure Access to Services The data downloaded from the Repositories in the process described above has to address also the some security requirements. These come principally from the fact that mapping a spatial region like a town or a whole country with the resolution of details requested by applications like TellMaris has an elevated cost so service offering them must
Figure 9. Registration and location schemes be protected from the access by unauthorized clients. Secondly, some information such as restaurant menu or shops special offers may be consider sensitive and to be protected as well. In general, the requirements that a system like TellMaris needs are: 1. Authentication: It must be possible for the TellMaris service provider to ascertain the identity of the client. As stated before, services may be paid for, thus it needs to be ensured that only authorized client requests are answered. It is also to be noted that sometimes the client sends sensitive information to the TellMaris service provider. In that case, also the latter needs to be authenticated. 2. Confidentiality: The content of the data transmitted must remain confidential; only the legitimate receiver should be able to read it. 3. Integrity: It should be possible for the receiver of the data to verify that it has not been modified in transit. In fact, if the data received is corrupted (by a malicious entity, for instance), the equivalent false map could still be validated by the system and thus create dangerous situations for the user. 4. Non-repudiation: A client should not be able to falsely deny later that he or she did not request a service, in order to avoid payments. Since the TellMaris Repository may change from time to time depending on the location of the client, a security set-up needs to be done every time the client establishes a new connection with a Repository. The following section describes such set-up. The security setup is the procedure in which the TellMaris parties agree upon the security level and all the necessary steps and messages are performed to create the se-
cure channel where all the data will pass across. The security setup can be void. In this case, no security mechanism is required and it happens, for instance, when the client and the Repository are well known to each other. In other cases, the security setup includes the achievement of one or more of the requirements presented above. The end-points must agree on the security properties of the communication. The parameters on the requirements could be negotiated to some extend. For example, one party could ask for an encryption link and the other could offer a choice between different encryption algorithms available. An agreement on the security properties should be reached at same point otherwise the communication aborts. Confidentiality is usually provided through encryption. Authentication, data integrity and non-repudiation are usually provided through digital signature and public key certificates. In the following section we give a conceptual example of a security setup.
Client
length of cryptographic keys. Messages are exchanged to negotiate the parameters. If and agreement is reached the setup continues. The client sends his authentication information to the Repository. For instance a digital-signed certificate is sent. The Repository checks the identity of the client verifying the information received. Because we are dealing with a mutual authentication, the Repository sends his credentials to the client as well. If the checks are passed, the mutual authentication is achieved. All data packets exchanged include message integrity checks. Cryptographic hashes, which include a secret key as a part of the calculation, may be used to verify that no changes to data have been made. At this point, the security setup phase is ended. An authenticated and tampering-proof channel has been created.
Client
Client-hello + security parameters Repository-hello + security parameters
Repository
parameter agreement
Client-hello + security parameters Repository-hello + security parameters
Client authentication information
parameter agreement Client authentication information check Repository authentication
Repository authentication information
Repository
check Client authentication
check Repository authentication
Repository authentication information
check Client authentication
Mutual authentication integrity key exchange Encryption channel established
Mutual authentication integrity
Figure 10. Establishment of a path with Authentication and Integrity Features
5.1 Security setup with Authentication and Integrity In this case, the security setup purpose is to establish a communication channel with the endpoints mutually authenticated. Moreover, the integrity feature assures nobody tampered with the data. The sequence diagram in Figure 10 illustrates graphically the actions and the messages sequence of this phase. An explanation of the diagram follows below. The parties exchange hello messages to identify themselves. It is a simple notification that the negotiation process is started. Included in the hello messages there are the parameters to be agreed upon for securing the connection. Examples of parameters could be: which TellMaris protocol version has to be used, which algorithms for authentication or algorithms for integrity check, or finally the
Figure 11. Establishment of a path with Authentication, Integrity and Confidentiality Features
5.2 Security setup with Authentication, Integrity and Encryption The case of security setup with authentication, integrity and encryption is similar to the previous diagram except for the last few steps. In this case, after the parties are mutually authenticated is necessary to proceed to the encryption of the channel. The block highlighted in gray in Figure 11 as ”key exchange” has the purpose to establish a common encryption key to secure the channel. The authentication phase has already been carried on and the information derived from it is used to create the common secret key. The common secret key may be distributed using a key exchange algorithm. Key exchange algorithms are a special class of public key algorithms that do not encrypt data but do allow two sides of transaction to figure out the same unpredictable number. In this document we are not
dealing with the choice of the algorithm used. The gray ”key exchange” box contains the messages necessary to create the common secret key used to encrypt and decrypt the data. The sequence is tightened to the particular key agreement algorithm chosen.
6 Summary The TellMaris project addresses a number of issues related to the development of 3D map applications for mobile users, including the segmentation of 3D maps and their dynamic download to the mobile terminal on user’s demand and the access to location-specific services related to a given map segment. TellMaris solution provides the mobile user with a 3D map that covers a square area centered to the current location of the user, and dynamically updates that map following the users displacements. To dynamically update the map, a TellMaris client downloads the appropriate map cells according to the users displacement from a TellMaris enables repository that contains location-specific data (3D map cells and their associated textures). To contact the right TellMaris enabled repository that contains the 3D map cells of the area where the mobile user is currently in, the TellMaris client contact a centralized directory service providing its current coordinates and looks up the address of the repository that contains 3D map cells of that area. In addition to the 3D map layout and textures, each cell downloaded from a TellMaris enabled repository contains also links to various services that are available in the geographical area covered by the cell. These service links are placed inside the 3D data in the form of annotations. Accessing these services can be restricted or fee-based depending the service provider. TellMaris does not force any model for accessing the location-specific services, but it does provide the necessary security support that allows the mutual authentication of services and users.
Acknowledgements The authors would like to thank the consortia of the TellMaris1 and the Robocop 2 projects.
References [1] G.D. Abowd, C.G. Atkeson, J. Hong, S. Long, R. Kooper, and M. Pinkerton. Cyberguide: A Mobile Context- Aware Tour Guide. Baltzer/ACM Wireless Networks, 3:421–433, 1997. [2] N. Davies, K. Mitchell, K. Cheverst, and G. Blair. Developing a context sensitive tourist guide. In Proceedings of the 1st Workshop on Human Computer Interaction with Mobile Devices, pages 64–68, 1998. 1 www.tellmaris.com 2 www.extra.research.philips.com/euprojects/robocop
[3] K. Cheverst, N. Davies, K. Mitchell, A. Friday, and C. Efstratiou. Developing a Context-aware Electronic Tourist Guide: Some Issues and Experiences. In Proceedings of ACM Computer-Human Interaction, pages 17–24, 2000. [4] I. Rakkolainen, T. Vainio, and J. Timmerheid. A 3D City Info for Mobile Users. In Proceedings of the 3rd International Workshop in Intelligent Interactive Assistance and Mobile Multimedia Computing (IMC’2000), pages 115–122, 2001. [5] I. Rakkolainen. 3D City Info – a Near-future Application of 4G Services. In Proceedings of the 2nd Wireless World Research Forum (WWRF, 2001. [6] N.G. Vinson. Design Guidelines for Landmarks to Support Navigation in Virtual Environments. In Proceedings of CHI’99, pages 278–285, 1999. [7] M. Blades and C. Spencer. How do people use maps to navigate through the world? Cartographica, 24(3):64–75, 1987. [8] E. Hunt and D. Waller. Orientation and wayfinding: A review. Technical report, Office of Naval Research, 1999. [9] C. Freksa. Spatial Aspects of Task-Specific Wayfinding Maps, chapter in Visual and Spatial Reasoning in Design (J.S. Gero & B. Tversky Eds.), pages 15–32. University of Sydney, Key Centre of Design Computing and Cognition, 1999. [10] GSM 03.71 version 7.3.0 Release 1998. Technical report, European Telecommunications Standards Institute (ETSI), 2000. [11] OpenGL. General Performance Techniques. http://www.sgi.com/software/opengl/kitchen/general/ perf/index.html. [12] W3D. Web 3D http://www.web3d.org.
Consortium
Homepage.
[13] M. Van Velsen and R. Fercoq. 3D Studio File Format (3ds). http://www.dcs.ed.ac.uk/home/mxr/gfx/3d/3DS.spec. [14] M. Satyanarayanan. Fundamental Challenges of Mobile Computing. In Proceedings of the ACM Symposium on Principles of Distributed Computing, 1995. [15] G.H. Forman and J. Zahorjan. The Challenges of Mobile Computing. IEEE Computer, 27(4):38–47, 1994. [16] A. Raposoand L. Neumann, L. Magalhaes, and I. Ricarte. Visualization in a Mobile WWW Environment. In Proceedings of the WebNet 97 – World Conference of the WWW, 1997.
