Spatial Authoring and Orientation-Based Aggregation of Annotated Information
Kengo Koiso, Takahiro Matsumoto, Katsumi Tanaka Graduate School of Science and Technology, Kobe University 1-1, Rokkodai, Nada,Kobe, 657-8501 JAPAN TEL/FAX : +81-78-845-7643 Email : koiso,matumoto,
[email protected] KEYWORDS : Spatial Authoring, Aggregation, Panorama
Abstract
Three dimensional representation of spatial data provides an ecient way of presentation for it gives us extensive information on what one can nd at a glance. Viewing three dimensional data such as perspective of streets and plazas would give us detailed and intuitive information but it might take hours if one doesn't know where to search, and even worse, one can get lost while in search. In this paper, we propose a way of authoring threedimensional space by aggregating annotated information based on orientation-based annotation model(OAM) to enable the retrieval of the wanted spatial data. By using OAM, one can annotate three-dimensional spaces or spatial objects depending on the direction from which they are viewed. By using the information stored in OAM, one can retrieve spatial data matching the viewer's interests and present the aggregated information. We rst discuss our basic concept and describe how we annotate spatial objects and three-dimensional spaces, aggregate the information, and present it using OAM.
1
Introduction
In a three-dimensional space, one can walk around or
y about, and view three-dimensional objects from di erent directions. Aspen Movie Map(Brand, 1987) produced by the Architecture Machine Group, Media Lab, MIT in 1979 is known as a system for generating a virtual tour of the town from the video data previously recorded. One characteristic advantage of the Aspen Movie Map over the conventional twodimensional map is that one can view the actual cityscapes such as facades of buildings whereas one can view them only as two-dimensional symbols on a map.
By managing data spatially, one can associate data manipulation to spatial locations and this provides a "clue"(Fields, 1977) in nding the wanted data. In the "tour", one experiences in the Aspen Movie Map, one views a sequence of di erent views of the town of Aspen. If such views could be annotated, it will be possible to search and retrieve the views one wishes to see. In this paper, we propose a way of spatial authoring using orientation- based aggregation of annotated information. Taking into account the fact that views of spatial objects and spaces di er when viewed from di erent directions, we propose to annotate them based on the direction from which one views them and aggregate the annotated information in order to describe a view.
2
Basic Concepts
2.3
Retrieval of Data by Annotated Information
2.1
Viewpoint and Guided Tours
A viewpoint can be described by its coordinates in the three-dimensional space and orientation. A viewpoint vi can be expressed as vi = (ci ; oi ) where ci is the coordinates (xi ; y i ; z i ) in the three-dimensional space, and the orientation oi is the direction towards which the user's view is directed. A guided tour can provide an useful guidance for it takes the user automatically for a tour and one doesn't have to walk with no idea where one is going. A guided tour can be de ned as a sequence of viewpoints, and a viewpoint can be described by its coordinates in the three-dimensional space and orientation. So a guided tour can be expressed as the sequence of viewpoints v1 ; v2 ; :::; v n (n 1). 2.2
By describing viewpoints and guided tours by feature vectors, it becomes possible to retrieve viewpoints or guided tours matching the interests of the user. The user may choose some samples re ecting his or her interests, and a query vector can be generated in order to search for viewpoints or guided tours having similar feature vectors. The retrieved data can be presented in the forms of individual viewpoints or guided tours. If the viewpoints were certain cityscapes in a city, retrieved data can be presented in the forms of a number of "photographes" capturing some buildings or a citeseeing tour introducing some touristic sites.
3
Orientation-Based
Aggrega-
Orientation-Based Annotation and
tion of Annotated Informa-
Aggregation
tion
In a three-dimensional space, spatial objects can be viewed from di erent directions, and it is reasonable to think that their descriptions di er when viewed from di erent directions. For example, Mt.Matterhorn located on the Swiss-Italian border looks very di erent depending on where you are. The Swiss people who call it Mt.Matterhorn associate it with their view of the mountain from north, while the ltalian people who call it Monte Cervino associate it with their view of the mountain from south. Their imppressions may di er, but it is still the same mountain. In this research, we give more than one annotation information to a spatial object or a space depending on the direction from which is viewed. In order to describe a viewpoint, it becomes necessary to aggregate the annotated information given to an object or a space. We propose to aggregate the information based on the orientation of the viewpoint such that priority is given to the information annotated in the direction which is closer to the direction from which an object or a space is viewed. Thus, in the example with Mt.Matterhorn, the name "Matterhorn" would look larger than the name "Monte Cervino" when viewed from Switzerland. Viewpoint data only contains their location and orientation. In order to retrieve viewpoints requested by the user, they need to be described by some annotated information. In this paper, we use feature vectors in vector space model[4]. We give more than one annotation information expressed by feature vectors to a spatial object or a space. We then aggregate the annotated information based on orientation in order to describe a viewpoint by a single feature vector. A guided tour which is a sequence of viewpoints can be expressed by a feature vector using the vectors of the viewpoints.
3.1
Orientation-Based
Annotation
Model
We propose to annotate a three-dimensional object based on orientation. In other words, you give different annotations to the same object depending on the direction from which you view the object. We used MBB(Minimum Bounding Box) for approximating spatial objects and a model by Ishikawa(Ishikawa, 1997) in which he de ned the rotation of an object as an attribute by specifying twenty six subdivided boxes adjacent to the surfaces of MBB to create a orientation-based annotation model.
Figure 1: Orientation-Based Annotation Model Using MBB In this orientation-based annotation model, the twenty six subdivided boxes adjacent to MBB are to be annotated as well as the MBB itself. Thus, the object approximated by MBB can be annotated from di erent directions.
3.2
Aggregation of Annotated Infor-
3.2.1 OAM Examples
mation
We will explain our method using some examples. We name the faces of the MBB A, B, C, D, E, and F as shown in Figure 2. In the examples below, we demonstrate how we calculate the weight wij in order to obtain the annotated information V (di ) from the viewpoint dj for the three cases: 1)viewing towards the apex, 2)viewing towards a side, 3)viewing towards a face.
This orientation-based annotation model may have di erent information annotated to each of the twenty six subdivided boxes. In order to present the annotated information, it is necessary to aggregate all the annotated information when viewed from a certain direction. We propose an orientation- based priority order determination rule. In this rule, we de ned a mechanism for deciding the priority order of presentation for the annotated information when viewed from a certain direction. When you look at a box, one to maximum of three surfaces can be viewed depending on the angle, and the visible area for the visible surfaces change according to the direction from which you look at the box. We used the relationship between the orientation of viewpoint and the faces visible as a factor for determining the priority order in this rule. Based on the twenty six subdivided boxes adjacent to the faces of MBB, there can be three cases of viewing it as follows. 1. Viewpoint is directed at a face of MBB (Only one face visible) 2. Viewpoint is directed at a side of MBB (Two faces visible)
Figure 2: Faces of Annotation Model 1. Viewing towards the apex
3. Viewpoint is directed at an apex of MBB (Three faces visible) The number of faces visible in each case is one, two, and three, respectively. We let the viewpoint located inside one of the twenty six subdivided boxes in Figure 1 be di (i = 1; 2; 1 1 1 ; 26) and the set of faces visible from the viewpoint di be face(di ), and express the annotated information from viewpoint di as A(di ). The set of annotated information visible from viewpoint di is expressed as
V (di ) = fwij 1 A(dj ) j j = 1; 2; 1 1 1 ; 26g . Here, we de ne each weight wij as,
wij =
j
face(di ) ^ face(dj ) j j face(dj ) ^ face(di ) j 1 j face(di ) j j face(dj ) j
The former part of this equation expresses how much of the faces of MBB visible from the current viewpoint is visible from the direction for which MBB is annotated. The latter part expresses how much of the faces of MBB visible from the direction from which MBB is annotated is visible from the current viewpoint. When i = j , or when the direction of viewpoint equals the direction from which MBB is annotated, the weight of the information equals the maximum value one. The weight of information annotated in the direction from which no face visible from the current viewpoint is visible equals the minimum value zero.
Figure 3: View towards apex As shown in Figure 3, we calculate the weight w1n (n = 1; 2; :::; 26) in order to obtain the aggregated annotation information V (d1 ) = fw11 1 A(d1 ); w12 1 A(d2 ); 1 1 1 ; w126 1 A(d26 )g when viewed from d1 . When face(d1 ) = fA; B; C g for d1 , the weight of the authored information is
w11 =
A; B; C gj jfA; B; C gj 3 3 1 = 2 =1 jfA; B; C gj jfA; B; C gj 3 3 jf
for face(d1 ) ^ face(d1 ) = fA; B; C g. When face(d2 ) = fB; C g for d2 , the weight of the annotated information is jfB; C gj jfB; C gj 2 2 2 w12 = 1 = 2 = = 0:66 jfA; B; C gj jfB; C gj 3 2 3
for face(d1 ) ^ face(d2 ) = fB; C g. When face(d3 ) = fB; C; F g for d3 , the weight of the annotated information is
w13 =
B; C gj jfB; C gj 2 2 4 1 = 2 = = 0:44 A; B; C gj jfB; C; F gj 3 3 9
jf jf
for face(d1 ) ^ face(d3 ) = fB; C g. When face(d4 ) = fB g for d4 , the weight of the annotated information is
w14 =
jfB gj 1 1 1 B gj 1 = 2 = = 0:33 jfA; B; C gj jfB gj 3 1 3 jf
for face(d1 ) ^ face(d4 ) = fB g. When face(d5 ) = fB; E g for d5 , the weight of the annotated information is
w15 =
jfB gj 1 1 1 B gj 1 = 2 = = 0:16 jfA; B; C gj jfB; E gj 3 2 6 jf
for face(d1 ) ^ face(d5 ) = fB g. When face(d6 ) = fB; E; F g for d6 , the weight of the annotated information is
w16 =
jfB gj jfB gj 1 1 1 1 = 2 = = 0:11 A; B; C gj jfB; E; F gj 3 3 9
jf
for face(d1 ) ^ face(d6 ) = fB g. When face(d7 ) = fD g for d7 , the weight of the annotated information is
w17 =
Figure 4: View towards a side The weight of annotated information from d3 ; forface(d3 ) = fB g, is
B gj jfB gj 1 1 1 = 2 = 0:5: B; C gj jfB gj 2 1
jf
w13 =
jf
The weight of annotated information from d4 ; forface(d4 ) = fB; E g, is
w14 =
B gj jfB gj 1 1 1 = 2 = 0:25: B; C gj jfB; E gj 2 2
jf jf
The weight of annotated information from d5 ; forface(d5 ) = fB; E; F g, is
w15 =
jfB gj 1 1 B gj 1 = 2 = 0:16: B; C gj jfB; E; F gj 2 3
jf jf
The weight of annotated information from d6 ; forface(d6 ) = fD; E g, is
w16 =
0 0 1 = 2 = 0: B; C gj jfD; E gj 2 2
jf
3. Viewing towards a face
0 0 1 = 2 =0 jfA; B; C gj jfD gj 3 1
for face(d1 ) ^ face(d7 ) = . 2. Viewing towards a side When viewing the object from direction d1 as shown in Figure 4(a), the weight of the annotated information V (d1 ) is
w11 =
2 2 1 =1 2 2
The weight of annotated information from d2 , for face(d2 ) = fB; C; F g, is
w12 =
B; C gj jfB; C gj 2 2 1 = 2 = 0:66: B; C gj jfB; C; F gj 2 3
jf jf
Figure 5: View towards a face As in the previous example, we dei ne the weight of the annotated information viewed from d1 as
w11 = 1
for face(d1 ) = fB g. The weight of the annotated information from d2 in Figure 5(b) is 1 1 2 = 0:5 1 2
w12 =
for face(d2 ) = fB; E g. Figure 6: "Quicktime VR" Annotation Model
The weight of the annotated information from d3 , face(d3 ) = fB; C; F g, and
w13 =
1 1 2 = 0:33: 1 3
The weight of the annotated information from d4 ; face(d4 ) = fA; C; Dg, and
w14 =
0 0 2 = 0: 1 3
As well as annotating spatial object(s) in the MBB, it is also possible to use this model to annotate the space surrounding the MBB by placing the viewpoint inside the MBB and annotating the twenty six boxes about the objects visible.
3.2.2 "Quicktime VR" Annotation Model The model described in the previous section is capable of aggregating information annotated in any direction. In considering the actual three- dimensional space such as urban space with streets and buildings, it is possible to use a simpler model. Quicktime VR generates a continuous panorama from twelve separate pictures taken at thirty-degree consecutive intervals. The image from one picture to the next is caliberated so the resulting panorama is continuous. Quicktime VR is used in generating a 360-degree view of an object or a 360-degree panorama around a given point. By introducing the concept of Quicktime VR, it is possible to construct a cylinder shaped model divided into twelve portions thirty degrees each. A certain point can be annotated in di erent directions depending on the objects visible. The annotated information can be aggregated in the fashion similar to the previous model. The di erence between the direction of annotation and view can be calculated and translated into the weight value which would be multiplied to the annotated information. As in the previous example, we express the set of authored information visible from viewpoint di as
V (di ) = fwij 1 A(dj ) j j = 1; 2; 1 1 1 ; 12g ,
and each weight wij as,
wij = .
j
face(di ) ^ face(dj ) j j face(dj ) ^ face(di ) j 1 j face(di ) j j face(dj ) j
This simpli ed model aggregates the information annotated 360 degrees around the viewpoint divided into twelve sections thirty degrees each named A(upper right) in Figure 6 to L respectively, so the number changes to twelve from twenty six. We de ne the angle of view as 150 degrees. The weight is determined in the similar manner. We demonstrate some examples below when the viewpoint d2 is located in the section B. When face(d2 ) = fL; A; B; C; Dg for d2 , the weight of the authored information is
w22 =
L; A; B; C; Dgj jfL; A; B; C; Dgj 5 5 1 = 2 =1 L; A; B; C; Dgj jfL; A; B; C; Dgj 5 5
jf
jf
for face(d2 ) ^ face(d2 ) = fL; A; B; C; Dg. When face(d4 ) = fB; C; D; E; F g for d4 , the weight of the authored information is
w24 =
jfB; C; D gj jfB; C; D gj 3 3 1 = 2 = 0:36 L; A; B; C; Dgj jfB; C; D; E; F gj 5 5
jf
for face(d2 ) ^ face(d4 ) = fB; C; Dg. Thus, the weight drops down from one as the di erence between the direction of the viewpoint and the direction of the annotated information increases. 4
Using
Feature
Describing
Vectors
Viewpoints
in and
Guided Tours
We propose to use feature vectors used in vector space model(Ingwersen, 1993) to express the annotated information of the three-dimensional objects. In the orientation-based annotation model described in the previous section, the orientation-based priority order determination rule was introduced to to determine the
weight of the information annotated in each direction. By using this rule to determine the weight and the annotated information expressed as a feature vector composed of a set of elelments, it is possible to annotate a three dimensional object viewed from a certain direction using feature vectors. 4.1
Feature
Vectors
for
Three-
Dimensional Objects and Spaces
We de ned the set of annotated information of a threedimensional object viewed from viewpoint di as,
V (di ) = fwij 1 A(dj ) j j = 1; 2; 1 1 1 ; 26g The annotated information A(dj ) can be expressed as feature vectors fj for the twenty six boxes. In order to aggregate the vectors based on orientation, we multiply them by the weight values wij and obtain the sum to get the feature vector Fi of the viewpoint di .
Fi (di ) =
Xw 26
fj (di )
(1)
Vectors
for
ij 1
j =1
4.2
Feature
In the previous section, we annotated a threedimensional object viewed from a viewpoint by the sum of feature vectors multiplyed by the weight determined based on the orientation. We should also annotate the guided tours in order to retrieve them by annotated information. We propose to annotate a guided tour by obtaining the sum of all the feature vectors of the viewpoints contained. By obtaining the sum of the feature vectors of all the viewpoints, we are able to obtain the feature vector Tr for the whole guided tour.
X F (d ) n
i
i
2. The user selects some samples. 3. The query vector is formulated from the selected samples. 4. The query is executed in the guided tour database. 5. The viewpoints or guided tours are retrieved. 6. The viewpoints or guided tours are presented to the user. Query vector is obtained by adding the feature vectors of one or more samples selected by the user and obtaining the average. The guided tours are tested for similarity by obtaining the cosine coecient value of the query vector and the feature vectors of the viewpoints or guided tours. Just as the similarity between the query vector and the feature vector of the text document is de ned as follows, cos(q; d) =
(2)
We assumed that each viewpoint is directed at a single three-dimensional object so the feature vector of the tour is the sum of the component elements of the feature vectors of all the objects viewed in the tour. Retrieval and Presentation of Spatial data 5.1
1
2
Retrieval of Viewpoints and Guided Tours
The wanted infromation can be retrieved in the form of idividual viewpoints or guided tours which are composed of viewpoints. The viewpoints or guided tours are retrieved according to the ow described below.
2
; t cos v 1
i
where t is an allowance for similarity, we de ne the similarity between the query vector and the feature vector of the viewpoints and guided tours. cos(q; T) =
pP qT pTP q 1
2
r
2
; t cos v 1
(3)
i
The tour satisfying the alllowance t, is retrieved as the viewpoint or guided tour matching the interests of the user. The query is formulated by adding the the feature vectors of objects selected by the user and obtaining the average.
i=1
5
pP qd pdP q i
Three-Dimensional Tours
Tr =
1. The user browses the 3D space for 3D objects of interest.
q=
1 n
X ^s n
(i)
i=1
;
S(i) S^(i) = (i) kS k
(4)
The feature vector of a given guided tour is calculated by obtaining the sum of all the feature vectors of the objects annotated at each viewpoint composing the tour. The elements of the feature vector of three-dimensional objects are rst multiplied by the weight wij decided by orientation, and they are added together to form the feature vector of the guided tour. The nature of such a feature vector is a ected by 1. the nature of the object annotated at each viewpoint 2. the orientation of the object(weight wij )
3. the nature of the component objects present on the viewpoints We assumed that the orientation of the object re ects the nature of the guided tour for the purpose of the tour is to view what is visible from that particular viewpoint. Since guided tour annotation described in this paper relies on the sum of the component elements of the feature vectors, how the nature of each object and how that particular combination of objects a ect the nature of the guided tour remain to be our research topic. Viewpoints can be retrieved separately but it is possible to retrieve them as a cluster. For example, let's say the "Quicktime VR" annotation model introduced in the previous section were placed along a street. The views of downtown would result in similar feature vectors, and those of suburban area would also result in similar feature vectors characteristic of the location. By retrieving viewpoints with similar vectors which are located close together, it is possible to retrieve an area of a city with similar features. 5.2
Presentation of Retrieved Spatial Data
5.2.1 Visualization Control by Orientation After the retrieval of a guided tour meeting the needs of the user, it is necessary to present the tour. The objects described in the viewpoints are visualized in presentation, and we have to consider a controlling mechanism for visualizing the annotated information for the three-dimensional objects which would be visible in the tour. Showing all the information annotated for each direction might cause confusion and also might exceed visualization capability. We use the orientation-based priority order determination rule used in annotation again in presentation to control the visualization. By using the weight value determined according to orientation, the visualization of the annotated information can be controlled by for example the size of the letters.
5.2.2 Visualization Control by Distance We introduce a method of controlling the visualization by the level of detail according to the distance between the user and the object. VRML (Hartman, 1996) has realized this function of LoD(Level of Detail). By using LoD, you can prepare a number of models of varying complexity for a single object and decide which model to visualize depending on the distance between the user and the object. Rendering speed can be accelerated by visualizing the coarce model when the user is far from the object and more detailed model when the user is near the object. By using the concept of LoD, it is possible to determine priority of information by the distance and therefore control the visualization
by distance so the information of the objects located near the user look more detailed than those of the ones located farther away. We de ne the distance between the object approximated by MBB and the user. The point halfway between the nearest and the farthest points from the origin is considered to be the centroid of the MBB. We de ne the distance between the centroid and the user's viewpoint as the distance between the user and MBB.
Figure 7: Distance between Viewpoint and MBB
5.2.3 Use of Threshold Values in Visualization Control We de ned two ways of controlling visualization of annotated information.
Visualization Control by Orientation
Visualization Control by Distance
By de ning a threshold value for each priority, it will be possible to control the visualization by the level of detail. Raising the threshold value results in retrieval of coarse information and lowering the threshold value results in retrieval of detailed information.
5.2.4 Connecting Viewpoints Since the guided tour is created by connecting the viewpoints which are described by objects visible, it is necessary to visualize the surrounding objects which are not described in the tour. 6
Annotation of Panoramas
The orientation-based annotation model with MBB surrounded by boxes and the "Quicktime VR" annotation model can be used to annotate a single spatial object located in the center. At the same time they can be used to annotate a number of spatial objects visible from a viewpoint within the MBB or in the center of the "Quicktime VR" annotation model. Considering the annotation of urban space, our "Quicktime VR" annotation model can be used to describe streets and plazas. Then we would be able to retrieve a certain cityscape by annotated information.
It will also be able to display various information in presentation using the visualization control based on orientation and distance. 7
Conclusions
In this paper, we introduced the orientation-based annotation model in order to annotate and aggregate information to a spatial object or a space based on orientation. We then applied vector space model to describe viewpoints and guided tours in order to realize retrieval of viewpoints and guided tours based on their characteristics. We are planning to implement a prototype system using the actual VRML data of a city capable of viewpoint or guided tour query, retrieval, and presentation. 8
Acknowledgements
This research was supported by the Japanese Ministry of Education Grant-in-Aid for Scienti c Research on Priority Area: "Advanced Databases," area no.275(08244103) and Research for the Future Program of Japan Society for the Promotion of Science under the project "Researches on Advanced Multimedia Contents Processing". We also thank Mr.Okajoh and Mr.Doi for developing the OAM and implementing a prototype system. References
[1] Fields,C., Negroponte, N., October 1977. Using New Clues to Find Data. Proceedings of Very Large Databases, Tokyo, pp156-158 [2] Hartman, J., Wernecke, J., Silicon Graphics, Inc. 1996. The VRML 2.0 Handbook(Building Moving Worlds on the Web), Addison Wesley Developers Press, Reading, [3] Ishikawa, M., Takakura, H., Uemura, T., 1997. Supporting Exchange of Information in Shared Virtual Space Based on Region De nition, Database System 113-50, pp.299-304 [4] Ingwersen, P., 1993. Information Retrieval and Interaction. Taylor Graham Publishing, London, pp.105-106 [5] Brand, S., 1987. The Media Lab. Viking Penguin Inc., New York, pp.191-194
