Virtual navigation fully controlled by head tracking 1 ... - CiteSeerX

Virtual navigation fully controlled by head tracking Patrick Bourdot, Martin Dromigny, Laurent Arnal LIMSI-CNRS UPR 3251 University of Paris XI BP 133, 91403 Orsay cedex - France tel: +33 (0)1.69.85.81.64 fax: +33 (0)1.69.85.80.88 e–mail: [email protected], [email protected], [email protected] http://www.limsi.fr/Recherche/IMM/theme2.html Abstract Many applications exploiting Virtual Reality (VR) require a hand-free navigation control. This forbides the use of devices such as Joystick, 3D mouse, ... That is generally because hand-based gestural interaction must be dedicated to specifical application tasks : sculpting or grasping the objects which compose a virtual prototype. With this aim, we present a fully head tracking approach which allows the system to determine the desired focus and the moving intention of the user during the navigational steps of a VR application. After we point out the limits of the classical solutions used to control virtual navigation, we present the conceptual and functional principles of our approach. Some of our considerations depend on physiological and ergonomic constraints. We conclude on the implementation state of a feasibility proof and the first application of our fully head tracking navigation controller. Keywords: 3D interaction, Head tracking, Navigational control, Virtual reality.

1 Introduction Any Virtual Reality (VR) application requires navigation control. To control a navigation in a virtual scene, the basic solution is the use of devices such as Joystick, 3D mouse and so on. But, as we already showed in a previous work on multimodal human-computer interface for CAD systems [1], it is really possible to reduce the amount of attention shifting if the gestural interaction is more dedicated to the prototyping tasks (sculpting, grasping, ...). If vocal commands are more suitable for functionnal requests than 3D mouse interactions on menus, speech is conversely not so intuitive to be entirely used to have a hand-free navigation control in a virtual world ! It is the reason why it is interesting to study fully head tracking approaches. In fact, except the “walking in place” metaphor [2, 7], many solutions combined events from head and hand trackers. On the other hand, if this kind of metaphor seems to be an appropriate VIRTUAL REALITY AND PROTOTYPING

June 1999, Laval (France)

solution for interactive walkthroughs, it has also several limits (small virtual world, virtual scenes structured as a real world,...). After a bibliographical overview, we explain how the valuation of a “Neutral Referential” can be used to manage a not restricted virtual navigation by a fully head tracking approach. With the help of a calibration stage our navigation control becomes independent to the spatial limitations of the actual tracking devices. Associated to an attenuation function the head tracking allows the system to determine the moving intention but almost the desired focus of the user during the navigational steps of a VR application. However, we will see that we make a distinction between a navigational mode and an observational one, while switching between these to modes could be usefully done by vocal commands. Finally we present a software demonstrator of our free-hand navigation controller and its first application.

2 Bibliographical overview 2.1 Visual devices and navigation control Two main classes of visual devices exist for VR applications : the first class is mainly “immersive” like Head-Mounted Display (HMD), while the second one can be considered as “semi-immersive” like Shutter Glasses (SG) associated to one or several stereoscopic displays [6]. Except for rear projection situations like in the CAVE [4], the main drawback of SG devices is to impose a kind of static use : even in Workbench case, any SG device requires to be almost in front of the stereoscopic displays. However, the HMD devices seem to be more dedicated to navigational applications. Unfortunately, these navigations are restricted due to the wires associated with this second type of devices. This short taxonomy shows that to enable large moves in VR worlds, i.e. movement greater than a user’s physically limited one, a specific approach needs to be settled. This limitation is usually overpassed by using other input devices in order to move specific part of interest in front of the user. Those devices can be Joysticks, Data gloves, Eyeball or any devices based on “HMD vehicle” techniques of DIVE [3], or on different interaction metaphors [8, 2]. In order to be efficient a 3D navigation requires commands with 6 degrees of freedom (6DOF) : 3 degrees for movement along X,Y, and Z axes and 3 degrees for rotations around the axes. 6DOF hand devices have been proposed by Ware et al. [8] and their ergonomic efficiency has been studied and proved by Zhai [9]. The “HMD vehicle” also manages 6DOF but divided in two groups : 3DOF for the orientation obtained from a head tracker and 3DOF for the direction given by a “flying mouse”. However, the use of this secondary device to help the navigation control is not always acceptable in many applications.

2.2 Vocal and gestural interactions for prototyping In a previous work on a 3D modelling system allowing reactive behaviour of the geometry for object simulations, we have studied the contribution of multimodal Human-Computer communication for interacting with the 3D objects built with our system [1]. No matter how much care is put into conceiving menus or cards, it is an acknowledged fact that the user of any CAD application has to shift his attention from 3D working areas to the

VIRTUAL REALITY AND PROTOTYPING

June 1999, Laval (France)

menu and card boxes. That was increased in our system by the fact that the high level of knowledge modeling implied deictic ambiguities when we needed to interact with some geometrical aspects of the objects (for instance, the simple vocal command “check this border” when clicking near a curve which describes a border of a surface is more easy than any menu command to activate the card which allows to manage the fitting properties of this border). So we provided the user with a set of vocal commands which could be substituted for menu and button actions. Interaction could then take place mainly in the 3D working areas and the user could apply commands whose corresponding button or menu item are not visible. Applied to a prototyping VR application, it is similarly possible to reduce the amount of attention shifting if the gestural interaction is more dedicated to grasping, modelling and sculpting tasks. That is because vocal commands are more suitable for functionnal requests than 3D mouse interactions on menus. Conversely speech is not so easy to be used for a hand-free navigation control in a virtual world, mainly because this control needs continuous, precise but almost unconscious requests.

2.3 Free-hand navigation control One of the interesting considerations of Brooks [2] was the “shopping-cart” metaphor which he firstly implemented with a real treadmill. A “virtual treadmill” was studied afterwards by Slater et al. [7]. This approach presents the advantage to replace the use of hands for navigational commands by a pattern recognition process using head tracking. This “walking in place” recognition is also a solution to control large moves in the virtual world when the user is spatially constrained. This kind of navigation command presents the three advantages of using a 6DOF input, freeing hand for application purpose and been fully intuitive in case of virtual world simulating real or future ones. But the use of a treadmill is a heavy equipment solution and does not seem to be a well-adapted metaphor for navigation in “pure” virtual worlds (for instance to visualize molecules or moving particles). Furthermore, “walking in place” is not appropriate for very large moves, in which case the operation should be more fatiguing than necessary. However, a head tracker is a 6DOF input device. So, taking into account physiological considerations, that will be discussed in section 3.6, a head tracker alone can be used to command 3D navigation.

3 Conceptual and functional principles 3.1 Basic concept The basic element of our approach is the “Neutral Referential”. This referential is defined by a software calibration of the head tracker when the user is supposed to place himself at the more frequent location and orientation he thinks he will have during his activity on the VR environment (for instance, his most comfortable position when he uses a SG device in front of a stereoscopic display). Intuitively, the main idea is : while the head tracker is far from the Neutral Referential, the user’s view point (for instance, the “Navigational Referential” of figure 1) must change proportionally to the relative move of the “Head

VIRTUAL REALITY AND PROTOTYPING

June 1999, Laval (France)

Yi

Head Tracker Referential (

Mi)

Zi

yi Pseudo scene center

zi

Xi

Navigational Referential (d mi)

Zn Yn

xi

Neutral Referential.

Xn

Figure 1: The 6DOF case. Tracker Referential”. In other words, while the user is not interesting by what he sees from the Neutral Referential, the scene keeps moving.

3.2 Neutral area criteria For physiological reasons the Neutral Referential is extended to a neutral area, defined by two tolerance volumes : one around the neutral position (the sphere of figure 1) and one around the neutral viewing direction (the cone of figure 2). From these considerations we apply the following algorithm :




=[

]

=[

]

let Mi x; y; z; ;  ;  , the Euler’s parameters of the relative move of the head tracker at the ti moment ;

dx; dy; dz; d; d ; d , the Euler’s parameters of the relative move of let dmi the user in the virtual scene at the ti moment ; if


M >  , 8j 2 f1; 3g j i

j

then compute the user’s move position dmji , 8j

if


M >  , 8j 2 f4; 6g j i

j

then compute the user’s view direction dmji , 8j

if


M <  , 8j 2 f1; 6g j i

2 f1; 3g ; 2 f4; 6g ;

j

then keep on the same view, update the Neutral Referential. We will explain later how we compute the dmi move from ever, this simple algorithm means :


M

i

(section 3.4). How-

a - We do almost nothing, if the relative move of the head tracker makes it staying inside the two tolerance volumes. VIRTUAL REALITY AND PROTOTYPING

June 1999, Laval (France)

Zi

zi

Yn

Zn

yi Yi

Xi

Xn

xi

Neutral Referential. Navigational Referential (d mi) Head Tracker Referential ( Mi)

Figure 2: The orientation 3DOF case. b - When the head tracker is inside the sphere tolerance volume but outside the cone tolerance volume, we provide a pure “directional” move (3DOF case) to overpass stereoscopic display viewing angle limitation (cf. figure 2). Accordingly, while the user stays in a neutral position, he can look around him as if he was in the center of the virtual world! c - If the head tracker is outside the sphere tolerance volume but inside the cone tolerance volume, we provide a pure “positional” move (the complementary 3 DOF case of the previous one). Keeping only the motion of translation, we can filter signal noise from the head tracker as well as user move imprecision. d - Finally, if the relative move of the head tracker is inside the two tolerance volumes, we have a complete 6 DOF navigational move.

3.3 Navigational vs Observational interactive modes However, the user of a VR application is not always navigating through the virtual scene. So, our navigational approach has to be considered as a particular interactive mode, that can be left to an observational mode and conversely. To switch between these two modes, keyboard events or vocal commands can be used. This observational mode is useful when the application tasks require that the user should stop paying attention to his head position according to the Neutral Referential. In this context, we manage two matrix systems to define the viewing move of the user in the virtual scene. As usually, we find for the observational mode (see figure 3) :

V ?1 = S ?1:N ?1 , where : S ?1 = S ??11 . i

i

i

i

i

For the navigational mode, the matrix system becomes (see figure 4) :

V ?1 = V ??11:N0 :N 0?1 , i

i

i

VIRTUAL REALITY AND PROTOTYPING

June 1999, Laval (France)

Si R

Sc

Ni

N0

Vi

∆ Mi Ti

T0

Figure 3: Matrix system of the observational mode. Si R

Sc

N0

Vi-1 Ni N’i

Vi

dmi Ci-1

dmi ∆ Mi

T0

Ci Ti

Figure 4: Matrix system of the navigational mode. where : Ni0 is an attenuated matrix computed from tion 3.4).

N0

and

N

i

matrix (see sec-

To switch from observational mode to navigational one, we only have to give Vi?1 as an argument of the navigational viewing procedure. For the converse switching, Si?1 has to be updated to give at U0 the view defined by the last Vi?1 of the navigational mode. In other words : Si?1 Vi?1 :Ni with Ni N0 .

=

=

3.4 Attenuation process







To determine the dmi move from Mi , it is necessary to make an attenuation on M . If we comeback to the previous notation, dmi is computed by finding Ni0 from N0 and Ni. However, we have to process separatly the position components from the orientation informations, because angles cannot be interpolated alike spatial trajectories. Moreover, this attenuation process has two aspects, the first one is the reduction of the movement to compensate the iteration of the navigational process, the second one is the forme response of this attenuation. To compute the new position a straightforward solution is to interpolate the referential origins of U0 and Ui by a parametric curve. For instance, if we choose a cubic B´ezier formalism, the evaluation of an attenuated position can integrate a non linear response for a chosen u parameter, the cubic polynomial providing a non linear response along a strait line trajectory. An independent attenuation of each Euler’s angles do not provide an optimal solution. VIRTUAL REALITY AND PROTOTYPING

June 1999, Laval (France)

Conversely, quaternions formalism allows a global interpolation of these angles between the two referentials U0 and Ui . In fact, the SLERP (Spherical Linear intERPolation) [5] has the property to give, on the shortest path to rotate from N0 to Ni , an attenuate matrix rotation Ni0 for a given u parameter.

3.5 Improvements and alternatives According to the position stage of this algorithm, when a user is approaching a specific virtual object, his successive positions describe a set of convergent viewing axes on the object (Z axis of the Navigational Referential on figure 1). This set can be used to determine a “pseudo center of scene” which is an approximation of the user’s focus position (for instance, from the intersection of the yi?1 zi?1 plan with the zi axis after it was moved into the referential of the virtual scene). This pseudo center can be proposed as a center of revolution candidate for temporary deterministic movements. If the pseudo center is not computable (ie : behind the user or at an infinite location), that simply means that the user’s interest on the virtual objects is changing. On the other hand, the attenuation of the rotation Euler’s angles can be formalized without quaternion. For instance, we can use a B´ezier patch of 3 x 1 degrees defined from a cubic B´ezier curve between xi?1 and Xi axes and from two linear segments whose directions are zi?1 and Zi axes. A simple use of the De Casteljau subdivision algorithm gives for an attenuation parameter k the wanted zi and a transversal vector which together describe a tangent plan of the B´ezier patch. Then, two cross produces are sufficient to define yi and xi !

3.6 physiological constraints The software calibration is not only necessary to define the Neutral Referential but is also useful to determine the maximum potential user’s moves. Those maxima (Kmax) define a volume fully dependent of human physiological anisotropic limitations. In order to compensate the anisotropy of this volume we normalize each components of the position moves by his particular Kmax. This normalization has to be done for the two directions of each spatial axis. About these physiological limitations, three remarks can be done. Firstly, the sphere of neutral position could eventually be defined as a percentage of this maximum reaching volume. Secondly, we process the physiological limitations introduced by the VR devices as an implicit component of the human physiological limitations. Finally, our algorithm takes into account an evolution of the Neutral Referential according to user fatigue. In this case, the Kmax coefficients have also to be updated.

4 Conclusion After a short bibliographical overview, we have presented in this paper a new solution to control virtual navigation. This fully head tracking approach is useful for many VR applications where gestural interaction must be dedicated to the user tasks (like it can be the case for prototyping). Our approach seems quite enough “natural” because the process gives the user the view point he expects just like he was actually moving in the virtual world. That forces VIRTUAL REALITY AND PROTOTYPING

June 1999, Laval (France)

the user to come back to the Neutral Referential as soon as the expected objects appear in front of him under the wished view point. To validate this, we have built a feasibility proof on simple virtual scenes. Our software demonstrator uses a server of events that we have fully developed at LIMSI-CNRS. This server handles classical X events as well as non standard events comming from VR devices (Logitec SG and its tracker, Polemus or Flock of birds trackers) or more semantic processes (such as voice or gestural recognition). The visual output of this demonstrator presents four windows. The first one represents the neutral and head tracker referentials and their relations with the tolerance volumes. The second window shows the user referential within the virtual scene. The third one displays a video of the user in the real world showning his behaviour. Finally the fourth window presents the user viewpoint on the virtual scene during his observation as well as his navigation stages. The first application of this free-hand navigation controller is the stereoscopic studies on databases of computational fluid dynamics. The next step will be to introduce more complex scenes to do several ergonomic studies. Most of these ergonomic protocols would be based on quality and coordination investigation techniques used by Zhai in his Ph.D. thesis [10].

References [1] P. Bourdot, M. Krus, R. Gherbi Cooperation between reactive 3D objects and a multimodal X window kernel for CAD. Lecture Notes in Artificial Intelligence (1374) on Multimodal Human-Computer Communication: Systems, Techniques and Experiments; H. Bunt, R. -J. Beun and T. Borghuis editors; Springer; October 1998. [2] F. P. Brooks, Jr. Grasping Reality Through Illusion : Interactive Graphics Serving Science. In Prooceeding of CHI’88 : Invited keynote address at Conf. on Computers and Human Interaction, pages 1-11, Washington, D.C., May 1988. [3] C. Carlsson, O. Hagsand. DIVE : A Platform for Multi-User Virtual Environments. Computers and Graphics, 17(6):663-669, 1993. [4] C. Cruz-Neira, D. J. Sandin, T. A. DeFanti, R. Kenyon, J. C. Hart. The CAVE : Audio visual experience automatic virtual environment. Communications of the ACM, 35(6):67-72, June 1992. [5] L. Downs. CS184: Using quaternions to represent rotation. Electronic version: http://http.cs.berkeley.edu/ laura/cs184/quat/quaternion.html [6] C. Hand. A survey of 3D Interaction Techniques. Computer Graphics Forum, 16(5): 269-281, December 1997. [7] M. Slater, A. Steed, M. Usoh. The Virtual Treadmill : A Naturalistic Method for Navigation in Immersive Virtual Environments. First Eurographics Workshop on Virtual Environments, Polytechnical University of Catalonia, ed. Martin Goebel, pages 7183, September 1993. Accepted for publication in a Springer Verlag book on Virtual Environments, after a further refereeing process, pages 135-148, 1995.

VIRTUAL REALITY AND PROTOTYPING

June 1999, Laval (France)

[8] C. Ware, S. Osborne. Exploration and virtual camera control in virtual three dimensional environments. ACM SIGGRAPH Computer Graphics, 24(2):175-183, March 1990. [9] S. Zhai. Investigation of feel for 6DOF inputs : Isometric and elastic rate control for manipulation in 3D environments. In Proceeding of the Human Factors and Ergonomics Society 37th annual meeting, 1993. [10] S. Zhai. Human Performance in Six Degree of Freedom Input Control. Ph.D. thesis, University of Toronto. Electronic version: http://vered.rose.toronto.edu/people/shumin dir/papers/PhD Thesis/top page.html

VIRTUAL REALITY AND PROTOTYPING

June 1999, Laval (France)