Addressing Challenges of Creating Room-Scale

Addressing Challenges of Creating Room-Scale Interactive Mixed-Reality Worlds by Using Off the Shelf Technologies Vlasios Kasapakis1, Damianos Gavalas2 and Elena Dzardanova2 1

Department of Cultural Technology and Communication, University of the Aegean, Greece, [email protected] 2 Department of Product and Systems Design Engineering, University of the Aegean Greece, {dgavalas, lena}@aegean.gr

Abstract. Mixed-Reality (MR) worlds are created by combining real and virtual worlds. While off the shelf solutions for creating MR worlds already exist, little is known on the methods followed and challenges faced not only towards the development of a room scale MR world using high-end technology, but towards the incorporation of real-object representation in such worlds. This paper showcases and records lessons learned when creating a room-scale MR world using widespread off the shelf object tracking technologies and 3D modelling techniques, allowing free user movement, accurate real-object representation in the virtual world, as well as interactivity between real and virtual objects. Keywords: Mixed Reality, Virtual Reality, Photogrammetry, 3D Scanning, Real-time object tracking.

1

Introduction

Virtual Reality (VR) refers to interactive synthetic worlds which often mimic the properties of real-world environments and allow user immersion. Notwithstanding, these synthetic worlds may also exceed the limitations of physical reality, since any law which naturally governs space, time, mechanics, material properties and so on, can be tempered with. A subclass of VR-related technologies, that is dedicated towards mixing real and virtual elements, is referred to as Mixed Reality (MR) [1]. Intensive research has allowed the implementation of MR in fields such us gaming [2], education [3], industrial design [4] and medicine [5]. In addition, advances in VR-related technology generated affordable and compact systems, such as Head Mounted Displays (HMDs), resulting to the commercial breach of VR systems back in 2015. These developments have triggered various VR and MR expansions, especially by independent developers and small laboratories. Based on ongoing research, it seems that the use of real objects in MR worlds, has a positive impact on users [6]. However, studies specifically addressing real-object usage in MR are scarce, whereas the techniques used to incorporate real objects into virtual worlds do not take advantage of recent technological standards which greatly increase user comfort and experience quality. In this paper, we showcase the design and development of a room-scale MR environment which will provide a testbed for future experiments. More importantly however, several issues related to MR implementation are addressed such as free user movement, accurate real-to-virtual object representation in the VR environment, as well as interactivity between real and virtual objects.

2

Related work

Several MR systems have been designed providing users with high quality immersive experiences [7, 8]. For example, MR MOUT (Mixed Reality Military Operations

2

in Urban Terrain) allows training simulation with the use of a video see-through HMD and a re-creation of urban landscape. Using a combination of bluescreen technology and occlusion models, MR MOUT allows a trainee to move around a courtyard and hide behind objects while real and virtual players emerge from portals and engage in close-combat battle [9]. Another example is Touch-Space, which is an ubiquitous, tangible, and social MR game-space, which whilst using the physical and social aspects of traditional gameplay, incorporates the real-world environment as an intrinsic and essential game element [10]. Touch-Space provides a full spectrum of game interaction as well as an embodied experience that begins in the real, physical surroundings (human to human and human to physical world interaction) and gradually transcends through augmented reality (AR) towards the virtual environment [10]. Several other MR systems, developed by widespread technology providers, blend the real world with virtual ones. The accelerating emergence of such systems is indicative of the field’s industrial and marketing appeal. Real Virtuality1 for instance, is a multi-user immersive platform, combining a 3D environment which can be seen and heard through a VR headset, with a real-life stage set. Users are tracked by a motion capture system, whilst being able to see their own bodies, move physically in the virtual environment and interact with physical objects. Whereas, VIRTSIM2 is a highly immersive virtual reality system which can pit squads of players against each other in a realistic virtual reality combat environment. The system utilizes full-motion body capture, VR headsets, gun props, and even shock-feedback. Even though MR systems like Real Virtuality and VIRTISM are not yet commercially available, room-scale VR experiences are; Oculus room-scale solution3 allows users to freely move in a room with the use of Oculus Rift. The technological advances presented above reveal a promising future for MR, where users will be able to freely move around real and virtual spaces simultaneously, while interacting with both real and virtual objects. However, the development of roomscale MR worlds like Real Virtuality and VIRTISM is a challenging process, especially since there are no guidelines in regards of technical setup, cost-effective solutions, or suggested methods of development. In addition, the aforementioned MR systems have not in fact included physiologically complex real-object representations into MR worlds. This study supports future developers by providing a detailed report of the steps followed towards creating a room-scale MR world with the use of widespread, off the shelf technologies. More specifically, in this paper we showcase the development process of a dungeon themed MR world which incorporates: • registration of user movement, performed in the real world, into the VR environment • accurate representation of complex real objects (i.e. a rock and a wooden stick) in the VR environment • registration of accurate user-interaction with real objects into the VR environment • interactivity between real and virtual objects The rest of the paper is structured as follows: Section 3 discusses the process followed to generate accurate virtual representations of real objects; Section 4 presents the

1

http://www.artanim.ch/blog/tag/siggraph/ https://www.roadtovr.com/virtsim-virtual-reality-platform/ 3 https://www.oculus.com/blog/oculus-roomscale-tips-for-setting-up-a-killer-vr-room/. 2

3

process followed to materialize the MR world along with a number of emerging issues, while Section 5 concludes our work and draws directions for future research.

3

Accurate real object representations

When designing an interactive MR world one of the most crucial components is accurate real-to-virtual object representation. Dimensions of the virtual object should of course be identical to the original, but also, its virtual ‘physiology’, simulated via appropriate textures and materials, should also be in accordance with the original object. Immersed users who are meant to touch a real object, virtually represented in the VR environment, should not experience inconsistencies in regard to tactile-to-visual correlating information. Therefore, realistic correlation of characteristics is a requirement which can be met via conventional 3D modelling methods (e.g. polygonal modelling) only for simple objects, in regards of geometry and texture. Otherwise, the modelling process is strenuous and time consuming, especially for complex objects like the rock used for the present development of a MR experience. To address this issue, the 3D modelling techniques of Photogrammetry and 3D object Scanning were used, using low-cost methods and equipment. 3.1.1

Photogrammetry

Photogrammetry, with the use of image measurement and interpretation, delivers the form and location of an object by using one or more photographs of that object [11]. Nowadays, various accessible software solutions simplify the process of creating 3D models via photogrammetry. In this work Autodesk Remake4 was used. In order to create an interactive 3D object with the use of digital imaging, a sequence of photographs is required, circling the perimeter of the object and alternating the capture angle at about 5 to 10 degrees between each shot so that the resulting photographic sequence has approximately 70 percent of overlapping information [12]. Moreover, all photographs need to be sharp and overall capture all possible angles of the object. Finally, the room/space, where the Photogrammetry process is taking place, needs to have proper lighting conditions to achieve diffused illumination of the object, whilst high contrast, intense shadows and the use of a flash light are prohibited. For the purposes of the present study the 3D model of a rock was generated using photogrammetry, to be later incorporated in a MR experience within a dungeon themed VR environment. Fig. 1a shows the preparation of the room where the sequential photographing of the rock took place. The rock was elevated with the use of a tripod to expose the maximum number of visible angles, whereas artificial lighting ensured diffused lighting conditions. The resulting sequence of photographs was imported into Autodesk Remake and the primitive 3D model was created. However, 3D objects, modelled via photogrammetry, often require additional tweaking, since surrounding objects may interfere with the extraction of object information through digital imaging (see Fig. 1b). Autodesk Remake offers a set of tools which allow alterations of the object and removal of excess information. The final 3D model representing the rock object is presented in Fig. 1c, while Fig. 1d shows the positions wherefrom the photos of the object were taken.

4

https://remake.autodesk.com

4

(a)

(c)

(b)

(d)

Fig. 1. (a) Photoshoot preparation (b) Initial photogrammetry results; (c) Final 3D model; (d) Photos positions.

3.1.2

3D Scanning

The second method utilized to create 3D models for the purposes of the present study, was the 3D scanning of real objects, with the use of relevant technologies, such as structured light or laser scanners, to create detailed 3D models of real objects [13]. To perform 3D scanning in this work, the low-cost, 3D depth camera-enabled, Microsoft Kinect5 sensor was used. The Microsoft Kinect sensor is based on an infrared light (IR) projector and an IR camera which allows the capture of changes on an IR pattern (see Fig. 2a). The Microsoft Kinect sensor utilizes the abovementioned process to produce depth images (see Fig. 2b). Finally, an RGB camera allows the capture of color [14]. After installing the Microsoft Kinect sensor using the Kinect SDK 6, the Skanect7 software was used to 3D scan objects. The object chosen to be 3D scanned was a wooden stick, also placed on a tripod to ease the scanning process. After feeding the dimensions of the object into Skanect, the 3D scanning process began, moving the Microsoft Kinect sensor around the wooden stick to capture as much geometry as possible (see Fig. 2c). Then, the colorize function of the Skanect software was used to analyze data captured by the RGB camera of the Microsoft Kinect sensor, creating a texture to be applied on the 3D model. Finally, the 3D model was exported from Skanect to the 3D graphics creation software, Cinema 4D8, where the additional unwanted geometry caused by surrounding objects was removed, finally producing the desired 3D model. However, the colorize function of Skanect produced low quality textures. Therefore, the wooden stick texture was replaced with one of higher quality, also representing a wood interface, created in Cinema 4D (see Fig. 2d).

5

https://developer.microsoft.com/en-us/windows/kinect www.microsoft.com/en-us/kinectforwindows 7 http://skanect.occipital.com/ 8 https://www.maxon.net/en/products/cinema-4d/overview/ 6

5

(a)

(b)

(c)

(d)

Fig. 2. (a) Microsoft Kinect sensor IR image; (b) Microsoft Kinect Sensor depth image; (c) 3D Scanning of object using Microsoft Kinect sensor; (d) 3D Scanning result (3D model on the left) and texture replacement into Cinema 4D (3D model on the right).

It should be noted that, based on the software available to this study, the 3D models produced with the process of photogrammetry were better suited for a MR experience, in terms of texture quality and lower number of polygons. However, using photogrammetry to create 3D models of objects with slim parts, such as the wooden stick, resulted in inaccurate models, rendering 3D scanning as a better option. Therefore, the choice of modelling method depends on various parameters and combination of software is mandatory for accurate real-to-virtual representation of objects, especially for MR worlds.

4

MR world materialization

4.1

Real object registration to the MR world

Vicon’s motion capture system, along with its complementary software, Blade, were the crucial components of the present study9. The Vicon motion capture system is built upon infrared cameras (see Fig. 3a), commonly placed along the perimeter of a room’s ceiling, allowing the tracking of retroreflective markers in real space (see Fig. 3b). Apart from full-body human motion capture, which is Vicon’s main purpose, tracking of prop objects is also possible by placing retroreflective markers around an object. To be able to track the position of the rock and the wooden stick objects in the real space, both were assigned with at least five (5) retroreflective markers (Fig. 3c) to ensure smooth tracking. Moreover, retroreflective markers were also assigned to an Oculus Rift Head Mounted Display (HMD), which was used by the user to navigate and experience the MR world (Fig. 3c). Following the marker assignment, the objects were placed on the floor and a data sample, including location and rotation of markers, was recorded using Blade. After reconstructing the data sample, the retroreflective markers’ positions appear in Blade, wherefrom they can be selected in groups. Finally, each retroreflective markers-group 9

https://www.vicon.com/products/software/blade

6

can be converted into a prop object, by assigning a prop name and then calibrating the prop using Blade’s built-in prop creation tool (see Fig. 3d).

(a)

(b)

(c)

(d)

Fig. 3. (a) Infrared camera; (b) Retroreflective marker; (c) Retroreflective markers assignment; (d) Prop creation in Vicon Blade.

The process described above allows the real-time registration (i.e. solving) of the position and rotation of the real objects to the props through Blade (see Fig. 4a & Fig. 4b). Here it must be noted that Blade calculates the position of a prop using a bone object, with the first marker selected during prop creation serving as the root of the bone and the second marker selected to represent the end-point of the bone. Fig. 4a, presents the rock item assigned with markers, while Fig. 4b shows the root, end-point and bone of the respective prop in Blade.

(a)

(b)

Fig. 4. (a) Objects location in the real world; (b) Solving of real object location to the props in Blade.

Continuing, to successfully materialize the MR world created in this work, the Unity 3D engine was used10. Unity 3D allows the creation of 3D interactive virtual worlds, whilst also allowing incorporation of Oculus Rift supported VR applications development. At first, a VR space representing a dungeon cell was created, having the same square dimensions as the real room where the user was whilst experiencing the MR world. Then, the 3D objects, created using photogrammetry and 3D scanning, were 10

https://unity3d.com/

7

imported in the VR space (see Fig. 5a). 3D objects are often affected by the software used when creating or processing them. Therefore, once they are imported into Unity 3D, their scale factor may require adjustment to secure proper dimensions in relation to the real objects. The following step regards data flow between Blade and Unity 3D, which is typically handled by Vicon’s Tracker11 or the Pegasus12 software. These however were unavailable; thus, a cost-effective solution was implemented by using Vicon’s Datastream SDK13, along with a custom script created for Unity 3D. The custom script was based on the script used to enable interaction between Unity 3D and Vicon Pegasus14. More specifically, the script collected information sent through Blade’s data stream function and used the names of the props to match the 3D objects’ and real objects’ location and rotation15. The Oculus Rift Core Package for Unity 3D 16 allowed preparation and usage of the Oculus Rift HMD in the MR world. This package includes tools which enable the integration of Oculus Rift HMD as part of a Unity 3D project, using simple drag and drop functions. Also, the Oculus Rift HMD comes with a separate sensor which tracks the HMD location to a certain distance. However, the tracking process only works when the HMD is directly in front of the sensor, thus limiting user’s range of movement whilst in the MR world. In addition, whilst trying to track the Oculus Rift prop location and rotation, the Oculus sensor interfered with ones provided by Vicon, and therefore had to be disabled. Even though Oculus Rift provides official instructions on how to implement a room scale solution, it is not suitable for real-time object tracking.

(a)

(b)

Fig. 5. (a) Objects location in the virtual world; (b) Objects location in the real world.

The processes described above result is an MR world where the user, in real-time, is free to move around, interact with real objects and their accurate virtual representations, and cause interaction between real and virtual objects17.

11

https://www.vicon.com/products/software/tracker https://www.vicon.com/products/software/pegasus 13 https://www.vicon.com/products/software/datastream-sdk 14 https://docs.vicon.com/display/Pegasus12/Using+Vicon+Pegasus#UsingViconPegasus-UnrealIntegration 15 The script used to register 3D objects location and rotation to the virtual space based on the props positions can be found at: http://zarcrash.x10.mx/Program.cs 16 https://developer.oculus.com/downloads/unity/ 17 An example of the MR world created in this work can be found at https://www.youtube.com/watch?v=Hy_wCD1LJYk 12

8

4.2

Real-time hand representation

Even though the above process was successful, several issues emerged when trying to implement hand interaction and subsequently test the accuracy of users’ touch with the real objects. First, Leap Motion18, an infrared sensor allowing precise hand tracking into VR environments was tested, to provide users with real-time hands visualization. However, the retroreflective markers placed on the real objects vastly affect Leap Motion’s performance. Fig. 6a illustrates the issue while using LeapMotion as the left hand of the user (while holding the rock object) is not accurately tracked in VR19. Therefore, the use of Leap Motion is not recommended.

(a) Fig. 6. (a) Leap motion hand tracking and occlusion issue.

4.3

Object tracking accuracy issues

When developing MR worlds with technologies such as the ones proposed in this paper, another issue to consider is proper adjustment of the 3D object’s pivot points (the center of the object’s rotational system), in order to enable accurate representation of real objects in the virtual world. The pivot point of every 3D object should be adjusted at the same location as its respective prop’s root point, to increase the tracking precision and also avoid incidents such as rotation and location offsets during user interaction with the real objects. To test the precision of this process we designed (using Cinema 4D) and 3D Printed a 10x10 cube, assigned it with retroreflective markers and finally registered it as a prop in Blade (see Fig. 7a & Fig. 7b). Moreover, the pivot point of the 3D model of the cube was assigned to the root of the prop item (see Fig. 7b and Fig. 7c). Finally, a coded script which acquired the markers positions from Blade and visualized them into Unity 3D, thus testing in detail the prop markers’ location both in real and virtual world (see Fig. 7d). This way we could accurately move and rotate the cube object20 (see Fig. 7e). To test accuracy of user touch on real objects and their virtual representations, we used a 3D hand model which represented a user’s real hand tracked by Blade in the virtual world (see Fig. 7f). The 3D hand had the same dimensions as the hand of the user. Tests examining interactivity of the virtual hand with both the cube21 and the

18 19

20

21

https://www.leapmotion.com/ An example of the LeapMotion performance can be found in https://www.youtube.com/watch?v=MoY7R02_ShE An example of the interaction between the user and the cube item can be found at https://www.youtube.com/watch?v=p-f7z2oUk7s An example of the interaction between the user hand and the cube item can be found at: https://youtu.be/nwnU8BiJX00

9

rock22 object showed successful levels of accuracy, since the user was able to touch each detail of the objects, both in real and virtual world.

(a)

(b)

(c)

(e)

(d)

(f)

(g)

Fig. 7. (a) 3D Printed cube assigned with retroreflective markers; (b) 3D Printed cube prop in Blade; (c) Pivot point adjustment; (d) Markers visualization in Unity 3D; (e) 3D Printed cube registration in VR; (f) Accurate hand interaction with the 3D Printed cube in VR; (g) Accurate hand interaction with the rock object in VR.

5

Conclusion and future work

This paper presents the process followed and challenges faced whilst creating an interactive MR world using off the shelve technologies. The MR world generated for the purposes of this paper, along with the implementation process, will serve as a testbed for further experimentation. However, the development process of the MR world, which can assist future developers, revealed useful directions such as: • • •

22

photogrammetry and 3D Scanning can be used to accurately represent real objects in the virtual world, object tracking accuracy is highly affected by the tracking method followed. Therefore, the pivot points and object scaling should be handled with care as the tiniest offsets may greatly affect user experience, hand tracking accuracy is important and LeapMotion is incompatible with systems using retroreflective markers.

An example of the interaction between the user hand and the rock item can be found at: https://www.youtube.com/watch?v=4b3QDFap2Ok

10

Future research aims at developing a prototype using Arduino23 and Flex Sensors24, which will allow finger interaction with virtual objects. We also wish to investigate the impact of real object usage (also incorporating 3D printed objects) as part of a MR experience and study its possible affects to users’ co-ordination and levels of immersion using real participants.

6

References 1. 2. 3. 4.

5.

6.

7.

8.

9.

10.

11. 12.

13.

14.

23 24

Milgram, P. and F. Kishino: A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems 77(12), 1321-1329 (1994). Cheok, A. D., Sreekumar, A., Lei, C., & Thang, L. N.: Capture the flag: mixed-reality social gaming with smart phones. IEEE Pervasive Computing, 5(2), 62-69 (2006). Pan, Z., Cheok, A. D., Yang, H., Zhu, J., & Shi, J.:Virtual reality and mixed reality for virtual learning environments. Computers & Graphics 30(1), 20-28, (2006). Fiorentino, M., de Amicis, R., Monno, G., & Stork, A.: Spacedesign: A mixed reality workspace for aesthetic industrial design. In Proceedings of the 1st International Symposium on Mixed and Augmented Reality. IEEE Computer Society, Germany (2002). Ferrari, V., Megali, G., Troia, E., Pietrabissa, A., & Mosca, F.: A 3-D mixed-reality system for stereoscopic visualization of medical dataset. IEEE Transactions on Biomedical Engineering 56(11), 2627-2633, (2009). Hoffman, H.G.: Physically touching virtual objects using tactile augmentation enhances the realism of virtual environments. In Proceedings of the Virtual Reality Annual International Symposium. IEEE, USA (1998). Davies, M. and V. Callaghan.: iWorlds: Building mixed reality intelligent environments using customisable 3D virtual worlds. In Proceedings of the Sixth International Conference on Intelligent Environments, IEEE, Malaysia (2010). Tecchia, F., Avveduto, G., Brondi, R., Carrozzino, M., Bergamasco, M., & Alem, L.: I'm in VR!: using your own hands in a fully immersive MR system. In Proceedings of the 20th Symposium on Virtual Reality Software and Technology. ACM, UK (2014). Hughes, C. E., Stapleton, C. B., Hughes, D. E., & Smith, E. M.: Mixed reality in education, entertainment, and training. IEEE computer graphics and applications, 25(6), 24-30 (2005). Cheok, A. D., Yang, X., Ying, Z. Z., Billinghurst, M., & Kato, H.: Touch-space: Mixed reality game space based on ubiquitous, tangible, and social computing. Personal and ubiquitous computing 6(5-6), 430-442 (2002). Luhmann, T., Robson, S., Kyle, S. A., & Harley, I. A.: Close range photogrammetry: principles, techniques and applications. Whittles (2006). Santagati, C., L. Inzerillo, and Di Paola, F.: Image-based modeling techniques for architectural heritage 3D digitalization: limits and potentialities. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences 5(w2), 555560 (2013). Tong, J., Zhou, J., Liu, L., Pan, Z., & Yan, H.: Scanning 3d full human bodies using kinects. IEEE transactions on visualization and computer graphics 18(4), 643-650 (2012). Zhang, Z.: Microsoft kinect sensor and its effect. IEEE multimedia, 19(2), 4-10 (2012).

https://www.arduino.cc/ https://learn.sparkfun.com/tutorials/flex-sensor-hookup-guide