Immersive Environment for Distributed Creative Collaboration Khanh-Duy Le
Morten Fjeld
Chalmers University of Technology Lindholmen, Kuggen 412 96, Gothenburg, Sweden
[email protected]
Chalmers University of Technology Lindholmen, Kuggen 412 96, Gothenburg, Sweden
[email protected]
Ali Alavi
Andreas Kunz
ETH Zurich Leonhardstrasse 21 8092, Zurich, Switzerland
[email protected]
ETH Zurich Leonhardstrasse 21 8092, Zurich, Switzerland
[email protected]
ABSTRACT While videoconferencing has been available for years, tools for distributed creative collaboration such as brainstorming have not yet reached professional use. Unlike regular conferencing, brainstorming relies on information exchange across multiple channels in shared task and communication spaces. If a participant joins a facilitated brainstorming session remotely through his/her mobile device, perceiving these spaces is challenging. By proposing an immersive environment for the remote participant, this tech note addresses how to provide him/her a stronger engagement. Offered as a client application for the remote participant’s tablet device, it enables him/her to see all channels and select which one to interact with. Our proof-of-concept client allows us to examine how this immersive environment for distributed creative collaboration can provide the remote participant with an increased engagement.
CCS CONCEPTS •Human-centered computing →Collaborative and social computing devices; Gestural input; Collaborative and social computing systems and tools;
KEYWORDS team work, immersion, mobile device ACM Reference format: Khanh-Duy Le, Morten Fjeld, Ali Alavi, and Andreas Kunz. 2017. Immersive Environment for Distributed Creative Collaboration. In Proceedings of VRST ’17, Gothenburg, Sweden, November 8–10, 2017, 5 pages. DOI: 10.1145/3139131.3139163
1 MOTIVATION Distributed creative collaboration, such as brainstorming, is an established way to generate ideas and handle problems efficiently. However, teams are often distributed so that remote participants Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from
[email protected]. VRST ’17, Gothenburg, Sweden © 2017 ACM. 978-1-4503-5548-3/17/11. . . $15.00 DOI: 10.1145/3139131.3139163
either need to travel or use off-the-shelf video conferencing. In the latter situation, the collaborative workspace can be subdivided into two complementary spaces [Kunz et al. 2014]. First, in the task space, participants interact with physical and digital artifacts using touch- or (deictic) gesture input. Second, in the communication space, participants use verbal or body language [Nicolas 2012]. These two spaces need to be captured, aligned, transferred, and presented to the remote participant. Their synchronous alignment is key; for instance, task space gestures must be presented in correct relationship to the task space artifacts [Kirk et al. 2007; Kirk and Stanton Fraser 2006]. To this end, videoconferencing does not allow for a more immersive integration of mobile participants into a facilitated brainstorming session, since important communication channels are not transferred properly or are completely missing. Many videoconferencing systems are capable of transferring the task space and a video signal from the persons interacting with it. On the remote side, these two information streams are displayed in separate windows, which destroys the spatial correlation between these two channels. Moreover, separate windows are still difficult to display on mobile devices and do not offer sufficient resolution to clearly show the content. Consequently, videoconferencing with mobile devices is frequently regarded as useless and people prefer traveling instead. In this paper, we introduce a concept to transfer and display all information channels correctly, also integrating the collocated group, the communication channel, and the facilitator as important sources of information. Our work is motivated by a growing interest in 360◦ cameras. For instance, while Polycom offers the RealPresence Centro [Polycom 2017] capturing everyone in the meeting room, their system requires a dedicated camera. Aiming for a low-cost meeting rooms solution while leveraging a mobile device camera, we make use of a low-cost Kogeto Dot 360◦ camera lens (Figure 2). Our setup leverages the wide-spread adoption of mobile devices for easier acquisition of the system. Besides, the design choice of placing the tablet equipped with a 360-lens on a table makes the system usable in any meeting room environment.
2
RELATED WORK
The importance of transferring task and communication spaces to the remote participant was addressed by Krueger [Krueger 1983] who introduced a system allowing the remote participant to see,
VRST ’17, November 8–10, 2017, Gothenburg, Sweden but not interact with, the shared task space. To allow for a collaborative workspace, Tang & Minneman [Tang and Minneman 1991a] introduced VideoDraw. It allowed drawing on the screen, which was captured by a camera and transferred to the remote side. It was the first system which transferred task and communication spaces correctly. Since the interaction space was limited, the authors later introduced VideoWhiteboard [Tang and Minneman 1991b]. It had a rear-projection screen where users were captured through the screen, resulting in a shadow as representation of the communication space. Commune used two transparent digitizing boards mounted on top of a screen [Bly and Minneman 1990]. Instead of capturing the hand, users could make gestures and add marks using the tablet’s stylus. Ishii et al. [Ishii and Kobayashi 1992] introduced ClearBoard, which allowed keeping eye contact in the communication space while drawing on the interactive surface in the task space. The system by Stotts et al. [Stotts et al. 2003] acquired each user’s face and displayed it on the collaborative task space. To share a personal task space with a remote participant, Wellner presented DigitalDesk [Wellner 1993] and Double DigitalDesk [Wellner and Freeman 1993]. Using a projector and a camera mounted above a regular desktop, digital artifacts were superimposed over real objects, while the user’s pointing gestures were captured by the camera. Hand gestures of users interacting with a digital whiteboard were captured by a camera within the VideoArms system [Tang et al. 2004, 2006]. Finally, a shared editing of a common whiteboard with a video overlay was introduced with the CollaBoard system [Kunz et al. 2010; Nescher and Kunz 2011]. However, none of these systems was able to offer the remote participant an immersive environment showing the collaborative task space together with the facilitator and the collocated team members. This paper introduces preliminary work towards an immersive environment that will allow the remote participant to better participate in the meeting.
3 CONCEPT FOR IMMERSIVE COLLABORATION In our approach, we preserve the spatial distribution of information, as it could for example stem from multiple whiteboards in the room. Moreover, studies show that it is also important for the remote participant to see the collocated team to gain a better understanding of the ongoing discussion. Figure 1 shows how the proposed system should thus enable the remote participant to see: • the common whiteboard from the position of the collocated team (view 1), • the collocated team from the facilitator’s position (view 4), • the whiteboard without the facilitator obstructing it, • the facilitator and his/her deictic gestures, • additional information screens (views 2 and 3). While this could be solved in principle by streaming four videos for the different orientations in a room, the remote participant could not see all these streams on a mobile device such as a tablet, due to its small screen and limited CPU/GPU performance. We thus propose a single stream setup, as shown in Figure 1. An image sensor of a tablet in the middle of the table captures a 360◦ video, from which mainly view 1 (the facilitator and the
D. Le and M. Fjeld and A. Alavi and A. Kunz
Figure 1: Setup of the immersive collaboration system.
Figure 2: A normal tablet with a 360-degree lens (here: Kogeto Dot 360 lens) attached on the front-facing camera.
whiteboard) and view 2 (the collocated team) are relevant. If necessary, views 3 and 4 could also show additional information spaces. While this panoramic video would already be sufficient to transfer the spatial situation in the meeting room to the remote participant, it does not allow any interaction by the remote participant, e.g. changing the content of the whiteboard. Moreover, in the raw video stream the facilitator’s body might also block regions of the whiteboard, or the content itself might not have sufficient resolution to be clearly readable by the remote participant. To overcome these shortcomings for the remote participant, we additionally capture the facilitator’s pose and deictic gestures by a Kinect depth camera. The captured data is then used to control the position and behavior of an avatar to represent the facilitator. We also reconstruct a virtual whiteboard representing the content of the real one. The avatar and virtual whiteboard are then added to the scene, replacing the real-life view 1. The remote participant is now able to fully navigate the content of the digital whiteboard, and can also manipulate the moderator’s avatar (e.g. by fading it out) to get an unobstructed view of the whiteboard. In order to avoid the circular view of the panoramic video, the other views (24) are mapped to a cube to resemble the typical shape of a room.
4
TECHNICAL SETUP
A tablet is equipped with a fisheye lens capturing the 360◦ video of the room with the facilitator, the whiteboard and the collocated team (Figure 2). It is placed on the table and the video stream is then transferred to the remote participant (Figure 3).
Immersive Environment for Distributed Creative Collaboration
VRST ’17, November 8–10, 2017, Gothenburg, Sweden
Figure 3: Remote participant client application: Panoramic video with indicated mapping separators (corners of the real room).
Figure 5: Remote participant client application: Avatar in front of the virtual whiteboard. Figure 4: Remote participant client application: Panoramic image captured by the 360◦ lens. On the remote participant’s tablet, this panoramic video is then used to build a virtual room, i.e. the video segments (Figure 3) are attached to the walls of a cubic virtual room to increase the spatial awareness of the remote participant. However, since the content of the whiteboard might no longer be readable to the remote participant, the video stream on that wall (Figure 3, center) is replaced by the content of the digital whiteboard (Figure 4). The video is evenly divided into four segments and mapped on four walls of the virtual room. The ceiling and floor of the room are synthesized by blending together the top and bottom 20% of the four walls to create visual coherence. The virtual whiteboard is overlaid on the top of the screen and the virtual avatar is also placed at the location of the facilitator. However, this comes at the cost of losing all non-verbal communication elements of the facilitator, such as deictic gestures that are very important in group meetings. To solve this, an avatar is placed in front of the virtual whiteboard. The appearance of the avatar is synthesized from the appearance features of the facilitator extracted from Kinect data such as height, hair color and clothing colors. The synthetization is performed only once for each user when he/she first interacts with the whiteboard. In terms of behaviors, this avatar is animated by the tracking signals stemming from the Kinect sensor. While the avatar might not look as realistic as a video image, its posture and pointing gestures correctly match those of the real facilitator (Figure 5). Since in virtual reality behavioral realism has a stronger effect than appearance realism to the anthropomorphic perception of users [Seyama and Nagayama 2007], we consider this visualization approach not problematic to users’ perceptions, especially on relatively small-sized devices like tablets. In addition, this visualization approach will require less
Figure 6: Remote participant client application: Interaction capabilities ‘swiping’ (left) and ‘pinching’ (right).
bandwidth to transmit the up-to-date movement of the facilitator compared to sending the whole Kinect point cloud, since the system just needs to send the positions of the avatar’s skeleton joints. This results in smaller latency and information loss, hence it is especially beneficial for mobile devices, where the internet connection is not reliable. Further, the remote participant is able to control his/her view in the virtual meeting room by horizontally swiping on the tablet’s screen. The system also supports pinching in order to zoom into the virtual environment (Figure 6). With these gestures, the remote participant can change the viewing perspective to see both the virtual whiteboard with the facilitator (avatar) as well as some collocated team members (Figure 7). By pressing the button ‘TO THROUGH-GLASS VIEW’ on the bottom left corner, users can switch to ‘Writing-on-glass’ view (similar to the metaphor of ClearBoard), where the virtual whiteboard is not occluded by the facilitator. The other collocated participants in the room are also shown behind the screen to let the user be aware of their presence and activities. To go back inside the room, the user needs to press the button ‘GO INSIDE ROOM’ on the bottom right corner (Figure 8).
VRST ’17, November 8–10, 2017, Gothenburg, Sweden
D. Le and M. Fjeld and A. Alavi and A. Kunz • Does the virtual collaboration help the remote participant to better participate in a brainstorming meeting? • Does the avatar have an equivalent or even better performance in integrating the remote participant? • Will the visibility of the collocated team help the remote participant to better participate in ongoing discussions? • Will the setup be able to transfer the spatial relationship of information to the remote participant’s 2D output device?
ACKNOWLEDGMENTS This research work was done within the EUREKA CELTIC project No. C2013/1-3 Figure 7: Remote participant client application: A combination of views 1 and 4 showing the avatar of the facilitator working at the whiteboard and a team member.
Figure 8: Remote participant client application: ‘Writing on glass’ mode.
We adopted touch-based interaction for our concept instead of utilizing VR headsets due to two main reasons. First, wearing a VR headset is still not widely socially accepted in public places (bus, train, station, airport, etc.), where users are very likely to work remotely with their team. Second, as we are working on the next stage of this concept, where remote users can interact on the virtual whiteboard such as pointing or even drawing. Here, touch input will be more precise and cause less arm fatigue than in-air interactions in a virtual environment when wearing a VR headset.
5 SUMMARY AND FUTURE WORK We introduced the proof-of-concept prototype of an immersive remote participation in a collocated team meeting. In particular, we focused on the correct spatial representation of information and collocated team members, since this is an important factor for the active participation of a remote participant. Integrating a virtual whiteboard together with an avatar allows optimal use of the screen size of the remote participant’s tablet in order to read the content of the whiteboard. Since the remote participant can actively control the viewing direction, it is also possible for him/her to see the collocated team during a discussion. Using this improved prototype, future work will also address the following research questions:
REFERENCES Sara A.. Bly and Scott L. Minneman. 1990. Commune: A Shared Drawing Surface. In COCS ’90 Proceedings of the ACM SIGOIS and IEEE CS TCOA conference on Office information systems. ACM, New York, USA, 184–192. https://doi.org/10.1145/91478.91514 Hiroshi A.. Ishii and Minoru Kobayashi. 1992. Clearboard: A Seamless Medium for Shared Drawing and Conversation with Eye Contact. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, USA, 525– 532. https://doi.org/10.1145/142750.142977 David Kirk, Tom Rodden, and Danea Stanton Fraser. 2007. Turn it This Way: Grounding Collaborative Action with Remote Gestures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, USA, 1039–1048. https://doi.org/10.1145/1240624.1240782 David Kirk and Danea Stanton Fraser. 2006. Comparing Remote Gesture Technologies for Supporting Collaborative Physical Task. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, USA, 1191–1200. https://doi.org/10.1145/1124772.1124951 Myron K. Krueger. 1983. Artificial Reality 2 (2nd. ed.). Wiley, New York, NY. Andreas Kunz, Ali Alavi, and Philipp Sinn. 2014. Integrating Pointing Gesture Detection for Enhancing Brainstorming Meeting Using Kinect and Pixelsense. In Proceedings of the 8th International Conference on Digital Enterprise Technology. CIRP, Paris, France, 205–212. https://doi.org/10.1016/j.procir.2014.10.031 Andreas Kunz, Thomas Nescher, and Martin Kchler. 2010. CollaBoard: A Novel Interactive Whiteboard for Remote Collaboration with People on Content. In Proceedings of the 2010 International Conference on Cyberworlds. IEEE, New York, USA, 430–437. https://doi.org/10.1109/CW.2010.17 Thomas Nescher and Andreas Kunz. 2011. An Interactive Whiteboard for Immersive Telecollaboration. International Journal of Computer Graphics 27, 4 (2011), 311–320. https://doi.org/10.1007/s00371-011-0546-2 Michinov Nicolas. 2012. Is Electronic Brainstorming or Brainwriting the Best Way to Improve Creative Performance in Groups? An Overlooked Comparison of Two Idea Generation Techniques. Journal of Applied Social Psychology 42, S1 (2012), E222–E243. https://doi.org/10.1111/j.1559-1816.2012.01024.x Polycom. 2017. RealPresence Centro. (May 2017). Retrieved June 26, 2017 from http://www.polycom.com Jun’ichiro Seyama and Ruth S. Nagayama. 2007. The uncanny valley: Effect of realism on the impression of artificial human faces. PRESENCE: Teleoperators and Virtual Environments 16, 4 (2007), 337–351. https://doi.org/10.1162/pres.16.4.337 David. Stotts, Jason Smith, and Dennis Jen. 2003. Clearboard: A Seamless Medium for Shared Drawing and Conversation with Eye Contact. In Proceedings of UIST 2003. ACM, New York, USA, 57–58. Anthony Tang, Carman Neustaedter, and Saul Greenberg. 2004. VideoArms: Supporting Remote Embodiment in Groupware. In Video Proceedings CSCW 2004. ACM, New York, USA. Anthony Tang, Carman Neustaedter, and Saul Greenberg. 2006. VideoArms: Embodiments for Mixed Presence Groupware. In Proceedings of HCI 2006. ACM, New York, USA, 85–102. https://doi.org/10.1007/978-1-84628-664-3 8 John C. Tang and Scott L. Minneman. 1991a. VideoDraw: A Video Interface for Collaborative Drawing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, USA, 313–320. https://doi.org/10.1145/97243.97302 John C. Tang and Scott L. Minneman. 1991b. VideoWhiteboard: Video Shadows to Support Remote Collaboration. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, USA, 315–322. https://doi.org/10.1145/108844.108932 Pierre Wellner. 1993. Interacting with Paper on the Digital Desk. Commun. ACM 36, 7 (1993), 87–96. https://doi.org/10.1145/159544.159630 Pierre Wellner and Stephen Freeman. 1993. The Double Digital Desk: Shared Editing of Paper Documents. XEROX Euro PARC Technical Report EPC-93-108 (1993).
