Move and Speak: Interaction with Multiple Surfaces Using Gestures and Voice Andreia Ribeiro
Carlos Duarte
LaSIGE, University of Lisbon FCUL, Edificio C6, Campo Grande, 1749-016 Lisboa, Portugal
LaSIGE, University of Lisbon FCUL, Edificio C6, Campo Grande, 1749-016 Lisboa, Portugal
[email protected]
[email protected]
ABSTRACT This paper shows the development of a Web application which will allow to study the gestures based interaction with voice commands. Gestures can be performed using touch in a tabletop, or without touch in a projection. These two surfaces are in continuous communication, which allows the user to transmit any information from one surface to the other. In this paper we explain which methods and technologies did we use to develop this application, as well as the decisions we took for the application to meet up to the study’s goals.
Categories and Subject Descriptors H5.2 [Information Interfaces and Presentation]: User Interfaces
General Terms Design, Human Factors
Keywords Gestures, Voice, Multitouch, Commands, Interaction, Surfaces, Touchless
1.
INTRODUCTION
The latest advances in technology made it possible for us to interact with devices in a different way than we are used to, without using a mouse, a keyboard or other controllers. In the last years, some devices have been launched to the market, like interactive tables with support for singletouch, rapidly evolving to multi-touch as it is the case of Microsoft Surface table. The same thing has happened with cell phones (or smartphones), where some of the devices are now multi-touch, with big screens, without keys to press, where the user just touches the object in the screen, drags it and manipulates it directly.
Copyright held by author(s). ISPD ’12 June 4-5, Porto, Portugal .
Other way to interact with the applications is using gestures, without using touch. This kind of interaction was explored in the paper “Put-that-there” [2], where the participants used gestures and voice to move an object. Almost two years ago, Microsoft released Kinect - a motion sensing input device - for their proprietary XBox 360. Earlier this year the final version of Kinect’s SDK has been released, which allows us to use it in our study. With Kinect, “touchless” gestures are now more accessible to being studied, without the need to use intrusive devices, like special gloves for hand tracking and sensors placed in the user’s body. This kind of evolution made it possible to interact with devices in a more natural and intuitive way [3]. Gesture-based interaction is becoming pervasive not only in smartphones, tablets and interactive tables, but also in video games. This work studies this type of interaction, which combines touch gestures and “touchless” gestures with or without voice in both cases.
2.
BACKGROUND AND RELATED WORK
When people communicate with each other they use speech and gestures. Gestures are a very important action because they are part of natural communication. Using this natural interaction with computers is still being studied. This work studies the gestural interaction with voice in multiple devices, because it is a growing area and it is quite appealing to be able to communicate with devices in a natural way. There is already some research in this area to help us better understand this new type of interaction: studies that show which touch based gestures are adequate for large screens [5], studies that show what “touchless” gestures with and without the help of voice commands are used [4, 2]. There are also some studies with interactive tables that show the gestural interaction through touch with the table [6] and show which gestures are most commonly used with them [7].
3.
OBJECTIVES
This work aims to explore the capabilities offered by gestural interaction with the addition of voice, both in direct contact interaction with the device and “touchless” interaction. It is necessary to understand how the users behave in different scenarios: - Interaction with the device using touch gestures; - Interaction with the device using “touchless” gestures, like zooming out (figure 1); - Interaction with the device using voice as an addition to gestures;
NodeJS [1]. We used HTML5 because it provided Web Sockets for easy communication, which we use to establish a connection between the browser and NodeJS. On the interactive table, we used TUIO protocol to recognize gestures. Figure 1: Representation of a user making a zoomout gesture.
Figure 2: Representation of a swipe gesture.
- Simultaneous interaction with two devices using “touchless” interaction on one device, and touch interaction on another device. Users will be able to swap information and data between the devices, as depicted in Figure 2, and it is fundamental to identify the difficulties encountered in all these processes, and which method the users naturally tend to use, to accomplish the tasks.
4.
5.
DECISIONS
For the application we used NodeJS, which allows for communication between Kinect and the browser, with the data flowing in real time. This way there is no delay between the user’s movements and the screen’s feedback. For the user’s data manipulation we had to calculate the distance between the shoulders and hands (given by the skeleton model available in Kinect’s SDK) and check which coordinates belonged to them, so that we could understand if the user wants to move, amplify or rotate an object. Initially for user testing the values were checked using only one skeleton. Now we can have as many skeletons as we want interacting with the surface as long as they are within the Kinect’s limit.
6.
FUTURE DEVELOPMENTS
The case study will be done with various users that will interact with this application, doing tasks previously assigned to help identify the important aspects of interaction, i.e the tasks have to force the user to interact with a combination of gestures and voice in multiple surfaces.
METHODS AND TECHNICAL DETAILS
To perform this study we decided to build a web application, since all users have access to a browser, regardless of what platform they use. This application allows us to study the gestural interaction, with or without touch and with or without voice interaction in the scenarios described in section 3. The application will allow users to transfer and manipulate information in various ways and between various surfaces (see Figure 3). All “touchless” gestures are obtained using Microsoft’s device for XBox 360: Kinect. For touch gestures, an interactive table is used. An application was created in Visual Studio 2010, using the programming language C# to recognize gestures and retrieve data from the Kinect, which are then transmitted to a simple JavaScript-based server: the
Figure 3: Application architecture.
7.
REFERENCES
[1] Nodejs site. http://nodejs.org/. [2] R. A. Bolt. ´’put-that-there´’: Voice and gesture at the graphics interface. In Proceedings of the 7th annual conference on Computer graphics and interactive techniques, SIGGRAPH ’80, pages 262–270, New York, NY, USA, 1980. ACM. [3] J. Y. Han. Multi-touch interaction wall. In ACM SIGGRAPH 2006 Emerging technologies, SIGGRAPH ’06, New York, NY, USA, 2006. ACM. [4] J. Neca and C. Duarte. Evaluation of gestural interaction with and without voice commands. In ˘ S IADIS International Proceedings of IHCI 2011 ˆ aA¸ Conference Interfaces and Human Computer Interaction 2011, Rome, Italy, 2011. [5] A. Neto and C. Duarte. A study on the use of gestures for large displays. In Proceedings of the 11th International Conference on Enterprise Information Systems ICEIS 2009, pages 55–60, 2009. [6] E. Tse, S. Greenberg, and C. Shen. Gsi demo: multiuser gesture/speech interaction over digital tables by wrapping single user applications. In Proceedings of the 8th international conference on Multimodal interfaces, ICMI ’06, pages 76–83, New York, NY, USA, 2006. ACM. [7] J. O. Wobbrock, M. R. Morris, and A. D. Wilson. User-defined gestures for surface computing. In Proceedings of the 27th international conference on Human factors in computing systems, CHI ’09, pages 1083–1092, New York, NY, USA, 2009. ACM.
