Hacking Nintendo Wii to paint virtual graffiti - Semantic Scholar

Hacking Nintendo Wii to paint virtual graffiti A low cost virtual tangible tool for users involvement

Paola Salomoni

Ludovico Antonio Muratori

Department of Computer Science University of Bologna Bologna, Italy [email protected]


Silvia Mirri

Francesco Pozzi


Corso di Laurea in Scienze e Tecnologie Informatiche University of Bologna Bologna, Italy [email protected]

Abstract—Any approach to the issues of virtual reality implies a general question about how to effectively plunge inside it. Despite a great variety of settings, the more feedbacks are the same as real ones, the more users feel involved on a likely situation. We can split such an assumption into the necessity of effective gestural recognition and likelihood of virtual tools to reality. In this paper we will present a system to draw virtual graffiti on the PC screen, as it were a wall. A virtual, but tangible aerosol bomb has been designed and built, by exploiting the Nintendo console gaming device and sensor system. Our object has the same shape and the same way of use as a common, real spray paint dispenser and it can be used on quite every available graphics application on a Desktop or notebook case. Besides the advantages due to the low costs and the good accuracy of gesture recognition, our system confirms the importance of similarity in simulated scenarios. Pervasive computing, interface, virtual tool.

I.

gestural

interface,

multimodal

INTRODUCTION

Users’ immersions on virtual environments are tightly connected to the similarity of simulated objects they use and gestures they can do, as regards to the real ones. High costs of sensors, transducers and computer-vision systems which can effectively detect and classify movements and actions, have somehow limited advance and trials on this field. The recent encroachment of gaming consoles, as pointed out in [1], on PC realm seems to make such costs more affordable and opens new perspectives on approaching issues and implementations for immersion on virtual realities. The same standards which are usual on a desktop case or a notebook, such as hard drives, graphics cards and communication devices are now embedded on the foremost game stations of

the market. In particular, all the power of interfaces, which have been designed to involve players as much as possible, can be exploited on PC platform because of they are compliant with well-known and documented standards. Bug fixing and enhancing (also cracking) which were done with the early, so-called mods by a small group of electronics insiders, have given way to hundreds of homebrew works (i.e. hackings of the consoles systems) by a large community of fans. Besides the producers’ documentation about consoles internals, standard specifications and official (or unofficial) SDK’s allows to explore and re-invent uses and capabilities of gaming devices. Input and feedback consoles systems, together with standard compliant characteristics let designing, building and assessing a wide range of low costs applications linked to gesture recognition. As for technological and scientific research, innovative features of the Nintendo console game pad – the Wiimote – and its input system, have been exploited to build instruments for rehabilitation [2,3], augmented reality [4], and, in general, for gestural interfaces. In this paper we will present a system and a gestural interface to draw virtual graffiti on a personal computer monitor or connected projector. It is made up of a virtual aerosol bomb, which has been built by utilizing the Wiimote device and its sensors system. Our goal has been investigating the potential of such a low cost equipment to realistically simulate human activities and assessing the limits of driving such a device as a PC peripheral. Unlike the original architecture of the Nintendo console, which uses the remote controller to receive red light signals from a sensor bar source and to transmits some kinetics parameters to the system via Bluetooth, our approach inverts the flow of information. Instead of using the remote controller as a

9781-4244-3941-6/09/$25.00 ©2009 IEEE

receiver, we chose to exploit it as a sensor system, recognizing red light emitters from the virtual aerosol bomb. Common graphics applications, such as The Gimp, Photoshop or MSPaint, can be used as virtual environments (the wall, the spray paint store, etc.) to draw. An approach oriented to device driving, which does not depend on the applications has been adopted so as to guarantee the interface portability. Since it is completely hand-in-glove as regard to the real one, the virtual aerosol bomb warrants a full likelihood with real gestures and proved the importance of the above hinted similarity for full immersion on virtual reality. The remainder of this paper is organized as follows: section II summarizes the background and related work about gestural interfaces and hacking approaches to the consoles world, section III detail the design issues of our system, by pointing out hardware and software features. Section IV reports some conclusions and discloses some future work we planned to do. II.

BACKGROUND AND RELATED WORK

Dan Saffer’s book “Designing Gestural Interfaces: Touchscreens and Interactive Devices” [5] has been defined “a testament to the fact that tangible, interactive interfaces that move beyond the mouse and keyboard have become an important part of product design”. [6] Everything in utilizing gestures to computer applications began with the PhD work of Ivan Sutherland [7], where he showed an early form of stroke-based gestures using a light pen to manipulate graphical objects on a tablet display. Since then, stroke-based gesture interactions has become more and more commonly used for text input on personal digital assistants (PDAs), mobile computing, and pen-based devices [8, 9]. Gloves augmented with electronic motion and position sensors were developed to enhance interactions with virtual reality applications [10, 11, 12]. A first (computer) vision-based system that enabled gestures to control the volume and channel functions of a television was presented on 1995 on [13]. Such a work presents the novelty of exploiting perceptual, device-free gestures, but it remains a technique restricted to laboratory studies [14]. Computer Vision (CV for the sake of brevity, from here), as a means to allow more natural gesture interactions with computers, has been an unexploited realm till a short time ago in comparison with direct-input devices such as mice [15], stylus [9, 16], or electronic gloves [17]. Such a shortage or delay is maybe due to the fact that CV wasn’t able to effectively satisfy accuracy constraints [14] and because of its high costs. Consumer electronics for entertainment has reduced such costs with its last generation of gaming consoles. Together with more and more power (in graphics, processing, interfacing with players) nowadays majors’ trends propose equipments more and more compliant to the PC standards [1] and provide CV systems to get players more and more involved on games. Sony Playstation II and III, as well as Microsoft Xbox have been enriched with cameras and systems for a whole-body gaming approach, so as to “shoot the Wii” market killer product by Nintendo (see

at http://news.cnet.com/8301-17938_105-10253586-1.html for details). Interests toward such systems are witnessed by some companies such as, as an example, Softkinetics, which promises the development of a “standard 3D real-time gesture recognition platform for Interactive digital entertainment, by means of a middleware which insulates game developers from the low-level technicalities” (at: http://www.softkinetic.net/Public/). Virtual tools, meant as object which look (in shape and way of use) like the real ones, can be considered as a must on enthralling simulations. Furthermore, they can significantly reduce the computation to recognize any gesture or action. As shown by Fuentes at al. on [18], general customization of the interface freedom-degrees, up to obtain a specific “virtual tool”, implies a lower amount of evaluations and processing to make it effectively work. Catalogues of gaming consoles accessories are filled up with virtual tools: from the guns for simulated shot, to the car or plane console-kits with cloche, steering wheel, pedals and so on. In particular, the Nintendo Wii console has proposed an innovative system utilizing a CV system (extremely simplified) and a quite universally adaptable device – the Wiimote - capable to be transformed into a specialized tool with few custom dap joint parts (at http://www.nintendo.com/wii/what). The Bluetooth standard communication protocol is used to connect the Wiimote and infrared light emitters are recognized and evaluated to provide kinetics parameters (position, speed, acceleration) to a CV system. Simplicity and standardization of Nintendo Wii (and Wiimote) provided a wide community of fans to design and build their own application of it as the so called homebrew hacs (at http://wiibrew.org/ for details). On the research field, Wiimote has been used for computer simulated arm and wrist therapy in stroke survivors in [2], low cost robotics for rehabilitation on [3] and other works exploiting accelerometers and gyroscopes. From the point of view of the CV system, Chung Lee has proposed some applications to use red light emitters for 3D space exploration, augmented reality interaction and object recognition on [4]. Virtual pens, head tracking glasses, and other “virtual tools” are shown as examples to effectively employ the Wii sensor (or CV) system. Lee’s approach points out the plenty of available software and documentation provided by the Wii homebrew community and the extremely low cost of equipments to be used. His basic, killer idea to reverse the system data flow, which the console was set for, has been taken into account here to design and build our virtual aerosol bomb. III.

DESIGN ISSUES

The availability of software tools and documentation about internals has made possible to design and build our tool, by hacking the Nintendo console remote controller. In the following, we will detail the scenario and all the phases of our project.

+Z Pitch

+X

-Y

Yaw +Y -X Roll -Z Figure 1. Wiimote freedom-degrees

A. Nintendo Wii at a glances After one year from its release, on November 2006, the 5th gaming console by Nintendo had sold 20 million pieces, despite it was (objectively) evaluated to be underpowered in comparison with its competitors, such as Sony Playstation and Microsoft Xbox. Nintendo’s market success is due to the innovative characteristics of its creature and, in particular, to the novelty of its console game controller: the Wiimote. It is a wireless handheld device communicating with the console via microwaves through the Bluetooth protocol. Besides some buttons, it contains a vibration engine to simulate trembling feedbacks, a speaker, a 3-axis accelerometer capable to trace inclination, speed and acceleration of the controller, and a high speed infrared camera. On Figure 1 the freedom degrees (x, y and z) and the inclination directions (Pitch, Yaw and Roll) detected by the accelerometer are depicted. The infrared camera (from the +Y side on figure 1) is capable to map up to 4 distinct light sources on a 2D space at a 45-degrees angle. Some infrared led emitters are grouped on a bar (called, with a sort of reversed term, sensor bar) which has to be placed in front of the players (usually over or upon the TV screen) and works as a referring system for the Wiimote camera. B. Wii Homebrewing As Chung Lee wrote on [4], official specifications of Wiimote are unpublished, but the wide community of the so called homebrew hackers has collectively made up for the lack of them. The most part of such an effort is available at http://wiibrew.org, a sort of de facto, open and official source of information about Wii internals. Furthermore, the standard Bluetooth protocol used to connect the Wiimote to the console allows using it with a personal computer without resorting to the development tools provided by Nintendo, which greatly limit their usage from a legal point of view. Software libraries and standalone device-driver applications have been written and are maintained for Microsoft Windows, Apple OS X and Linux platforms, to

connect the remote controller to a PC, so as to recognize and map all the possible signals. In general, they exploit The Bluetooth HID (Human Interface Device) specification and have provided some solutions to make up the partial incompatibility of Wiimote as regards to the standard. While libraries can be used inside some code to drive any specific application, device-driver programs allow to map the Wiimote input/output data for general-purpose, by setting a certain group of on-demand parameters. We chose such a second paradigm to interface our virtual tool in order to guarantee the portability of it as a peripheral for the state of the art applications, instead of writing dedicated code. Specifically, we referred to the GlovePie project by Carl Kenner (see http://carl.kenner.googlepages.com/glovepie for details) which has been written as a customizable devicedriver for the Wiimote (besides the mapping of any other device as a mice or a keyboard). Details about such software and its utilization for our goals will be disclosed in the following of this paper. C. Reversing the flow As for the intention of Wii designers, the CV system of Nintendo console expects the Wiimote to detect infrared light emitters from the sensor bar and then to send their position via Bluetooth to the console for processing, like any other parameters from buttons or from the accelerometer. Moving the Wiimote provides a different point of view for its camera as regards to the emitters on the still sensor bar, so as to trace the user’ displacements. Figure 2 shows the above scheme. The console dataflow system can be easily reversed, by placing and driving (i.e. switching) infrared light emitters as they were mobile parts of a tool and by putting the referring system in charge of the still Wiimote. Figure 3 synthesizes such a reversed approach. This way, the remote controller will identify and detect the displacement of really moving objects and, above all, it will be possible building devices which are essentially made of switch and infrared lights. Our virtual aerosol bomb has been built by exploiting the advantages of reversed data flow. Also a system based on the direct, default approach has been designed, and we will hint about it in the following.

Moving

microwave

Wiimote console infrared Still Sensor Bar Figure 2. The original flow of data on Nintendo Wii

Moving tool

infrared Figure 4. The first prototype of virtual aerosol bomb.

console Still Wiimote

microwave

Figure 3. The reversed approach to data flow on Nintendo Wii.

D. Hardware The Wiimote controller is able to map up to 4 distinct light-point on a 2D space. It means that the appearance and disappearance of some of them can be used to detect some kind of binary action (on/off) and the others can be in charge of tracing the object position. Furthermore, given an initial setting, i.e. a default distance, a couple of light point emitters can be utilized to trace the actual closeness or remoteness from the still Wiimote. All this features have been taken into account to design and build our virtual aerosol bomb. A very first prototype has been carried out by placing three infrared light led emitters on a box. Figure 4 shows the result: the external blue bulbs are the light points for mapping the device position and distance and they are switched on by the button on the left, while the central one is switched by the button on the right so as to simulate the sprinkling. Some experiments with this prototype have shown that single leds are not effectively detected because of their low emission of light and the presence of mutual interferences. Due to the goal of producing a tool which really looks like the real one, in shape and way of use, a common spray paint dispenser has been chosen as the cover or shell for it. Three holes on the cylindrical surface of the spray can have been filled with groups of infrared light emitters. A button to switch on the external groups of lights has been put on the bottom, while another switch has been connected to the actuator at the top of the can. Figure 5A shows one of the construction phases: as we can see, each light-point is made of 9 leds instead of 1 at the external holes and of 5 leds in the central hole, so as to provide enough, distinguishable infrared emission. Figure 5B shows the final result. A very simple circuit has been assembled to control the lights. Figure 6 reports its schema: (b) is the switch for the; (d) and (f) are the groups of lights; (c) is under the can actuator to switch on the (e) group of lights and, finally, (a) is the group of batteries powering the system.

Figure 5. The can under construction (A) and the final result (B).

All the functionalities of our virtual aerosol bomb are provided by processing signals coming from infrared lights. In order to exploit the buttons placed on the Wiimote, a direct approach and a different virtual tool has been build. It consists of a box which contains the remote controller with a mirror at its top. As shown on Figure 6, lights from a still sensor bar are deflected on the camera and one of the Wiimote buttons is used to simulate the sprinkling. Such a solution has been discarded because of the resulting object is not so hand-in-glove as the above one we described. E. Software The final goal of our project is utilizing the virtual aerosol bomb to draw virtual graffiti on a PC screen or a projector connected to a desktop or notebook case. In particular, this activity will have to be possible with widespread graphics applications such as MsPaint, Photoshop, TheGimp, which let the user choose to freehand sprinkling. Based on our approach, driving our virtual tool is equivalent to processing the signals coming from the infrared camera on the (still) Wiimote and re-transmitted to the PC via Bluetooth. The parsing and elaborating process of such a data flow can be done by means of libraries linked to the applications which use the Wiimote. Wiiuse by Michael Laforest, (on http://www.wiiuse.net/) is a notable instance of

such software tools, available to the wide community of Wii homebrewers. Nevertheless, this paradigm of implementation implies a dedicated use of the device, strictly connected to the application which has been written for, and so it is impossible or very complex to use it for our goals. In order to surmount this limit, it is necessary to place a suitable set of controls at the device-driver layer as it happens for mouse, keyboard, and all the common PC peripherals. A satisfactory solution for this approach is GlovePie by Karl Kenner. It is a sort of meta-device driver, letting to detect and associate data coming from a wide variety of devices (mice, keyboards, pads and the Wiimote). In particular, GlovePie is a standalone application, overriding the device drivers of the operative system (only MsWindows at the time the authors write). Once launched, it captures events coming from peripherals and can process them, or associate such events to others. All the rules and the processing functions are taken from a script, written in a c-like syntax, which users can modify or create. Global or local variable can be created to set up suitable parameters, as well as conditions or loops can be stated on the strength of received events and so on. GlovePie scripts are to be conceived as event-driven programs to response any peripheral change of state. This software is capable to associate events coming from the Wiimote so that they simulate other events from an alreadydriven peripheral like mice or keyboards. Since the latter ones are typically used to draw with the above mentioned graphics application, such features meet our issues. For our purposes, we chose to associate events coming from the Wiimote via Bluetooth, to the mouse movements and to some couples of keys which are typically used on graphics application to change the size of brushes (and aerographs). By exploiting the features of GlovePie, we firstly defined some global variables to detect the screen resolution and the actual state of the detected light sources. Such an operation is necessary to compute the position of the virtual aerosol bomb on the screen and to iteratively control the lights. The midpoint position and relative distance of the two external groups of leds are used to set the actual position of the spray can and the distance (i.e. the size) of it from the screen whenever the sprinkling is done. The central group of lights is used to simulate the sprinkling, once they are switched on by pressing the actuator. var.ScreenW = Screen.DesktopWidth; var.ScreenH = Screen.DesktopHeigth; … if wiimote.dot1vis then var.detect=1 end if if wiimote.dot2vis then var.detect=2 end if if wiimote.dot3vis then var.detect=3 end if Code 1. Resolution parameters setting and lights switch control code fragment.

if (var. detect ==2) then var.CursoreX=(1024((Wiimote1.Dot1x+Wiimote1.Dot2x)/2))/1024*var.screenW var.CursoreY=(var.screenH-(((Wiimote1.Dot1y +Wiimote1.Dot2y)/2)/768*var.screenH)) mouse.x = var.CursoreX/var.screenW mouse.y = var.CursoreY/var.screenH end if if (var.rilevati!=3) then mouse.LeftButton = false end if Code 2. Midpoint and mouse movement association.

Some chunks of script code about all this operations are summarized in Code 1: the first two statements assign the resolution parameters (known in GlovePie) to the global variables; the if…then…else statement controls the number of lights switched on, referring to the dot1vis and dot2vis as the ones tracing the spray can position and to the dot3vis as the sprinkling simulator. As the Code 2 shows, the computed midpoint is associated to the mouse movement (first if…then statement) and the lighting of the central group of leds to the right-click (second if…then statement). In this Code, Wiimote1.Dot1x and Wiimote1.Dot1y are the coordinates of the first group of leds on the can GlovePie is able to automatically detect, while Wiimote1.Dot2x and Wiimote2.Dot1y are the coordinates of the second one. Any change of the relative distance between the two external lights groups can be associated to the shortcuts for changing the brushes size. In particular, it has been tested with the couple “alt and +/-” as in TheGimp, and the couple “shift and ]/[ ” as in Adobe Photoshop. F. System usage and limitations Once the switch on its bottom has been turned on and GlovePie has been launched with our script, the virtual aerosol bomb can be used to move the cursor (sprinkling) on any image canvas inside a graphics application. By keeping pressed the actuator, users can trace their spray paint on the screen. The virtual tool has been tested with some widespread graphics applications: first of all with MsPaint, which is released together with Microsoft Windows. Due to the lack of shortcuts or other direct ways to change the sprinkling size, simulation of drawing is severely limited. By utilizing Adobe Photoshop, after setting up the associated shortcuts (which are “[“ and “]”) to the relative distance between external lights, the system is effectively usable. Also with TheGimp, where once we have set up the keys for simulate closeness and remoteness, actions appear to be likely to reality. Despite our efforts to empower the light emission and adjust the Wiimote sensibility (lowering the threshold of detection for each single point), wide angle variations as regards to the Wiimote, or interferences coming from external sources of light, condition the system. The similarity

of our object to the real one has involved users much more than in using mice or graphics tables to spray draw, even if users are forced to keep the can to the Wiimote and this hampers the natural movement. This constrain is due to the change of distance between external light, which is used to detect shortcuts in order to adapt the brushes size. IV.

CONCLUSIONS AND FUTURE WORK

Our system has been designed and built to provide users with a tangible object, hand-in-glove as regards to the real one, to simulate a common activity like graffiti drawing. Low cost technologies coming from consumer electronics for entertainment made such a work possible, together with the availability of software tools and documentation by a wide community. Despite its simplicity the Nintendo console remote controller has shown to be an appreciable tool for effective assessments. Users which have tested the prototype system have confirmed the importance of likelihood of virtual tools to reality for a much more involving experience. At the time we are writing we are conducting usability tests with users (through cognitive walkthrough and questionnaires). They will provide data about better involvement with realistic tools, performance, accuracy and precision. Other experiments have been made and are in progress: from a virtual xylophone, to a glove for haptic interfacing.

[5] [6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15]

REFERENCES [1] [2]

[3]

[4]

Voth, D. “Evolution in gaming” Pervasive Computing, IEEE Volume 6, Issue 2 pp. 7-10. April-June 2007. Leder, R.S. et al. “Nintendo Wii remote for computer simulated arm and wrist therapy in stroke survivors with upper extremity hemipariesis”, in Proceedings of Virtual Rehabilitation, 2008 pp. 7474, 25-27 August 2008. Spencer, S. J., et al “A Low Cost Parallel Robot and Trajectory Optimization Method for Wrist and Forearm Rehabilitation using the Wii”, in Proceedings of Biomedical Robotics and Biomechatronics, 2008. BioRob 2008. 2nd IEEE RAS & EMBS International Conference pp. 869-874. 19-22 Oct. 2008. Lee, J.C. “Hacking the Nintendo Wii Remote” Pervasive Computing, IEEE Volume 7, Issue 3, pp. 39 – 45 July-Sept. 2008.

[16]

[17]

[18]

Saffer D., Designing Gestural Interfaces Touchscreens and Interactive Devices (2008) O’reilly ISBN 10:0-596-51839-0. Diana C., “A Kiss Is Just a Kiss; A Sigh Is Just a Deselection: A Review of Designing Gestural Interfaces”. Interactions Volume 16, Issue 1 pp. 45-47 January-February 2009. Sutherland, I. E. “Sketchpad: A man-machine graphical communication system” in Proceedings of the AFIPS Spring Joint Computer Conference pp. 329–346. 1963. Buxton, W., Fiume, E., Hill, R., Lee, A. & Woo, C. “Continuous hand-gesture driven input” in Proceedings of Graphics Interface ’83, 9th Conference of the Canadian Man-Computer Communications Society. pp. 191–195. 1983. Cohen, P. R., et al. “Quickset: multimodal interaction for distributed applications” in Proceedings of the fifth ACM international conference on Multimedia. ACM Press, pp. 31–40. 1997. Sturman, D. J., Zeltzer, D., Pieper, S. “Hands-on interaction with virtual environments” in Proceedings of the 2nd annual ACM SIGGRAPH symposium on User interface software and technology. ACM Press pp. 19–24. 1989. Wexelblat, A. “An approach to natural gesture in virtual environments” ACM Trans. Comput.-Hum. Interact. 2(3) pp. 179– 200. 1995. Quek, F. K. H. “Toward a vision-based hand gesture interface” in Proceedings of the conference on Virtual reality software and technology. World Scientific Publishing Co., Inc., pp. 17–31. 1994. Freeman, W., Weissman, C. D., “Television control by hand gestures” Tech. rep., IEEE Intl. Wkshp. on Automatic Face and Gesture Recognition. 1995. Karam, M. PhD Thesis: “A framework for research and design of gesture-based human-computer interactions. PhD thesis, University of Southampton 2006. Moyle, M. & Cockburn, A. “Gesture navigation: an alternative ’back’ for the future” in Proceedings of CHI ’02: CHI ’02 extended abstracts on Human factors in computing systems. ACM Press, New York, NY, USA, pp. 822–823. 2002. Ou, J., Fussell, S. R., Chen, X., Setlock, L. D., Yang, J. “Gestural communication over video stream: supporting multimodal interaction for remote collaborative physical tasks” in Proceedings of the 5th international conference on Multimodal interfaces. ACM Press, pp. 242–249. 2003. Goza, S. M., Ambrose, R. O., Diftler, M. A., Spain, I. M. “Telepresence control of the nasa/darpa robonaut on a mobility platform” in Proceedings of the 2004 conference on Human factors in computing systems. ACM Press, pp. 623–629. 2004. Fuentes, O. Nelson, R.C. “The virtual tool approach to dextrous telemanipulation”, Robotics and Automation, 1996. Proceedings., 1996 IEEE International Conference, Publication Date: 22-28 April 1996, Volume: 2 On page(s): 1700 - 1705 vol.2.