2012 International Conference on Virtual Reality and Visualization
Enhancing Touch Screen Games Through a Cable-driven Force Feedback Device Zhu Hou1 Yuru Zhang1 Yi Yang1 1 State Key Laboratory of Virtual Reality Technology and Systems, Beihang University 37 Xueyuan Road, 100191 Beijing, China
[email protected] [email protected] [email protected]
Despite of the advantages of nature interaction on touch screens, touch screens games also inherit their disadvantages. When users touch on a flat touch screen, they feel no differences over the slippery glass. The haptic feedback that accompanied with physical interactions is missing. Users have to rely solely on their vision to interact with the touch screen. The lack of haptic feedback on touch screens causes more errors and user frustration as compared with their physical counterparts [1, 2]. To solve this problem, researchers have investigated several technologies to add tactile feedback to touch screens [3, 4]. Although these technologies are promising in improving users’ performance in certain tasks, little has been done to enhance interactive games that take a significant part of touch screen applications.
Abstract An increasing number of touch screen interfaces have promoted plenty of applications. A great part of the applications is game. However, the lack of haptic feedback on touch screens also degrades the enjoyment of touch screen games. We present the design of a new cable-driven force feedback touch screen to enhance touch interactions especially in games. The implementation and principle of the force feedback touch screen are presented. We evaluate our design in a Mole Attack game. A lateral force was applied when the user tapped on a mole. The finger acceleration was measured when tapping on the force feedback touch screen as well as stroking on a physical button. The magnitude of the force feedback was adjusted accordingly to simulate the physical interaction. Players reported that the force feedback enhanced tapping was similar to that on a physical button and they preferred the haptic enhanced game which brought them more fun. Keywords: touch screen, game, force feedback
1. Introduction Touch screens are prevalent user interfaces due to its intuitiveness and directness for interacting with digital information. The economic success of typical touch screen devices such as the Ipad and Iphone makes touch screens to be the majority configurations in modern consumer electronics. Among thousands of touch screen applications, a significant part of the applications is game. Typical touch screen gestures such as tapping and dragging are applied to catch fish or cutting fruits in touch screen games. These intuitive gestures allow part of users’ body (fingers) to be involved in gaming thus making the games more interactive and attractive. Moreover, gesture-based touch screen games are controller-free. They do not need extra physical controllers which would take away part of the touch screen real estate. 978-0-7695-4836-4/12 $26.00 © 2012 IEEE DOI 10.1109/ICVRV.2012.14
Figure 1. The force feedback touch screen and the Mole Attack game
The “Mole Attack” used to be a classic game on tabletop game console. In the game, the player whacks the moles with a rubber hammer. When this game is developed for touch screen, the hammer is replayed by a player’s finger. The player taps directly on the mole to score. However, the inherent force feedback when whacking the mole is not presented. Moreover, the player also fails to tell if she or he has hit the mole or the 56
background. As a result, the excitement and fun obtained when operator whacks the moles with a hammer is significantly deteriorated. Our objective is to add force feedback to touch screen games in order to make them more engaging. Compared with tactile feedback, force feedback has wider magnitude bandwidth and can be used to deliver more realistic feedback in games. To realize force feedback in touch screen games, we apply a planar cable-driven force feedback device in the application. Compared with linkdriven force feedback device, cable-driven force feedback device has advantages of fast reaction speed, low inertia, and large workspace [5]. A typical series of cable-driven haptic devices were developed by Sato and his colleagues, named the SPAIDAR series [6]. The SPIDAR haptic devices were developed for interaction with virtual objects in a 3D space, including 3D exploration [7], grasp [8] and bimanual manipulation [9]. In our touch screen applications, however, we only need to realize force feedback in planar interactions. Moreover, we also intend to simplify the design for portable touch screen devices. Therefore, we applied a planar cable-driven device which used four cables to realize 3 Degree-Of-Freedom (DOF) force feedback. In this paper, we present the design and implementation of this new force feedback touch screen. We demonstrate its application in a Mole Attack game, as shown in Fig 1. The paper is organized as follows. Section two presents system design, describing the equipment configuration in detail. Section three presents the kinematics analysis, and statics analysis is presented in section four. Section five introduces an application of the device and section six is our conclusion and future work.
receivers is “off”, it presents that the interface is in contact with the touch screen. A contact signal is sent to the computer. The infrared frame can also detect the position of the input interface by checking which receiver was off. However, the resolution of the position detection depends on the sizes of the emitter and the receiver. The infrared frame we use only has a resolution of 2 mm which is not sufficient for precise finger position detection. As a consequence, we only use the infrared frame to detect contact and use the force feedback device to track the interface’s position.
Figure 2. Schematic diagram of the infrared frame
2. System design The force feedback touch screen consists of three parts: a LCD monitor, an infrared frame and a cable-driven force feedback device. In this section, we demonstrate how each part works and explain the principle of this touch screen.
2.1. Hardware A touch screen can detect the contact status and the contact position when an input interface (a finger or a stylus) touches the screen. We apply an infrared frame to detect the contact status so that it is compatible with all types of interfaces. As a result, we can change the input interface according to the requirement of games. The infrared frame combines two groups of infrared emitters and receivers, as shown in Fig.2. When the input interface (e.g. a finger) blocks the infrared ray, the receiver shifts its status from “on” to “off”. The receivers’ status are updated every ten milliseconds. If any of the
Figure 3. Actuation module
The cable-driven force feedback device is the core part of the system. The cable-driven force feedback device is chosen for its transparent workspace which is critical for touch screen applications. The activation module is composed of a motor, an encoder and a pulley. One side
57
of the cable is connected with the ring, while the other side of the cable wounded on the pulley. The encoder tracks the change of the cable length while the motor controls the out torque of the pulley. The activation module can be re-assembly to recreate the size and shape of workspace to satisfy different applications and screen sizes. Figure 4 presents the system configuration diagram. The information and signal flow are presented.
In Fig 5, the red point represents the mole which is displayed on the screen. The green point is the corresponding model in the virtual environment. Assuming that the distance from eye to screen is 700mm and height is 250mm, the transform equation is deduced as y0' y0 (1) 250 y0 700 x0 x0'
y0 700 700
(2)
Where x0' and y0' is the click position on the screen, and can be calculated by the kinematic of the force feedback device (present in section 3), and ( x0 , y0 ) is the mole’s position in virtual environment. As soon as the mole is successfully hit, a force feedback is provided to the player’s finger. The force feedback control thread communicates with motor controllers at 1000 Hz. This fast thread prevents the device from vibrating, ensures that the output-force is smooth and responded quickly. In the following sections, we present the kinematics and force control of the device.
Figure 4. The system configuration diagram
2.2. Software All of the force feedback touch screen’s software is developed in C++. The software mainly serves to display the virtual environment and to control the force feedback. When the screen scanning frequency is greater than 24 Hz, an image is smooth for our eyes. Therefore, we set the display frequency at 30 Hz at the first step: not only to reduce image latency to improve fluency, but also to save computer resource. The second, we uses the Direct3D to display a three-dimensional virtual environment. The virtual environment is developed based on a horizontal plane while the computer screen is vertical. A mapping relationship from screen to virtual environment should be established, as shown in Fig. 5.
3. Kinematics Planar haptic interfaces are attracting the interest of many researchers [10-19]. In our application, we intended to add force feedback when a player interacts on the touch screen. The position of the finger is tracked by the force feedback device through its kinematics. We applied a comparatively simple configuration to simplify kinematics analysis and avoid singularities. Figure 6 presents the configuration.
T2
T3
p1 ( x1 , y1 ) p2 ( x2 , y2 )
x0'
b
y0'
L2
L3
ring ( x, y) M
L1
L0
x0
y
y0
T1
T0
x
a
Figure 6. Planar haptic interface Figure 5. Mapping relationship
58
According to the matrix operations,
We use the forward kinematics to deduce the position of the ring. Firstly, we calibrates the initial location of the rings with a ball which has identical diameters showed on the screen, obtains the initial cable lengths. Secondly, after the ring is moved, we calculate the current cables length L0 , L1 , L2 , L3 with encoders’ reading. After we
f = A+ F
where A + is the Moore-Penrose pseudo inverse of A . The result of f is the force required to control the torque of motors.
obtain the cables length, the third step can start with the parameters to calculate the location of the ring. L2 + a 2 - L12 T 0 arccos 0 (3) 2aL0
T2
arccos
L22 + a 2 - L23 2aL2
5. System evaluation The force feedback generated to the finger is to simulate a key-click effect. The lateral force yielded on the finger resulted in an accelerated movement along the force direction. Generating accelerated movement on the finger has been shown to be an effective method to simulate key-click feedback on touch screens [20, 21]. To compare virtual click with physical click, we measure the acceleration of different operation modes as a standard to judge the fidelity of the simulation. We use the comparative result of virtual key-click and a click on physical keyboard to evaluate the performance of the force feedback touch screen. We used a digital accelerometer ADXL345 which was connected to a MCU via I2C bus to measure the click acceleration, and transferred the data through a COM port from MCU to the computer. In the experiment, the accelerometer was worn on the operator’s click-finger and followed the finger’s movement. The acceleration value was transferred to the computer via the MCU in real time. After the experiment, data were plotted as a curve and analyzed. The acceleration signal was collected in the click process on the physical keyboard. The result is presented in Fig 7.
(4)
p1 is the upper point connecting with L2 and L3 , and p2 is the nether point connecting with L0 and L1 . x1 = a L2 cos T 2 , y1 = b L2 sin T 2 (5) x2 = L0 cos T 0 , y2 = L0 sin T 0 (6) y1 y2 x1 x2 , y= (7) x= 2 2 y y M atan( 1 2 ) (8) x1 x2 where a is the bottom side length of the workspace, and M is the torsion angle of the ring, which will be used for torque calculation. Thus, the locations of finger ring have been calculated.
4. Statics The other important parameter of the planar haptic interfaces is force feedback: when a player hits the mole, the device provides a force feedback at once to simulate the click force as in the physical world (a vertical upward force in the experiment). The force feedback is generated by controlling the cable tension. The resultant of four cable tension yielded 3-DOF force feedback ( Fx , Fy , TR )
Acceleration of physical keyboard 0.8 Acceleration 0.6
¦f
Acceleration(g)
in the screen plane. According to the static equilibrium, the force and torque exerted by the environment should be balanced by the resultant force and torque of the cables. The statics equations are: 3
3
i
¦t
= FR
i=0
i
= TR
(11)
(9)
0.4
0.2
0.0
i =0
The static Jacobian equation is deduced from equation (9) F = Af
-0.2
-0.4
(10)
0
200
400
600
800
1000
1200 1400 1600
1800 2000
2200
Time(ms)
Where F = ª¬Fx
Fy
Figure 7. Acceleration of physical keyboard
T
TR º¼ is the wrench of the end
effector, A is the static 3 u 4 Jacobian matrix, and the cable tension vector is f = >f0 f1 f 2 f 3 @T .
The acceleration in the overall process was relative flat, and the only abrupt change existed when the finger was
59
moving to the bottom of the keyboard. The abrupt change was generated due to an acceleration whose direction was contrary to the direction of movement to slow down and eventually stopped the motion. When the finger stopped moving, the acceleration then decreased to zero. Consequently, we can see a peak on the map followed by an increase and a decline in the acceleration curve. Figure 8 presents the acceleration curve collected from the click process on the touch screen enhanced by force feedback.
6. Conclusion and discussion We presented the design and preliminary evaluation of a new force feedback touch screen. The objective of this research was to enhance touch screen games with haptic feedback. The haptic feedback was provided by a planar 3-DOF force feedback device. We demonstrated its application in a Mole Attack game. Lateral force feedback was triggered when the player tapped on a mole successfully. The finger acceleration was measured when the lateral force feedback was implemented. The acceleration was then compared with that in a key stroke on a physical button. With these measurements, we could adjust the magnitude of the force feedback to emulate a physical interaction in real world. Preliminary results showed that players preferred the force feedback in the game. In this paper, we only demonstrated the force feedback in a tapping interaction on a touch screen. The force feedback could also be applied to enhance dragging-based interaction as well. These haptic enhanced games can be used to hand rehabilitation or education domain in order to make trainings more interesting. Several new haptic enhanced games are underdeveloped and more thorough evaluation will be conducted in the future to further evaluate the system.
Acceleration and force of virtual keyboard 1.2
3.5 Acceleration Force
1.0
3.0 2.5 2.0
0.6 1.5 0.4
Force(N)
Acceleration(g)
0.8
1.0 0.2 0.5 0.0
0.0
-0.2 0
500
1000
1500
2000
2500
Time(ms)
Figure 8. Acceleration of virtual keyboard
Acknowledgment
The information in this figure indicates that the acceleration curve is consistent with the process of the physical button click, and the magnitude of the peak acceleration is the only difference. This is due to the fact that we set a greater force feedback than that on the physical button. As a result, this force made the upward acceleration greater so that the deceleration process became faster. Consistent with the physical key-click, we adjusted the value of the force feedback, and the result compared with physical experiment was improved accordingly. At the same time, we also asked three operators to evaluate the experience when click on the physical keyboard and on the force feedback enhanced touch screen. All of the operators considered that the virtual button-click was very similar to the physical button. However, the force feedback we applied to the key-click was a step function according to the click status – on or off. There was no corresponding force varied with the compressing depth as in the physical key. As a result, the force rendering of the virtual button should be further improved. Eventually, this force feedback device was applied to the Mole Attack game: when the player tapped on the target mole, he obtained a vertical upward force resulted in an experience as if whacking on the mole with a hammer. This effect made players felt immersive and entertaining.
This work is supported by the National Natural Science Foundation of China under grant No. 61190120 and No. 61190125. Their support is greatly appreciated.
References [1] B. Banter. "Touch screens and touch surfaces are enriched by haptic force-feedback", Information Display, vol. 26, 2010, pp. 26-30. [2] S. Brewster, F. Chohan and L. Brown, "Tactile feedback for mobile interactions", San Jose, CA, United states, 2007, pp. 159-162. [3] M. Levin and A. Woo, "Tactile-feedback solutions for an enhanced user experience", Information Display, vol. 25, 2009, pp. 18-21. [4] K. Dong-Soo, Y. Tae-Heon and C. JoonYeon, "Trend & prospects of haptic technology in mobile devices", in Industrial Electronics (ISIE), 2010 IEEE International Symposium on, 2010, pp. 3778-3783. [5] Kim, Ishii, Koike, and Sato, "Development of tension based haptic interface and possibility of its application to
60
[19] D. Mayhew, et al, “Development of the macarm: a novel cable robot for upper limb neurorehabilitation”, in Proc. RR'05, 2005, p229.
virtual reality", The ACM symposium on Virtual reality software and technology, 2000, pp. 199-205. [6] Sato, M., "SPIDAR and virtual reality", Automation Congress, 2002 Proceedings of the 5th Biannual World, 2002, pp. 17-23.
[20] "Touch sense Tactile Immersion Corporation".
[7] Hirata, Y., and Sato, M., "3-dimensional Interface Device for Virtual Work Space", Intelligent Robots and Systems, 1992, pp. 889-896.
Feedback
Overview,
[21] K. Tashiro, Y. Shiokawa, T. Aono, and T. Maeno, "A Virtual Button with Tactile Feedback Using Ultrasonic Vibration", Virtual and Mixed Reality, vol. 5622, 2009, pp. 385-393.
[8] Kim, S., Hasegawa, S., Koike, Y., and Sato, M., "Tension based 7-DOF force feedback device: SPIDARG", Virtual Reality, 2002, pp. 283-284. [9] Walairacht, S., Yamada, K., Hasegawa, S., Koike, Y., and Sato, M., "4 + 4 fingers manipulating virtual objects in mixed-reality environment", Virtual Environment, Vol.11, No.2, 2002, pp. 134-143. [10] C. Bonivento, et al, “WireMan: A portable wire manipulator for touch-rendering of bas-relief virtual surfaces”, in Proc. ICAR'97, 1997, p13. [11] R.L Williams II, “Cable-suspended haptic interface”, INTJVR, 1998, pp.13–21. [12] Gallina.P, Rossi,A, “Planar Cable-Direct-Driven Robots, Part II: Dynamics and Control”, in Proc.DETC'01, 2001, DAC-21146. [13] R.L. Williams II, P. Gallina, “Planar Cable-DirectDriven Robots, Part I: Kinematics and Statics”, in Proc.DETC'01, 2001, DAC-21145. [14] P.Gallina, G. Rosati and A. Rossi, “3-d.o.f.Wire Driven Planar Haptic Interface”, J INTELL ROBOT SYST, 2001, 32: pp.23–36. [15] P. Gallina, G. Rosati, “Manipulability of a planar wire driven haptic device”, MECH MACH THEORY, 37, 2002, pp.215-228. [16] R.L. Williams..II, P. Gallina, “Planar Cable-DirectDriven Robots: Design for Wrench Exertion”, J INTELL ROBOT SYST, 2002, 35: 203–219. [17] R.L.Williams II, P. Gallina, J. Vadia, “Planar Translational Cable-Direct-Driven Robots”, J ROBOTIC SYST, Vol.20, No.3, 2003, pp.107-120. [18] V. Richard, "Doctoral thesis: Analysis of the Workspace of Tendon-based Stewart Platforms", 2004.
61
