MagiThings: Gestural Interaction with Mobile Devices Based on Using Embedded Compass (Magnetic Field) Sensor Hamed Ketabdar Telekom Innovation Labs, TU Berlin Deutsche Telekom Laboratories Ernst-Reuter-Platz 7 10587 Berlin
[email protected]
Amin Haji-Abolhassani Centre for Intelligent Machines McGill University 3480 University Street H3A 2A7 Montreal (QC) McConnell Engineering Building
[email protected]
Mehran Roshandel Telekom Innovation Labs Deutsche Telekom Laboratories Ernst-Reuter-Platz 7 10587 Berlin
[email protected]
ABSTRACT The theory of around device interaction (ADI) has recently gained a lot of attention in the field of human computer interaction (HCI). As an alternative to the classic data entry methods, such as keypads and touch screens interaction, ADI proposes a touchless user interface that extends beyond the peripheral area of a device. In this paper, we propose a new approach for around mobile device interaction based on magnetic field. Our new approach, which we call it “MagiThings”, takes the advantage of digital compass (a magnetometer) embedded in new generation of mobile devices such as Apple’s iPhone 3GS/4G, and Google’s Nexus. The user movements of a properly shaped magnet around the device deform the original magnetic field. The magnet is taken or worn around the fingers. The changes made in the magnetic field pattern around the device constitute a new way of interacting with the device. Thus, the magnetic field encompassing the device plays the role of a communication channel and encodes the hand/finger movement patterns into temporal changes sensed by the compass sensor. The mobile device samples momentary status of the field. The field changes, caused by hand (finger) gesture, is used as a basis for sending interaction commands to the device. The pattern of change is matched against pre-recorded templates or trained models to recognize a gesture. The proposed methodology has been successfully tested for a variety of applications such as interaction with user interface of a mobile device, character (digit) entry, user authentication, gaming, and touchless mobile music synthesis. The experimental results show high accuracy in recognizing simple or complex gestures in a wide range of applications. The proposed method provides a practical and simple framework for touchless interaction with mobile devices relying only on an internally embedded sensor and a magnet.
Keywords: Around Device Interaction, Mobile and Tangible Devices, Embedded Compass (Magnetic) Sensor, Gestural Interaction, Touchless Data and Command Entry.
1.
INTRODUCTION
Compass, a human made navigational tool, has been widely employed to facilitate the navigation difficulties in the past centuries. An ordinary compass, by itself, is nothing more than a magnetized needle that pivots on an axis and tends to stay aligned with the earth’s north-south magnetic field. Recent developments in electronics, has introduced compact, cheaper and high-
Figure 1: Gestural interaction with a mobile device by a magnet taken (or worn) around a finger, based on using embedded compass sensor. performing electronic devices such as magnetometer, gyroscope, and accelerometer. In recent years, digital compass, along with other genre of sensors such as GPS, accelerometer and dual camera have been embedded within the cell phones to enhance the functionalities of the phone. Digital Compass along with GPS has been used to provide navigation to the user. We however show that the usability of the digital compass can be extended beyond navigational applications, providing a new user interaction approach with mobile devices (Ketabdar et al., 2010b). The electronic magnetic sensor in a mobile device acts like a regular compass. Any slight displacement of the device with respect to the earth’s magnetic field is sensed and registered by the device. A similar type of influence can be imposed upon the magnetic field of the sensor if we slide a permanent magnet around the device. Specifically, a small magnet that moves around the device affects the magnetic field around the sensor, and therefore generates a temporal pattern which changes along the x, y and z axes depending on the movement of the magnet. This pattern can be used to establish a touchless interaction framework as a mean of interaction between the user and the device (Figure 1). In other words, the user generates a specific gesture while moving the magnet, which creates a temporal pattern of change in the magnetic field sensed by the compass sensor. This pattern can then be compared against the pre-recorded templates or pre-
trained models in order to recognize the gesture and interpret it as a command. This touchless input method addresses some of the limitations commonly associated with traditional input methods such as keypads or touch screens interaction. One of the main restrictions in designing miniature electronic devices is the size of the user input interface that needs to be large enough to comply with the human physical characteristics. A small properly shaped magnet, e.g. in shape of a rod, ring or pen though, can freely move in the 3D space around the device which is considerably broader than the surface of hand held device screen. By properly shaped magnet, we mean a magnetic material that can be taken or worn in finger comfortably and naturally. This can make design of very small handheld devices yet with a proper user interface mechanism feasible. Moreover, the 3D characteristic of the proposed method opens new door for augmented and virtual reality applications on mobile phones. Additionally, since the magnetic field can penetrate through occluding objects, it allows for interactions, even though the device is concealed by other objects, or while the device is inside the user pocket or handbag. For instance, the user may be able to dial a number, enter a pin code, or select a music album without taking the mobile device out of his pocket/bag. Additionally, by the same reason, space at back of the device can be also freely used for interaction (Figure 2). This is in contrary with touch screens where interactions are only possible when the device is in a direct contact with the user. The compass (magnetic sensor) is a cheap, small sensor which can be internally embedded in the hardware. Acquiring such utility does not impose any change in the physical specifications of a device which is a notable advantage for small mobile devices. Replacing keypads or touch screens with such data entry technique in small devices allows saving cost and reduces complexity in design. Compared to touch screen, magnetic sensor can be much simpler, smaller and cheaper, and can be internally embedded inside the device. The proposed methodology (Ketabdar et al., 2010b) can be applied using multiple magnets allowing for concurrent multigesture or multi-user interaction with a mobile device. If the magnets used come with different shape or polarity, their influence in the magnetic field can be potentially separated.
Figure 2. Back of device interaction based on magnetic interaction framework. The magnetic field can pass through many covering fabrics, allowing interaction even when the hand is not in the line of sight of the device (Ketabdar et al., 2010b).
The proposed method opens up a variety of possibilities for touchless interaction with mobile devices in different context and applications. In this paper, we review a few of these applications so far developed within the framework of our research. As it is shown in the next sections, the proposed approach can be used as a mean for gesture based interaction with user interface of a mobile device (Figure 3). This can be for instance turning pages in an e-book or a photo gallery, zooming, answering or rejecting a call, etc. All this is done by simple gestures in the space around the device. We also show that the method is precise enough to be used for touchless text entry. This is done by drawing a character (digit) shape gesture in the space around (in front) of the device. Moreover, we also introduce a new concept in mobile security which is called “Magnetic Signatures”. The user simply signs using a magnet in the 3D space around the device, and this is used as a basis for being identified or authenticated by the device. Finally, we also talk about using the method for mobile entertainment, including gaming and music synthesis on mobile devices. We show that the method can provide a new way for playing different music instruments in a touchless manner. For instance, we explain implementation of an AirGuitar which is a guitar that can be triggered in air. The proposed touchless interaction method can also be useful in assistive technologies. The fact that such data entry approach does not entail user’s visual recourses makes it a pragmatic communication solution. This can be important for visually impaired people, interaction in a vehicle, and interaction in darkness. Regular gesture based recognition techniques based on computer vision methods cannot perform in darkness. The proposed method can also be suitable in scenarios where a direct touch is not favorable, such as entrance doors in public places, industrial and scientific experiments, etc. In the next sections, we first provide further details on theoretical aspects of the interaction based on magnetic field and compass sensor. We also compare the proposed method with the state of the art ADI methods. Different modeling and recognition approaches used for identifying gestures are presented. We continue with a review of our studies and implementations for the use of the gesture interaction method in different context and applications. We show that the proposed interaction framework
Figure 3. Interaction with user interface of a mobile phone using space around the phone based on change in magnetic field (Ketabdar et al., 2010b).
can be efficiently employed in different applications. The paper ends with some discussion on other potential applications and variants of the technology.
2. MAGNETIC INTERACTION METHODOLOGY Around Device Interaction (ADI) has been recently investigated as an efficient interaction method for mobile and tangible devices. ADI techniques are based on using different sensory inputs such as camera (Starner et al., 2000), infrared distance sensors (Kratz and Rohs, 2009, Hinckley et al., 2000, Butler et al., 2008 and Howard and Howard), touch screen at the back of device (Baudisch and Chu, 2009), proximity sensor (Metzger et al., 2004), magnetic field (Harrison and Hudson, 2009), electric field sensing (Theremin and Petrishev, 1996 and Smith et al., 1998), etc.
Figure 5. The external magnet can be taken in the hand or worn around a finger.
In this paper, we are mainly concerned about ADI with a mobile phone. For mobile phones, mainly optical techniques (camera or infrared sensors) are proposed. Comparing with camera based techniques, extracting useful information from compass is algorithmically simpler than implementing computer vision techniques. Our method does not impose major change in hardware specifications of mobile devices, or installing many optical sensors (e.g. in front, back or edges of the device). It is only based on an internally embedded compass in new generation of mobile devices. In contrast to the compass which is internally embedded, installing optical sensors occupies considerable physical space which may be a critical issue in small devices.
In this article, we suggest influencing the embedded compass sensor in mobile devices using motion of an external magnet for ADI purposes. We call our proposed approach as “MagiThings”. Digital compass (magnetic) sensor embedded in mobile devices contains a 3-axis Hall Effect sensor which registers the strength of magnetic field along the x, y and z axis. The Hall Effect sensor produces a voltage (Hall potential VH) proportional to the
Our approach is not influenced by illumination variation and occlusion problems. Employment of optical techniques can be limited when the camera or sensor is occluded by an object including the body of the user. Occlusion is not a critical problem in our approach, as magnetic field can pass through many different materials. Since the back of a mobile device is usually covered by hand, optical ADI techniques (e.g. camera and infrared based) can face difficulties for capturing interactions at the back of the device. The space at the back of the device can be efficiently used in our method, as magnetic field can pass through the covering hand (Figure 2). In addition, interaction is yet possible even if the device is not in the line of sight, or when it is covered (e.g. mobile device in a pocket or bag). The user can for instance accept or reject a call, or change a music track, without taking the phone out of his pocket/bag.
Sliding a permanent magnet across the peripheral area of a device deforms the default magnetic field patterns surrounding the device. Hence, by recording the momentary values of the magnetic flux density along the x, y and z coordinates, it is possible to obtain a sequence of 3D vectors that reflects the temporal pattern of field deformation due to the movement of the magnet by the user. As mentioned before, the magnet can be taken in fingers or worn as a ring (Figure 5).
magnetic flux density (B in Tesla) due to the Hall Effect. The output of the sensor is provided in x, y and z coordinates of the device (Figure 4). For iPhone 3GS platform, the range of values for these axes varies between ±128 µT.
In the method presented here, we use this interaction between the magnet and the embedded compass to send gestural commands to the mobile device. When the magnet taken or worn in hand is moved in the shape of a certain gesture, it causes a certain change in the pattern of magnetic field sensed by the compass sensor. Analyzing the pattern of change can lead to recognizing the gesture which can be then interpreted as a command for the device. The overall effect of the magnet trajectory on the device will be recorded in the form of a sequence of vectors where each element contains an instantaneous sample of the senor values along each coordinate. The resulting vector sequence can be used by the device to infer the user’s command or data. In this section, we present some methods for analyzing and interpreting the output of magnetic sensor.
Figure 4. A magnet affecting embedded sensor readings of a mobile device along different axis.
The compass sensor is constantly under the influence of earth magnetic field. This is an undesired effect which plays no role in interpreting gestures created by an external magnet. In most of the cases, it would simplify the rest of processing steps if the effect of earth magnetic field is removed. The effect of earth magnetic field can be considered as an almost constant (DC) component in the output signals. Therefore, one can think of using a high pass filter for removing it. The high pass filter highlights high frequency
components of the signals caused by motion of the external magnet, and removes the effect of earth magnetic field. In order to achieve the high pass filtering effect, we normally apply a time derivative function on the output signals. Besides earth’s magnetic field, there can be other sources of magnetic noise in the environment. However, in practice we have realized that such sources can have a very minor and neglectable influence on the performance of the proposed technology. The main reason is the fact that magnetic field strength decays very rapidly with distance with respect to the source of the field. Therefore, unless a very strong external magnet comes as close as e.g. 5 cm to the mobile device, the influence of noise associated with it is neglectable. If the output of magnetic sensor along x, y, and z axis is shown by x(t), y(t) and z(t) respectively, the output vector can be written as
p(t ) [ x(t ) y(t ) z(t )] And the time derivative operation is obtained as:
v(t )
p x y z [vx (t ) v y (t ) vz (t )] t t t t
In practice, we have observed that interpreting v(t) can be simpler than p(t). In the rest of the paper, whenever we refer to magnetic signals, we mean v(t) components, i.e. time derivative of sensor readings, unless explicitly clarified.
2.1
Measurement-Based Recognition
The measurement-based recognition (MBR) of gestures is an approach to recognize a human gesture based on the movement’s aggregate measurements. In this approach, we measure deviation of magnetic signals along different axis (or some simple features extracted from them), in order to interpret the gesture as a certain command. This approach can be useful for interpreting simple and short gestures which are usually composed of a simple motion of the magnet along one of the axis. The number of gesture classes should be also small in this case. For instance, if the aim of gestural interaction is to turn pages left and right in a calendar or text view application, two simple gestures composed of moving hand (finger) with magnet towards left and right can be defined. The two gestures (left or right motion) can be recognized and distinguished by comparing values of x-axis readings with a predefined threshold. In simple words, the magnitude of signals can show the amount of movement, and the sign of the signals can indicate direction of the motion. Some simple features usually based on average and variance of signals in a certain interval can be also used for interpreting simple gestures. For instance, as it is presented in Section 3.4.1, a click or triggering action in air (rapid motion of magnet in an arbitrary direction) can be recognized by detecting a high variance in magnetic field signals for a certain time window frame. Average magnitude of magnetic signals can be also a good feature for detecting significant movements of the magnet. The magnitude of the magnetic field is defined as Euclidian norm of magnetic field components along different axis. Changes in magnitude of the field obtained in this way can be good indicator of the magnet movements independent of its direction. The measurement based approach can be cheap in terms of implementation and computational resources required on mobile devices, and can be very efficient for recognizing a small set of
simple gestures in real time. This is an essential feature, if the interaction framework is used for instance in gaming applications. However, the approach described above can not be used for recognizing gestures with complicated patterns or large number of gesture classes due to huge sources of variability that can possibly happen in different trials of a gesture. For such cases, as presented in the following subsection, more sophisticated statistical machine learning or pattern matching approaches should be used.
2.2
Pattern Based Gesture Recognition
As it will be presented in the next sections, we used the magnetic interaction framework for recognizing not only simple gestures but also for a large number of gesture classes, or for complicated gesture patterns such as characters and signature shaped gestures. In such cases, a simple measurement of the magnetic signals can not be sufficient for interpreting the gesture. In this section, we study a number of pattern matching techniques that we have used for analyzing magnet based gestures. Dynamic Time Warping (DTW) (Ten Holt et al., 2007), Neural Networks (NN) (Minsky and Papert, 1969) and Binary Decision Trees (BDT) (Trevor et al., 2001) are three types of template matching techniques we have investigated in our studies for analyzing magnetic signals and gesture recognition. In short, the above techniques compare pattern of magnetic signals with a model that is created for gestures or some pre-registered patterns of gestures. The models can be created in a training phase, using several samples for each gesture. When a new gesture sample is presented, it is matched against the models for gesture classes, and a score is calculated showing the matching level. 2.2.1
DTW
Dynamic Time Wrapping (DTW) is a template matching approach for measuring similarities between two signal sequences. DTW generates a similarity measure which is based on the distance measurements between two sequences. It can operate with a limited number of templates, and still generate very accurate results. In order to apply DTW, first we have to collect a number of templates associated with each gesture class. Subsequently in the recognition phase, DTW calculates a distance between a new sample and each class template. The new sample is classified with the label of the template providing the smallest distance. In this work we have developed a multi-dimensional DTW that takes the Euclidean distance of the signal points into account. DTW technique can be very effective in practical applications in which the amount of training data available is very small. For instance, if the user needs to setup a gestural interaction by presenting only 1 or 2 samples of each gesture, DTW can be very effective. Many other statistical machine learning approaches may fail in this case, as they usually require a huge number of training samples (templates) which can be an issue for practical applications on mobile devices. 2.2.2
Neural Networks
Neural Networks (NNs) are non-linear statistical data modelling tools composed of a dense interconnection of computational units called artificial neurons. They are usually used for discovering complex patterns in data. In this work, we have used a certain type of NN called Multi-Layer Perceptron (MLP) for gesture classification.
For statistical machine learning approaches such as NNs and Binary Decision Tree, usually the raw magnetic signals can not be used directly as input. Raw signals can present redundant information which may not be necessarily useful for distinguishing between classes. In these cases, usually a feature extraction step from signals is required. Features which are extracted from signals have much less redundancy compared to raw signals, and can be more efficient for discriminating between different classes. In the following, features used in our study are listed. These features are extracted over a time window of signals in which the signal can be assumed stationary. • Average field strength along x, y, and z axes. • Variance of field strength along x, y, and z axes. • Average of Euclidian norm of field strength along x, y, and z. • Variance of Euclidian norm of field strength along x, y, and z. • Piecewise correlation between field strength along x-y, x-z, and y-z. Using these parameters we can extract a 15 dimension feature vector that characterizes the gesture trajectories. The MLP model has to be trained first in order to be used for classification. In order to train the MLP, several samples of each gesture class is collected from different users, and features are extracted for each sample. Features are presented to the MLP in a training phase, so that a model for the gesture classes can be obtained. For the classification, a new gesture sample is presented to the MLP model (after extracting features). The MLP then measures the similarity between the new sample and different classes and provides a similarity measure for each class. The gesture class with highest similarity is selected as recognition output. 2.2.3
Binary Decision Tree (BDT)
Binary Decision Tree (BDT) is a class of supervised learning approaches represented as a binary tree (two-way split at each node) that shows how the value of a target variable (gesture classes) can be predicted by using the values of a set of predictor variables (features). Same features as used in case of MLP can be also fed to the BDT for classification. At each node, a binary rule divides the input features into two classes and this refinement continues until a decision about which class of gesture does the features represent (with a certain confidence) can be made. During the training phase, we obtain a set of criteria based on which, we can classify the training data into their corresponding gesture classes. The same as MLP case, the training is performed using a set of samples obtained for each gesture class from different users.
3
Applications and Implementations
We have studied the proposed interaction framework within different gesture recognition contexts and applications. These vary from basic gesture recognition for controlling user interface of a mobile device to character recognition, signature verification and entertainment. As it is presented in this section, the proposed methodology can be used for tasks such as controlling a music player in a mobile device (loudness, music track change), turning pages, etc. This involves recognizing basic gestures indicating certain commands for the user interface. In addition, we have
investigated more complicated and detailed gesture patterns such as digits. In this case, the user is asked to write a textual command or a digit in air! The device can then recognize the command or the digit. Moreover, we have proposed what we call as “3D Magnetic Signatures”. In this case, a user is authenticated based on a signature that he makes in air! We have also investigated the proposed methodology for entertainment purposes including music synthesis and gaming. In the rest of the section, we summarize our studies and results for the mentioned application scenarios. 3.1 General Gesture Recognition We have investigated the application of our proposed magnetic based interaction to infer simple user gestures by monitoring the movements of the magnet held in the user’s hand (Ketabdar et al., 2010b). The main motivation behind this experiment is to serve as a proof-of-concept to our proposed idea of magnetic interaction. The gestures studied in this case can be used for interaction with the user interface of a mobile device. These gestures mainly comprise basic hand or finger motions (Figure 6). These gestures could be used for communicating some basic commands to the device; such as, turning pages in a reading application, changing a music track, or controlling the music volume. In order to investigate accuracy of the gesture recognition system, we have setup some experiments using gestures presented in Figure 6. In these experiments, we have invited 6 subjects to make 8 simple hand gestures while holding a rod shaped permanent magnet in hand. The gestures are simple movements such as vertically moving a magnet in front of the phone, moving the magnet from the backside of the phone and so on, as illustrated in Figure 6. Every subject is asked to repeat each gesture 15 times. We have used MLP for gesture modelling and recognition (see Section 2.2.2). We have also used a 10 fold cross-validation scheme to avoid over-fitting. The result of this experiment yields an accuracy of 91.4% on average for recognizing different gestures. The confusion matrix in Table 1, shows that in each row the highest recognition rate occurs in the column corresponding to the correct gesture (where the column number equals the row number). The value in the other columns would be the probability (normalized frequency) of misclassifying a gesture as an occurrence of the gesture corresponding to that column. As can be seen in the matrix, the highest level of confusion is between gestures 3 and 6, as well as 1 and 7, because of somewhat similarities between these corresponding gestures. Gesture 3 can be regarded as being similar to gesture 6 (circle) if the right-left trajectory in this gesture is different from the left-right trajectory. Also gesture 7 can be interpreted as quick repetition (two times) of gesture 1 (double click vs. click). One critical factor in obtaining the results presented here is how to detect the beginning and the end of the gesture trajectories. This can be highly influential on the accuracy of the results. For the current setup, the start and the end of the gestures are detected by comparing the magnitude of the magnetic field with a threshold. In other words, the beginning of the gesture is when the magnet is brought close enough to the device and the end is when the magnet is away from the device. The acceptable results obtained from general gesture recognition experiments encouraged us to further try more complex gesture patterns as presented in the next subsections.
Table 1: Confusion matrix for gesture recognition using the MLP classifier. It shows the actual gesture entries (rows) and the classification results (columns). The numbers in each row are normalized so that the sum of values in each row becomes one (Ketabdar et al., 2010b). Gesture Index
1
2
3
4
5
6
7
8
1
0.89
0.02
0.01
0
0
0
0.08
0
2
0.01
0.93
0.03
0.01
0.01
0.01
0
0
3
0.01
0.01
0.86
0.01
0.02
0.08
0.01
0
4
0
0
0
0.90
0.06
0.04
0
0
5
0
0.02
0.01
0.03
0.92
0.02
0
0
6
0
0.03
0.01
0.04
0
0.91
0
0.01
7
0.02
0
0.03
0
0
0
0.95
0
8
0
0
0.01
0.01
0
0.02
0
0.96
We have also developed several demonstrators based on simple gestures for Apple iPhone 3GS (Ketabdar et al., 2010b). They mainly demonstrate interaction with user interface of a mobile device for turning pages in a photo view application, zooming and un-zooming, and controlling an audio player (volume and track change) using simple gestures. As the gestures are simple and few in terms of classes, we have mainly used measurement based approach for interpreting the gestures. For instance, in order to detect a left-to-right gesture, we evaluate the amplitude of the magnetic field along the x axis. A negative value can indicate a right-to-left motion, and a positive value indicates the reverse. 3.2
User Authentication based on 3D Magnetic Signatures Mobile computing devices are frequently used to store and access sensitive information during daily life. Hence, user authentication/identification seems to be essential part of these devices for granting access to certain information or services. While conventional password-based authentication methods can be easily copied and distributed, there is a rapidly growing demand for new security systems for mobile computing devices that should be fast, easy and reliable. In (Ketabdar et al., 2010c), we have introduced MagiSign, a new touch-less, gesture-based authentication solution based on 3D magnetic signatures created in the space around a mobile device.
Figure 7. Using a magnet as user entry medium for authentication (Ketabdar et al., 2010c). The idea is that the user moves a permanent magnet (e.g. a pen or a ring) by hand in the space around the device along an arbitrary 3D trajectory to create a unique 3D signature (Figure 7). The embedded magnetometer in the device captures the changes in the temporal patterns of the magnetic field around the device as the 3D magnetic signature. Subsequently for user authentication, temporal pattern of a new magnetic signature is compared against the models or templates of the registered signature. We conducted an experiment to evaluate the accuracy of the proposed technology for user authentication/identification. We invited 15 subjects for the experiments and asked each of them to make an arbitrary 3D magnetic signature 15 times. We utilized MLP as classifier and we used a 10 fold cross-validation for training and testing the data. The results show an overall accuracy of 95.2% for user identification. Furthermore, for user authentication, we measured the Area Under the Curve (AUC) in the ROC, True Positive (TP) rate, and False Positive (FP) rate averaged over all users. The authentication results show a good trade-off between true and false alarms as it can be seen in Table 2. Magnetic authentication can have several potential advantages such as higher level of security and more flexibility over classical authentication methods. Unlike regular signatures, it is very difficult to replicate such a signature because it is created in air with no track remaining! As magnetic signature can be flexibility drawn in 3D space, it provides a wider choice for user authentication. Moreover, the proposed authentication method does not impose changes in hardware or physical specifications of mobile devices, and can be especially useful for authentication in very small mobile devices. Unlike face recognition based authentication, our method does not suffer from illumination and occlusion problems. Table 2. User authentication, averaged over all users (Ketabdar et al., 2010c).
Figure 6: Different gestures used in general gesture recognition studies. Gestures 7 and 8 can be interpreted as quick repetition (twice) of gestures 1 and 3, respectively (as in double click vs. click) (Ketabdar et al., 2010b).
Measure
AUC
TP rate
FP rate
Value
0.991
0.952
0.003
digit class which is showing a higher score (i.e. better match) is selected as recognized digit.
Figure 8. MagiWrite: Entering text (digits) using magnetic interaction (Ketabdar et al., 2010a). The proposed method can be further enhanced by providing a second layer of security and personalization. The idea is that a user can employ a magnet with a personalized shape and polarity for creating the signatures. In this way, the data that is registered by the magnetic sensor will be a function of both the shape of the magnet, as well as the way it is moved in the air. This can be considered as using a physical key in addition to the movementbased signature for authentication process. Finally, we have implemented a demo application based on the proposed touch-less, gesture-based authentication framework for Apple iPhone. The demonstrator allows a user to register a few templates of his 3D Magnetic signature using a magnet in fingers, and then try the authentication process by a new sample of his 3D signature. Although here the authentication method is designed for mobile devices, the application can be extended for user authentication/identification at ATM machines, auto-lock systems at the entrances, etc. 3.3 Touchless Character (Digit) Entry An important part of interaction with mobile and tangible devices involves entering textual data e.g. for sending a text message, dialing a number, etc. Text entry generally requires direct physical interaction with mobile and tangible devices via keypads or touch screens. As the actual size of these devices is getting smaller, it may not anymore be comfortable to operate small buttons or touch screens for text entry. Recently in (Ketabdar et al., 2011 and Jensenius, 2008), we have presented a new technique (MagiWrite) based on the proposed touchless interaction framework for text entry that can overcome limitations of existing keypads and touch screen input interfaces. Our method expands the text entry space beyond physical boundaries of keypads and touch screens, and uses the space around the device for entering textual data. The user simply writes characters or textual commands in air and the device recognizes them. In MagiWrite (Ketabdar et al., 2011 and Jensenius, 2008), we mainly focused on entering digit data for simplicity. However, the same methodology can be applied for entering other characters, symbols, or textual commands. This is done by drawing gestures similar to digits (in terms of shape) in the 3D space around the device (Figure 8) using a properly shaped magnet taken in hand. The user can register one or a few templates for each digit in the training phase. The template for each digit is stored as a time sequence of magnetic signals sampled along x, y, and z directions. During a digit entry (testing the system), a new digit gesture drawn in the space around the device is compared against registered templates available for different digits using DTW. The
In order to evaluate the accuracy of the proposed digit entry approach, we invited 8 subjects to participate in our experiments. We asked each subject to draw 10 digits (from 0 to 9) in space around the mobile device by holding a magnet in hand. For each digit, we collected 15 different templates per each subject. We have run an 8 fold cross validation on each subject’s sampled data and on each fold we have increased the number of templates by 1. Figure 9 represents the experimental results of using DTW for digit recognition. Each curve in the graph corresponds to the data obtained from one user. It can be seen that DTW recognition results converges very fast after having only 3 or 4 templates for each digit. It means that even if the number of templates is limited, a high accuracy can be obtained with the proposed method. This feature is especially very important in practical mobile applications where the user may not want to enter many templates for each digit. According to our experiments, the average accuracy and standard deviation of the proposed digit entry approach is 0.8535 ( 0.075) using only 3 templates for each digit. Since the interaction in this technique is based on magnetic field which can pass through different materials, the device does not need to be necessarily in the line of sight or in hand for entering textual data. Data entry can be potentially possible even if the device is in a pocket or bag. For instance, the user may be able to dial a number, enter a pin code or textual command without taking the mobile device out of his pocket/bag. A demo application on Apple iPhone 3GS has been developed to demonstrate the effectiveness of digit entry application (Ketabdar et al., 2010a). Despite the fact that the text (digit) entry experiments indicates the accuracy of the approach in recognizing more complicated gestures, a usability study of the concept should be performed as well. According to our interviews with the subjects, they all fancy the concept of writing in air for sending commands to the device. However, they also mentioned that they may use it only in certain circumstances; such as dialing a default number, rejecting a call with a default rejection message, etc.; where a casual text entry is required. 3.4 Magnetic Interaction for Entertainment Purposes We have also investigated the application of the proposed technology in the context of mobile device entertainment, mainly for digital music synthesis and gaming. Nowadays, mobile devices have also become popular digital instruments for musical performance (Gillian et al., 2009, Wang, 2009, Wang et al., 2008 and Ketabdar et al., 2010d). Numerous applications in music have been developed using the touch-screen and accelerometer for interaction, in order to simulate the traditional instruments on mobile devices. However, playing musical instruments on the surface of mobile devices is limited and usually required both of user’s hands on a small screen. Moreover, the applications that use accelerometer require the user to repeatedly turn and tilt the device to generate certain sound which makes the user lose direct sight to the screen.
Figure 9. Accuracy vs. number of templates. Each curve corresponds to the data obtained from one user (Ketabdar et al., 2011). In order to address these limitations, in (Ketabdar et al., 2010d) we presented a novel approach for music performance applications based on the proposed touch less interaction methodology. The proposed magnetic based interaction framework provides higher degree of flexibility for music performance, because the interaction space is extended to 3D space around the device. This allows users to play music on mobile devices using highly intuitive hand gestures as what they do with real instruments. Playing musical instruments usually requires harmonic movements of hands with the instruments. In the proposed musical performance method, we establish a mapping between motions of hand (or fingers), and movement of a magnet (taken in hand) in the space around the device. Position, movement, shape, and orientation of the magnet can be used as an input for altering or adjusting parameters and characteristics of the music. In the rest of this section, we elaborate on some music synthesis applications implemented for mobile devices based on the proposed technology. 3.4.1 AirGuitar Traditional guitars are played by two distinct actions. A user presses pitches on different strings alongside the neck of the guitar with one hand, while with the other hand he periodically scratches the guitar strings. The combination of these two actions produces different tones. In the current mobile phone touchscreen based guitar applications, both of these actions (i.e. holding notes and scratching strings) is performed on the small screen of the device. Such user interface setup can be limited, as it requires using both hands on a small screen. In our proposed guitar application (AirGuitar) (Ketabdar et al., 2010d), the holding action remains on the touch screen, but the touch screen based strumming action is replaced with a 3D gesture in the space around the device. The user can periodically move his hand (holding a magnet) in the space around the device (equipped
Figure 10. Gestural interaction with AirGuitar application using magnetic interaction around a mobile device. with embedded compass) imitating the natural strumming actions in a real guitar. The rapid movements of hand in the air (carrying a magnet) in a similar fashion as strumming, create rapid temporal changes in the magnetic field around the device. This can be detected by comparing the variance of magnetic field (estimated over an interval/window) with respect to a pre-defined threshold. One or multi tones are played, if their corresponding pitches are held on the screen with the other hand of the user. An Apple iPhone demo has been developed for our proposed music performance application as it is shown in Figure 10. When the AirGuitar application is launched, the user can see guitar strings on the screen. The user holds the strings on the screen with one hand, and strum in the 3D space around the device using the other hand with a magnet. If a string is touched, the corresponding sound of the string is played.
3.4.2 Drum-Kit We have also developed an AirDrum based on the proposed magnetic interaction framework. Two factors are important in playing a drum: the strength of the hit and the radios of the hitting point to the center of the drum surface. Several drum applications have been introduced for mobile devices, however in these apps only location of the hit can affect the generated sound. The strength of the hit cannot be detected by touch screen. Using our framework, the strength of the hit can be captured as a part of the hitting gesture (Ketabdar et al., 2010d). The drum application that we have developed (Figure 11) interprets the second time derivative of the Z component signal of the magnetic sensor as the strength of the hit. The higher the strength (energy) is, the generated sound will be louder. On the other hand, the X component of magnetic sensor output represents the radial distance to the center of the drum surface.
easy to integrate, and cheap interaction possibility especially for small mobile devices. We presented several use cases for the proposed technology studied for mobile devices. This includes gestural interaction with user interface of a mobile device, character (digit) entry, 3D Magnetic Signatures, and “Air Instruments”. We showed that initial studies and experiments yield satisfactory results in terms of accuracy for gesture recognition. Although we have investigated a few initial applications for the proposed interaction framework, there are several other possible applications which can be further studied in this framework.
Figure 11. User is playing the Drum-Kit application on Apple iPhone by moving a magnet. Moreover, we have developed several other musical instrument applications such as Harmonics, Theremin, and several sound composition applications like sound modulation and effecting, and DJ interface application (Ketabdar et al., 2010d), in order to show the wide range of potential applications that can be developed based on the proposed magnetic interaction framework.
The fact that our data entry approach does not entail user’s visual resources can make it a pragmatic communication solution, especially for visually impaired users. This feature can also be adopted in situations such as driving a vehicle, where the visual system of the driver is occupied with critical tasks. Another assistive application would be for patients with excessive shakes in their hands whose tremors make typing accurately difficult or impossible. In this regard, the patients can enter a shape of their own choice to represent a template to the device. This easy-toreplicate shape, then, can be used later on to refer to a character, command or any other definition that the template represents to the device. Regular users can still benefit from the proposed method for interaction in darkness or when the mobile device is covered in a pocket or bag. In those scenarios, camera based gesture recognition techniques may fail.
Besides the music performance applications, the proposed touch less interaction framework can be employed for mobile entertainment applications such as gaming (Ketabdar and Yüksel, 2010). The major challenge facing mobile gaming applications design is that interaction with small size touch screen and buttons on the mobile devices is not user friendly and convenient. Besides, the current user interfaces limit the interaction speed of players. In (Ketabdar and Yüksel, 2010), we have presented a new technique based on the proposed touch less interaction framework for mobile gaming applications. The framework extends the interaction space to 3D space around the device, thus leads to faster and more flexible interaction. User can employ hand/finger gestures to control actions of a character in a mobile game application. Moreover, the user is able to use multiple magnets with different shapes and polarities to perform multiple commands simultaneously. Thus, the proposed framework can potentially enhance the usability and playability of mobile games by employing more natural, intuitive and faster method of interaction.
The framework can be extended for multi-users. An initial approach for realizing this idea is to use magnets of different shapes and/or polarity for different users. In this way, motion of each magnet can create a different deviation in the magnetic field around the device. This can allow two or multiple users to interact with e.g. a mobile gaming application or digital music instrument.
4
6
Summary and Future Work Possibilities
In this paper, we elaborated on a novel magnet-based interaction framework. Our work has been motivated by the space constraints on small handheld device, and the limits which result from the use of touchscreen, accelerometer, IR-sensor and joystick modalities. Using the proposed magnetic interaction framework, the input space extends beyond the physical dimensions of a device, and therefore offers not just a pragmatic solution to the space constraint, but also a more natural and flexible means of interacting with mobile devices. The proposed method relies only on a magnet and a compass sensor which is embedded in new generation of mobile devices. Therefore, it does not require extensive hardware setup or modifications in physical specifications of the device. It can be considered as an alternative,
The magnetic interaction framework can be extended beyond mobile device platforms. In such cases, a magnetic sensor or sensor package should be integrated or provisioned in the devices to be controlled (Williamson et. al. 2007). This enables the application of magnetic based gesture recognition framework in different cases, such as entrance gates and ATM machines.
5
Acknowledgements
The authors would like to thank Deutsche Telekom Laboratories for supporting this work under SR Project “MagiThings”. The Authors also thank Sebastian Möller, Michael Rohs, and Sven Kratz for helpful discussion.
REFERENCES
Baudisch, P., & Chu, G. (2009). Back-of-device interaction allows creating very small touch devices. Proceedings of the 27th international conference on Human factors in computing systems (pp. 1923-1932). Boston, MA, USA: ACM. Butler, A., Izadi, S., & Hodges, S. (2008). SideSight: Multi“touch” interaction around small devices. Proc. of UIST’08 (pp. 201-204). Monterey, CA, USA: ACM. N. Gillian, S. O'Modhrain, and G. Essl. Scratch-off: A gesture based mobile music game with tactile feedback. In CHI, volume 3, pages 234-240, 2009.
Harrison, C., & Hudson, S., (2009). Abracadabra: Wireless, HighPrecision, and Unpowered Finger input for Very Small Mobile Devices, UIST’09 (pp. 121-124). New York, NY, USA: ACM. Hinckley, K., Pierce, J., Sinclair, M., & Horvitz, E. (2000). Sensing techniques for mobile interaction. In Proc. of UIST '00 (pp. 91-100). San Diego, CA, USA: ACM. Howard, B. & Howard, M. G. (2009): Ubiquitous Computing Enabled by Optical Reflectance Controller. Whitepaper. Lightglove, Inc., Retreived June 25, 2009, from http://lightglove.com/WhitePaper.html A.R. Jensenius. Some challenges related to music and movement in mobile music technology. In 5th International Mobile Music Workshop, Vienna, Austria, 2008. H. Ketabdar and K.A. Yüksel. Magientertain: entertainment interaction based on magnetic field. 2010.
Mobile
H. Ketabdar, M. Roshandel, and K.A. Yüksel. Magiwrite: Towards touchless digit entry using 3D space around mobile devices. In Proceedings of the 12th international conference on Human computer interaction with mobile devices and services, pages 443-446. ACM, 2010a. H. Ketabdar, K.A. Yüksel, and M. Roshandel. MagiTact: Interaction with mobile devices based on compass (magnetic) sensor. In Proceeding of the 14th international conference on Intelligent user interfaces, pages 413-414, 2010b. H. Ketabdar, K.A. Yüksel, A. Jahnbekam, M. Roshandel, and D. Skirpo. Magisign: user identification/authentication based on 3D around device magnetic signatures. In UBICOMM 2010, The Fourth International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies, pages 31-34, 2010c. H. Ketabdar, K.A. Yüksel, and M. Roshandel. Towards digital music performance for mobile devices based on magnetic interaction. In Haptic Audio-Visual Environments and Games (HAVE), 2010 IEEE International Symposium on, pages 1-6. IEEE, 2010d. H. Ketabdar, A. Haji-Abolhassani, K.H. JahanBekam, and K.A. Yüksel. Maginput: Realization of the fantasy of writing in the air. In The Fifth International Conference on Tangible, Embedded and Embodied Interaction, 2011. S. Kratz and M. Rohs. Hoverow: expanding the design space of around device interaction. In Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services, page 4. ACM, 2009. C. Metzger, M. Anderson, and T. Starner. Freedigiter: A contactfree device for gesture control. In Wearable Computers, 2004. ISWC 2004. Eighth International Symposium on, volume 1, pages 18-21. IEEE, 2004. M. Minsky and S. Papert. Perceptrons: An introduction to computational geometry. The MIT Press, Cambridge, Mass, 11:189-208, 1969. J. Smith, T. White, C. Dodge, J. Paradiso, N. Gershenfeld, and D. Allport. Electric field sensing for graphical interfaces. Computer Graphics and Applications, IEEE, 18(3):54-60, 1998. T. Starner, J. Auxier, D. Ashbrook, and M. Gandy. The gesture pendant: A self-illuminating, wearable, infrared computer vision system for home automation control and medical monitoring. In
Wearable Computers, 2000. The Fourth International Symposium on, pages 87-94. IEEE, 2000. G.A. Ten Holt, M.J.T. Reinders, and E.A. Hendriks. Multidimensional dynamic time warping for gesture recognition. In Proc. of the conference of the Advanced School for Computing and Imaging (ASCI 2007), 2007. L.S. Theremin and O. Petrishev. The design of a musical instrument based on cathode relays. Leonardo Music Journal, pages 49-50, 1996. H. Trevor, T. Robert, and F. Jerome. The elements of statistical learning: data mining, inference and prediction. New York: Springer-Verlag, 1(8): 371-406, 2001. G. Wang. Designing smules iPhone ocarina. In Proceedings of the International Conference on New Interfaces for Musical Expression. Pittsburgh, 2009. G. Wang, G. Essl, and H. Penttinen. Do mobile phones dream of electric orchestras. In Proceedings of the International Computer Music Conference, 2008. J. Williamson, R. Murray-Smith, S. Hughes. Shoogle: Multimodal Excitatory Interaction on Mobile Devices, Proceedings of ACM SIG CHI Conference, San Jose, 2007.
