KITTY: Keyboard Independent Touch Typing in VR - CiteSeerX

KITTY: Keyboard Independent Touch Typing in VR Carsten Mehring Dept. of Mechanical and Aerospace Eng. University of California, Irvine, CA 92697 {cmehring}@uci.edu

Falko Kuester, Kunal Deep Singh and Michelle Chen Dept. of Electrical Eng. and Computer Science University of California, Irvine, CA 92697 {fkuester,singhk,mchen}@uci.edu

Abstract A new hand- or finger-mounted data input device is presented, using traditional touch-typing skills as method of (alphabetic) character input thus providing an ultra-portable solution for “quiet” data input into portable computer systems. The presented keyboard independent touch-typing device (KITTY) offers high data input rates and a virtually zero learning curve for new users.

1

Motivation

With recent advances in portable computing and in particular the design of pocket PCs [13] and eye-glass displays [14], the development of new augmented reality (AR) environments has been made possible. However, one of the current shortcomings in these AR systems is the lack of intuitive input devices that provide users with complete control over their workspace. In particular in environments where voice input is undesirable or infeasible, touch-typing capabilities have to be available to allow for intuitive access to possibly complex data.

2

Introduction

The size, layout and weight of traditional keyboards make them poor candidates for use in mobile AR applications. In particular, if integration with mobile computing devices, such as personal digital assistants (PDAs) or pocket personal computers (PocketPCs) is desired, a new generation of natural handheld input devices is required. Today’s mobile computing devices frequently include a physical or digital miniature keyboard, that may be integrated with the device, worn on the user’s wrist or simply be part of the display area. Unfortunately, these keyboards generally require the use of both hands, one holding the device and the other for finger-based or stylus-based data input. The selection of individual keys on a hand held display-based keyboard, for example, is a rather inefficient process and requires the user to look at the keyboard devise. Handwriting

Figure 1. Prototype glove system illustrating initial contact placement.

recognition can increase input rates after extensive training but is still very limiting. This performance gap is problematic if the portable computing devices are to be operated at the level suitable for today’s computational platforms. Other devices for data input into portable computing units, include chording devices, such as a hand-mounted keypads that require the user to press different key combinations to generate the characters found on a standard keyboard. Further alternatives include microphones in combination with voice recognition software. The former input devices typically do not meet the data input rates desired for full-scale interaction and typically require the user to learn a chording/coding language particular to the specific device. On the other hand, privacy concerns or ambient noise frequently limit the use of voice recognition devices. Possible scenarios include input of confidential or sensitive data via voice recognition in conference settings, meetings, or crowded public places, such as airports.

3

Related Work

With a demand for decreased keyboard size and increased portability, while maintaining the traditional keyboard layout, new device concepts are required. A first attempt was

the development of foldable keyboards that allowed users to carry familiar input devices with reduced physical footprint. While these devices are certainly more compact than traditional full-size keyboards during transport, a folding keyboard is still relatively large in the unfolded state in which its surface requirements match that of a traditional full-size device. Other devices have tried to reduce the number of keys rather than reducing the overall size of the keys in order to achieve size reductions. For example, one-handed keyboards use only half of a traditional keyboard by mapping two characters on each key. This requires the addition of a switch (keyboard identification) key, enabling the user to toggle between two sets of characters, each representing one-side of a full-size keyboard. Even though “half-keyboards” may have a very small footprint, they commonly still require a support surface. The “body-wearable keyboard” and similar the so-called “Gesture-Pad”, which is under development by Sony Inc., place “keyboard-keys” on the users clothing or a skeletal structure worn on the users body. Even though this increases portability, it greatly restricts the users arm-movement and flexibility for data input. Many of the existing keyboard- or keypad-less data input devices for portable computing applications, use finger or hand mounted sensors in order to generate the signals that would normally be generated by pressing a keyboard key. The ”finger-mounted computer input device” [11], the ”fingerless glove for interacting with a data processing system” [15] and the ”Virtual Keyboard” by Senseboard Technologies [12] use pressure sensors on the users fingers or sensors (mounted on gloves) measuring flexure of the fingers in order to determine data input via finger positions with respect to a physical or virtual reference surface. Generally, these devices assume that the reference surface functions as virtual touch-typing environment. As for folding keyboards, the need of a physical support surface greatly limits the mobile computing capabilities of these devices. In response to this limitation, hand- or finger-mounted input devices have been proposed, such as the ”Chording Glove” [16] and ”Twiddler” [1]. The Chording Glove has contacts at various positions on the fingers and thumbs and a combination of contacts (chord) is pressed simultaneously to generate a specific keyboard event. Although chording devices remove the need for a support surface, their use requires substantial training as the user has to learn a new coding language. Vaughan R. Pratt describes a device-independent digital sign langue (Thumbcode), using keys on the users fingers that are activated by the thumbs of the same hand [10]. In order to generate different input signals with the same key, the user has to pair neighboring fingers according to a prescribed coding language. As with chording, thumbcoding requires a

user to essentially learn a new coding language in order to enter data into the computing device. The functionality of the so-called Lightglove by Howard [4, 3], is based on the generation of a ”light-matrix” below the users hands. Penetration and location of penetration of this matrix by the users fingertips is detected and electronically mapped to an overlay assigning each position a character according to the layout of a traditional keyboard. This operation of the device involves simultaneous scanning of the matrix with various light emitting diodes or lasers, evaluation of the scattered electromagnetic waveforms by various light sensitive detectors and signal filtering in a bandpass filter in order to reject non-correlated ambient signals. Generation of the optical matrix and detection of finger position in this matrix is fairly complicated. Furthermore, the system has to be calibrated (possibly in real-time) for specific ambient conditions, such as strong ambient illumination or electromagnetic interference. The lack of tactile feedback (providing confirmation of data input as is the case for touch-typing on a keyboard) may also result in lower data input rates. The use of an optical reflection matrix to detect signal input by predicting finger position with respect to the optical matrix below the user’s hands does not allow for free finger motion without the risk of data input. Instead, the user has to hold the hands and fingers in a more-or-less stretched out position in order to omit interference with the optical matrix. The involuntary generation of signals is a common problem with these devices and has to be addressed. Finally, many of the devices described in this section are large, counter-intuitive, and too cumbersome for fast data input. Most users when confronted with having to learn an entirely new coding language in order to enter information will likely decide against using such a device. This is particularly true for good VR or AR applications that should require only very limited user training. For VR/AR and mobile computing applications, intuitive, flexible, and efficient, keybaordless input devices are urgently needed. KITTY, a device prototype that addresses these objectives is presented in this paper (Figure 2).

4

KITTY Interface

The presented KITTY interface overcomes the limitations described in the previous sections by providing intuitive access to common touch-typing skills. To accomplish this, the device provides multiple contacts on each finger that are only activated when the appropriate finger and thumb contacts are combined. The layout of these contacts follows the traditional QWERTY keyboard layout, allowing anyone with previous touch-typing experience to immediately use the device. A significant advantage of this device is that arbitrary finger motion is allowed without data input, as long as finger contacts and thumb contacts on a single hand do not meet. In other words, KITTY allows the user complete freedom for

! 1

@ 2

Q

Esc Caps Lock

# 3

W

A

$ 4

E

S

Shift

^ 6

R

D

Z

Ctrl

% 5

X

T F

& 7

Y G

C

V

* 8

U

H B

(

)

9

I

J

O K

N

< ,

M

: ;

D E F D0E0 F0

A C B

1 23

A0 C0 B0

4

Back

} ] " ’

>

?

.

/

Alt

+ = { [

P

L

~ ‘

Alt

_ −

0

Ins

| \ Enter

Shift

u

l

d

Del

r

678 5

A   3F /  ./!   ,# (  ! 055 %          P'  "  bottom

    

 top row  (dashed) and row (dash-dot) characters.     
       @ # $ % ^ & * ( ) _ + Back 2 3 4 5 6    7 9  − =  !137" 02    is referred to as   the top row and 8the row0 below the base row { } | Q   W E R
"
5" T Y 9"95" U I %"%5 O P   Esc   [ ] \ referred (ZXCV ...) is annotated with a short dash: line and " "  Caps A

"F Figure  K illustrates   S  D G H  L Enter ; ’ the Lock to as the bottom row. 3J also enumer    $"$5" 8"85" .

T2-6 T3-Shift T4-Space

T2-5 T3-Shift T4-Space

T2 T3 T4

TI-8 T2-I T3-K T4-< ,

TI-0 T2-P T3-" ' T4-? /

T1 T2 T3 T4

Right Hand

Figure 7. Keyboard to glove mapping of wireless KITTY implementation.

According to the alphanumeric mapping summarized in Table 1 and illustrated in Figure 7 each finger contact is linked to the column of characters it would press on a keyboard using touch-typing. For example, the middle finger of the left hand, LM, is linked to ”e”, ”d”, and ”c”, the same keys this finger would normally press on a keyboard using touch-typing. The three thumb contacts match the three letter rows on a normal keyboard. The contact at the tip of the thumb, T2, corresponds to the top row of the keyboard, T3 to the base row, and T4 to the bottom row. Using the left middle finger again as an example, LM would yield ”e” when touched to T2, ”d” when touched to T3, and ”c” when touched to T4. In this modified key-map, each index finger has two more contacts on the side of the finger, LI2, LI3, RI2, and RI3, spaced above and below the top knuckle. The contact above

Numeric input is provided by simply placing another contact, T1, on the back of the thumb on the nail. The finger contacts are touched to T1 consecutively in keeping with the traditional keyboard assignments. This configuration for numbers works also well for non-touch-typists since it follows a traditional method of finger counting. Starting at the left pinky, LP touches T1 to generate the number 1, and then LR, LM, and LI1 follow for numbers 2 to 4. The number 5 is the missing number that uses the LT3 to T2 combination. Although an odd combination, it still follows the flow of the series as the tip of the thumb is used for number 5 like the finger pads for the other numbers. The right hand picks up number 6 in a mirror series to the left hand, with 6 to 0 moving through RT3, RT1, RM, RR, and RP in that order.

As with the key-map discussed in the previous section, a drawback to the described key-to-glove mapping is the bottom row key on the pinky. Touching LP or RP to T4 has an “awkward feel” to it. A momentary slight strain is felt in stretching the pinky towards the lower pad of the thumb. Like learning an awkward chord on a guitar, it is possible that the strain will lessen with prolonged use as the hand adjusts towards this unaccustomed position. One could argue that the keys ”Z” and ”? /” are used so infrequently that there is no concern regarding continuous straining of the tendons. If it is found, however, that the awkwardness and discomfort of the particular finger combination becomes an issue, an additional contact could be placed on the sides of the middle fingers to offer an alternative and a more comfortable combination.

5

Hardware Implementation

Two different implementation strategies were investigated supporting a wired and wireless configuration of the KITTY device.

5.1

Wired Prototype

Initially, the KITTY interface was implemented with minimal effort using the “Virtually Indestructible” foldable keyboard by GrandTecT M (see Figure 8). The mentioned keyboard was chosen due to the 100 % correspondence between the circuit layout on the two membranes of the keyboard and the key-mapping described in Section 4.2, with fingercontacts representing keyboard columns and thumb-contacts representing keyboard rows. Wires were attached to the corresponding locations on the keyboard membrane circuitry using self-adhesive copper tape. Wirewrap was used to keep the design sleek. The wire ends were then collected and attached to two golf gloves according to the layout shown in Figure 4.

Figure 8. First prototype system utilizing traditional roll-up keyboard as the keyboard encoder.

Clearly, the wiring or printed circuitry on the two membranes of the ”Virtually Indestructible Keyboard” by GrandTec perfectly matches (without modifications) the mapping proposed in Section 4.2. The GrandTecT M keyboard uses two membranes, membrane A onto which the wires for the finger contacts are attached and membrane B onto which the wires for the thumb contacts are attached. The contacts for the letters on membrane A are connected in a column format and the contacts on membrane B are connected in a row format.

5.2

Wireless Interface

Once a finger-thumb contact has been established according to the mapping schemes discussed earlier, various methods can be used to generate the signals. For example, wires from the contacts may be connected directly to a keyboard

encoder. The keyboard encoder may be located on top of the user’s hand. For example, the keyboard encoder may be attached to a glove, such as the one shown in Figure 9. The signal that is generated can then be transmitted to the computing device via wire or wireless, e.g., infrared (IR) or the like. The computing device can then display the character represented by the signal. Keyboard Encoder – A 8x12 keyboard encoder would suffice for the keys mapped according to Section 4.3. However, a 8x16 matrix is preferred; the larger matrix allows future functionality for additional keys such as macro keys. The smallest possible platform and low power consumption are other factors in deciding which keyboard encoder to use. A small platform is necessary to keep the wearable device unobtrusive and comfortable to wear for the user. Since power would be supplied by batteries, it is important that power consumption be minimized for longer battery life. For the above reasons, the Motion Encoder ME1001A keyboard encoder was selected for the wireless KITTY interface. Thumb contacts are wired to the row pins of the encoder, while the finger contacts connect to the column pins. The encoder transmits a single keystroke as a TTL serial signal at 2400 bps with no parity, 8 data bits and 2 stop bits. A hexadecimal number from 0x00 to 0x7f (decimal 0 to 127) indicates which key on the matrix was selected. This number is transmitted as part of a byte and occupies bit positions 0 to 6. The seventh bit indicates the key state of key down or up. The recommended 6MHz resonator is attached to the encoder. Other additional components that must normally be added to other encoders, such as pull up resistors, are internal to this encoder and helps keep the size of the device as small as possible. Wireless Signal Transmission – The KITTY interface was created with portability and convenience in mind. Accordingly, it was our goal to add wireless signal transmission capabilities to a second generation KITTY interface providing data-input via gloves which are physically disconnected of the computing device itself. From the wide range of wireless technologies available, BluetoothT M technology has been used. BluetoothT M wireless technology is being used extensively in wireless and portable devices like cell phones and PDA’s, and has become a wireless standard. It provides plug and play type connectivity between any two or more BluetoothT M devices. One of the goals behind the ongoing development of KITTY is to provide an input device that could work with PC’s, PDA’s, Pocket PC’s and any other device that requires alphanumerical input. Hence the use of BluetoothT M wireless technology was appropriate for this application. Within the new wireless KITTY data-entry system, the serial output of the encoder is fed into a BluetoothT M RS232/UART DCE Terminal that is configured to communicate serially with the encoder. For this prototype the Wire-

less Futures BlueWAVE RS232 PCB DCE Terminal is used that connects to a generic BluetoothT M USB dongle/adapter that supports the BluetoothT M serial port profile. This DCE terminal was chosen for sake of programmability, low power consumption and its size. The prototype has been developed for a Windows 9x and Windows NT platform. The adapter used in this case is the D-Link DBT-120 BluetoothT M USB adapter. It does not require any external power supply, but receives all necessary power from the USB port. The DCE terminal requires a supply voltage that is between 3.3V to 9V DC, which is accomplished by using four AA Ni-MH rechargeable batteries that provide 1900mAh each and a total potential difference of 5.2V. The terminal’s average current consumption is 1.8mA when it is in the idle mode and 60.9mA while it is transferring data. The DCE terminal measures about 40mm x 30mm x 5mm, and the antenna extends about an inch over that. The voltage is applied across pin 1 and pin 12 of the DCE terminal with pin 1 connected to the positive terminal and pin 12 connected to the negative terminal of the 5.2V battery pack. The serial data is connected to the TxD pin, which is pin 6 of the DCE terminal. Pins 3 and 4 are connected to each other in order to disable any hardware flow control. The serial data signal and the voltage supply are connected to the DCE terminal using a 0.05” (1.27mm) pitch ribbon cable. Both the adapter and the DCE terminal conform to the BluetoothT M v1.1 standard. When power is switched on for the DCE terminal it will become “discoverable” by other BluetoothT M devices. Since it is a DCE terminal it can only be used as a slave module. The BluetoothT M serial port profile creates a virtual COM port on the PC, which is used for connecting the two devices. The virtual COM port is similar to a physical COM port, but instead of having a wire for transferring the data, the data is transferred wirelessly. The serial data from the encoder is received by the terminal and is transmitted to the adapter. The virtual COM port is also configured to the same setting as the encoder and the terminal. Then a software driver decodes the data received and produces the appropriate keystroke. The information is transmitted at 2400MHz to 2483.5MHz frequency, as specified by the BluetoothT M standard. The driver is responsible for configuring the virtual Com port, waiting for the data, decoding the data and producing the appropriate character using the keybd event() system call.

5.3

Design Concepts

Various designs have been considered for implementing the described KITTY interface. The employed prototype design shown in Figure 1 and Figure 9 uses contacts attached to gloves (only one glove is shown in the figures carrying only contacts for alphabetic character input). The contacts are connected by wires to the encoder and (for the wireless implementation) transfer electronics. Preferably, the gloves

are fairly thin and lightweight, thereby allowing the user to easily manipulate his fingers. An alternative configuration also shown in Figure 9 has contacts mounted on clips or half-rings. The clips are attached to a skeletal structure. The keyboard encoder and transfer electronics (e.g., wireless transmission electronics) are also mounted to the skeletal structure. Figure 9 shows an exemplary skeletal structure for the left hand. The skeletal structure rests against the user’s palms (or alternatively on the back of the user’s hand) and the contacts are positioned on the user’s fingers by placing the clips (on which the contacts are mounted) around the user’s fingers and thumbs. The clips can be attached to the skeletal structure in a way that allows adjustment of the positions of the clips in order to accommodate different finger and thumb lengths. The skeletal structure can also be adjustable in order to accommodate hands of varying sizes. The main purpose of the skeletal structure is to preserve the integrity of the overall system.

Figure 9. Top: Glove-based prototype design. Bottom: “Click-on” skeletal-structure design.

The designs discussed here can also be configured to be used on one of the user’s hands to simulate a “HalfKeyboard”. The location of contacts on the hand being used (either right or left) would be the same as in the configuration of contacts when using finger/thumb based touch-typing in the two-handed configuration described above.

6

Results

With recent advances in portable computing and in particular the design of pocket PCs and eye-glass displays, efforts aimed at the design of portable input devices and techniques have increased. While voice recognition is becoming

a strong contender for next generation input technology, privacy concerns and ambient noise problems are still a challenge and alternative approaches are required. The presented KITTY interface consists of a finger-mounted device mimicking finger movements similar to traditional touch-typing in order to provide alphanumerical data input. This low cost device guarantees a rapid learning curve for everyone experienced in touch-typing while offering significant flexibility. Two prototypes were build one using a PSII and one using a USB version of the GranTec keyboard. Priced at around $50 US for parts the KITTY device proved to be a very cost effective solution. The proof-of-concept implementation of the KITTY device was tested with wearable devices and desktop systems and demonstrated its effectiveness.

References [1] Handykey Corporation. www.handykey.com, 2003. [2] M. E. Horn. Single-hand mounted and operated keyboard. U.S. Patent 5,552,782, September 1996. [3] R. B. Howard. Wrist-pendent wireless optical keybaord. U.S. Patent 6,097,374, August 2000. [4] Lightglove. www.lightglove.com, 2003. [5] MachineBrain.com. www.machinebrain.com, 2003. [6] E. Matias, I. S. MacKenzie, and W. Buxton. Half-QWERTY: Typing with one hand using your two-handed skills. In Proceedings of ACM CHI’94 Conference on Human Factors in Computing Systems, volume 2 of INTERACTIVE EXPERIENCE, pages 51–52, 1994. [7] E. Matias, I. S. MacKenzie, and W. Buxton. A wearable computer for use in microgravity space and other non-desktop environments. In Proceedings of ACM CHI 96 Conference on Human Factors in Computing Systems, volume 2 of INTERACTIVE POSTERS: CHI in Space, pages 69–70, 1996. [8] Matias Corporation. www.halfkeyboard.com, 2003. [9] R. P. McCarty. Wearables central: Softwear. wearables.blu.org, 2003. [10] V. R. Pratt. Thumb-code: A device-independent digital sign language, July 1998. [11] K. R. Prince. Finger-mounted computer input device. U.S. Patent 5,581,484, December 1996. [12] Senseboard Technologies. www.senseboard.com, 2003. [13] Stanford Wearables. wearables.stanford.edu, 2001. [14] The MicroOptical Corporation. www.microopticalcorp.com, 2003. [15] S. Vance, L. Migachyov, W. Fu, and I. Hajjar. Fingerless glove for interacting with a data processing system. U.S. Patent 6,304,840, October 2001. [16] A. Wozniak. Pic-key chorded keyboard. http://mudlist.eorbit.net/ adam/pickey/, 1999.

Finger-Thumb Contact 1-B 2-B 3-B 4-B 4-B0

Character A S D F G

(a) Left-hand base-line characters.

Finger-Thumb Contact 1-A 2-A 3-A 4-A 4-A0

Character Q W E R T

(b) Left-hand top-line characters.

Finger-Thumb Contact 1-C 2-C 3-C 4-C 4-C0

Character Z X C V B

(c) Left-hand bottom-line characters.

Finger-Thumb Contact 5-E0 5-E 6-E 7-E 8-E

Character H J K L Z

(d) Right-hand base-line characters.

Finger-Thumb Contact 5-D0 5-D 6-D 7-D 8-D

Character Y U I O P

(e) Right-hand top-line characters.

Finger-Thumb Contact 5-F0 5-F 6-F 7-F 8-F

Character N M . ?/

(f) right-hand bottom-line characters. Table 2. Prototype Key-Map for KITTY device.