A Hands Gesture System Of Control For An Intelligent

2007 4th International Conference on Electrical and Electronics Engineering (ICEEE 2007) Mexico City, Mexico. September 5-7, 2007

A Hands Gesture System Of Control For An Intelligent Wheelchair. R. Posada-Gómez1, L. H. Sánchez-Medel1, G. Alor Hernandez1, A. Martínez-Sibaja1, A. Aguilar-Laserre1. L. Leija-Salas2 1

Departamento de Posgrado e Investigación, Instituto Tecnológico de Orizaba Av. Instituto Tecnológico No.852, Col. Emiliano Zapata, 94320 Orizaba, Veracruz, México 2 Departamento de Ing. Eléctrica, Sección Bioelectrónica, CINVESTAV, México. Av. Instituto Politécnico Nacional 2508 Col. San Pedro Zacatenco, 07360 México, D.F. Phone 01(272) 7244096 E-mail: [email protected] Additionally, works have been developed in the detection of user's face for controlling the wheelchair, detecting the iris position with a webcam located in front of the user [7], as the location of iris is very dependant of the camera’s position, a user's face control is also proposed [8]. However this type of IWs needs that user sees only in the direction of displacement, and can not be used by quadriplegic people with problems at neck level.

Abstract –– When an unfortunate event affects the motor capacity of a person, it is necessary to use devices like wheelchairs that offer a means o displacement for patients with motors problems of the lower limbs. Some patients that cannot manipulate the wheelchair with their arms due to a lack of force or psychomotor problems in the superior members, request electric wheelchairs, frequently manipulated with joysticks; however the joystick manipulation is even not practical and frequently it must be handle with the mouth. The present article presents the partial results in the development of a wheelchair controlled by an intuitive interface, where the instructions are given by hand gesture instructions. The advances are presented in the realization of the control software using a Webcam and some distances and presence sensors controlled by a PIC microcontroller that establishes the communication with a program developed in Labview.

In this article are presented the advances in the development of a control based in tracking of an object, the object in question is the users hand, so the proposed algorithm of control must in first instance recognize the hand of user, a skin detection is implemented to discard other objects in the scene, then the form and position of user’s hand determine the wheelchair direction. In the same way the general structure of the proposed wheelchair is presented, results for controlling the mobile using hands gesture recognition is finally presented.

Keywords –– Digital image processing, hand gesture control, Labview, wheelchair control.

I. INTRODUCTION II. METHODOLOGY The Wheelchair is an important way of transfer for handicapped and aged people. Many researchers have been developing intelligent wheelchairs due to the increasing requirement of safer and more comfortable wheelchairs. Kinds of developed wheelchairs uses electric power (EPWs) and are controlled by a joystick [1]; when sensors and computers are added to EPWs an intelligent wheelchair (IWs) is obtained [2].

To realize this project a mobile robot has been developed (Fig. 1), in future works a real wheelchair will be used. The robot was designed using a PC for implement the algorithms of skin detection and hand gesture detection; the motor’s control as well as sensors of obstacles were implemented using a PIC 16F877A. The robot’s hardware is designed to establish computer communication; additionally the economy and availability of components were an important factor to consider. The feasibility for implement in a real system and for new algorithm implementations does the PIC an adequate platform for this project.

The IWs can include different kinds of improvements as design or operation including devices that allow to go up the IWs to a provided car of special harness where the IWs takes the most convenient position to be able to ascend to the vehicle. An array of sensors and cameras are included as much in the automobile as in the IWs [3]. Additionally, some works propose determine the IWs route analyzing the environment looking for borders of walls [4].

The software of control is developed in Labview, due to the capability of design offer by this software; One of the first stages has been the recognition of a defined form in a constrained scene to be able to control the movement of the robot trough the serial port of a computer port, the communication between PIC and PC is done by the RS232 port.

Previous works propose to control the IWs by speech [5] or even identifying the form and position of lips [6]; however both systems are susceptible of undesirable movements during a conversation while the seat is operated, or when the IWs is located in noisy environments.

1-4244-1166-1/07/$25.00 ©2007 IEEE.

68

IEEE Catalog Number: 07EX1762C ISBN: 1-4244-1166-1 Library of Congress: 2007923398


LCD 2x16

D Port Power stage

RC0 RC1 RC2 RC3

RC5 RC4 RA4

Keyboard

B Port

RC6

Wire Selector

PIC16F877A RC7

Wireless

A, B Port

Sensors

Fig. 2. Block diagram of PIC implementation. a). Image Acquisition

Skin Detection

Propose Direction b).

received. For practical semiduplex communications aims, a direct connection using an UTP 8 line was implemented. For practical simplex communication aims, a radio frequency receptor was used for communication between PC and PIC. The first implementation uses a simplex communication from the computer to the robot, the direction instructions are given in one byte as shown in Table 1. Forward versions would use a semiduplex communication so that the PC would decide if is possible to advance or not. Table 1 shows the sent word of control, bytes B6 and B7 are zero logic.

Form Detection

Form Tracking

PC Sensors N

Is propose direction obstacle free

The power stage is controlled by the microcontroller itself, to being possible to use a PWM for the motor control speed. For future implementation in a real wheelchair the PWM could be substituted by relay or a suited H bridge.

Obstacle Detection

Y

PIC16F877A

IWs Move

II.2. Software implementation.

Fig. 1. a) Robot designed to implement the control by software b) Block diagram of the robot.

The hand gesture identification system (Fig. 3.) is divides in the next stages: Image acquisition and representation, Color processing and skin detection, Image processing and segmentation, Extraction of geometric characteristics and Euclidean distance classifier.

II.1. Hardware implementation The robot has 2 kinds of analog sensors: presence sensor and light sensor and a digital sensor of bumper type. The values of analog sensors are digitalized by the PIC16F877A using an ADC microprogram. The digital sensor is directly coupled to microcontroller PORTB (Fig. 2) to generate an interruption when an obstacle has been present.

For acquiring the image a webcam is used, due its availability. Once the image is acquired, the color serve to recognize the skin [9], then the image is segmented for keep only the hand shape, the geometric parameters of object is extracted and finally compared with a database of geometric parameters to fin which control gesture is represented.

The presence sensor determines if the robot can advance without crash with an obstacle or found a stair. Light sensors establish how much light is onto the object; this can help to accelerate the vision algorithm process to determine the best way to process the intensity of light in the image. Finally the robot counts with a keypad that leaves to choose the microprogram to run from a same microcontroller, so a simplex or semiduplex communication between PIC and PC can be chosen. A LCD helps to select the microprogram to run, besides to shows the sensors value.

TABLE 1 BYTE ARRAY SENT AND RECEIVED B# B0 B1 B2 B3 B4 B5 B6 B7

The computer communication is done with the USARTRS232 protocol because the PIC16F877A has special interrupts for determining when a byte is send or

1-4244-1166-1/07/$25.00 ©2007 IEEE.

69

Sent Byte Frontal sensor Rear sensor Left sensor Right sensor Left counter Right sensor Useless Useless

Received Byte Direction PWM 1 PWM 2 Useless Useless Useless Useless Useless



II.2.C. Geometric recognition and direction Image acquisition Real world scenario

Image processing

Database

Object recognizing

Representation and description

The localization of an object is made by a geometric matching. When the hand is isolated of the scene, the next stage is to obtain a vector of patterns or characteristic that represent to each sign, these will be called discriminants. The Hu invariants were chosen as discriminants, because they are very useful in the recognition for two dimensional images, since they are robust under transformations like scale changes, rotations and translations.

Color processing

Morphologic processing

The Hu invariants derive of the geometric moments, for the case of a continuous function f(x, y), the one first order moment (p+q) is given as:

Segmentation

∞

Fig. 3. Block diagram of the hands gesture recognition system.

m pq =

When a valid object is located (corresponding to an object in database) their coordinates are determined in the created image. Once the coordinates determined, the limits of the region are establish in the image. Finally the direction through the object is moving define the direction to take by the robot (up= forward, down=backward, right= turn right, left= turn left).

∫x

p

−∞

y q f ( x, y ) dxdy

(1)

For p,q = 1,2,3,4… the infinite group of moments determines in unique form each function f(x,y), in a reciprocal way to the group of moments (mpq) determine an only function f(x,y), then the central moments was defined as:

µ pq =

II.2.A. Image acquisition

∞ ∞

∫ ∫ ( x − x) ( y − y ) p

q

dxdy

−∞ −∞

To acquire an image in Labview, the method shown in the Fig 4 was used; which consist of the next stages: • Generate a list of available cameras. • Choose a camera by means of a index. • Open a session and configure it.

x=

Subsequent a new image is created in which the webcam generated will store the image taken and a snap is taken of it, where the object will be recognized.

This corresponds to the center of mass of the object. And for the case of a digital image they are

II.2.B. Skin detection

µ pq = ∑∑ x − x

In this case:

m10 m ; y = 01 m00 m00

x

Skin detection in images is based on probability distributions in color spaces [9]. Typically the decision for whether a pixel belongs to a skin region is made on the basis of that pixel's color-value alone. A large set of training data is created by labeling skin and non-skin regions of many regions. Then an estimate of the color probability distribution is made from the training data, converting the RGB image in an HSV (separate dimensions of Saturation, Hue and Value of Intensity) color representation because it shows a better continuity. The Hue and Saturation data are pull out into new separate variables and only scene values that are located within the pre-defined ranges associated with target skin values are indicated as being skin, the range of Hue and saturation is fixed as 0- 0.11 and 0.2-0.7 repectively.

1-4244-1166-1/07/$25.00 ©2007 IEEE.

(2)

y

(

(3)

) ( y − y) p

q

f ( x, y )

(4)

Where (x, y)∈ R, being R the region of the object of interest. Once the form identified in image, the x,y coordinate’ mass center is took as the position of image, then a change in x,y coordinate values will indicate the movement of the recognized object. Fig 3 shows the correct form to implement this tool. Being the output of this the position in which the object of interest is found. Once the object is located, by means of an overlay is locked to observe the coordinates of the object founded using the coordinates in the matching.

70



Fig. 4 Machine vision program

recognize object where the success depends of the angle of contraction and light conditions that affect the image captured bye the webcam. This algorithm both with Labview presents an optimal way of control machine vision because is easily to configure it. Until this moment geometrical shapes has been recognized successfully, however when similar objects are in the scene the system tends to be unstable, so it’s necessary to develop a robust algorithm for duplicity of object of interest.

II.2.D. Sending data by the RS232 port Once the direction is determined the data must be organized to be sent to the robot. To locate the region of the object of interest first is asked if there isn’t any correspondence; if exist one, the declaration would be false and depending the coordinates of the match this will be send the data. Once the array of bits is set, is necessary to send it trough the protocol RS232, so the port must be configured to 9600 bauds, without parity, 1 stop bit. For it the data are sent assuring the MSB nibble is set 0 logic, however at being despicable and the robot doesn’t recognize this part of the array is ignored by the robot, but the array it must be filled in order to complete the bye.

REFERENCES [1]. Amundson JS, Amundson SG., “A joystick controlled wheelchair.”, Biomed Sci Instrum . 1991;27:131-3. [2]. T. Gomi and A. Griffith, “Developing intelligent wheelchairs for the handicapped,” Lecture Notes in AI: Assistive Technology and Artificial Intelligence, Springer, vol. 1458, pp. 150– 178, 1998. [3]. Humberto semeno-Villalta and jhon Spletzer, “Vision-based Control of a Smart Wheelchair for the Automated Transport and Retrieval System (ATRS)”, p.1, Department of Computer Science and Engineering Lehigh University, 2005. [4]. Trahanias, P. E. and et al. 1997. ”Navigational support for robotic wheelchair platforms: an approach that combines vision and range sensors” Proceedings of the 1997 IEEE International Conference on Robotics and Automation, Albuquerque, NM, 1265-70. [5]. Pacnik, G., K. Benkic and B. Brecko, Voice operated intelligent wheelchair — VOIC, Proc. of the IEEE International Symposium on Industrial Electronics, vol.3, pp.1221-1226, 2005. [6]. Ulrico Canzler, Karl-Friedrich Kraiss, “Person-Adaptive Facial Feature Analysis for an Advanced Wheelchair User-Interface”, Conference on Mechatronics & Robotics 2004, Volume Part III, pp. 871-876, September 13-15, Aachen, Sascha Eysoldt Verlag. [7]. L.M. Bergasa, M. Mazo, C. T. San Juan, J. A. Herradas. “Guiado de unmóvil mediante los movimientos oculares”. I Jornadas de Inteligencia Artificial, Control y Sistemas Espertos, E.U. Politécnica. Universidad de Alcalá, pp 25-33. 1996 [8]. P. Jia and H. Hu (2005), “Head Gesture Based Control of an Intelligent Wheelchair”, p.1, The 11th Annual Conference of the Chinese Automation and Computing Society in the UK (CACSUK05), Sheffield, UK,10 September 2005. [9]. M. Jones and J. Rehg: Statistical color models with application to skin detection. Computer Vision and Pattern Recognition (CVPR), Pages 274-280, 1999.

III. RESULTS Table 2 shows the time that take the system to recognize the object of interest, this times are taken once the camera is already on (the time for the recognition of the first object at the first acquisition is 1 second). The system has limits for the contraction angle, in future work the aim will be recognize an object spite its perspective transformation. IV. CONCLUSION It has been presented the first advance to a IWs controlled by the acknowledgement of an object using the technique of geometric recognition, which is useful to TABLE 2 Angle contraction vs. Time Angle of contraction −30 −20 −10 0 +10 +20 +30

Time (ms) 200 202 191 197 194 192 191

1-4244-1166-1/07/$25.00 ©2007 IEEE.

71
