Assistive Glove for Pakistani Sign Language Translation Pakistani Sign Language Translator Kehkashan Kanwal', Saad Abdullah+, Yusra Binte Ahmed¥, Yusra Saher'l' & Ali Raza Jafri§ Department of Biomedical Engineering NED University of Engineering & Technology Karachi, Pakistan '
[email protected],
[email protected], ¥
[email protected], "'
[email protected], §
[email protected]
Abstract- This paper presents system to translate gestures of Urdu Sign language using an instrumented wearable glove. This system contributes first ever attempt in terms of fabrication of Pakistani Sign Language translating glove which is portable
for glove fabrication, sensors mounting on glove and methodology used for our project is described then the results and discussion are given in section IV, followed by conclusion in section V.
as well as cost effective. As the sensor values from the glove vary from person to person, this system was made to use pattern
II. LITERATURE REVIEW
recognition approach. In order to accomplish the task, Principal
Gesture recognition has found its major application in this area to enable deaf and mute people to communicate others with more ease. Tremendous amount of work has been done worldwide for sign language translation. However no such work could be traced for Urdu Sign Language and approximately 0.2 Million deaf and mute Pakistani individuals have no assistive device at their disposal. The aim of the designed system was to bring this technology for deserving people of Pakistan. There are two major technologies that are being used for hand posture or gesture recognition. One is computer vision based approach that takes images of signer and translates them using image processing protocols. The other is use of sensor based glove [5]. Our system has been devised using the later approach.
Component Analysis (PCA) was employed for feature extraction and Euclidean distance as classification technique. Up till now the system has ten static gestures in its library and it perfectly judges nine gestures out of targeted ten gestures which are commonly used in Urdu Sign language.
Keywords-Instrumented
glove;
Pattern
Recognition;
Principal component Analysis; Euclidean Distance; Urdu sign language
I. INTRODUCTION Visual communication methods have been around for thousands of years. Nowadays there are hundreds of different types of sign languages in use, that is a language which instead of acoustically conveyed sound patterns, uses visually transmitted sign patterns to convey meaning by simultaneously combining hand shapes, orientation and movement of the hands, arms or body, and facial expressions to express fluidly a speaker's thoughts[J]. No international system is all inclusive that provide encyclopedic way for deaf people communications across the world. There exists American Sign Language, British Sign Language, Spanish Sign Language, and probably sign language in every country in order to carry out communication efficiently and rapidly among deaf community and normal people without use of paper and pencil [2]. According to World Federation of the Deaf, approximately 72 Million people in the world [3] are experiencing hearing impairment problems and according to Pakistan association of deaf over 0.2 Million people are deaf in Pakistan [4]. The deaf individuals in Pakistan use the Pakistani Sign Language (PSL) to communicate. Like all other sign languages, PSL follows the rules of linguistics; similar to the spoken Urdu Language, it has its own grammar, letters, words, expression, sentence structures, distinct vocabulary of distinct signs and syntax subject to improvement and growth, like any other sign language system around the world. With the blend of other regional languages (Sindhi, Pushto, Punjabi, Balochi), it has many regional variations. PSL has evolved over time and now developed into a full-fledge language. The paper is organized as follows: in section II, we have described the reviewed literature and the work done for Sign Language translation globally. In section III the material used ISBN:
978-1-4799-5754-5/14/$26.00 ©2014
IEEE
Mohamed A. Mohandes used two CyberGloves, two hand tracking devices and data acquisition system to take signal for each hand gesture [6]. He took 56 sensor values from both hands and divided duration of sign into ten segments. For each segment mean and standard deviation of sensor values was calculated. Owing to large data set, he used Principal Component Analysis for feature extraction and formulated Support Vector Machine for classification. Olguin et al. [7] used P5 virtual reality glove to fabricate Mexican language interpreter. Kramer [8] used patented CyberGlove and a look up table to recognize English alphabet. Jose et al. [9] used Accele glove bearing accelerometers to design pattern recognition system to translate American Sign Language. Shahabi et al. used cyber glove for haptic user interface and C4.5 Decision Tree, Bayesian Classifier and Neural networks for classification of data in order to translate ASL to spoken words [10]. Waldron and Soowon used DataGlove for obtaining hand shape and position data and used them as input to two stage neural network [11]. In 2012 Ukarine based students designed a glove called EnableTalk and won Microsoft: Imagine cup. They used an instrumented glove to record the values of sensors and windows phone application to recognize each gesture [12]. III. MATERIALS AND METHODS The system explained in this paper was designed with an aim to cater variation in hand sizes of different individuals; i.e. design a universal glove that could translate sign language through hand of every individual who wears it.
173
The next issue was portability. Almost all gesture recognition systems using computer vision approach requires some camera attached computers to take images of the hand and to judge the appropriate gesture. Therefore glove based system was designed that seems feasible and comfortable to user and thus all the necessary circuitry is designed in such a way that it is worn by user using wrist and chest belt. Moreover our target was local population of Pakistan so instead of going with expensive devices, we preferred cost effective criteria for components selection. A.
II
Proposed System
The glove was designed using artificial leather and net. The material was chosen to get balance of stretchiness and stiffness to accommodate various hand sizes. Five flex sensors were used to obtain information regarding bending of fingers and a triple axis accelerometer to get data about position of hand in space. With help of these eight variables, PSL was translated into audio and visual outputs. Smart phone application was avoided. Developing a smart phone application is relatively easier to cater the task, however since we targeted the local population of Pakistan; it was ensured that majority of these individuals don't have access to smart phones or they can't use it efficiently. Therefore the system uses GLCD for the visual outputs and a playback module for the audio output. The flex sensors were being sown on each finger of the glove while the accelerometer MMA 7361 was sown just in the middle of the glove. The glove design i s represented in Fig. 1. The flex sensors were used in VDR configuration. Arduino Mega 2560 was used for processing. This microcontroller takes input from the eight sensors, digitize them, and drive GLCD and playback module WTV020 SD in order to display and playback the appropriate word as signed by the user.
Fig. 2.
Final Hardware
Once the glove was fabricated, the flex sensors were calibrated on hand of 10 different individuals (5 males and 5 females) to obtain minimum and maximum flex sensor voltages when a person fully bends and stretches the fingers of hand. These values were averaged and the mean was then used to get all readings from the flex sensor. This calibration and averaging was first step to normalize variation in hand sizes that exists naturally among different people. Initially only 10 gestures were taken from PSL since the aim was to get best possible accuracy and obtaining universality. Sampling was then performed for these selected gestures. Samples from 30 different individuals (15 females and 15 males) were taken. The ages of the subjects varied from 13 years to 45 years. For each sign, the set of all sensor values were being taken and their average was sued as sensor reading of the individual for that particular sign. So the master data set had dimensions of 300*8. Where the rows correspond to observations (30 individuals and 10 gestures, 30*10=300) and the columns correspond to number of sensors (variables). Another data set has been constructed in which mean of sensor values from all 30 individuals for each sign was calculated. B.
Feature Extraction
Any pattern recognition system first generates a description of the system and then classifies different objects based on the description. Sampling yielded 2400 data points in original data set therefore to reduce dimensions feature extraction was employed. Many algorithm choices were available for feature extraction. Principal Component Analysis (PCA) has been used for the described system. PCA seeks to reduce dimensionality while preserving as much of the class discriminatory information as possible [13].
Fig. I.
174
Glove Design
PCA coefficients of training set were obtained. The input to PCA was 300*8 (m*n) data Matrix in which rows correspond to observations i.e. 300 for training set and columns to variables i.e. 8 in glove. Output of PCA was 8*8 (n*n) matrix of PCA coefficients. These coefficients were used to transform both the saved mean of sensor values for each gesture and real time inputs. Equation (1) & (2) summarize the transformations.
Transformed set for saved gesture = saved mean of gesture * PCA coeficients
(I)
�
Transformed real time input = real time input * PCA coeficients
(2)
Principal component analysis (PCA) is a mathematical procedure that uses orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components. MATLAB 7.9.0 (R2009b) was used for calculating PCA coefficients. MATLAB has built- in commands to calculate these coefficients. Princomp command was used that calculates covariance matrix of the data and takes its Eigen values [14].
Obtain data from sensors of glove and save them in matrix [1*8]
l Multiply matrix data with obtained PCA coefficients [1*8 x 8*8]
, Subtract above matrix from transformed mean of gesture
After feature extraction we had 64 data points (PCA coefficients of dimension 8*8) that could fully represent original data set.
� Applying Euclidean Distance formula to fmd relevant gesture
Calibration (Min & Max Values of flex sensors)
, Display the corresponding gesture on GLCD and play it through audio module
,
Sampling (For 10 gestures of PSL on 30 individuals [300*8])
-
Delay
Fig. 4.
Real Time Operation
IV. RESULTS AND DISCUSSION
Pattern recognition (PCA coefficients [8*8] & Transformation of saved mean of gestures [1*8])
Fig. 3.
Project Development
C. Classification The PCA coefficient matrix and these transformed saved means were used in Arduino sketch for classification of real time inputs. For gesture recognition, each real time input is transformed using the transformation explained above and calculated Euclidean distance of each transformed saved mean along to transformed real time input. The smallest Euclidean distance corresponded to most valid gesture. E. D
=
�xf
+
x�
+
xj
+
xl
+
xg
+
x�
+
x�
+
x�
(3)
In (3); xl through x5 are the subtracted values of five flex sensors and x6 through x8 are the x, y, z subtracted coordinated for palm orientation. Real time operation of our system is shown through flowchart in fig. 4:
The results of this project yielded desired task after finalizing the mounting of all the components, modules and securing all the remaining connections of the project design. When an individual made any one of the gestures, the trinuner circuit converted the resistance of flex sensors into voltages and the output from the accelerometer were sent to the analog pin of Arduino. The audio module and GLCD (for the audio and visual output respectively) were connected with digital pins of Arduino. When the program was executed, it transformed each real time input using the transformation method, Principal Component Analysis (PCA), calculated Euclidean distance of each and saved its mean along with the transformed real time input. 'The smallest Euclidean distance corresponded to most valid gesture. Simultaneously controller triggered the audio module and GLCD, so that the statement corresponding to the gesture appeared on the GLCD, as well as played through speaker. The system detected the nine gestures accurately and misclassified only one of them, during each test. Thus, nine out of ten gestures were recognized accurately. As shown by (4), (5) & (6); %Error
=
%Error
=
%Error
=
Actual value-Measure value Actual value
10
-9 X
10
100
X
100
(4) (5)
10%
175
O/OAccuracy
=
100
%Accuracy
=
900/0
-
%Error
(6)
We realized that the cause of misclassification was very high similarity in values pertaining to those gestures; they were exceedingly comparable in shape, with the only difference of orientation of hand. Also, the total sum of sensor values was nearly equal.
grateful to Mr. Bilal Ahmed Usmani (Asst. Prof. NEDUET), who helped us to understand basic concept of Principal Component Analysis and different aspects of its application. REFERENCES [I]
Sign language and deaf communication methods and infonnation. Disabled-world website. [Online] [Cited: 10 10,2013]. Available: http://www.disabled-world.com
[2]
Chapter Three Bolty Hath. Yumpu. [Online] 2013. [Cited: II 5,2013]. Available: https:llwww.yumpu.com
[3]
Deaf people and human right report. World Federation of Deaf Website.
[4]
Deaf Statistics. Pakistan Association of the Deaf [Online] [Cited: 10 15,
[5]
Laviola, Jr. Joseph J. A survey of hand gesture and posture recognition
[Online] [Cited: 10 15,2013] Available: http://www.wfdeaforg 2013]. Available: http://www.padeaf.org/deaf_statistic.html
II
techniques and technology. Rhode Island: Department of Computer Science, Brown University,1999. [6]
Mohandes, Mohamed A. Recognition of Two-handed Arabic Signs
[7]
Olguin-Carbajal Mauricio, Zarte-Rivera Israel, Pimentel-Cruz Jesus.
using the CyberGlove. 2010. pp. 124-129. Language Interpreter for the deaf core using a virtual reality data glove, Journal of Theoretical and Applied Information Technology, 30th November,2011. pp. 135-141. [8]
J. Kramer, L. Leifer. The Talking Glove; An Expressive and Receptive "Verbal" Communication Aid for the Dead, Deaf-blind and Nonvocal, SIGCAPH,1988. pp. 12-15.
[9]
Jose
L.
Hernimdez-Rebollar,
Robert
W.
Lindeman,
Nicholas
Kyriakopoulos. A Multi-Class Pattern Recognition System for Practical Finger Spelling Translation. Pittsburgh, 2002. pp. [85-190. [10] C. Shahabi, L. Kaghazian, S. Mehta, A. Ghoting, G. Shanbhag and M. McLaughlin. Analysis of Haptic Data for Sign Language Recognition. Fig. 5.
Detectedgesture on GLCD
V.
CONCLUSION
Deaf individuals are members of language minority. They possess different kind of communication ways. This article explains fabrication of an assistive device for deaf and mute individuals that can help them to communicate with others with more ease. The novelty of the project lies in the fact that it interprets Pakistani Sign Language (PSL) whereas most of the work done in sign language translation targeted American Sign Language (ASL). This assistive glove do not need to be connected with computer, also portable display along with bearable weight of electronic circuitry board makes it utterly portative. It is easily adjustable and repairable too. Moreover most currently made devices like EnableTalk use smart phone applications for audio and visual outputs. We considered the fact that majority of Pakistan's population do not have economic power to buy such phones or are unable to use them efficiently. Therefore the device has been designed in such a way that it is cost effective and also operates stand-alone without use of any external independent device (e.g. Mobile phone). ACKNOWLEDGMENT We would like to acknowledge with much appreciation the crucial role of Eden Lion Club (Faisalabad) and Mr. Adnan Azam Solatch for providing us funds for the project. We would also like to thank all the students of LEJ campus and faculty members who patiently participated in our anthropomorphic data collection. We would also like to thank Mr. Iftikhar Imam Naqvi (Former Chairperson Department of Chemical Sciences and Dean Research, Jinnah University for women, Karachi) who helped us in gaining access to his department since we needed students for sampling of the project. We would also like to thank Mr. Mohsin Tiwana, Ms. Sumera Munir and Ms. Maheen Rafique for their valuable suggestions for fabrication of glove; which is the backbone of our project. We would like to thank Team SignSpeak (NUST) for their recommendations regarding pattern recognition approach. We are extremely 176
New Orleans, Louisiana: 1st
International Conference on Universal
Acces in HCI,20 II. pp. 441-445. [11] Manjula B. Waldron, Soowon Kim. Isolated ASL Sign Recognition System
for
Deaf
Persons.
S.1.
Rehabilitation
Engineering, IEEE
Transactions, September 1995. pp. 261-271 [12] Enable
Talk.
Enable
Talk
website.
[Online]
Available:
http://enabletalk.com. [13] S. Balakrishnama, A. Ganapathiraju. Linear Discriminant Analaysis; A brief
Tutorial.
The
music.
Mcgill
website.
[Online]. Available:
http://www.music.mcgill.ca [14] Department of Statistics. Carnegie Mellon University Website. [Online] [Cited: 4 11,2014]. Available: http://www.stat.cmu.edu
