Android Application for handwriting segmentation

Android Application for handwriting segmentation using PerTOHS theory Hanen Akouaydi Sourour Njah Adel M. Alimi REGIM-Lab: Research Group on Intelligent Machines Laboratory, University of Sfax, National School of Engineers, BP 1173, 3038. Sfax, Tunisia { hanen.akouaydi.tn, sourour.njah, adel.alimi} @ieee.org Abstract The paper handles the problem of segmentation of handwriting on mobile devices. Many applications have been developed in order to facilitate the recognition of handwriting and to skip the limited numbers of keys in keyboards and try to introduce a space of drawing for writing instead of using keyboards. In this one, we will present a mobile theory for the segmentation of for handwriting uses PerTOHS theory, Perceptual Theory of On line Handwriting Segmentation, where handwriting is defined as a sequence of elementary and perceptual codes. In fact, the theory analyzes the written script and tries to learn the handwriting visual codes features in order to generate new ones via the generated perceptual sequences. To get this classification we try to apply the Beta-elliptic model, fuzzy detector and also genetic algorithms in order to get the EPCs (Elementary Perceptual Codes) and GPCs (Global Perceptual Codes) that composed the script. So, we will present our Android application M-PerTOHS for segmentation of handwriting. Keywords: PerTOHS theory, Perceptual codes, On-line handwriting segmentation, Beta-elliptic model, Fuzzy theory, Genetic algorithms, Android, PHP.

1. INTRODUCTION The excessive use of handheld devices which accept handwritten data as input created a demand for applications which analyze and recognize data efficiently. New applications are appearing to eliminate the using of keyboards and facility the storing of users notes. These applications have attracted the intention of many users but some require that notes should be in computerized form. Handwriting recognition is one of challenging fields in research in image processing and pattern recognition. In this respect fits our work, which consists to develop an Android application for handwriting segmentation and recognition. The main developed part in this paper is to present a new perceptual mobile theory entitled M-PerTOHS which use PerTOHS theory, Perceptual Theory for On-line Handwriting Segmentation, based on the fact that our human perceptual system is based on basic features called perceptual codes divided elementary and global ones. This new approach for improving handwriting segmentation via perceptual codes used the Beta-elliptic model for the generation of on-line handwriting scripts, the fuzzy set theory to detect the elementary perceptual codes and the genetic algorithms for the global perceptual ones [1] [7]. The organization of this paper is as follows: in section 2, we present an overview of mobile applications handwriting. In section 3, we describe the developed Android application for handwriting based on PerTOHS theory. Then, we detail elementary and global perceptual codes used to segment handwriting. In section 4, we examine some experimental and simulation results on on-line handwriting. Finally conclusions and future works are mentioned.

2. OVERVIEW Handwriting is a human skill that facilitates communication. It is artificial and graphical marks on a surface. Humans can read and understand any type of writing in different constraints. In order to create a system as the human one, that recognizes handwriting in any condition, it is important to study the process of handwriting. Interfaces of mobile phone are based mainly on keyboards or on special pens for writing. Those approaches of writing are sometimes cumbersome. To get a desired output, you have to do several presses. The limited numbers of keys in keyboards and their smallness harm the comfort of users. Many techniques are used to analyze this phenomenon. Some studies were made in order to understand mechanism of handwriting. The aim of this part is to introduce general theoretical approaches in mobile applications. In the literature, we mark the existence of different techniques treating the handwriting [5].

There are different methods for on-line handwriting recognition has been proposed in the literature. Dynamic time warping (DTW) is one of the most popular solutions for recognizing handwritten characters But, it is a computationally quite an expensive approach for a mobile phone, where the computational resources are limited [4] [5]. There are many approach for handwriting recognition, for example Zhengxing Sun has an adaptive online multi-stroke Recognition approach based on Hidden Markov Model (HMM) and Chebanyuk O.V. proposed handwritten recognition based on Analysis of motion vectors. The majors applications of recognition are used in the field of storing contacts and sending messages, they give to user’s suggestions of the word or letters they write or as applications for storing notes they transform them from handwritten form to digital form. Others mobile applications try to recognize some special languages as Myanmar, Tamil, Hindi [2]. As example the application “Multilingual Online Handwriting Recognition System” which uses the Artificial Neural Network as technique for recognition. It allows the user to select any language from the languages supported. By digitization of the pen/stylus movements as an array of the (x, y) coordinates which called stroke, the handwriting is collected. It extracted features putted in feature vector that are passed as input to an artificial neural network (ANN) classifier. This classifier compares the input data with the trained data and finds the nearest prototype from the database [2]. Although many techniques have been proposed to improve text input on touch screens, the majority of this research ignores handwritten input for Arabic language. In the proposed system, segmentation is done by using Perceptual theory for On-line handwriting segmentation. The recognition consists of generation of character schema and searching of the character in knowledgebase by analyzing the user input. In order to develop an android application treating the case of handwriting we propose a system based on a perceptual theory for handwriting segmentation.

3. ANDROID APPLICATION FOR HANDWRITING As mentioned, the aim of this part is to present an Android application theory for mobile handwriting M-PerTOHS. The Android application uses the PerTOHS theory which based perceptual one for the segmentation of handwriting. So, we will present it in the next section. 3.1 PerTOHS: Perceptual theory for On-line handwriting segmentation PerTOHS, Perceptual Theory for Online Handwriting Segmentation. It is a perceptual theory for handwriting segmentation is based on the following facts: all we perceive is a form, all we see are forms, and handwriting is forms. Our perceptual theory considers that handwriting is a special form. So we divide the processing of recognition it in three steps: sensation, perception and recognition. The first step is the sensation of the script, corresponding to the acquisition data step. Then, we detect the perceptual codes composing handwriting in the step of perception. By the segmentation of the script, we detect global perceptual codes (GPCs) composed of sets of elementary perceptual ones (EPCs), which correspond to the step of segmentation in the developed system. This step of perception is divided into two layers: in the first one, we detect the elementary perceptual codes (EPCs), and in the second one we detect the global ones (GPCs). In the stage of recognition, the set of perceptual codes detected must be gathered in the brain in order to recognize the script [1] [3]. Handwriting is defined as a sequence of perceptual global codes (GPCs) and elementary perceptual codes (EPCs) as illustrated in figure 1. We note that GPCs are the combination between EPCs in different sizes and orientations. After analysis on reading-writing models we have 4 EPCs, and 10 GPCs.

Figure 1. Perceptual Codes: EPCs and GPCs.

PerTOHS theory proposes that handwriting is a graphical mark that composed from perceptual codes grouped together in order to obtain a character. Beta-elliptic model, an on-line script is characterized by Beta-elliptic parameters and segmented into different strokes. These strokes are modeled with elliptic arcs [1] [3]. With the elliptic strokes we will classify the handwriting into elementary perceptual codes by using the extracted parameter θ which corresponding to the deviation angle of each ellipse by the horizontal axis. The beta-elliptic model consists of the detection of the perceptual codes. For each elliptic stroke, we assigned an Elementary Perceptual Code (EPC).According to the to the deviation angle θ, we have identified four types of strokes: Shaft, Valley, Left oblique shaft, Right oblique shaft). The number of strokes of an appropriate handwriting trace is predefined by the Beta-elliptic model and this latter is equal to the number of EPC [1] [3]. After segmented the on-line script into sequence of elliptic strokes, we will apply the fuzzy detector which associate with each elliptic stroke an EPC with a certain membership degree. As we say the handwriting is a sequence of EPCs and by the combination of this one, we obtain global ones GPCs (Global Perceptual Codes). A GPC is a combination of a set of elementary perceptual codes (EPCs) according to well defined criteria. In other words, just like a script is defined as a sequence of GPC, each GPC may be defined as an alternating sequence of EPCs. These different GPCs are the bases to identify and recognize handwriting. So to detect these GPCs from an on-line script, we will use genetic algorithms. For example, figure 2 shows the result of applying genetic algorithm in order to obtain GPCs composing the letter “L” [1] [3].

Figure 2. GPCs of letter “L”.

3.2 M-PerTOHS: Mobile-Perceptual Theory for On-line Handwriting Segmentation We develop an Android application. For Android, it is a mobile operating system developed by Google. It used in touchscreen mobile devices, such as cell phones and tablets. Its design allows users to manipulate mobile devices intuitively, with phone interactions such as pinching, swiping, and tapping [6]. Interested on developing Android application for mobile handwriting, we study the technology solutions available to use for handwriting recognition on the mobile phone. This operating system is used in research domains such as recognition handwriting, image processing, optimization… Android platform, adapted for those domains libraries: iTk (for medicine), Gesture (handwriting and motion detection). After having done a state of art about the techniques using in handwriting recognition, we note that handwriting segmentation is an important step for handwriting multi-language recognition systems, we proceed by creating an Android application which send the (x, y) coordinates of the handwritten writing to MATLAB algorithms in order to get the EPC form of the input. Then the output of the MATLAB algorithms, this one will send to Android in order to be shown. Our mission consists in developing an Android mobile using PerTOHS theory [1], so we draw in canvas, space of drawing, then we send the (x , y) coordinates of o touched up or touched down the on screen and we send 0 if it’s touched up and 1 if touched down, so we have this file. In the next step, HTTP connection is created between Android application and the server and we store the file by calling PHP script. The server receives data from Android under String form and stores the (x, y) coordinates in text file. Besides, we call MATLAB algorithms in order to construct a plot from the (x , y) coordinates stored in the text file and then we try to apply the PerTOHS theory in order to detect EPCs, the Elementary Perceptual Codes, by applying the Beta-elliptic model and the fuzzy perceptual detector and also genetic algorithm to detect GPCs (Global Perceptual Codes). The application currently runs on tablets with Android version 4.4.2 or higher and the machine were located the server should install MATLAB. The different steps of the proposed system are illustrated in figure 3.

Figure 3. Architecture of the Android application for handwriting.

Since M-PerTOHS algorithms would require high computation complexity so, the project implemented is based on client-server communication model [5]. The client side, Android Device, is in charge of getting the matrix of handwritten text which drawn by user and displaying result to user, while the server side is mainly responsible for most processing jobs. The major stages in the pipeline include: First, we get matrix of written script: Using Android device to get matrix of script written and send it to the server. In the next step, several processing methods are applied to the matrix. These methods include the Beta-elliptic model for the generation of on-line handwriting scripts, for each elliptic stroke, we assigned an elementary perceptual code (EPC). According to the to the deviation angle θ, we have identified four types of strokes: Shaft, Valley, Left oblique shaft, Right oblique shaft) then the fuzzy set theory to detect the elementary perceptual codes and the genetic algorithms for the global perceptual ones. Finally, we get the GPCs of the handwriting.

4. EXPERIMENTATION To experiment the proposed application, more than 100 examples are produced. We tested on various examples such as digits, letters, words in Arabic and Western language. It gives 80% of optimized GPCs. We deal with the variations of the writing style of different individuals: so we tested with different kinds of persons. Our developed Android application was evaluated on a Samsung Galaxy J1Ace and on Hexabyte tablet with version of Android 4.4.2. The average duration to write digits and letters was around 2 seconds and the communication between server and device was around 1minute to have the final output. A flow chart for the pipeline of the interfaces of the application is shown in figure 4: the segmentation of the Arabic letter “‫”س‬. This letter “‫ ”س‬is written by 3 different segments where a segment is a part of handwriting handling between hand touched down and hand touched up. It is composed of three successive GPCs which are “Up half occlusion” in different size.

Figure 4. Interface of M-PerTOHS: Segmentation of letter “‫”س‬: (a) Script wrote by the user, (b) Initial form, (c) Elliptic strokes of letter “‫”س‬, (d) EPCs of letter “‫”س‬.

In figure 5, we illustrate the segmentation of the digit “5”, which is composed by three GPCs corresponding to “Valley”, “Shaft” and “Left half occlusion”.

Figure 5. Interface of M-PerTOHS: Segmentation of digit “5”: (a) Script wrote by the user,(b) Initial form ,(c) Elliptic strokes of digit “5”,(d) EPCs of digit “5”.

Figure 6 shows the segmentation of the character “V”. The letter “V” is written by 1 segment. This letter is composed by the two GPCs which correspond to “Right oblique shaft” and “Left oblique shaft”.

Figure 6. Interface of M-PerTOHS: Segmentation of letter “V”: (a) Script wrote by the user, (b) Initial form, (c) Elliptic strokes of letter “V”, (d) EPCs of letter “V”. .

5. CONCLUSION AND FUTURE WORKS In this paper, we present an Android application treating the field of handwriting based on a new M-PerTOHS theory for segmentation of mobile handwriting which uses PerTOHS theory. This theory is based on the fact that all we perceive is forms even handwriting, so handwriting is a defined as a sequence of elementary perceptual codes (EPCs) composing global ones (GPCs). To validate this Android application and M-PerTOHS theory, we opt to test it on different languages and writers, so they obtained encouraging results which are similar to those produced by the human perceptual system. As future work, we go to develop a multilingual Android application for handwriting recognition which recognizes notes of users based on M-PerTOHs theory.

References [1] S. Njah, M. Ltaief, H. Bezine and A.M.Alimi, “The PerTOHS Theory for On-Line Handwriting Segmentation”, IJCSI International Journal of Computer Science Issues, Vol. 9, 142-151, (2012). [2] T.R. Indhu, V. Vidya and V. K.Bhadran,“ Multilingual Online Handwriting Recognition System: An Android App”, 5th International Conference on Advances in Computing and Communications (ICACC), IEEE, 33 -36, (2015). [3] S. Njah, H. Bezine and A.M. Alimi, “ A fuzzy-genetic system for segmentation of on-line handwriting: application to ADAB Database”, IEEE Symposium Series on Computational Intelligence, 5th IEEE International Workshop on Genetic and Evolutionary Fuzzy Systems, Paris-France, 95-102, (2011). [4]: W. Du, “Code Runner: Solution for Recognition and Execution of Handwritten Code”, Standford University, 1-5, USA (2012). [5]: J. Hannuksela, P. Sangi and J. Heikkila, “Motion-Based Handwriting Recognition for Mobile Interaction”, University of Oulu, Finland, IEEE, (2006). [6]: http://www.investopedia.com/terms/a/android-operating-system.asp. [7]: H. Bezine, M. Kefi, A.M.Alimi, “On the Beta-elliptic model for the control of the human arm movement”, International Journal of Pattern Recognition and Artificial Intelligence (IJPRAI), Vol 21, n°1, 5-19, (2007).