Handwriting Command Recognition and Digital ...

D

Journal of Communication and Computer 13 (2016) 164-170 doi:10.17265/1548-7709/2016.04.003

DAVID

PUBLISHING

Handwriting Command Recognition and Digital Operation Using Digitalized Pen Naoya Toyozumi, Junji Takahashi and Guillaume Lopez Department of Integrated Information Technology, Aoyama Gakuin University, 1-10-5, Fuchinobe, Chuo-ku, Saganihara, Kanagawa, 252-5258, Japan Abstract: We aim to realize a novel pen based interface system that provides various digital operations for both low-tech and high-tech people. In this paper, we developed an algorithm that associates drawn shapes, such as a circle and a rectangle, with operating commands, such as message input, address input. Using a commercially available digital pen, we confirmed that our proposed command recognition and operation algorithm worked well and outperformed usuale-mail sending operation in time. As a result, the accuracy of the command recognition is 93.8% and the response time of 90% strokes are within 100 ms. Key words: Digital operation, shape recognition, human computer interface, handwriting.

1. Introduction ICT (information and communication technology) has provided more convenient life to us with their development. Recently, SNS (social networking service) such as LINE, Twitter and Facebook, has become part of our life. However, some people cannot use digital services and systems efficiently. One capital reason is that they feel difficulty to use conventional interface systems, such as a keyboard, a mouse, a touch-pad, a flick-input device, and so on because learning them takes long time. So, we focus a spotlight again on hand-writing motion, which was invented as soon after the birth of human race. Many people in the world have basic literacy skills, because anyone is able to master literacy skills as long as he/she receives enough education. As a user interface writing with a pen has many advantages for outputting ideas. First, a pen is good in portability. Second, it excels in instancy. Third, it excels in intuitiveness. So, if digital operations such as talking-memo, file manipulation, mail-sending, tweeting, and so on, Corresponding author: Junji Takahashi, Ph.D., assistant professor, research fields: mechatronics, distributed autonomous robotic systems, factory automation, and bio-signal processing.

could get executed using a pen based interface, not only low-tech people but also high-tech people can take advantage of it. Therefore, we aim to realize a novel pen based interface that provides various digital operations for both low-tech and high-tech people. The rest of this paper is as follow: Section 2 presents related works of this paper; Section 3 provides a quick overview of our concept of pen based interface system that enables us to enforce digital operation such as, talking a digital memo, sending an e-mail or a short message, and tweeting by just writing a correspond command; Section 4 explains details of each command and an algorithm that recognizes commands and messages; In Section 5, we evaluate accuracies of a recognition and a response time as an application.

2. Related Works Yeo et al. [1] proposed a handwriting message system. Their system needs smart phone operations to activate the system. Saund and Lank [2] proposed sketch-based interfaces. Their system can switch between two modes: a drawing mode and an editing mode by handwriting trajectories. However, in their system, the detected operation is within the drawing

Handwriiting Command Recognitiion and Digita al Operation Using Digitallized Pen

software. Koolberg and Magill M [3] proposed networrked appliances control c system using a digital d pen annd a paper. Theirr system allow ws users to coontrol appliannces only a diggital pen, hoowever the control metthod requires an exclusive e papper and takess costs more than a regular one. o Matulic and Norrie [4] presenteed a document-auuthoring ennvironment for interacctive tabletops, inn which severral common document-edi d iting operations are a supportedd by gestures instead of, or o in addition to widgets. w How wever, the geesture by pen and touch may complicate user interfacce due to many m operations. In handwritiing gesture study, s Wang and Chuang [5] presented p diggit and gesturre recognitionn for accelerometer-based digiital pen, the recognition raate is very high. However, thheir method is needed high h performancee computer to t train the neural netw work. Aggarwalet al. [6] proposed a feature representaation for the handdwritten data captured usiing depth sennsors such as Leeap Motion and Kinectt. They shoowed alphabet chharacters recoognition algorithm. A large movement is needed to draw d the charracters. Amm ma et al. [7] develloped an inpuut system whhich allows us u to write characcters in the air a using a data d grove. Their T system is noot required anny handheld devices, d how wever it is difficuult to use it in daily life. Tian et al.. [8] presented peen tail gesturres, a techniqque using the pen tail while thhe pen tip is occupied by other tasks. It is

Fig. 1

1655

not intuitive beccause users aare not familiar to movee pen n tail. We have h developped pen bassed interfacee systtem for digitaal operation [9].

3. Writing W Baased Digitall Operation n We W provide a quick overviiew of our neew concept off pen n based interfface system. T This system enables e us too enfo force digital operations ssuch as, takiing a digitall mem mo, sending an e-mail oor a short message, m andd tweeeting by juust writing a correspond d command.. Acttually this system needs a smarttphone withh Blu uetooth conneection to make a digital op peration. However, H a usser does not nneed to touch h any buttons,, the screen or the t smartphoone itself. Th he necessaryy opeerations are inputted by digital pen via writtenn trajectory recognnition algoritthm. So, it works w even iff the smartphone is in his/heer bag. We have h alreadyy dev veloped an e-mail transmission op peration thatt con nsists of comm mand series aas an examplle (Fig. 1). Inn order to send an a e-mail meessage, the sy ystem has too reco ognize what is i the messagge, who is thee destination,, and d when the syystem executees the transm mission. Manyy add ditional appllications of writing based digitall opeeration beside the (1) m message transsmission, (2)) notee sharing betw ween studentts and teacherrs in a schooll, (3) web searcch service with handw writing, (4)) auth hentication forr net shoppingg. Therefore, itt is consideredd

The concept c of digittal operation through pen baased interface system. s

166

Handwriiting Command Recognitiion and Digita al Operation Using Digitallized Pen

that a pen baased interfacee has many poossibilities.

4. Commaand Recogn nition Algorrithm 4.1 Commannd Definition This sectiion describess commands for f transferrinng a message. Eaach commandd is drawn with a single strroke. We define a circular trrajectory as a MC (message M is drawn on a paper,, the command). When the MC a an inside area of the circle is extracted and saved as image file on the devvice storagee. We definne a rectangular trajectory ass an AC (adddress commaand).

ment. The forth comm mand is a TC (tweett segm com mmand), whicch is recogniized when a check c sign iss draw wn near MC C. A tweet message attaached to thee imaage using a tw witter accounnt that has been registeredd in th he mobile device. 4.2 Algorithm Fllow Fig. F 2 show ws a flow chart of th he commandd reco ognition alggorithm. Thiis algorithm receives a trajectory and cllassifies it intto one of the 5 categories.. Thiis process is executed e wheen the pen getts away from m a paaper.

A is drawn on o the paper, the t inside areea of When the AC the rectangle is extraccted and anaalyzed by OCR O Tesseract-OC CR [10]). Thhen, the obtaained algorithm (T characters are used for f matchingg of registtered addresses inn a mobile device. Wee define a line segment, whhich links thhe MC and the t AC, as a SC (send comm mand). The SC C is a triggerr for sendingg the message to the address. This commaand is recognnized M and the AC A are conneected with a line when the MC

Fig. 2

Comm mand recognittion flow.

4.3 Stroke Divisiion A digitalized pen can deteect statuses su uch as a Penn Dow wn (contact with w a paper),, a Pen Movee (drawing onn a paper) and a Pen P Up (gettting away fro om a paper).. Han ndwriting traajectories arre representeed series off draw wing points. The drawinng points aree divided too stro okes based onn this status cycle and sto ored as a listt form mat (Fig. 3).

Handwriiting Command Recognitiion and Digita al Operation Using Digitallized Pen

1677

4.6 Vertex Detecction and Anggle Measurem ment

Fig. 3

Pen status cycle.

Fig. 4

The minimum m distaance search.

The T stroke reccognized as cclosed figure is calculatedd inteerior angles and a judged aas MC, AC or Message.. Thee drawing pooints tend to be denser in n around thee verttices becausee a pen speeed becomes slow downn therre. The densse points maay give bad effect to thee folllowing anglee measuremennt. Thereforee, the pointss are extracted so that each disttance is longeer than Td. Inn add dition, for eassy to extract tthe vertices, the t points aree searrched in ordder of the disstance betweeen the pointt and d the gravityy point. We define that the distancee betw ween startinng and gravvity point is Ds,g andd extrracting param meter is Pex. Then Td is calculated c byy folllowing formuula,

Td  Pex Ds, g

4.4 Calculatting TSS and TDS In this prrocess, the TSS T (time spaan of the stroke) and the TD DS (travel distance d of the stroke) are calculated. We W define thhe starting tiime and the end time of the stroke as t0 and tN-1, resspectively. Using U them and thhe sampled poositions (xi, yi) (i = 1, 2… …N), TSS and TDSS are calculatted by follow wing formula,

TSS S  t N 1  t0

(1)

Fig. F 5 shows examples oof vertices deetection withh variious-shaped figure such as ellipse, reectangle, andd starr. The dense points p are rem moved by thiss process. Next, N a figuree composed of the extraccted points iss con nsidered n-siided polygoon, the angles betweenn neig ghbor points are calculateed. The points are definedd pi(i = 1,2…N), we define thhe vector com mposed of pi and d pi-1 is A, thee vector com mposed of pi and a pi+1 is B,, inteerior angle θi is described aas follows,

N

TDS   ( xi  xi 1 ) 2  ( yi  yi 1 ) 2

 i  cos 1

(2)

i 1

When a useer draws a command, c TS and TDSS are TSS expected a large l value. Therefore, T the stroke satissfied TSS>Ttss andd TDS>Ttds is extracted. 4.5 Judgemeent of Figure Closing In this process, p a minimum m disttance betweeen a starting andd end point of o the strokee is calculated in order to judge whether thhe stroke is closing c or noot. A closed figurre is consideered that starrting point grroup and end poinnt group are near. Let Dmin m and Tcf denote the distancee and the threshold, reespectively. The stroke whosse distance saatisfied Dmin