Network-based Context-Aware Input Method Editor

Network-based Context-Aware Input Method Editor Shinji Suematsu, Yutaka Arakawa, Shigeaki Tagashira, Akira Fukuda Department of Computer Science and Communication Engineering. Graduate School of Information Science and Electrical Engineering, Kyushu University. 744 Motooka Nishi-ku Fukuoka, Japan {suematsu, arakawa, shigeaki, fukuda}@f.csce.kyushu-u.ac.jp

Abstract—In this paper, we propose a context-aware Input Method Editor (IME) as a ubiquitous service for improving the input of text on mobile devices. In our proposed IME, it estimates the context of users according to a position information from GPS (Global Positioning System) sensor or a presence information from the Internet. Simultaneously, it generates a personal context-aware dictionary dynamically from the keywords gotten via some APIs in the Internet. Currently, the information of user’s context is also provided by NGN. In this paper, we explain the overview of our proposal and prototype implementation of Japanese. Keywords-ubiquitous; context-aware; IME;

I. I NTRODUCTION In a recent research in Japan[1], it is turned out that over fifty percent of Internet users access the Internet from mobile devices. And among them, over eighty percent users access the information by using not hierarchical menu in official site but search engines such as Google. Moreover, current mobile devices can use not only text messaging but also web-mail such as Gmail. It indicates that one has an opportunity to input a long text. The increase of text input on mobile devices drives the demand for improving a text input method. In general, one uses multi tap input method for inputting text on mobile devices. For example, Japanese language has fifty characters called ”kana”, and five characters are allocated to each of the digit keys on mobile devices. The key ”1” has following kana ”あ, い, う, え, お”. In the case of inputting Japanese kana ”え”, a user pushes the key ”1” four times. And if the user wants to transform kana into kanji such as ”江”, the user pushes a space key, and selects the desired kanji from the candidate list. The following approaches are conceivable as a typical method of saving input, 1)improvement of procedure of text input, 2)expansion of dictionary, and 3)prediction of input word. Some example of 1), T9[2] has been proposed in order to reduce the number of pushes. In multi tap method, an average of three pushes is necessary for inputting a character. However, T9 only needs an average of one push because the system predicts the desired word from the combination of characters. In addition, a flick-style input method which is operated on touch screen devices such as iPhone in Japan

has been proposed. The five characters are input by touching the key and keeping the finger there or sliding to four directions. Some examples of 2), one can download fieldspecific dictionaries such as a famous person’s name. In addition, Social IME[3] has been proposed to share a word registration dictionary on the network. Some examples of 3), POBox[4] has been proposed. In this method, a user inputs a small part of the word, and the system searches a dictionary for candidate words and suggests them to the user for selection. For example, when a user inputs ”g”, the system predicts the words such as ”Good”, ”General” and ”Great”. And if the user selects ”Good”, the system suggests the words such as ”morning” and ”job”. The input methods which predict input words like this are widely used in Japan. Besides, it is now general to suggest words based on usage history and its frequency by learning the input and selection of user. However, it is difficult to predict a input word with high-precision from only usage history and its frequency of use. Because, the input word varies from moment to moment with the user’s context (e.g., location or presence) and the purpose (e.g., mailing, searching transfer guide or around information). Incidentally, in this paper, the word ”context” means real information around users. Therefore, some context-aware input methods have been proposed recently. Among them, iWnn[5] predicts words based on time and the receivers’ profile gotten from address book (we define them as local contexts). However, local context is not enough because one’s context varies with time in a mobile computing environment. For example, a place-name input for the first time or a niche landmark name (e.g., building or crossroads or restaurant name) which is not included in a dictionary can’t be suggested to user. Therefore, we propose the new method to predict with high accuracy for mobile devices. The difference from existing prediction methods is that our system accesses the network constantly and uses the resource on it. In particular, we use the location information and presence information (we define them as global contexts), and get the name of landmark, nearby station, restaurant from some APIs. In this way, our proposed context-aware IME can suggest useful words, based on user’s location, presence, and time. We

explain the process as follows. First, it estimates the situation from both local contexts such as time acquired from a local device and global contexts such as user’s location acquired from a network (NGN). Next, it generates the dynamic personal context-aware dictionary by the collected words from APIs. At last, it displays words as the candidate list from the dictionary when the user input some characters. Furthermore, we achieve personal IME by obtaining feedback of repetition of guessing words and user’s selection, and associating words with each of context. We expect the following effects. • When a user runs a navigation application, nearby place names are already registered in the dictionary. • In the scene of inputting current location, a city name which starts with a ”C”- ”Caracol” and ”Cancun” are sorted by the location and suggested preferentially. • The system recommends same words at a similar place, similar time. Our proposed system can be applied to many different languages. In this paper, we explain the related research of our system in next section. In the section3, we describe the overview of our proposal. And prototype implementation for Japanese language is explained in the section4. The paper ends in section5 with conclusion and future work. II. R ELATED R ESEARCH In this section, we explain conventional word prediction methods which utilize context of a user. In addition, we describe elemental technologies related to the proposed system, which are expansion of dictionary, estimation of user’s context from several sensor data, and mashup services. A. Word Prediction Methods Which Utilize Context of User There are enough researches about a word prediction method which utilizes context of a user. Hattori et al.[6] have proposed a method of query modification and guessing of search keywords based on Real-World contexts, such as user’s geographic location or the real objects surrounding the user. Also, the assistant system for writing a document on the web has been proposed[7]. This system gets attribute values from the Internet and indicates them for a user. For example, when the user writes landmark name, it guesses its address. However, these methods are not usable in a general text input scene. Because, it only work on the specific input box prepared by researchers. As a Japanese IME to utilize context of a user, there is iWnn. It dynamically modifies guessing words based on the local context of a user. For example, changing the priority of these words ”good morning” and ”good night” based on time, or switching conversation expression and honorific expression based on the profile of the receiver. On the other hand, Kobayashi et al.[8] have focused on the communication scene and have proposed a word prediction

method through the use of context information. In this system, some words in the message from companions during mail and chat are suggested preferentially. B. Expansion of Dictionary for Japanese IME The research has been taken to predict inputting words by not only using local dictionary data but also accessing the network. Okuno[3] have proposed Social IME which shares dictionary on the network and allows everyone to resister vocabulary into the dictionary. In this way, everyone is able to use wide variations of prediction words such as IT terminologies and trendy term. Our system also sets the dictionary data on the network like this system. C. To Estimate Context of Users from Several Sensor Data Our proposed system is one of context-aware services. There are researches to estimate a context of a user from several sensor data and apply it to other services. A method of Saruda et al.[9] estimates common movements such as sitting, standing, writing, drinking from the feature quantity of several sensor data. Morioka et al.[10] also judge up-anddown movement of elevator and common movements such as walking and running. As a context-aware service, it has been proposed that the system pushes useful contents for user utilizing user’s context[11], and the system supports to keep some objects as a certain context[12]. However, these services are realized by each of specific context estimation engine. A middleware has been proposed as a flexibility design for unified access to real world information.[13][14]. Also, ”Synapase” is a context-aware services platform[15]. It is designed in such a way as to learn of relation between sensor data and service in theory of probability, and create services automatically reflect user’s like and habit. D. Mashup Services Recently, it has been taken actively to create a new web service by mashup. Mashup means to combine several services. The development environment is improved by releasing some web services with APIs. In using APIs, developers can use SOAP or REST protocol and translate data written in XML. Many kinds of API has been released such as weather information, geographical information, station name and product information. For example of mashup services, showing contents related to geographical information on the map[16], and showing the product information from shopping site and the blog site mention the product at the same time. III. T HE OVERVIEW OF OUR P ROPOSED S YSTEM The schematic chart of the proposed system is shown in Figure 1. The context-aware IME is implemented on the left side server in this figure. We design it to have three functions. First, it estimates the context of user from local

Data for context estimation - Schedule API - Presence API - Context Aggregation - Context Estimation - Dynamic Dictionary Update - Context-Aware Select & Sort

Transit Market

Station name

Station

Restaurant

Meeting Carrier

Dinner

Restaurant name

Amusement

New Area

Landmark, Shops

Shopping

12:34 PM Web Page Title

http://host.domain.tld

The internet

Pub

Figure 2.

320 x 480 Vertical Resolution

Short phase

Facemark

Product name

User’s contexts and necessary words

Data for making dictionary - Yahoo API - Google API - Amazon API

place are just suggested, normal words are suggested in other place. Figure 1.

A system schematic chart

contexts such as time and acceleration sensor data acquired from local devices and global-contexts such as schedule and presence information acquired from the network. Next, it generates and updates the dynamic personal context-aware dictionary from the collected words gotten via APIs. At last, it displays words from dynamic dictionary as the candidate list which are sorted by user’s context. We explain each of functions as follows. A. How to Estimate the Context of User In our proposed system, it estimates what situation the user is placed and indicates the useful words as predictive candidates based on the user’s context. As input of user’s context estimation, our system uses not only local contexts such as time or address data like iWnn, but also global contexts obtainable from a network such as location information, presence information, and user’s schedule. Also, we have a plan to integrate each sensor’s data (e.g., GPS, acceleration, geomagnetism) obtainable from mobile devices, and design more integrated estimation engine. The server collects these information and estimates context of user. The method of estimation is to define some typical behavior models and to learn the words used by user. There are related researches[10][13] about the method for estimating a user’s context. An example of typical behavior models in a certain place is shown in Figure 2. It indicates the importance of words varies with a location (i.e., user’s context). For example, nearby station name is used at stations, landmark name is used at a new places, product name is used at bookstores and electronic retail stores. At a station, the system suggests a nearby station name. If a user uses the word, the system judges it may be used at the same place and raises the its weight. If the word is not used, it judges the word is not important at this area. By repeating these learning processes cyclically, it is realized that the words related to a certain

B. The Creation of Dynamic Personal Context-Aware Dictionary As a conventional approach to suggest useful words, a method sorts the candidate list by setting the relation between user’s contexts and words which is already included in a dictionary. However, it is difficult to set the relation for all words. As a different approach, our proposed system creates the dynamic dictionary utilizing user’s context. The dictionary is set on the network like Social IME[3]. Also, it is created from the words gotten via some APIs related to the user’s context. Currently, a lot of APIs released and the words can be obtained easily from it. For example, nearby station API, Yahoo Local Search API, Amazon API, and Google Map API are released. Especially in Japanese, our system saves the words and its kana into the dictionary by using Japanese language morphological analysis. The reason of saving kana is that Japanese text is mixture of kana and kanji, and kana is necessary to transform to kanji. However, the response of Japanese language morphological analysis is slow. For preventing our whole system from delaying, we have made some artifices in implementation. C. Algorithm for Sorting the Candidate Words Our proposed system uses a learning algorithm for sorting. It utilizes a user’s context and usage history of personal context-aware dictionary. In the initial phase, it replaces a part of usual words in the candidate list by the suggested words from personal context-aware dictionary. And it raises the importance of words if user uses it. At the same time, it records the context of the user and guesses the same words when the same context appears. It achieves the recommendation of words based on one’s pattern of behavior by obtaining the feedback of the repetition of suggesting words and user’s selection. IV. P ROTOTYPE I MPLEMENTATION We now assume that input word related to the user’s context. On the other hand, it is important for developing

Internet Local device

Internal server

Context updater

Global context

Other sensors Feedback

Asynchronous

Google API Schedule API Amazon API

Tabelog API

For making dictionary

ATOK

Japanese language morphological analysis (MeCab)

Direct plugin

IME

API access module (XML parser)

Roman character

Context-Aware Select & Sort Engine

or

Location API

Yahoo API

Input Hiragana

Schedule API Presense API

For estimation

Acceleration sensor

Local context

API access module (XML parser)

Context Estimation Engine

GPS sensor

External server

GuruNavi API

Output mixture of Chinese characters and Japanese phonetic characters

"Personal Context Dictionary"

As a general dictionary kana-kanji conversion API

Figure 3.

Figure 4.

The display of prototype（HTML version）

The architecture of prototype system

a user’s context estimation algorithm and sort algorithm to analyze the strength of the correlation between input word and a user’s context. Therefore, we implement the prototype system of Japanese for the preliminary experimentation. Currently, we are transporting this prototype to the Google android device. In this section, we explain our prototype developed on PC. The architecture of prototype system is shown in Figure 3． It is composed of three parts, local device, our server on the Internet, and general web services on the Internet. The local device has various sensors such as GPS and acceleration. In our prototype system, we use a PC as local device and adopt the Google Maps API as GPS sensor for setting user ’ s location visually. The internal server in the center of Figure 3 is a main part of our proposed system. It collects information and estimates of user’s context, creates the dynamic dictionary, and suggests the words by utilizing user’s context. These functions are possible to construct on local device. However, we set it into the server on the global network because it is important not only accuracy of estimation algorithm but also processing speed. Besides, we architect it works asynchronously to collect sensor information by the system and to input text on local device. The dictionary is updated whenever location is varied. As a result, local device only searches pre-constructed database when text is input. This architecture enable the system to prevent the processing speed from slowing down when web external servers increase. The external servers in the right of Figure 3 are not our servers but provided by several companies. In the case of this prototype system, it cooperates with the Yahoo Local Search API, Google Maps API, and Gurunavi API (provide

restaurant information). Some words provided by these APIs are materials of personal context-aware dictionary. We develop the two interfaces on PC to work prototype system, using HTML form and using ATOK Direct Plug-in. ”ATOK” is one of the major IME in Japan such as MSIME. The former displays the words variation of the personal context-aware dictionary according to the location. The latter can be used on other applications such as MS-Word. The prototype using HTML form is shown in Figure 4. The center point in the map shows a current location and it can be moved. The lower left form shows some landmark names existing within one-kilometer radius. These words provided by Yahoo Local Search API. The upper right form is entered the character ”て”, and the words which start with ”て” are listed in lower form. These words are not sorted because we don’t implement yet. This figure shows the words can be suggested, which are not recorded normal dictionary such as around building name and detailed addresses. Also, Figure 5 shows the prototype implemented on ATOK. The ATOK has a function called ”Direct Plug-in” which can import one’s own plug-in. We create the plug-in communicate with our server. In this way, proposed system can be used not depends on application. V. C ONCLUSION AND F UTURE W ORK In this paper, we propose context-aware IME as a new approach to predict an input word on mobile devices. And we implement the prototype of Japanese to generate dictionary dynamically by using some APIs. This prototype indicates that dictionary can be created based on user’s location, and proposed system can be integrated to general IME seamlessly.

[4] T. Masui, POBox: An Efficient Text Input Method for Handheld and Ubiquitous Computers, ser. Lecture Notes in Computer Science. Springer Berlin / Heidelberg, January 1999, vol. 1707/1999, pp. 289–300. [5] OMRON SOFTWARE. (2009) iwnn. (in Japanese). [Online]. Available: http://www.omronsoft.co.jp/SP/mobile/iwnn/ (last access:2009/12/1) [6] S. Hattori, T. Tezuka, and K. Tanaka, “Query modification based on real-world contexts for mobile and ubiquitous computing environments,” Mobile Data Management, IEEE International Conference on, vol. 0, p. 77, 2006. [7] K. Nakamura, “Suggestion of referntial information for writing documents on the web,” Master’s thesis, Japan Advanced Institute of Science and Technology, Mar. 2007, (in Japanese). [Online]. Available: http://hdl.dandle.net/10119/3589 (last access:2009/12/1)

Figure 5.

The display of prototype（ATOK Direct Plug-in version）

We consider the effect of saving input time depends on the strength of a correlation between user’s context and input words. Therefore, we will conduct a feasibility study to define the relation. In this research, we record a input word and a user’s context simultaneously, and analyze the relation of these for developing the algorithm of context estimation and sorting. Also, we have a plan to evaluate quantitatively how our system improves the input environment. Nevertheless, the text input environment will be improved if the system considers a user’s context and correctly predicts input words by using not only local information but also information provided from all over the world. ACKNOWLEDGMENT The work is carried out by the joint research program of the NTT Service Integration Laboratories and the National Institute of Informatics. It is performed using the facilities provided by them. R EFERENCES [1] rTYPE. (2009) 「the access of the internet is by mobile devices rathar than pc.」the more than half person who access the internet during five years or more has answered so. — rtype, marketing research opening to the public by ishare online research service. (in Japanese). [Online]. Available: http://release.center.jp/2008/11/0502.html (last access:2009/12/1) [2] Grover,D.L., King,M.T., and Kuschler,C.A., “Patent no.us5818437, reduced keyboard disambiguating computer,” Tegic Communications, Inc., Seattle, WA, 1998. [3] Y. Okuno and M. Hagiwara, “Japanese input method based on the internet,” in Natural language processing society of IPSJ, May 2009, (in Japanese).

[8] H. Komatsu, S. Takabayashi, and T. Masui, “Context-aware predictive text input method using dynamic abbreviation expansion,” Transactions of Information Processing Society of Japan, Nov. 2003. [9] Y. Saruta and T. Tomii, “A user context estimation method in the ubiquitous environment using acceleration sensors and rfid,” The Database Society of Japan(DBSJ) letters, vol. 6, no. 3, pp. 13–16, Dec. 2007, (in Japanese). [10] H. Morioka, N. Iwata, and Y. Niitu, “Context presumption method using plural sensors,” in Lecture collection of The Institute of Electronics, Information and Communication Engineers(IEICE), 2008, p. Lecture Number.94, (in Japanese). [11] T. Hayashi, Y. Kawahara, H. Tamura, H. Morikawa, and T. Aoyama, “Design and implementation of a context-aware content delivery service using tiny mobile sensors,” IEICE technical report. Information networks, vol. 104, no. 691, pp. 149–154, Feb. 2005, (in Japanese). [12] S. Mugishima, R. Shimizu, and T. Tomii, “Action upport with users’ concepts in ubiquitous environment database,” Journal of the Database Society of Japan(DBSJ), vol. 7, no. 1, pp. 221–226, Jun. 2008, (in Japanese). [13] S. Amano, Y. Kawahara, H. Morikawa, and T. Aoyama, “B7-168 design of a middleware for unified access to real world information,” vol. 2006, no. 2. The Institute of Electronics, Information and Communication Engineers, Mar. 2006, p. 264, (in Japanese). [14] H. Morikawa, “Real world oriented ubiquitous networks,” Journal of IEICE, B, vol. 88, no. 11, pp. 2137–2146, 2005, (in Japanese). [15] H. SI, Y. Kawahara, T. Igakura, T. Tonouchi, H. Morikawa, and T. Aoyama, “A hybrid context-aware service platform based on stochastic and rule-description approaches,” in Annual International Conf on Mobile and Ubiquitous Systems: Networks and Services (MOBIQUITOUS) Demo session, Jul. 2006. [16] N. Nishio, N. Sakamoto, and I. Arai, “Real-world-oriented contents management system mashed up with google street view,” in Demonstration at Pervasive, Mar. 2009, (in Japanese).