An Open Machine Translation System for Augmentative ... - CiteSeerX

0 downloads 0 Views 114KB Size Report
adapted to form a translation system for the average AAC user of an alternative ... be satisfied from a software engineering point of view include expandability, ... Makaton. Picture of an apple. Greek. English. Spanish. French. Written form ... implementation of the database is very straightforward using a contemporary DBMS.
An Open Machine Translation System for Augmentative and Alternative Communication Constantinos Viglas and Georgios Kouroupetroglou Deptartment of Informatics and Telecommunications National and Kapodistrian University of Athens Panepistimiopolis, Ilisia, GR-15784, Athens, Greece Tel: +30 10 7275617, Fax: +30 10 6018677 {cviglas, koupe}

Abstract. In this paper we describe the characteristics of an open system we designed and implemented that adds Machine Translation (MT) capabilities into Augmentative and Alternative Communication (AAC) applications. In particular, we present the basis for an open MT framework that can easily integrate with component-based developments especially regarding interpersonal communication and information presentation, either in standalone or web-based applications. We concentrate on the system’s core, an expandable database of multimedia elements, which offers uniform management of multiple representations of concepts in both natural languages and symbolic communication systems, allowing concept-by-concept omnidirectional translation. Applications of the system are also presented, along with brief pointers on current work for possible extensions and improvements.

1 Introduction Users of AAC systems have experienced many benefits over the years, especially with the introduction of modular computer-based aids [7],[8] in the Computer Mediated Interpersonal Communication (CMIC) domain. Although such devices offer expandable vocabularies, speech output, and customizable interfaces, they don’t do as much in allowing the disabled to communicate transparently with other people (disabled or not) regardless of the communication system they use. Unlike the common translation problems, when it comes to AAC additional effort is required. Our solution is realized by the design and implementation of an open and scalable system whose properties, characteristics, and applications will be presented hereafter.

2 Interpersonal Communication and Machine Translation for Assistive Technology Machine Translation (MT) and Natural Language Processing (NLP) are being used in a number of computer-based applications [1],[2] and often cross the domain of Interpersonal Communication [6],[13],[14]. Nevertheless, they need to be properly K. Miesenberger, J. Klaus, W. Zagler (Eds.): ICCHP 2002, LNCS 2398, pp. 699–706, 2002. © Springer-Verlag Berlin Heidelberg 2002


C. Viglas and G. Kouroupetroglou

adapted to form a translation system for the average AAC user of an alternative communication system. 2.1

Contemporary Communication and Translation

Interpersonal communication as far as assistive technology is concerned, relies heavily on the use of alternative communication systems known collectively as symbolic communication systems (such as BLISS, MAKATON, PCS, etc.) [5]. They use graphic symbols (available in all sorts of formats ranging from cards, boards and tablets, to portable libraries of symbols in the form of computer viewable graphic images) in order to depict concepts and convey meanings from and to their users. However, little has been done to provide standardized and compact ways of dealing with symbolic systems and communication aids as a whole, covering aspects like learning graphic symbols, using symbols with a communication aid for everyday communication needs, either in face-to-face situations or remotely (e.g. over the telephone or a computer network). Another of their prominent characteristic is that they offer speech output (either synthetic or digitized) since most disabled are incapable of producing intelligible speech. However, if the disabled wish to converse with other people (disabled or not), some kind of automated process leading to a common transcription is necessary be that speech, text or plain graphical symbols. Moreover, such a transcription is needed for the disabled to access and retrieve information offered in mediums like the World Wide Web (WWW), according to the Web Accessibility Initiative (WAI) regarding access to information [18]. To communicate or review information in a form or language different from what one uses is a complicated issue that becomes more challenging for AAC. A more flexible approach should provide translation and transcription between the various languages, symbolic systems and the representations of the associated symbols and concepts. On the other hand, research on MT has produced a number of significant results over the years [4], with its realizations focusing mainly on bi-lateral natural language translation, although there have been multilingual, language independent systems built around specific lexicons [3],[16], with efforts ranging from simple word-byword systems to more complex ones providing correct translation of sentences and phrases in terms of overall meaning, grammar and syntax in the target language. MT has already found application in both stand-alone and web-based translation systems (e.g. SYSTRAN [17]) that provide either off line or “real-time” translation of documents. Moreover, MT has found new ground of application in conjunction with NLP, especially concerning specific or restricted vocabularies leading to major applications of translation of scientific and technological writings through lexical analysis of written text in the language of interest. Under similar notions other areas of application have been multi-lingual document generation in a given language using large lexical knowledge acquisition. Systems under both of the latter two cases have been built for medical purposes [12],[16]. 2.2

Requirements and Specifications for an MT Framework for AAC

Our goal is to integrate MT techniques into contemporary Computer Mediated Interpersonal Communication (CMIC) aids. The modular character of such devices

An Open Machine Translation System


resulting from their Component Based Development (CBD) nature, call for a scaleable degree of MT integration. At the same time, desirable qualities that need to be satisfied from a software engineering point of view include expandability, adaptability, reusability and accessibility. Further requirements and properties for the implementation of a specific MT framework, would satisfy some –if not all- of the following: support for direct translation mode (i.e. a word-by-word or symbol-bysymbol translation), language and media independence (all possible representations for a given language like text, pictures, graphics, or voice should be addressed), effective information presentation regarding data transmission and bandwidth overload in all communication modes (e.g. local, face-to-face, remote, real time, or asynchronous), usage of restricted or expandable lexicons, context of translation, adoption of analysis, augmentation, and/or synthesis stages, consideration for possible solutions to lexical ambiguity and synonym meanings, potential applications, etc. In the light of the above, in principle we aim to provide an open and compact environment that handles natural languages, symbolic systems, symbols and related vocabularies in a uniform manner, so as to prove beneficial for an adaptable communication aid, and to go beyond simply providing symbol libraries of communications systems associated with a couple of -usually hardwired- natural language interpretations. This system can be the cornerstone of the whole communication process interconnecting various up to now isolated entities, allowing system integrators to build customizable systems in a cost-effective way, ultimately supporting omni-directional translation between the concepts of all interconnected languages. Moreover, our system would be incorporated into modular communication aids that target on the international market, dealing with not converging user needs in various counties, with different cultural backgrounds and different languages. Therefore, multilinguality (with reference to both natural languages and symbolic communication systems) in all levels of encoding and presenting meanings, definitions and information is imperative. Adaptability is also desirable, in terms of selecting the proper language or communication system and associating it with user specific and context dependable vocabularies, meeting the changing-with-time user needs. Last but certainly not least, to maintain effective communication between partners that “speak” different languages a common ground needs to be determined based on a common interconnecting mechanism that relates languages and corresponding meanings using proper techniques to overcome mismatching vocabularies [4], thus enabling correct concept-by-concept translation. Moreover, translation should not only be limited to verbal terms, but allow for transcription rather than just translation between multiple output modalities and media representations, such as speech, audio, visual, text/printed, etc. 2.3

System Realization

The design specifications, made it clear that we needed a structured means of organizing, storing, managing and retrieving the information, mechanisms and media involved. Hence, the choice to build it around a properly engineered database that can hold data of a more diverse nature than just the graphic symbols that make up a symbol library and their associated meanings in a given natural language.


C. Viglas and G. Kouroupetroglou

Symbolic Communication Systems BLISS


Picture of an apple

Natural Languages Greek ìÞëï



Written form apple manza na Oral form 

French pom me 

Fig. 1. Example representations of the concept “apple” in various languages and systems

Database Structure and Design. Databases have always been considered the best way to store information in a well organized, structured and efficiently accessible manner even over computer networks and the Internet in particular (via client-server or web interfaces). The database design is based upon previous experience and implementations as well as contemporary studies, experience and domain knowledge [3],[10],[11],[12],[15]. Primarily, we were concerned with the aspects of the database design that deal with media representation, since a variety of media need to be considered (e.g. speech, text, icons and pictures, video, etc., depicted in Fig. 1) and associated with the various languages and systems. Translation is facilitated by the database by means of relating the corresponding words and symbols of the various languages and systems via the entity of Interlingua concepts, namely by the definition of an independent, stand-alone, natural-like language (the Interlingua) that primarily gets all supported concepts defined, categorized, marked for synonyms, and indexed. After defining the required languages and symbolic systems into the database (through the corresponding Language entity), utilizing the proper structures to best incorporate their properties and values, concepts are also defined for each language or system including any associated corresponding media representation is needed. The Interlingua concepts are then mapped onto the elementary communicative blocks of the defined languages (i.e. their words) and symbolic systems (i.e. their graphic symbols) with which they share the same meaning. In this way we ensure a simple first degree of concept-byconcept translation. In this way all languages (even artificial ones) that have a formal definition can be included, and their properties can be encoded into the meta-entities of our translation capable database (provided that these meanings can be associated to Interlingua concepts). By way of inclusion of language representation entities, we made sure that all modalities for a given concept would be addressed for both natural languages (e.g. text literal, definition, phonemic transcription, any regional related pictures or videos, etc.) and symbolic systems (the graphic symbol, tutoring pictures or animations, etc.). Other entities and internal mechanisms allow the definition of groups of concepts into user-oriented modifiable vocabularies that adapt to the user’s abilities and needs evolving with time and place (proper selection sets can be accommodated by the definition of categories based on a variety of criteria such as graphical, thematic, grammatical, contextual, etc., per language or system). The open

An Open Machine Translation System


design of the database also includes meta-entities. They are generic in nature, may relate to specific languages and systems and include non-communicative items (NCI) per language, various language element categorizations, and linguistic information (such as grammar or syntax rules that can be utilized for MT and NLP purposes to address for instance synonymity, context and ambiguity). Additional structures include built-in rules and procedures, triggers and views that have to do with database integrity and customisation. User database production criteria Management

Import/Export Rules Users-Vocabularies-Selection Sets Views

Concepts Languages Lang. Properties Meta-elements Users

Central Database of multimedia elements

Vocabularies Selection sets User-Vocabulary Relations

Local Management for User Database

User Databases

Vocabularies Selection sets

Views ...

User Database


Fig. 2. Management options and production of user databases. Management is downsized for the user databases, which constitute a subset of the central database

Implementation Issues. The system we developed is to be used especially by computer-based interpersonal communication aids to which will be a major component. It has taken into account software building technologies particularly regarding database connectivity and web-enabled access. In this respect, the implementation of the database is very straightforward using a contemporary DBMS. Storing and manipulation of data is ensured via proprietary software tools that deal with adding, deleting, updating and reorganizing data in a transparent and secure way (see Fig. 2 for available management options). Connectivity and presentation is attainable for local and web access using ODBC or JDBC drivers, while adopting CBD recommendations so that the developed database can integrate in a modular application environment. Particular measures were taken in order to accommodate all media related information in a text form, namely storing all binary object related data concerning image or sound files as pointers to local storage directories and not as embedded binary objects. Such a technique allows transparent modification of the binary objects without affecting database content, as well as transparent transmission of concept codes when communication and translation is needed. The database itself may co-exist in a number of instances varying in location, size, content and utilization, but still bearing the same internal structure of entities and relationships. A main instance can be a central or reference database, bearing literally


C. Viglas and G. Kouroupetroglou

any available information on the structures and entities incorporated. When used as a module for CMIC component-based systems it can produce new customized databases (referred to henceforth as “user databases”) to be used in specific interpersonal communication applications. “User databases”, are smaller parts of the central database, user specific (personalized) database versions according to specific needs and requirements reflected into interpersonal communication aids (concerning natural language and symbolic system to be used, vocabulary and selection sets definitions, gender specifics for user personalization, etc.). The “user databases” can be stored locally alongside the application that uses it, and can be easily adapted to reflect evolving user needs by proprietary tools that modify vocabulary and selection sets for a local user, and proper import/export mechanisms in conjunction with the central reference database, when languages or systems need to be added modified or removed, or whole vocabularies need to be adapted with the provision of new or modified concepts. Table 1. Number of concepts defined for some of the Natural Languages and Symbolic Communication Systems included in the database (the Interlingua has 2064 distinct concepts) Natural Language Greek English

Number of defined concepts (with synonyms)  

French Spanish 2.4



Number of defined concepts  


Field Tests

The presented framework has been field tested in a number of cases related directly or indirectly with interpersonal communication. A central version of the database acts as the core of an on-line translation dictionary for natural languages and symbolic communication systems (called the PolyLexicon), and can operate either locally or over the Internet, using a web interface offering: full database search abilities for concepts in various natural languages and symbolic systems, and concept based translation capabilities between symbolic systems and natural languages (Fig. 3 depicts the access for translation workflow used in the PolyLexicon, while Table 1 offers a comprehensive summary of the defined concepts per natural language and symbolic system in the database). A (“user”) database module for interpersonal communication aids has been implemented and successfully utilized under the ULYSSES framework for building component-based CMIC applications [7], for both face-to-face and remote communication allowing even for vocabulary synchronization between the conversation parties [9],[15]. Another domain of application had to do with teaching symbolic systems, since the framework can provide with translation from the symbolic system’s graphical symbols to natural language equivalents and vice versa, thus being a valuable tool to both teachers and learners (the application is simple, yet web based, using a user database to teach the BLISS system to Greek disabled students).

An Open Machine Translation System


Translation Request Internet Web Access Central Database of multimedia elements 1


ASP web page

Fig. 3. Sample translation over the web. A request is made and the results come through accessing the central database and presented in a dynamically generated web page using ASP

3 Conclusions – Plans for Further Work We have demonstrated an open system that was built to serve as the basis of a concept-by-concept, omni-directional MT system between natural languages and symbolic communication systems, through the unified management of multiple representations of concepts via a properly constructed expandable and customisable database. It is web enabled, and can easily become integrated as a component of modular applications. Currently, we investigate the potential of augmenting the database design, in order to include further linguistic metadata so that translation can be more effective regarding synonymity, multiplicity of meanings and definitions in relation with context information, and grammar/syntax rules for the purpose of semantically correct sentence translation. Another area of investigation deals with embedded database-driven user interfaces –at least- in all defined natural languages within the database, so that online content (utilized by presentation applications like the translation dictionary) can be automatically presented in the proper interface language.

Acknowledgements. Part of the work reported in this paper was carried out within the framework of the AENEAS project (contract 98AMEA19), funded by the EPET II Programme of the Greek General Secretariat of Research and Technology.

References 1.

Antona, M., Stefanidis, C., Kouroupetroglou, G.: Access to Lexical Knowledge in Modular Interpersonal Communication Aids. Augmentative and Alternative Communication, 15, (1999) 269–279

706 2. 3. 4. 5. 6.

7. 8. 9.

10. 11. 12. 13.

14. 15. 16. 17. 18.

C. Viglas and G. Kouroupetroglou Boualem, M.: Translation Engines: Techniques for Machine Translation. Speech Communication, 32 (3,) (2000) 220–222 do Amaral, M.B., Satomura, Y.: Structuring medical information into a languageindependent database. Medical Informatics, 19 (3) (1994) 269–282 Dorr, B.J.: Machine Translation: A View from the Lexicon. MIT Press (1993) Fuller, D., Lloyd, L., Schlosser, R.: Further development of an augmentative and alternative communication symbol taxonomy. Augmentative and Alternative Communication , 8, (1992) 67–84 Karberis, G., Kouroupetroglou, G.: Transforming Spontaneous Telegraphic Language to Well-formed Greek Sentences for Alternative and Augmentative Communication. Methods and Applications of Artificial Intelligence, Lecture Notes in Artificial Intelligence, Vol. 2308, Springer-Verlag, (2002) 155–166 Kouroupetroglou, G., Pino, A.: ULYSSES: A Framework for Incorporating Multi-Vendor Components in Interpersonal Communication Applications. C. Marincek et al (Eds) Assistive Technology, IOS Press (2001) 55-59 Kouroupetroglou, G., Pino, A.: A New Generation of Communication Aids based on th ULYSSES Component-Based Framework. Proc. of the 5 Int. ACM Conf. on Assistive Technologies, ASSETS 2002, July 8–10, Edinburgh, Scotland (2002) Kouroupetroglou, G., Pino, A., Viglas, C.: Managing Accessible User Interfaces of MultiVendor Components under the ULYSSES Framework for Interpersonal Communication Applications. C. Stephanidis (ed.) Universal Access in HCI, Lawrence Erlbaum Ass (2001) 185–189 Kouroupetroglou, G., Viglas, C., Metaxaki-Kossionides, C.: Web Enabled Teaching Aid for Non-orthographic Languages. Proc. of the SITE 2000, Feb. 8–12, San Diego, CA, (2000) 1962-1967 Kouroupetroglou, G., Viglas, C., Anagnostopoulos, A., Stamatis, C., Pentaris, F.: A Novel Software Architecture for Computer-based Interpersonal Communication Aids. Proc. of ICCHP’96, July 17-19, Linz, (1996) 715–720 Lowe, H.J.: Image Engine: an object-oriented multimedia database for storing, retrieving and sharing medical images and text. Proc Annual Symposium Computer Applications in Medical Care (1993) 839–843 Sutton, A., Gallagher, T., Morford, J., Shahanaz, N.: Relative clause sentence production using augmentative and alternative communication systems. Applied Psycholinguistics 21 (4) (2000) 473–486 Teich, E., Hagen, E., Grote, B., Bateman, J.: From communicative context to speech: Integrating dialogue processing, speech production and natural language generation. Speech Communication, 21 (1–2) (1997) 73–99 Viglas, C., Stamatis, C., Kouroupetroglou, G.: Remote Assistive Interpersonal Communication Exploiting Component Based Development. Proc. of ICCHP’98, 31 August – 4 Sept. 1998, Vienna – Budapest, (1998) 487-496 Wagner, J.C., Solomon, W.,D., Michel, P.A., Juge, C., Baud, R.H., Rector, A.L., Scherrer, J.R.: Multilingual natural language generation as part of a medical terminology server. Medinfo, 8 (1) (1995) 100-104 SYSTRAN translation system Web Accessibility Initiative