W3C MMI Architecture as a Basis for Enhanced ...

W3C MMI Architecture as a Basis for Enhanced Interaction for Ambient Assisted Living António Teixeira, Nuno Almeida, Carlos Pereira, Miguel Oliveira e Silva [email protected]

Overview

• Our motivations – AAL – Interaction

• …and goals

• Information on usage of MMI architecture in AAL projects University of Aveiro/IEETA is involved – AAL4ALL – Smartphones for Seniors – AAL Paelife

• Some movies and demo(s)

Motivations:

• Multimodal Interaction • Speech centric • Interaction for AAL

Human interaction

• Dialog • Speech … – Speech + gestures + expressions …

5

5

Long term goal • Development of Robust, Useful and Easy to Use HumanMachine Interaction • Using speech and language, • Supporting Portuguese native speakers – We have been involved in spoken interaction since the 90s 6

6

AAL Ambient Assisted Living

Motivations - Active Ageing • Based on the recognition of the human rights of older people and the United Nations (UN) principles of independence, participation, dignity, care and self-fulfillment; • Is the process of optimizing opportunities for health, participation and security in order to enhance quality of life as people age; • Refers to continuing participation in social, economic, cultural, spiritual and civic affairs • Allows people to realize their potential for physical, social, and mental well-being throughout the life course and to participate in society, while providing them with adequate protection, security and care when they need 8

Motivations – Tech Innovation • The introduction of technological solutions can facilitate the daily life of the elderly, fighting isolation and exclusion, increasing their pro-activity, work capacity and autonomy. • Addressing Accessibility – There is a need to overcome the gap between an individual's capabilities and the technological resources, • Addressing Usability – a term used to denote the ease with which people can employ a particular tool or other humanmade object in order to achieve a particular goal. 9

AAL and multimodality

• Strong focus on contextual usage, usability, pervasiveness, and the user

– Leading to highly heterogeneous environments

• Multimodality can provide an important push in shortening the gap between the user and its ambient • AAL challenges:

– necessity of including a wide number of devices – distributed focus of AAL – requirement of constant connection

10

W3C MMI architecture

• The architecture has the potential to solve some of our issues for interaction in AAL: – Heterogeneity • now a smaller issue

– Autonomy is increased – Usability through choice

Information on concrete uses

Example 1 - AAL4ALL •

AAL4ALL project has a specific task (under the leadership of University of Aveiro) to define the architecture and develop multimodal support for AAL Applications

•

W3C Architecture Adoption followed several phases –

–

–

the

task

First phase »

Some experiments with EMMA

»

Creation of an MMI interoperability test report

Second phase »

–

within

Tests with Interaction Manager based on SCXML (on the right)

Third phase »

First ‘blackbox’ modalities supporting Lifecycle Events communication – ASR, TTS with Interaction Manager iteration

»

Test framework based only on EMMA and Lifecycle Events

»

First integrated demo

Ongoing »

Developing and integrating more modalities (ex: avatar)

»

Work on fusion and fission capabilities

13

AAL4ALL Demonstrator - July 2012

• To prove the potential of W3C multimodal architecture, a cooperation was made between University of Aveiro and Microsoft Language Development Center (in Lisbonne). • Development of a simple tecnological demonstrator making use of input and output modalities developed in scope of the project. • The demo consisted in an multimodal integrated water heater controller. • It was presented at project general meeting in July 2012, Porto, Portugal 15

AAL4ALL Demonstrator - July 2012

16

Example 2 – Smartphones for Seniors System Architecture

• Windows Phone 7 • Web services to overweight features • Several modules to control modalities and intelligence

19

Multimodal Output

•

Graphics and Text – Panoramic view – Medication images generation – Medication details

• Speech – TTS (e.g. “The side effects of medication are…”) – Use of Natural Lang Generation • Ex: “You should take one tablet of Aspirin”

20

The movie …

21

Example 3 – PaeLife Personal Assistant

•

Goal: – Accessible and easy-to-use Assistant that integrates a set of services aiming at increasing social activity and socialintegration of the elderly, making use of multimodal interaction.

22

System architecture overview Platform

• Project adopted a system composed of both a stationary main unit and a portable unit. Home Portable Unit

Portable Unit (Premium Tablet)

Portable Unit (Low Cost Tablet)

Internet Main Unit authentification services Screen kinect PLA Database

Automatic Speech Recognition Office 365

Voice Search Text to Speech

Home computer

23

System architecture overview Platform: main unit

• Main unit integrates the mentioned big screen – supporting graphical output modalities and also sound and speech output

• Home computer • And input devices such as Microsoft Kinect – providing the means for the body gestures input modality 24

System architecture overview Platform: portable unit

• Integrated multi-touch support in the operating system makes possible the touch input modality • Integrated microphone and speakers will enable the speech input and output modalities • The display will support graphical output modalities 25

Interaction with PLA

• Since the PLA will be composed of two units each one being independent yet simultaneously interoperable • The following types of interaction are possible: – Only through the main unit – Only through the portable unit – Both through the main and portable units 26

Architecture

• The Paelife multimodal framework is directly based on the “standards” defined by the W3C – This choice is justified by the architecture’s open standard nature. – This architecture provides an answer to the previous requirements easing the creation and integration of new modules 27

Modalities

• Speech input/output modalities were already created

– Using Microsoft’s speech platform – Easy support for the several languages and future language packs for Elderly Speech ASR

• Input Modality for Body Gestures

– Uses the PaeLife Kinect Framework built for the PaeLife project for recognition of user gestures

28

Proof-of-concept demonstration of the Framework • To serve as a test for the produced framework, a demo application was created. • The application recreates a news reader.

DEMO

29

Acknowledgments • This work presented is part of the COMPETE - Programa Operacional Factores de Competitividade and the European Union (FEDER) under projects AAL4ALL (www.aal4all.org), Smartphones for Seniors (www.martphones4seniors.org) and Living Usability Lab for Next Generation Networks (http://www.livinglab.pt/). • Authors also acknowledge the funding for project PaeLife (www.paelife.eu) from AAL JP and national agencies: Part of the work presented was funded by FEDER, COMPETE and FCT in the context of AAL/0015/2009 and IEETA Research Unit funding FCOMP-01-0124-FEDER-022682 (FCT-PEstC/EEI/UI0127/2011). • Special thanks to all our partners