Web-Based System for Automatic Reading of Technical Documents for ...

Web-Based System for Automatic Reading of Technical Documents for Vision Impaired Students (an introduction of project ARET)

Jindˇrich Matouˇsek, Zdenˇek Hanzl´ıˇcek, Michal Campr, Zdenˇek Krˇ noul, Pavel Campr, Martin Gr˚ uber Faculty of Applied Sciences, Department of Cybernetics, University of West Bohemia, Plzeˇ n, Czech Republic

September 1 - 5, 2011

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

1 / 29

Outline 1

Introduction

2

System description System back-end System front-end Text-to-Speech technology Project-specific issues

3

Examples

4

Conclusion

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

2 / 29

Outline 1

Introduction

2

System description System back-end System front-end Text-to-Speech technology Project-specific issues

3

Examples

4

Conclusion

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

3 / 29

Introduction project ARET I

Automatic Reading of Educational Texts for Vision Impaired Students (Automatick´e ˇcten´ı uˇcebn´ıch text˚ u pro zrakovˇe postiˇzen´e studenty)

I

september 2009 – july 2012

solvers (partners) I I I

University of West Bohemia, Department of Cybernetics Primary School and the Kindergarten for the vision impaired in Pilsen firm SpeechTech, s r.o.

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

4 / 29

Introduction aim of the project I

innovation and enhancement of schooling of vision impaired students & facilitation of their self education F

I

Mathematics and Physics – ISCED 2nd level (5th - 9th grade)

development of a special system for automatic reading of technical (educational) texts F F F F

web interface (accessible via internet browsers, optimized for Firefox) back-end for educational texts administration (by teachers) front-end for educational texts studying (by students) our own text-to-speech system employed (cooperation with third-party screen readers possible, not implemented yet)

current state of the project I I

fully-functional system implemented many educational texts created and made available for students

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

5 / 29

Outline 1

Introduction

2

System description System back-end System front-end Text-to-Speech technology Project-specific issues

3

Examples

4

Conclusion

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

6 / 29

System description

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

7 / 29

Outline 1

Introduction

2

System description System back-end System front-end Text-to-Speech technology Project-specific issues

3

Examples

4

Conclusion

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

8 / 29

System back-end interface for educational texts administration available at http://ucebnice-admin.zcu.cz main text editor based on TinyMCE equation editor derived from DragMath Equation Editor

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

9 / 29

System back-end interface for educational texts administration available at http://ucebnice-admin.zcu.cz main text editor based on TinyMCE equation editor derived from DragMath Equation Editor

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

9 / 29

System back-end

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

10 / 29

System back-end

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

11 / 29

System back-end

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

12 / 29

Outline 1

Introduction

2

System description System back-end System front-end Text-to-Speech technology Project-specific issues

3

Examples

4

Conclusion

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

13 / 29

System front-end public web interface for displaying and reading the educational texts available at http://ucebnice.zcu.cz audio (MP3s with speech) generated by a web Text-to-Speech server MP3s played by JPlayer & Adobe Flash

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

14 / 29

System front-end public web interface for displaying and reading the educational texts available at http://ucebnice.zcu.cz audio (MP3s with speech) generated by a web Text-to-Speech server MP3s played by JPlayer & Adobe Flash

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

14 / 29

System front-end

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

15 / 29

System front-end

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

16 / 29

Outline 1

Introduction

2

System description System back-end System front-end Text-to-Speech technology Project-specific issues

3

Examples

4

Conclusion

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

17 / 29

Text-to-Speech technology Speech synthesis I I

built-in TTS system ARTIC cooperation with screen readers (matter of the user’s choice in the future - screen reader vs. built-in TTS system)

The goal ⇒ to generate speech signal meeting phonetic and prosodic requirements from the input text ARTIC system I

I I

variability in voices: 2 male, 2 female so far (it is going to be extended in the future) variability in speech rate: slower, faster the system is being improved constantly

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

18 / 29

Text-to-Speech technology Czech TTS system ARTIC (Artificial Talker in Czech) developed by Dept. of Cybernetics @ UWB and firm SpeechTech corpus-based concatenative speech synthesis method

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

19 / 29

Text-to-Speech technology Text processing I I

transcription of mathematical and physical formulas text filtering, normalization, phonetic transcription, ... Známe 3 Newtonovy zákony, ten 1. se nazývá "Zákon síly".

Známe 3 Newtonovy zákony, ten 1. se nazývá Zákon síly.

Text filtering

známe tři ňůtnovy zákony, ten první se nazývá zákon síly

Word substitutions

Phonetic transcription

Phonetic transcription

zna:me tQ\i Ju:tnovi za:koni

ten pr=vJi: se nazi:va: za:kon si:li

Phonetic filtering

Phonetic filtering

zna:me tP\i Ju:tnovi za:koni

ten prvJi: se nazi:va: za:kon si:li

Text normalization

známe tři newtonovy zákony, ten první se nazývá zákon síly

Synthesizer

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

20 / 29

Text-to-Speech technology Czech TTS system ARTIC (Artificial Talker in Czech) developed by Dept. of Cybernetics @ UWB and firm SpeechTech corpus-based concatenative speech synthesis method

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

21 / 29

Speech generation F0 Explicit prosodic description

TTS system ARTIC corpus-oriented concatenative speech synthesis method

#

d

;uu>:

m

Symbolic prosodic description:

50

70

120

preceding phone succeeing phone position in syllable position in word prosodeme type ...

d ...

# u"": P B 1-1 ...

d m N M 1-1 ...

90 ;u: # K E 1-1 ...

# duration (ms) 50 m ...

Acoustic unit inventory

... (Prosodic and/or spectral modification)

(Smoothing)

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

22 / 29

Outline 1

Introduction

2

System description System back-end System front-end Text-to-Speech technology Project-specific issues

3

Examples

4

Conclusion

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

23 / 29

Project-specific issues automatic reading of mathematical entities (formulas, expressions, notations) I I

I

I

transcription into corresponding word forms mathematical entities represented by a simple text or MathML code (more complex mathematical structures) developed system based on a special context-dependent rules for conversion from MathML to word form system simple extensible with new operators, reading exceptions etc.

text processing I

I

generally, technical texts contains many non-standard words (numbers, variables, symbols, abbreviations etc.) → conversion into gramatically correct reading form text filtering and normalization, word substitution, phonetics transcription and filtering

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

24 / 29

Outline 1

Introduction

2

System description System back-end System front-end Text-to-Speech technology Project-specific issues

3

Examples

4

Conclusion

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

25 / 29

Examples

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

26 / 29

Outline 1

Introduction

2

System description System back-end System front-end Text-to-Speech technology Project-specific issues

3

Examples

4

Conclusion

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

27 / 29

Conclusion

system is already employed and tested within classwork developed system is general and flexible - can be used for reading texts from other specific domains (with some modifications) future work I I I

new educational texts (Mathematics & Physics @ ISCED 2 level) enhancing system functionality (e.g. individual settings for each user) compatibility with other tools for vision impaired (cooperation with screen-readers)

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

28 / 29

Thank you for your attention. Project ARET CZ.1.07/1.2.00/08.0021 is co-funded by the European Social Fund and the State Budget of the Czech Republic.

J. Matouˇsek et al (UWB)

ARET

September 1 - 5, 2011

29 / 29