ivy – interpreting in virtual reality

Deliverable 6.3

IVY – INTERPRETING IN VIRTUAL REALITY

Deliverable 6.3 Pedagogical evaluation report Maria Tymczyoska [[email protected]] Marta Kajzer-Wietrzny [[email protected]] Sabine Braun [[email protected]] Catherine Slater [[email protected]] Nicholas Botfield [[email protected]] Margaret Rogers [[email protected]] February 2013

Project coordinator: Sabine Braun The IVY project has been funded with support from the European Commission. This report reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein.

© IVY – February 2013 511862-LLP-1-2010-1-UK-KA3-KA3MP

1

Deliverable 6.3

1

INTRODUCTION ....................................................................................................................... 4

2

RATIONALE ............................................................................................................................... 5

3

EVALUATION WITH INTERPRETING STUDENTS: OVERALL APPROACH AND METHOD..................................................................................................................................... 8

3.1

General procedure .................................................................................................................................. 8

3.2

Participants and procedure at each site .................................................................................................. 9 3.2.1 Adam Mickiewicz University (AMU).................................................................................................... 9 3.2.2 University of Surrey (SUR) ................................................................................................................. 10 3.2.3 University of Cyprus (UCY) ................................................................................................................ 10

4

OUTCOMES OF THE EVALUATION WITH INTERPRETING STUDENTS ................ 13

4.1

Quantitative analysis ............................................................................................................................ 13 Adam Mickiewicz University ............................................................................................................. 13 University of Surrey ........................................................................................................................... 14 University of Cyprus .......................................................................................................................... 14

4.2

Qualitative analysis: AMU .................................................................................................................... 15 4.2.1 Student journeys (AMU) ................................................................................................................... 15 AMU Case Study Student 1 – Frequent user ..................................................................................... 16 AMU Case Study Student 2 – Infrequent user ................................................................................... 17 AMU Case Study Student 3 – Moderate user .................................................................................... 17 Conclusions: Using IVY in blended teaching/learning ....................................................................... 18 4.2.2 Analysis of E-diaries and observations from tutorials....................................................................... 19 Self-perceived learning success ......................................................................................................... 19 IVY suitability for interpreter training ............................................................................................... 21

4.3

Qualitative analysis: Surrey .................................................................................................................. 23 4.3.1 Student journeys ............................................................................................................................... 23 SUR Case Study Student 1 ................................................................................................................. 23 SUR Case Study Student 2 ................................................................................................................. 24 SUR Case Study Student 3 ................................................................................................................. 26 4.3.2 E-diaries and observations from tutorials ......................................................................................... 27 Accessing the environment ............................................................................................................... 27 Use and choice of material ............................................................................................................... 27 Coping with task overload and need for guidance ............................................................................ 30 Development of learning strategies .................................................................................................. 31 Sense of presence and appreciation of 3D capabilities ..................................................................... 35 Collaborative work ............................................................................................................................ 36

4.4

Key outcomes of the evaluation with interpreting students ................................................................. 37

4.5

Effectiveness of evaluation methodology ............................................................................................. 40


2

Deliverable 6.3

5

EVALUATION WITH USERS OF INTERPRETING SERVICES ...................................... 42

5.1

Overall approach .................................................................................................................................. 42

5.2

Hands-on Workshop ............................................................................................................................. 43 5.2.1 Method ............................................................................................................................................. 43 5.2.2 Outcomes of the Evaluation Workshop ............................................................................................ 44 Participant profiles (demographic questions) ................................................................................... 44 Positive overall comments ................................................................................................................ 44 Concerns and critique........................................................................................................................ 45 Suggestions for improvement ........................................................................................................... 45 Usefulness for interpreter clients ...................................................................................................... 46 Effectiveness of design model ........................................................................................................... 46 System Usability Scale ....................................................................................................................... 47

5.3

Online Survey ....................................................................................................................................... 47 5.3.1 Method ............................................................................................................................................. 48 5.3.2 Module results .................................................................................................................................. 53 Respondent profiles (demographic questions) .................................................................................. 54 Pre-test .............................................................................................................................................. 57 What is interpreting? ........................................................................................................................ 58 Interpreting pitfalls ........................................................................................................................... 59 Impressions of the IVY environment ................................................................................................. 60 Post-test ............................................................................................................................................ 65 5.3.3 Key outcomes.................................................................................................................................... 69 Module respondents ......................................................................................................................... 69 Provision, content and delivery of training material ......................................................................... 70 Assessment of the IVY environment .................................................................................................. 71

6

CONCLUSIONS ........................................................................................................................ 73

7

REFERENCES ........................................................................................................................... 75

APPENDIX 1: INTERPRETING PRACTICE E-DIARY ............................................................. 76 APPENDIX 2: IVY EVALUATION QUESTIONNAIRES – INTERPRETING STUDENTS .. 80 APPENDIX 3: IVY EVALUATION QUESTIONNAIRE – CLIENTS ......................................... 83


3

Deliverable 6.3

1 Introduction The aim of the IVY project is to exploit the affordances of a 3D virtual learning environment and to integrate audio content and a comprehensive set of practice materials for interpreting students into that 3D VLE. Alongside this, the project seeks to raise a client-side awareness of what is involved in working with an interpreter, which is crucial in business and community settings. To this end, the IVY project has targeted a very heterogeneous group of end-users composed of students of interpreting, and (potential) clients of interpreting services. Work Package 6 of the project has focused on evaluating the usefulness of the IVY solution for interpreter and client training. The evaluation combines a functional and a pedagogical dimension. The Functional Evaluation (see Deliverable 6.1) provided an assessment of the design and functionality of the IVY environment and the user experience with the environment as a whole, including such elements as the ease of use and navigation, the experience of working with the audio content and learning materials, and the perceived ‘realism’ of the virtual environment (i.e. the sense of presence). The specific aims of the pedagogical evaluation included evaluating the quality of users’ experience (userfriendliness, sense of presence, 3D capabilities), the suitability of the IVY environment for training interpreting trainees (including interpreting students’ self-perceived learning success), and the usefulness of the IVY tool for raising the client-side awareness of working with interpreters. A multi-tier strategy was used for the pedagogical evaluation in order to account for the heterogeneity of the target groups. The students of interpreting first participated in induction sessions, and then conducted a guided self-study evaluation phase followed by questionnaires. The use of E-diaries gave students the opportunity to observe and reflect on their learning progress over a period of time. Two different approaches were used to reach (potential) clients of interpreting services and to conduct evaluation of the IVY solution from the clients’ perspective. This strategy enabled the consortium to reach a relatively large group of target group users. This report first provides a rationale for the pedagogical evaluation (Ch. 2). It then describes the approach to, and outcomes of, the evaluation with interpreting students (Ch. 3-4), followed by the approach to, and outcomes of, the evaluation with users of interpreting services (Ch. 5). The conclusions highlight the major insights that emerge from this evaluation, point to limitations of the current study, and outline areas for future study.


4

Deliverable 6.3

2 Rationale The idea of Computer Assisted Interpreter Training (CAIT) emerged in the mid-1990s as a spin-off from Computer Assisted Language Learning (CALL) and centred at the very beginning on providing e-learning materials for interpreter training. The need for such a development was clear, as it was long ago discovered that no interpreter training programme at any institution could guarantee the success of its participants without a sufficient self-study on their part. At first, CAIT assumed the form of speech banks, such as IRIS (University of Trieste 1999) or MARTIUS (University of Granada 2003), which were in fact well organised databases of audio, video and textual materials usually classified by the suggested interpreting mode or stage of interpreter training. There were also successful attempts for the development of dedicated courseware for interpreting students, e.g. Interpr-It (University of Hull 1995). All of these tools were prepared with the self-directed learning of individual students in mind, whereby the teacher served as a facilitator. Two decades later, CAIT now focuses more on blended learning, perceived as an indispensable complementary element of traditional interpreter training which provides opportunities for autonomous yet guided self-study. The focus has shifted towards more collaborative learning revolving around the students rather than around the teachers. The role of the latter would rather now be shifted “to help*ing+ students feel a part of the technology and not just its mere receivers” (Tymczyoska 2009). In a socio-constructivist approach, students immerse themselves in a learning environment and acquire knowledge more autonomously than was envisaged in the earlier transmissionist model. The computer has now become a much more powerful tool than just a mere database of audio and video materials. CAIT has entered the era of intelligent speech repositories (such as the SCIC multilingual speeches accompanied by SCICrecTM), conference system simulators (Melissi Black Box, ETI), and virtual learning environments (IVY) that also provide opportunities for group collaboration and teacher-supervised mock conferences (Sandrelli 2012). It has to be said, however, that in its current form CAIT mostly meet the criteria of the cognitive constructivist perspective on learning, which assumes that "[r]ather than dissecting skills, knowledge, or instructional content for easy digestion, (...) learning [should] be authentic and productive, learning tasks need to remain embedded in their larger, natural complex of human activity (Kiraly 2000:43). The IVY 3D environment has shifted the focus to meeting the principles of social constructivist approaches, which highlight the importance of the social embedding of learning for the process of knowledge construction. This has been achieved by combining “authentic situated action” (Kiraly 2000: 3), e.g. through the virtual interpreting scenarios, with “the collaborative construction of knowledge” (ibid.), e.g. in the Live mode, and “personal experience” (ibid), e.g. in the Interpreting practice mode. It should, however, be emphasised that the IVY environment is not intended as a replacement for in-class practice, as tempting as this may be in view of reduced contact hours at many interpreter training institutions and in view of the ongoing attempts to brand new types of distance (only) learning – especially massive open online courses (MOOCs) – as the ‘next big thing’ that will ‘revolutionise’ education. The IVY environment is firmly rooted


5

Deliverable 6.3

in the concept of blended learning, and the pedagogical evaluation was conducted with this concept in mind. The IVY solution is unique compared to other CAIT tools in that it provides students with a virtual environment in which they are faced with tasks, questions, and problems to be solved. By mirroring or recreating real-world communicative events (interpreting situations and conditions), this environment not only fosters student autonomy, i. e. the ability to take charge of their own learning process (cf. Holec 1981, Benson 2001), but is also conducive to collaborative learning and situated project-based learning (cf. Kiraly 2000, 2003, 2005) because it has the potential to engage groups of students working to attain the same goal. In a collaborative, student-centred framework, instructors are required to take on the role of “organisers, advisers, and sources of information” (Horváth 2007: 104), so they increasingly act as guides and counsellors assisting students in their learning process (Samson 2005: 102). In other words, guided autonomy rests on students’ ability to pace and direct their own learning process based on the tutors’ clear formulation of student learning outcomes. It is necessary to specify “how learning will empower or enable students, reflect intentions that guide teaching and learning, indicate how students can demonstrate skills and knowledge, and suggest how other types of learning such as values and attitudes might be inferred from student choices or actions” (Angelelli 2006: 35). The principle of guided autonomy has underpinned the pedagogical evaluation phase of the IVY environment with students of interpreting. For example, students have been asked to keep E-diaries during their self-study evaluation phase which encouraged them to share individualised and detailed comments about their interpreting practice. The information they provided has been analysed to assess their overall learning experience in the IVY environment with a focus on (perceived) learning success. E-diaries have been complemented by face-to-face tutorials which offered students an opportunity to discuss their use of the IVY environment and their approach to practising interpreting. In this manner tutors were able to resolve problems, elicit feedback and encourage self-reflection, while students had the opportunity to acquire and develop selfassessment skills, and take an active part in their learning processes. A self-study phase with E-diaries combined with face-to-face tutorials enabled evaluators to look into students’ perceived learning success. For the purpose of this project, this concept was initially defined as students’ perceptions of success following their IVY learning process. However, during the evaluation phase it transpired that students gradually developed an increased awareness of the interpreter’s role, and the duties and responsibilities which are integral to this profession. Finally, students said that the IVY tool helped them to develop and improve their self-assessment skills. Therefore, students’ perceived learning success has been re-defined as comprising students’ opinions and comments relating to: a/ interpretingrelated skills, b/ professional awareness, and c/ self-assessment skills. During the evaluation phase it also became apparent that the students’ perceived learning success was strongly related to their degree of satisfaction from working in the VLE created by the so-called sense of presence. Compared to traditional e-learning, the immersive properties of SL which create the sense of presence are increasingly seen as an important


6

Deliverable 6.3

educational affordance of the environment leading to greater cognitive engagement. For example, Sobkowiak (2012: 50) contends that the issue of immersion/presence is important for learning success in SL (see also Deliverable 3.1, Review of Best Practices, for a more detailed discussion). Therefore, we also relate students’ learning success with the sense of presence. Finally, in addition to the notions of guided autonomy and collaborative learning, students’ cognitive and emotional interaction with the SL environment could also be related to the long-established student approaches to learning identified in higher education pedagogy (e.g. Entwistle and Ramsden 1983; Biggs 1987). Approaches to learning are traditionally used to refer to the intentions that students have when they carry out academic tasks, as well as the strategies they apply to achieve their learning outcomes. Entwistle and Peterson (2004: 9) list typical features of the three approaches: 

 

Deep learners are characterised as meaning-seekers who analyse academic tasks both holistically and serially: they are able to uncover the underlying principles and relate them to previous knowledge and experience, while also being critical and monitoring their learning progress cautiously. Surface learners, on the other hand, are seen as reproducers of content who display a general tendency for memorising facts and limiting themselves to fulfilling the syllabus requirements. They also tend to feel pressure and anxiety towards academic tasks. The strategic/achieving approach to learning has emerged as an important concept from further research into student study patterns. It is understood as organised studying in the sense that it combines a focus on the academic content characteristic of the deep approaches with an effort to satisfy the demands of the evaluation system which is considered typical of the surface approach. Strategic/achieving approaches centre on effective management of the time and effort put into academic tasks in order to achieve personal and syllabus-related goals.

Important caveats to the distinctions presented above include the fact that approaches to learning are not stable characteristics but rather describe the relative prominence of a given approach to certain academic tasks which is content- and context-dependent. Although these distinctions do not do justice to the complex ways of studying adopted by individual students, they are nevertheless taken to be indicative of differences between the level of student engagement and the effort they put into their interpreting practice sessions in the IVY environment.


7

Deliverable 6.3

3 Evaluation with interpreting students: overall approach and method The pedagogical evaluation with the students of interpreting was carried out by three project partners (the University of Surrey, the Adam Mickiewicz University, and the University of Cyprus). The evaluation took place between October 2012 and January 2013. To ensure that the maximum resources were available to participants taking place in the evaluation at each test site,, the evaluation phase was staggered in the following way:  (AMU): from 24th October to 9th November 2012 at the Adam Mickiewicz University,  (UCY): from 17th November to 17th December 2012 at the Aristotle University of Thessaloniki, and  (SUR): from 02 November 2012 to 10 January 2013 at the University of Surrey. This Chapter describes the approach to the evaluation. The results are then reported in Chapter 4.

3.1 General procedure The evaluation procedure was similar across the three evaluation sites: students first participated in induction sessions where they were shown the IVY tool and how to operate it. They then conducted a self-study evaluation phase during which they were encouraged to use the IVY environment for their interpreting practice and were asked to note down introspective comments in their E-diaries (see Appendix 1). At the end of the evaluation phase, the students completed a questionnaire to sum up their IVY experience. The use of Ediaries gave students the opportunity to observe and reflect on their learning progress over a reasonable period of time. At SUR and AMU students received additional guidance from their course tutors in the form of tutorial sessions, during which field notes were taken by the IVY members. In general, the evaluation with the students of interpreting focused on eliciting comments about their selfperceived learning success which comprised students’ opinions about the degree of improvement of interpreting-related skills (including self-assessment skills) and a sense of professional awareness. The qualitative data also yielded important information about the development of learning strategies and the degree of student satisfaction from working in the VLE (sense of presence and appreciation of 3D capabilities). To collect more information about the participants, questionnaires (see Appendices 2 and 3) were used at all evaluation centres. This allowed the researchers to gain insight into the participant’s perceived computer literacy and prior gaming experience, as well as their evaluation of the helpfulness of the tool for the development of interpreting-related skills. The students completed the questionnaires at the end of the evaluation phase. A quantitative analysis and a qualitative analysis were carried out by the IVY team on the basis of the different types of data elicited during the evaluation phase.


8

Deliverable 6.3

3.2 Participants and procedure at each site 3.2.1 Adam Mickiewicz University (AMU) A group of 18 MA Interpreting students took part in this evaluation (their working languages were: DE, EN, FR, PL). They were students of the Conference Interpreting Programme which in the first semester consists of the following courses: Introduction to conference interpreting – Public Speaking, Interpreting Strategies, Note-taking, Interpreting as a profession. The evaluation was embedded in 2 core modules: ‘Interpreting Strategies’ and ‘Note-taking’. The first four weeks of the modules provide students with a generic introduction to interpreting, mainly covering active listening, mnemonic strategies and note-taking skills. After week 4, students begin their language-pair specific liaison and consecutive practice. The induction to the IVY environment took place in week 4, i.e. at the transition from the generic to the language-pair specific part of the module. The students received an induction (24th October) which lasted about 90 minutes. During the Induction phase, students were shown the IVY environment and its capabilities, and they were asked to create their own avatars at home and to communicate their ID details so that they could be admitted to the IVY island. They were then asked to come to individual tutorial sessions. During those sessions students were instructed how to work in the environment (working modes, pedagogic materials) and they were also guided how/when to complete the E-diary (see Appendix 1). The materials that students used included English, German and French monologues, and dialogues with English, German and French B-turns. Students were encouraged to interpret into their mother tongue first. They had also tried interpreting Polish monologues and dialogues into their respective B-languages by the end of the evaluation phase. The pedagogic materials available to them at the time included: transcripts of all Polish monologues and dialogues with Polish as the B-language, and learning activities. The learning activities included generic preparatory and reflective exercises (in English and in Polish) and language-specific preparatory and reflective exercises. In the case of Polish, language-specific activities were created for four dialogues and two monologues. During the self-study period (24 October-09 November), students worked in the IVY environment (interpreted and worked with pedagogic materials) and commented on their experience and self-perceived learning success in the E-diaries. They were able to contact their course tutors at any time if they encountered difficulties. Later it became clear that students mostly needed help to resolve technical problems. The final evaluation stage at AMU comprised a tutor-moderated in-class discussion where students had the opportunity to share their comments on the IVY experience, and then filled in a short questionnaire (see Appendix 2). Altogether 17 out of 18 students returned the questionnaires.


9

Deliverable 6.3

3.2.2 University of Surrey (SUR) A group of 22 MA Interpreting students took part in this evaluation (their working languages were: EL, EN, ES, FR, IT, ZH). The evaluation was embedded in a core module, ‘Applied Interpreting Skills’. The first five weeks of this module comprise a generic introduction to interpreting, mainly covering active listening, memorisation and note-taking skills. After week 5, students begin their languagepair specific practice in smaller groups. The induction to the IVY environment took place in week 5, i.e. at the transition from the generic to the language-pair specific part of the module. The students received 2 inductions (24 and 26 October 2012), each lasting about 1 hour. In the first induction session, the IVY environment and its capabilities were demonstrated to the students. Students were shown how to create an avatar and were asked to create their avatar, and to communicate their ID details so that they could be admitted to the IVY island (ideally before the second session). The second session was used to show students how to work in the environment and how/when to complete the E-diary (see Appendix 1). After these sessions, a number of small-group tutorials were scheduled (4 tutorials for each student), and the students were asked to send their E-diaries before each tutorial session so that the Surrey IVY team could read the E-diaries, draw out any common themes and address any technical/pedagogical issues. The sessions were held on the following dates: Session 1: 02 November 2012 Session 2: 16 November 2012 Session 3: 27-28 November 2012 Session 4: 10 January 2013 The purpose of holding these sessions was two-fold: firstly, as a way of evaluating the students’ use of the IVY Second Life environment and secondly, as a way of monitoring students’ progress and providing feedback on their self-learning approaches. The evaluation period therefore lasted from 02 November 2012 to 10 January 2013, i.e. a period of approximately 8 weeks. This period included four weeks of the Christmas vacation during which there were no classes, but the students had access to the IVY environment. Finally, the interpreting students at Surrey filled in a questionnaire (see Appendix 3) summing up their IVY experience. Out of the 22 students who took part in the evaluation, 16 students completed and returned the evaluation questionnaire.

3.2.3 University of Cyprus (UCY) The initial plan had been to use the students of the MA Programme in Conference Interpreting at the University of Cyprus. As the programme was cancelled due to budget cuts, UCY contacted the coordinators of the MA Conference Interpreting Programme at the Aristotle University of Thessaloniki and those of the Translation and Interpretation programme at the Hellenic-American Union in Athens. Athens did not have any interpreting students in 2012/13. However, students from the Aristotle University of Thessaloniki took part in the evaluation, under the guidance of UCY. A group of seven MA Interpreting © IVY – February 2013 511862-LLP-1-2010-1-UK-KA3-KA3MP

10

Deliverable 6.3

students from Thessaloniki completed this evaluation (their working languages were: DE, EL, EN, FR). The evaluation at the University of Thessaloniki was embedded in the MA in Conference Interpreting programme within the Faculty of Philosophy of Aristotle University of Thessaloniki, Greece. The programme reflects the core curriculum of the European Master’s Programme in Conference Interpreting,1 with components in Theory of Interpretation, Practice of Interpretation (Consecutive Interpretation, Simultaneous Interpretation), Linguistics, and the EU and International Organisations (including European history and culture, and the institutions of the EU). The induction to the IVY environment took place in month 9 of the programme. By that time students had completed the following modules: memory exercises, voice coaching, notetaking, content analysis, consecutive interpreting (with and without notes), sight translation, simultaneous interpreting (with and without text), information technology (interpreting resources, information retrieval, and conference preparation techniques), elements of linguistics, institutional structure and practices of the European Union and other international organisations. The students received a 1-day induction to IVY (18th October). In the morning students were shown SL and the IVY environment. In the afternoon students received personal inductions: they practised with Greek materials and did general and specific preparatory and reflective exercises. They were also instructed how/when to complete the E-diary (see Appendix 1). During the self-study period (17 November-17 December), students worked in the IVY environment (interpreted and worked with pedagogic materials) and commented on their experience and self-perceived learning success in the E-diaries. The materials that students used included Greek, English, German and French monologues and Greek, English, German and French B-turn dialogues. The pedagogic materials available to them at the time included transcripts of all Greek monologues and dialogues with Greek as the B-language, and learning activities. The learning activities included generic preparatory and reflective exercises (in English) and language-specific preparatory and reflective exercises. In the case of Greek, language-specific activities were created for four dialogues and two monologues. In the final evaluation stage, students were asked to fill in a short questionnaire (see Appendix 2). Five out of seven students returned the questionnaires. It should be noted that the small number of participants at this particular evaluation site was due to staff and student strikes. The temporary unavailability of university hardware coupled with students’ limited co-operation under the circumstances prevented the evaluators from carrying out SL inductions and tutorials with students as planned. Additionally, the evaluation period was hampered by technical difficulties both on the IVY 1

Developed by several European universities in collaboration with the Joint Interpreting and Conference Service of the European Commission (SCIC), the Directorate for Interpretation of the European Parliament, and DGXXII, in the context of the Thematic Networks Project (Area of Languages). © IVY – February 2013 511862-LLP-1-2010-1-UK-KA3-KA3MP

11

Deliverable 6.3

Induction Day and throughout the evaluation period. Despite instructions to bring their laptops, not all of the students did so on the IVY Induction Day, and some of the students could not access the university’s WiFi (authentication problems, weak WiFi signal). The afternoon session therefore took place in a nearby cafeteria that offered WiFi. Moreover, during the self-study evaluation phase, the poor quality of the students’ own hardware prevented them from installing and using the Second Life client software. Therefore, despite the remote support offered by the IVY team, most students accessed the IVY materials (audio content, learning activities) via the URL of the IVY Heads-up Display (HUD) rather through the Second Life software, thus missing out on the 3D capabilities of SL/IVY and on situating the monologues and dialogues in the virtual interpreting scenarios. Finally, it was difficult for UCY to monitor the students’ progress as all of them lived and worked in Thessaloniki. The UCY evaluation results should therefore be considered against this background.


12

Deliverable 6.3

4 Outcomes of the evaluation with interpreting students Of the types of analyses that were carried out with the evaluation data, the quantitative analysis, which will be reported in Section 4.1, is based on the answers given by the students at each evaluation site, and the qualitative analysis, reported in Section 4.2, focuses on the ‘journeys’ of individual students, in the form of case studies, and on an analysis of students’ E-diaries, along with an analysis of the observations from tutorials and student observation.

4.1 Quantitative analysis With regard to the basis for the quantitative analysis, it has to be stressed that not all students participating in the evaluation completed the questionnaire given to them at the end of the evaluation period. The general number of respondents per test site ranged from 5 at the Aristotle University of Thessaloniki to 22 at Adam Mickiewicz University. This makes the analysis presented in this section an indicative one, and the results should be considered together with the qualitative analysis. Adam Mickiewicz University There were altogether 17 respondents: 14 female and 3 male. The age of the Polish students ranged from 22 to 25. They assessed their computer skills on average slightly higher than intermediate (i.e. 3.35 on the scale from 1 to 5). The students identified themselves as novices at computer games. The Polish students at AMU spent on average 356 minutes in the IVY environment. As regards the overall usefulness of the tool, on the scale of 1 to 5 the Polish students rated it as 3.56. They reported that the use of IVY was most helpful for them in the development of comprehension skills (4) and improving their memory (4). It seems that they also benefited from IVY almost equally as much in improving their production skills (4) and their selfmonitoring skills (4). The usefulness of the tool for the development of preparation skills received a slightly lower rating, but was still considered it helpful (3.5). The students did not perceive the IVY environment as particularly useful in the acquisition of stress management skills (2.5). Altogether, the Polish students gave IVY much better scores than students in the remaining centres, suggesting that the slightly higher computer literacy of the students may have had a positive effect over the entire learning experience. As the tool was still in its testing phase, not all learning materials were available at the time of the evaluation period, which might be the reason why the students found IVY only a little helpful in the development of preparation skills. The fact that the students did not perceive IVY as particularly helpful in stress management might be related to the fact that from the outset the tool was designed to make the learning experience pleasant and “unstressful”.


13

Deliverable 6.3

University of Surrey Out of the 22 students (3 male and 19 female) who took part in the evaluation, 16 students completed and returned the evaluation questionnaire. Most students (10) indicated that they had not interpreted before starting the Masters course, one had done a semester of consecutive interpreting at seminars, two had interpreted in formal professional settings, and three had interpreted for informal or ad hoc occasions. The majority of students at Surrey were not native speakers of English. English was the most frequent B-language. The language combinations used in the IVY environment were FR-EN (6), EN-ZH (4), EN-IT (3) and EN-EL (3). Regarding the average time spent in the IVY environment, students indicated that they spent 211 minutes: in the learning activity mode (80 min) and in the interpreting mode (131 min) respectively. It should be noted, however, that students also worked outside of IVY: according to their own estimates, they spent about 70 min interpreting with IVY materials and about 260 min interpreting with non-IVY materials. The Surrey students thought that IVY was quite helpful for developing lexis and terminology in their A and B languages, and improving the rendition of features typical of spoken language (e.g. dealing with sloppy pronunciation, fast delivery, different accents, repetitions, incoherence in the source texts, etc.). Moreover, the students rated IVY as being quite helpful in the development of interpreting skills (e.g. memory training, note-taking, etc.). Although students perceived IVY as helpful with regard to the development of target text delivery skills (e.g. improving delivery and fluency of the target text, etc.), the results are less clear with regard to whether IVY was helpful in terms of improving their use of grammar in their A and B languages or in target text production (e.g. accuracy, completeness and coherence of target text). Students were also asked to rate the IVY Second Life environment as a place to practise interpreting and perform live interactions. In both respects students rated IVY as fair. As regards system usability, students considered IVY to be neither difficult nor easy. On average, students rated the environment as quite realistic. The audio quality of the monologues and dialogues was rated as quite good. University of Cyprus The number of questionnaires filled by the Greek participants does not allow any farreaching conclusions and the results of the quantitative analysis are therefore treated as indicative only. It also needs to be borne in mind that not all of the Greek students were able to access the IVY 3D environment, i.e. that some of them worked with the URL of the HUD (see above). There were altogether five respondents: three females and two males. The age of the Greek students ranged from 22 to 30. The students assessed both their computer skills and their gaming experience as almost intermediate (2.75) on a five-point Likert scale, ranging from 1 = ‘inexperienced’ to 5 = ‘very experienced’.


14

Deliverable 6.3

On average, the Greek students spent 146 minutes using the IVY environment or the HUD, which is lower than the time spent by the students at the other two sites and probably explains why the Greek students found the tool relatively less helpful than the students in other evaluation centres: on the scale of 1 to 5 the Greek students rated the overall usefulness of IVY as 2.30. It seems that the tool was relatively helpful for the Greek students in developing their self-monitoring skills (3), comprehension skills (2.75) and production skills (2.75). At the same time, they saw little or no benefit of IVY in their development of such skills as preparation (1.8), memory (2.1) and stress management (1.4). Such results are not surprising given the fact that the Greek students did not have the chance to get fully acquainted with IVY due to a relatively short exposure time and the technical difficulties they experienced (see Section 3.2.3). This confirms that the benefits from using a new tool like IVY can only be expected after the students familiarise themselves with the environment, which for technical reasons could not happen at the University of Thessaloniki. It must be also stressed that the evaluation was carried out in a period of great instability in the country. The fact that the Greek participants demonstrated a relatively low level of computer literacy and were slightly older than the students in other centres might have also had an impact on their IVY experience.

4.2 Qualitative analysis: AMU Students at AMU, UCY/Thessaloniki and SUR formed three heterogeneous groups with regard to their language and interpreting-training backgrounds. For example, students from the Aristotle University of Thessaloniki were in the ninth month of their programme, while students at the University of Surrey were still at the initial stage of their interpreter training programme. To draw out the different uses that the students made of the IVY environment as a consequence of this, a decision was made to analyse the journey of individual students in a case-study fashion using the data from the two evaluation centres where the pool of student data was the largest, i.e. at AMU and at SUR respectively. The case studies represent different experiences and approaches to learning. The analysis of case studies also led to a number of observations related to using IVY in blended teaching/learning contexts. In addition to the case studies, we also analysed all E-diaries with the aim of identifying comments relating to students’ self-perceived success, and students’ comments about IVY suitability for interpreter training.

4.2.1 Student journeys (AMU) The qualitative analysis of 18 students’ E-diaries at AMU has made it possible to identify three student profiles based on the amount of time they spent in IVY and the nature of their interpreting practice: FU – frequent user, MU – moderate user, and IU – infrequent user. These profiles were related to the deep, strategic/achieving and surface approaches to learning identified in higher education pedagogy (cf. Section 2, above). Below is a short summary of students’ interpreting practice sessions in relation to student profiles and their approaches to learning.


15

Deliverable 6.3

Table 1: Summary of student profiles, students’ interpreting practice sessions and learning approaches Student 1

Student 2

Student 3

student profile/ approach to learning

Frequent user (FU)/ deep approach

Infrequent user (IU) surface approach

Moderate user (MU)/ strategic approach

gender

F

F

F

total duration of interpreting practice

780 min

270 min

360 min

number of sessions

6 (in 6 days)

3 (in 2 days)

3 (in 3 days)

no. of dialogues practised

4

-

1

no. of monologues practised

3

3

2

The three students tried different types of interpreting depending on the perceived level of complexity of a given monologue/dialogue: they mostly did liaison practice but they also practiced consecutive interpreting with and without notes. With regard to the use of learning activities and transcripts, FU and MU used generic preparatory exercises and transcripts (whenever available). All three students also used the BACKBONE site to prepare for their interpreting sessions. The three students differed with regard to motivation, goals they set for themselves and skills they wanted to practice. There are also some interesting comments to be made about the structure of their interpreting sessions in relation to the three approaches to learning. Finally, the students included a description of problems they encountered in their interpreting practice and their different ideas about how to overcome them. AMU Case Study Student 1 – Frequent user In line with the defining features of the deep approaches to learning, FU was interested in studying challenging topics, learning new vocabulary, and handling difficult foreign accents in interpretation. This student focused both on practising active listening and memory skills, and on improving the fluency of delivery in the mother tongue. FU started with extensive preparation characteristic of the deep approach: she studied generic preparation activities and compiled a glossary based on her predictions from the brief and analysis of BACKBONE materials. She interpreted after listening to each fragment once only. She then analysed her performance by comparing the recording of her interpretation to transcripts. FU pointed out problems with interpreting fragments which included numbers and proper names but she said that listening again to problematic fragments and referring to transcripts solved most problems. FU also mentioned that transcripts helped her to assess the quality of her interpretation with regard to accuracy and terminological consistency. They also enabled her to detect false friends in her interpretations. Preparatory activities were considered helpful in predicting the level of difficulty (terminology, information content) © IVY – February 2013 511862-LLP-1-2010-1-UK-KA3-KA3MP

16

Deliverable 6.3

and anticipating the register and the topics to be discussed. This means that the student approached the learning process in a conscious and critical way and tried to understand and solve problems referring to previous academic knowledge and skills. AMU Case Study Student 2 – Infrequent user In contrast, IU (surface approach) worked mostly with familiar topics and materials which did not present any major linguistic challenges. Her main goal was to improve her memory skills and the quality of target language delivery, which included such aspects as fluency and elimination of empty gap fillers. She did not make any explicit comments about working to improve her comprehension and analysis skills. IU’s sessions lacked a clear structure and were quite superficial: it is unclear whether or not her preparation included term search using BACKBONE resources; she confessed to poor anticipation skills based on her analysis of interpreting briefs; and it seems she did not listen actively enough and was quite distracted by note-taking and learning new vocabulary because she had to listen to most fragments twice. It is also unclear whether she recorded her interpretations. Finally, she also wrote that she abandoned interpreting to return to language-specific preparation exercises, and that she did language-specific reflective exercises even before she started interpreting. IU had trouble recalling proper names, abbreviations and numbers, as well as remembering fragments which exceeded 100 seconds. She tried to address this problem by noting down some information but she did not say how she solved the problem of memorising the structure of source language fragments which she perceived as long. She also mentioned not being able to manage spontaneous meta-comments while interpreting. It appears that she did not find any solution to this problem. Finally, she said that she would get upset when she could not find a word or a phrase in the TL but she did not mention using paraphrase or any other strategy to help her overcome such lexical retrieval problems. Overall, her E-diary does not include much reflection on the development of learning strategies. It is for these reasons that her approach has been characterised as surface-based. AMU Case Study Student 3 – Moderate user MU (strategic/achieving approach) worked with topics she perceived as interesting but familiar. Her objectives in the individual interpreting sessions were to improve her interpreting skills by extending the length of interpreted chunks, to test the influence of preparation on the quality of interpreting, and to improve notation techniques. Altogether she focused on practising the following skills: preparation for interpreting assignments, anticipation, concentration, notation and good intonation at delivery. MU’s session procedures developed over time, especially in terms of preparation for the interpreting assignment. In general the sessions consisted of three stages: preparation, interpreting and assessment. At the very beginning (session 1), her preparation for the interpreting assignment was limited only to the familiarisation with the brief. Later she decided to extend this stage to preparation with the tools available in the BACKBONE corpus and additional external online dictionaries (session 2). Finally, her preparation included also a search in Wikipedia and other online resources (session 3). The preparation stage was


17

Deliverable 6.3

followed by the actual interpreting practice in which she tried to interpret longer and longer chunks (in each session). The student then analysed the quality of her interpreting, although it was only an impressionistic evaluation since she did not record herself. Thus, we can observe a clear progression towards a well-organised practice session (i.e. organised studying, a feature typical of the strategic approaches). MU identified two major problems during her interpreting practice i.e. memorising longer utterances (chunks) and comprehending difficult fragments in the source text. Initially, she attempted to divide the utterances into shorter fragments. Over time she discovered that it would be a better strategy to try to memorise a longer utterance that she could comprehend because this would enable her to render the entire message, including as many details as possible while still paying attention to a consistent and cohesive delivery. She also realised how helpful a good note-taking technique was and that it had to be accompanied by good concentration. After the first listening her interpretation practice would be followed by self-assessment and another interpreting attempt. The student noticed as well that she needed to pay more attention to the linguistic aspects of her interpreting, such as a good intonation at delivery and sufficient terminological preparation. Conclusions: Using IVY in blended teaching/learning Based on the AMU case studies, several remarks can be made with regard to using IVY in blended teaching/learning contexts. 





From a tutor’s perspective, it can be seen that students tried to improve too many skills at once and set themselves goals that were too ambitious, and that they paid considerable attention to the quality of their output at the expense of developing active listening and analysis skills. This highlights the importance of tutor guidance in the blended learning experience. Tutors should set a hierarchy of skills to be developed and a set of strategies that can be used to attain those skills. For example, before students attempt note-taking they can be advised to listen to identify the main ideas, and then listen to identify supporting information and evidence to create a mental map of the macro- and microstructure of the spoken text. It seems that students draw greater benefit from well-structured sessions which start with the development of good preparation skills using the IVY learning activities, transcripts and online resources such as BACKBONE. Students also think that transcripts are a useful tool in the self-evaluation phase because they allow them to compare their interpretations to the source text. This means that it would be pedagogically sound for tutors to suggest ways in which interpreting sessions in IVY should be structured in order to allow students to benefit most from the IVY environment. For example, students can be advised to try active listening and reformulation before attempting liaison interpreting, or they can summarise and note down the main ideas of a speech before attempting to interpret it consecutively using notes. Overall, students tried to develop some strategies to help them deal with proper names and numbers and to remember longer, more complex utterances. Strategies for handling such interpreting-related challenges have been presented in the generic skillsbased exercises (cf. Appendix 2 to “Learning Activities”, the Report accompanying Deliverable 5.3) which were made available to students in the IVY environment.


18

Deliverable 6.3

However, it appears that students who tend to adopt strategic/achieving and surface approaches to learning would benefit from more tutor guidance, particularly as regards mnemonic and note-taking strategies. Such guidance can be provided in traditional faceto-face classes, for example as part of a note-taking course.

4.2.2 Analysis of E-diaries and observations from tutorials In addition to the case studies presented above, we also analysed all AMU E-diaries with the aim of identifying students’ comments relating to the IVY environment. The analysis of the E-diaries shows that the students’ comments mostly relate to their self-perceived learning success and the suitability of the IVY environment for interpreter training. Self-perceived learning success Self-perceived learning success was discussed by the students in terms of their development of interpreting-related skills, professional awareness, and self-assessment skills. With regard to the development of discrete interpreting skills, the students reported that their work in IVY has helped them improve their interpreting performance with regard to preparation, comprehension, memory, and production. The following comments illustrate each aspect.









 

  

Preparation “I realised that anticipating and preparing useful terms and expressions beforehand has helped me to reduce the unpredictability of the original speech.” and “Thanks to preparation I learned new terms in the SL without which I would have been unable to comprehend the speaker and thus fail to interpret the message.” (AMU Student 12, session 2); "I learned about the traveller community in Ireland and I learned a few new terms" (AMU Student 5, session 2); “I practiced anticipation and I actually managed to anticipate the speaker successfully on a number of occasions” and "I learned new medical terms and phrases (names of hospital wards)" (AMU Student 10, sessions 1 and 3 respectively). Source text comprehension “I managed to focus on active listening and identifying the structure of the speech” and "Despite difficulties I tried to convey the sense of the message after the first listening" (AMU Student 12, sessions 1 and 3 respectively) Memory “This time I managed to take better notes. This exercise also helped me to memorise symbols better” (AMU Student 1, session 4) "I noticed the importance of concentration and the necessity to share attention between listening and note-taking (…) Note-taking was very helpful in this interpreting. It took the burden off my memory and enabled me to interpret numbers and proper names" (AMU Student 12, session 3) "With every new fragment I would remember longer utterances “ (AMU Student 16, session 5) “I can now share my attention more efficiently” (AMU Student 6, session 3) Target text production "I think I managed to achieve my goals. After listening to my interpretation I managed to identify some minor mistakes but I realised that my output in Polish was quite fluent and that I spoke more slowly and calmly" (AMU Student 17, session 5)


19

Deliverable 6.3

 

 



“I think I have improved the cohesion of my interpreting output. I managed to avoid empty fillers" (AMU Student 12, session 3) “Because I studied the vocabulary before interpreting this dialogue, my interpretation was more fluent and I had a better control of my voice. (...) I know that if I encounter similar vocabulary in the SL I will be able to interpret it well” (AMU Student 1, session 3). "Thanks to previous sessions I realised that I am getting more focused and more fluent.”(AMU Student 4, session 6) “I learned to select information. (...). I realised that notes help me to construct my interpretation.” (AMU Student 6, session 2); "I overcame the problem of irregular breathing, I successfully controlled my stress reaction. (…) I realised that only the most important information should be noted and that I should trust my memory more.” (AMU Student 6, session 3); “I gradually improved the fluency of my delivery.” (AMU Student 8, session 4)



“I realised that I became more focused and found it easier to choose appropriate words. I became more aware of the importance of choosing the right words.” (AMU Student 13, session 1)



"With every fragment I noted a greater fluency and a better control of emotions” (AMU Student 16, session 3)

These quotes show that overall, students perceived improvement in terms of subjectspecific knowledge (wider range of lexical choices), listening comprehension (attentive listening, anticipation), efficiency in selecting information and analysing the structure of SL utterances, note-taking (new symbols), concentration and memory skills (split attention), fluency and control of prosody, and a better control of emotion (stress management). There were also a number of comments in students’ E-diaries which pointed at the process of developing an awareness of the interpreter’s role and responsibilities. For example, there was a student who realised the importance of generalising when lexical retrieval proves impossible or takes long enough to impair the reception of the interpretation by the audience: “there were phrases which I found difficult to interpret and which led to long pauses in my output but this made me realise that sometimes one has to interpret some ideas more generally in order to avoid long pauses in the interpretation” (AMU Student 7, session 2)

Another student commented on the importance of preparation for professional interpreting assignments: “I know that in the future I would have to prepare for every interpreting assignment so in this case I should have read about Cannes before I started interpreting” (AMU Student 17, session 6).

The following student’s opinion does not really apply to trainee interpreters in the beginning stages of training, particularly since the student reports interpreting fragments she did not fully comprehend: "I assumed that in professional situations I would not have a chance to listen to the original speech twice, so I decided to interpret right after the first listening even if my comprehension was not satisfactory" (AMU Student 5, session 1)

However, it certainly suggests that this student tried to behave professionally from the very start of her interpreting practice. Her subsequent interpreting sessions were more geared


20

Deliverable 6.3

towards training discrete skills, such as understanding the SL message, and she became more focused on analysing its structure. In terms of self-assessment, most students reflected on the quality of their performance. Some students were quite critical of their interpreting performance, others were more understanding, but most of them included constructive comments about how they think their performance should improve. We selected the following comments because we consider them to be very important from the point of view of students’ perceived learning success: "I realised that when I interpret I speak very fast and I use a lot of empty fillers. Moreover, I tend to make too many gestures. (...) My goal right now is to speak more slowly and to use fewer gestures.” (AMU Student 14, session 1) “Listening to my interpretation a couple of times and comparing it to the original speech (using transcripts) helped me to detect and eliminate many mistakes” (AMU Student 15, session 1) “As I listened to my interpretation I realised I should work on my grammar more” (AMU Student 6, session 2); “As I was listening to my interpretation I spotted a tendency of repeating the same words and phrases. I need to widen my vocabulary (also in my mother tongue) and try to look for synonyms. (…) Interpreting the interview in the vinyl records shop made me reflect on how to handle incoherent utterances in the SL” (AMU Student 6, session 3)

This means that autonomous practice can help students to develop self-assessment skills. Students found the transcripts of monologue and dialogue materials very useful because they could compare them to their recorded interpretations (students used Audacity to record themselves). In fact, 72% of students reported using transcripts in their E-diaries, 39% said they used the generic learning activities (mostly for preparation), and 33% wrote that they found the language-specific learning activities useful (both preparatory and reflective). It seems that students would also benefit from peer feedback in the IVY Live mode, but this mode was not fully operational at the time of testing. Nevertheless, AMU students regularly gave and received feedback from their peers in face-to-face interpreting and general English classes. Otherwise it would have been more difficult for them to reflect on their own performance. IVY suitability for interpreter training During the evaluation phase it became apparent that students’ self-perceived learning success was related to their degree of satisfaction from working in IVY. Satisfaction, in turn, was related to the immersive properties of the environment which create a sense of presence. Negative comments on the perceived sense of presence can, in most cases, be traced to insufficient computer hardware (see below). A number of students, especially those who did not experience hardware problems, commented positively on their IVY experience saying that this environment provided them with a range of different monologue and dialogue materials with native speakers who spoke naturally. They praised the orality of IVY materials (pauses, speaker’s self-corrections, etc.)


21

Deliverable 6.3

and they compared them favourably to scripted speeches. They also said that the life-like character of the IVY environment made it easier for them to prepare for professional practice. There were also comments suggesting that the SL environment was perceived as immersive enough to create a sense of presence that would lead to greater student involvement in the evaluation period. In order to benefit fully from the learning environment and experience the sense of presence, IVY users had to master a number of basic Second Life operations. These included setting up their profiles and choosing avatars; moving between different locations (walking, flying, and teleporting); socialising and communicating with different SL users and other students on the IVY island via groups, instant messaging, text chat, and voice chat. Some students needed tutor support to master these skills. As mentioned above, one of the problems that students had to tackle related to hardware deficiencies. The idea behind IVY has been that students practise interpreting from home and in their own time. However, it turned out that some students did not have personal computers or that their personal hardware was insufficient for the SL client to run smoothly (poor graphics or RAM parameters). Very frequently access to broadband was a problem. There were times when the SL platform itself generated technical obstacles. Sometimes the platform was unstable which made it impossible to log in to the IVY island (in such cases users received notifications saying ‘This region may be experiencing trouble’). Finally, a few technical difficulties resulted from temporary imperfections of the testing version of the IVY island. Students reported that occasionally recordings or exercises were missing from the learning activities. Occasionally the recordings ended abruptly, in awkward moments, or overlapped. A few remarks concerned player functionality or missing robots in interpreting locations. Irrespective of the issues reported above, students appreciated the following aspects of IVY:  diverse materials in many language combinations  interesting topics  different speaking styles and accents  transcripts useful in spotting mistakes  good learning activities  appropriate length of recordings  not stressful  opportunity to compare the same dialogues recorded in different languages  opportunity to learn more about interpreting, to practise interpreting in working languages, to learn more about different cultures. Based on these comments it seems that the benefits of the IVY solution tend to outweigh the negative comments. Problems related to personal hardware will disappear gradually as students acquire better equipment. While the issue of temporary unavailability of the SL platform (maintenance) is unavoidable, tutor support is bound to eliminate problems stemming from student’s insufficient computer skills.


22

Deliverable 6.3

4.3 Qualitative analysis: Surrey As in the section above reporting the results from AMU, the present section will first present three different student ‘journeys’ to illustrate individual uses of the IVY environment by students in Surrey, before reporting the outcomes of the analysis of the Surrey students’ E-diaries, which will give more global insights into the Surrey students’ experience of training in the IVY environment.

4.3.1 Student journeys From the total number of 16 students who took part in the pedagogical evaluation at the University of Surrey, the following three students were chosen as sample case studies as they represent a cross-section of the students’ spectrum of experience at this site: Student 1 struggled from a technical point of view; Student 2 could use the environment but did not really make (best) use of it for self-study; Student 3 was able to use the environment and used it relatively extensively. Below is a short summary of the students’ profiles and their experience in IVY reports on the students’ self-perceived previous experience with online platforms and virtual worlds, the total duration of interpreting practice, number of attended tutorials, encountered hardware problems and their overall feeling about the IVY experience. Table 2: Students’ profiles and experiences in IVY environment Student profile

gender self-perceived previous experience with online learning platforms and virtual worlds total duration of interpreting practice (including learning activities) no. of sessions no. tutorials attended hardware problems overall IVY experience

Student 1 technical problems with the environment, ineffective use F extensive

Student 2 no technical problems, ‘flat’ use

Student 3 without technical problems, effective use

F good

F none

510 min

660 min

600 min

4 3 yes mixed

3 4 no positive

2 3 no positive

SUR Case Study Student 1 This student was familiar with computer-based learning programmes, online learning platforms and virtual environments and said she had a lot of experience with these tools. The student completed three E-diaries (sessions 1, 2 and 4) and the questionnaire, and attended 3 of the 4 tutorials.


23

Deliverable 6.3

Her first practice session focused on accessing the environment and, while she was able to get into the IVY environment, she could not get the player to work. She commented that she “liked the look of the material on the menu” but was “a bit concerned about running speed, usability, time and effort”. For her, access to the audio was of much greater importance than being in the environment. She received significant technical support in the period between the first tutorial and the second E-diary. In the second E-diary she reported that she could hear the sound but that the quality was “ok but variable”. There were clear hardware problems as she said that “the system seems to freeze and slow down randomly, buffering for moving around is disruptive” and that the “system seems very clunky”. This was followed by further audio player problems. Not surprisingly, she expressed concern that she was taking time and energy away from other tasks “with higher priority”. In the final E-diary she listed further technical issues but also mentioned that an aim of the session was to “practise two dialogues”. The student is clearly still attempting to work in SL but she reports that “nothing actually loaded or ran”. Her responses in the questionnaire confirm her experiences with the IVY environment. She comments that she had “several attempts to use SL that did not succeed were not converted into some other strategy immediately”. She described these attempts as “very stressful and disruptive” and as “very frustrating”. As mentioned above, however, this student rated the environment as “very realistic”, even though she rated the environment as being very unhelpful for all aspects of interpreting practice. For this student, accessing and working in the IVY environment was not successful and she therefore had a negative attitude towards and perception of the environment. The availability of the audio outside SL was considered by her to be “the most useful item and should be the focus rather than virtual reality, furniture, etc”. In other words, the access problems prevented this student from experiencing the most important features of the 3D environment. This student’s experience highlights the importance of providing students with access to appropriate hardware for the SL environment, which may still constitute a challenge for some institutions.2 SUR Case Study Student 2 This student was also familiar with computer-based learning programmes, online learning platforms and virtual environments (e.g. University-level grammar software) and said she had quite a lot of experience with these tools. Her profile was therefore similar to that of Student 1. The student completed three E-diaries (sessions 1, 2 and 3) and the questionnaire, and attended all 4 of the tutorials. In her first E-diary she recorded two practice sessions. In the first session she did not seem to use the Second Life IVY environment or any of the IVY resources. She practised notetaking skills by listening to a speech, taking notes and then trying to repeat what was said 2

At the University of Surrey, however, appropriate hardware was available, and it is not entirely clear why the student chose not to use these (other than the argument that she would obviously have preferred to work with the environment from home and in her own time). © IVY – February 2013 511862-LLP-1-2010-1-UK-KA3-KA3MP

24

Deliverable 6.3

without changing languages. The student was able to identify what her main problem with note-taking was (“I couldn’t always note enough so I sometimes had to miss out important details when I was repeating what was said”) but did acknowledge the progress she was making (“It is getting easier as I get better at using symbols and abbreviations”). In her second practice session, she used the IVY bookshop dialogue and asked someone to read the transcript aloud which she then interpreted. She included a comment reflecting on her practice session in which she stated that she found it “useful because I found certain terms which I didn’t know and it helped to practice interpreting into as the majority of practice I had been doing recently was note taking”. The notes on sessions 1 and 2 are narrative entries which mention websites and specific online articles relating to the topic of the materials she was working with. She writes about her approach to subject-specific research but does not make use of the space provided to detail resources or URLs used. Her second E-diary also contained entries on two practice sessions. The first entry is brief but she indicates that she has used the Second Life IVY environment and the IVY resources (one of the prepared monologues and the general preparation exercise). She comments that she found this useful because “it was helpful to go over the types of aspects that should be researched before doing an interpreting assignment”. It should be noted, however, that this practice session was timed 30 minutes and that the student does not state whether she actually interpreted the monologue or not; her practice session could simply have consisted of doing the preparatory exercises. Her second session reported in this e-diary is similar, but she had changed materials and used a dialogue instead. Her notes to the sessions included links to websites and some terminological/lexical items in a short glossary but she simply notes equivalence and there are no usage notes to accompany the resources. The third E-diary consists of just one practice session lasting two hours. She reports on a group practice session that did not make use of the IVY environment or the IVY resources and explains how the session was organised. In the following tutorial, the students in this group said that they did not prepare in advance, perhaps showing a difference between what an individual student does during individual practice sessions and what happens prior to a group practice session. In the questionnaire, this student indicated that she had used the monologues and dialogues in the IVY SL environment for a total of 7 hours for the whole evaluation period. In general terms, she rated the environment and the resources quite positively and rated the helpfulness of the environment in practising and developing interpreting skills as “very helpful”, although she somewhat disagreed with the statement that the IVY environment helped her to achieve a more fluent delivery. She neither agreed nor disagreed with the statement “I would recommend the IVY environment to other trainee interpreters”. It would seem that this student used the environment successfully in terms of access and being able to work with the materials but her responses in the questionnaire indicated that she thought the environment was perhaps better suited to some tasks than to others, e.g. very good for memory training, note-taking, etc., but not very beneficial for improving target text delivery.


25

Deliverable 6.3

SUR Case Study Student 3 Student 3 had not had any experience with computer-based learning programs, online learning platforms or virtual environments. The student completed two E-diaries (sessions 1 and 2) and the questionnaire, and attended 3 of the 4 tutorials. In her first E-diary this student reported on a four-hour practice session where she worked in the IVY environment using two monologues (Irish Primary School System and Santa Fe Chamber of Commerce). Her E-diary entry is extremely comprehensive, focusing on the aims of the session and what she did, and evaluating the usefulness of the session. Her aims, however, were very broad for a first practice session (“to focus on active listening, comprehension and also note-taking”). She concluded that the session was useful for listening, memorising and note-taking, but that it was not very useful for her for practising the target language. This is perhaps due to a misunderstanding that the IVY environment could be used for recording the students’ target text output as she says she “did not manage to register my turn, probably because of a technical problem”. Nevertheless, in her second E-diary this student reports using a recording device to record her output. Also in the second E-diary this student took part in group interpreting practice sessions as well as individual practice. She did some work in the IVY environment although this time found the sound quality “a little poor”. Her assessment of her sessions shows an awareness of her own progress and an ability to contrast what she was doing in earlier practice sessions with a more refined approach to these sessions. Some of her comments suggest a focus on efficiency e.g. “I am trying to eliminate the stress of note-taking as much as I can and I’m trying to find the right way to get the best out of note-taking”. Her E-diaries seem to suggest that she is actively thinking about and reflecting on what she is doing and why. She used the notes sections of the e-diaries for subject-based knowledge e.g. facts about the school system in Ireland that she had learned and for summarising what research she had done. Resources and information sources were not recorded in the e-diary. This student spent 7 hours in the environment (4 hours on monologues, 2 hours on dialogues and 1 hour in the live mode) and 3 hours working with the learning activities. In addition, she spent 2 hours per week using the IVY materials outside of the SL environment. Overall, she rated the IVY environment positively, and either somewhat agreed or completely agreed with all the statements relating to the various ways that the environment could support her interpreting development and she completely agreed that she would recommend the IVY environment to other trainee interpreters.


26

Deliverable 6.3

4.3.2 E-diaries and observations from tutorials Early discussions in the project, the focus groups, and the analysis of the E-diaries and tutorial observations identified a number of themes. These are used here to report the main findings of this analysis. Accessing the environment Given the steep learning curve reported in the virtual world pedagogy, the students were given two induction sessions to show them how to access, use and learn in the environment. There did not, however, seem to be any issues with ease of use (perhaps disproving the point often made in the literature; our results of the functional evaluation showed the same tendency). The E-diaries completed for the first tutorial did show that many students struggled with access and problems with functionality. For example, one student showed some frustration at the functionality of the system/player. Another student was not able to move beyond the reception area and saw everything coloured in pink, although he did not show any frustration with these initial problems as he was very keen to use the environment. It was therefore of vital importance that adequate technical support was available to the students at this induction stage. A drop-in technical support helpdesk was organised to assist students with any access or functionality issues they experienced. Typical problems that the students encountered were slowness (mainly due to the fact that the students’ computers did not have the technical specifications required to run Second Life) and not being able to hear the audio. These issues were addressed in the first tutorial sessions and there were minimal technical problems reported in the second set of tutorials. It would be fair to say that, excluding three students whose laptops were unable to support Second Life, all students were able to access the environment by the end of the second set of tutorials (mid-November 2012). The students who did encounter functionality issues, however, did persevere but were ultimately put off by the difficulties (see Case Study for Student 1). Use and choice of material 

The feedback made it very clear how important it is for students to have a good choice of material. In their e-diaries, several students commented on why they chose materials to practise with, e.g. “I used the Bristol monologue because being university-related it sounded like something easy to start with.” *SUR Student 4, E-diary session 2] “*I+ practised the dialogue ‘Drama at school’ from Greek to English because I found it very interesting and something that is missing from Greek schools.” *SUR Student 5, Ediary session 1] “*I+ chose the Irish primary school system monologue because at the beginning it seemed a rather accessible theme and since this was my first training session I felt the need to start with something that would make me feel more comfortable.” *SUR Student 3, E-diary session 1] “I started with this video as it was one we had done in class and more or less remembered what the topic was about.” *SUR Student 6, E-diary session 1]


27

Deliverable 6.3

“I used the bookshop dialogue *…+, I did this because in groups we had looked at about half of the dialogue and I wanted to go through the rest.” *SUR Student 2, E-diary session 1]



The importance of having ready-made content in the IVY environment can also be seen from some (anxious) comments about missing languages (which were not available in the IVY environment at that time). “I did interpreting practice on my own without using the IVY environment because there is no Chinese-English material for me to practise” *SUR Student 7, E-diary session 1] “no Italian monologues for practise into English which would be direction for exam.” [SUR Student 8, E-diary session 4]



Students also requested more and different materials to be included in the IVY environment, e.g. one student wanted more specialised material (such as business/ economics topics) and commented that the IVY materials were too general [SUR Student 4, session 4 tutorial]; however one student [SUR Student 7, session 4 tutorial] commented on the difficulty in specialised fields and understanding what is going on in the source text before being able to interpret. One student asked for further materials to be added to the environment as they were getting “used up” *SUR Student 8, E-diary session 4]. This comment can perhaps be contrasted with the approach taken by another student who re-used the monologue and dialogue transcripts outside Second Life and in other practice contexts, therefore maximising the different ways the resources can be used [SUR Student 9, session 4 tutorial]. These observations highlight three aspects: firstly, the students’ desire to have more practice material suggests that they adopted and accepted the IVY environment quickly, that they practised intensively (i.e. intensively enough to “use up” the available materials); the creation of a sufficiently large pool of digital educational content will remain a challenge even for European-wide partnerships with a multilingual ‘production force’; but also that students need further guidance on how to make best and multiple use of the available content.



With regard to the last point, it is noteworthy, however, that the students did not restrict themselves to accessing the audio in SL but made use of the materials (e.g. learning activities) and transcripts in other ways, e.g. printing out the transcripts and one group member reading the text aloud for others to practise interpreting skills or to interpret [SUR Student 7, E-diary session 1+, using the transcripts to check the student’s rendering: “I managed to listen and take notes on long passages but I also used the transcripts to be sure that I covered all points”. *SUR Student 3, E-diary session 1]



The students made use of the learning activities (generic, skills-based and reflective exercises) to varying extents, and to a greater or lesser degree of systematicity. For example, some students used the learning activities outside Second Life when working with other materials, some students used the activities when working with the BACKBONE videos (although this would normally have been triggered by the instructions in the learning activities themselves, as these frequently ask students to use the


28

Deliverable 6.3

BACKBONE videos, e.g. for preparing to work with the IVY monologues and dialogues), some accessed the learning activities for the IVY dialogues and monologues via the SurreyLearn online course management system rather than through the IVY SL environment [SUR Student 4, E-diary session 2]. None of the students reported using the skills-based activities in their E-diaries or during the tutorials. 

The audio quality was also mentioned by a couple of students: “the sound quality was a little poor” *SUR Student 3, E-diary session 2] “The sound on IVY was not so good” *SUR Student 5, E-diary session 3]

It is important to note, however, that in the questionnaire the audio quality was rated as quite good or very good by 10 out of 13 responses. The student who said the “sound on IVY was not so good” did not complete the questionnaire and the student who said that “the sound quality was a little poor” rated the audio quality as “quite good”, suggesting that there might be individual materials with poorer sound quality but overall the sound quality was acceptable. 

In terms of turn length, there were some different views about the length of the turns in the IVY materials: “It was helpful *practising interpreting+ in Second Life because the clips were short and allowed us to practise, think and rephrase.” *SUR Student 10, E-diary session 3]

One student commented that it was not necessary to take notes when the sections are so short [SUR Student 11, session 3 tutorial], although another student [SUR Student 12, session 3 tutorial+ commented that she was “sometimes distracted by the shortness of the utterances” and would like longer sections of text to work with. This difference in the student evaluation of the turn length suggests that turn length could either be a matter of personal user preference, user ability or the extent to which the turn length suits the activity or skill being practised. Most importantly, however, it points to the necessity of discussing this more explicitly with students so as to make them aware of characteristic turn lengths (and required skills in terms of note taking and/or working from memory) for different interpreting situations. 

It should also be reported that, not surprisingly, the IVY materials were not the only materials used for interpreting practice during the evaluation phase. Other materials included videos found on YouTube, the Technology, Entertainment, Design (TED) talks, the BACKBONE videos, news bulletins, and materials given the students during their interpreting classes (cf. also section below on Development of learning strategies). There was evidence that some students developed a certain degree of systematicity in their approaches, although by tutorial session 3 (week 9 of their MA course), many sessions were still either ad hoc or unprepared. Some students demonstrated changes in approach depending on whether they were working in a group or individually (e.g. one student [SUR Student 12] reported in tutorial session 3 that she found the general preparatory exercises “really helpful” and would prepare for individual practice sessions in the IVY environment but reported not preparing in advance of group practice sessions “as the topics we were covering were all current affairs it wasn’t necessary to do much


29

Deliverable 6.3

research, just to keep up with the major stories in the news especially in France and England”). Another student *SUR Student 7, E-diary session 3] developed different steps to her practice sessions depending on the type of material she chose to use. For example, she reported that when using the new materials in the IVY environment, she would “listen, search and interpret*ing+ because the player offers an advantage that I could pause whenever I want” but when using the TED materials her strategy was “1) to listen to the talk, 2) try to remember as many [much] as possible, 3) listen to the talk again, take notes and 4) interpret it into Chinese”. This shows a variety of approach which could be seen as testing which method works for her or as demonstrating an at least intermediate level of learner autonomy in that the student is clearly able to select activities that match the different materials used or skills being practised. Coping with task overload and need for guidance 

Some of the comments given in the e-diaries and during the tutorials highlight the need for clearer/more direct guidance in this complex environment, e.g. in order to make pedagogically useful choices of material and to select appropriate tasks. Although it had been explained to the students (in class, in the induction) that it would be best to start with individual interpreting skills (e.g. listening comprehension and repeating or summarising as opposed to actual interpreting) or with interpreting into A language, some students started with real-time two-way interpreting in addition to exploring the IVY environment and focused on their actual output, making the task more complex.



As late as the third tutorial, one student was still struggling with note-taking and commented that she didn’t know “a good way to do it” *SUR Student 13, session 3 tutorial]. Given that the IVY environment offered comprehensive instructions on how to practise note-taking, this comment demonstrates a need for more detailed guidance on how to use the environment to practise different skills, and underlines perhaps that a blended learning approach is required to ensure that the students’ practice is directed and productive.



Another example of the need for further guidance can be seen in a question asked in a tutorial relating to which dialogues/monologues would be easy/appropriate for practice without note-taking [SUR Student 4, tutorial session 2]: “I thought I would work on my memory if the turns were short enough but I had to start taking notes after a few minutes”.

This shows that the student is indeed focusing on one task rather than trying to focus on practising multiple skills at same time and that the student had a clear aim for the practice session and then tried to match the available resources to that aim. This is an issue that can be addressed in the Guidelines and Best Practice document (Deliverable 8.2). 

Some students also pointed out that they wanted some more information about which mode was best suited for which aspect of practice [SUR Student 14, E-diary session 1]. This is also useful feedback for the Guidelines and Best Practice.


30

Deliverable 6.3

Development of learning strategies The development of learning strategies during the evaluation period can be broadly categorised into the following areas: 

Aims of interpreting practice session



Preparation



Reflection



Self-evaluation

If these elements are drawn together, it can be seen that some students developed a certain degree of systematicity in their approach to their practice sessions, although the e-diaries were not always written in such a way as to elicit a full description of the practice session. Aims of interpreting practice session 

In terms of aims, the students realised that a) it is advisable to use the materials in the ways suggested in the learning activities (e.g. preparing a topic before starting to interpret) and b) they should give themselves time to practise different skills separately at the beginning of their Masters course. The point about practising different skills individually was reinforced during the tutorials. The following extracts from the e-diaries show that the students had aims in mind when starting their practice sessions, although they are not always achievable or realistic aims: “*My aim was+ to focus on active listening, comprehension and also note-taking” *SUR Student 3, E-diary session 1] “The aim of the practice session was to improve my memory skills and practise two-way consecutive interpreting” *SUR Student 5, E-diary session 1] “I used one material for at least three times to ensure I can make progress step by step.” *SUR Student 7, E-diary 2] “I used the IVY dialogues to practise interpreting.”

The third quote above shows that the student stated her intention to “progress” and that she felt one way to do this was to repeat the same material, but there is no mention of the skills she is practising in order to progress (this could in fact be a weakness in the e-diary as the students select what information to present). This is similar to the final quote above where the aim is very broad and demonstrates a lack of focus in the practice session. Preparation 

The students were advised during the tutorials (notably tutorials session 3) that preparation should be a key part of an interpreting practice session. Many students commented on how they prepared and the benefits of this preparation. It is interesting to note, however, that preparation was still not always carried out before each practice session. This was particularly true of the group practice sessions where there was little or no preparation; individual practice sessions featured more preparatory work.


31

Deliverable 6.3

Preparatory exercises were done “in order to familiarise ourselves with the topic and find relevant vocabulary”. *SUR Student 5, E-diary session 3]

The only evidence of carrying out preparatory work in a group practice session demonstrated the use of the BACKBONE corpus which gave learners “the opportunity to watch the speakers”. *SUR Student 5/SUR Student 15+

These students prepared the topic of the UK education system, building a glossary and collating terms, and then watched three BACKBONE videos from the English corpus which related to the UK education system. This would seem to be an efficient strategy for structuring the interpreting practice session. It is interesting to note, however, that the students: “paused the video every 3-5 minutes and each time one [student] was doing the interpreting, the other two [students] were adding information where was necessary and then we commented on our notes.”

This is a creative blended use of the IVY and BACKBONE resources in a group practice setting. There is unfortunately no explicit comment in the students’ e-diaries about the success or otherwise of the structure of this group practice session. 

Another student commented on preparing before a practice session: “Secondly, I changed it (sic) to another material we had not used in class to say whether I could do it well without any preparation beforehand. It turns out that even though I had some unfamiliar words, but I could pause at any time to check information whenever it is needed.” [SUR Student 7, E-diary session 2 using Gaelic football material]

This shows that the student was carrying out research during the interpreting practice which is not reflective of a realistic interpreting assignment. The strategy of pausing turns in order to carry out research is not making best use of the IVY player (the pause function is there to replay a turn when necessary rather than as a matter of course). Eliminating preparation before the practice because the pause function is available could be advised against in the Guidelines and Best Practice document. This approach can be contrasted with that of another student [SUR Student 2, E-diary session 2] who prepared before the interpreting practice by doing the general preparatory learning activities, producing a small glossary and carrying out web-based research, with a clear path shown in the e-diary from general research to specific research. Although the specific research was often carried out during the actual practice session, there is evidence that the student is starting to develop a working methodology for her own practice sessions. 

One student [SUR Student 16] used the Audio Description material because she had done a lot of research on this topic in preparation for an Interpreting Studies essay which made use of this material. This shows that the student made good use of learning/knowledge from other activities. However, she came to a different conclusion about the usefulness and importance of preparing for an interpreting practice session.


32

Deliverable 6.3

She concluded that the preparation did indeed help with her target text production but then commented that: “we will not *be+ given time to do research in real situations, thus accumulation of terminologies and specific knowledge in various field[s] shall be done in everyday study.” *SUR Student 16, E-diary session 3]



One student used the BACKBONE videos as preparation for working in Second Life but also attempted to take notes at the same time: “I revised on the relevant vocabulary before the ‘session’ and did some additional research on the topic. I saw the fair trade video on backbone and took notes on it as an exercise but I didn’t come up with new symbols on the topic of fair trade beforehand (with the exception of FT as fair trade). I started with this video as it was one we had done in class and more or less remembered what the topic was about. Nevertheless, I had problems with the speaker’s speed. I thought that it would be better if I didn’t have to take notes as I had a hard time summarising and focusing on the speaker and take notes all at the same time. I think I failed to mention a chunk if things as I interpreted the text into Greek, not because I didn’t understand the SL, but because I couldn’t combine the multitude of skills needed. Second Life and the fair trade dialogue was a piece of cake compared to that. Maybe next time I should start with Second Life” *SUR Student 6, E-diary session 1]

This student started with a topic and dialogue that had already been looked at in class, i.e. she made an informed choice for her first session in the IVY environment, keeping the complexity down. She also understood that it is necessary to prepare an assignment and do additional background research before attempting to interpret the given dialogue. The student first tried to use/practise too many skills at the same time. She used the video, which was meant to be watched only as a preparation for the Fairtrade dialogue in the IVY environment, for preparation as well as note-taking practice. However, she reflected on this and probably learned from this for her next session. (Such points were also taken up and discussed in the tutorials.) Regardless of her initial problems with task overload, it seems that the suggested way of learning with the dialogue was successful for her: she realised that choosing a dialogue which had already been used in class as well as her preparation e.g. by using the BACKBONE video (as described in the IVY Learning Activities) made the interpretation of the Fairtrade dialogue in the IVY environment a manageable task for her. Reflection upon interpreting challenges and interpreting performance 

There was little indication in the e-diaries of the use of the IVY reflective learning activities. One student commented in her e-diary that she had “started looking at the post-interpreting exercises on *Surrey’s VLE+ SurreyLearn” *SUR student 4, tutorial session 2] and this was picked up in the following tutorial where the student commented that: “there is a lot of useful material in these exercises” *SUR Student 4, comment during tutorial 2]


33

Deliverable 6.3



This lack of comment on the reflective learning activities suggests that the students did not reflect in detail on the challenges of the interpreting tasks and on their performance. It is also possible that some students, especially students without prior study experience in language-related subjects are not used to the detailed analysis of language-based challenges (i.e. source-text based analysis) and find this difficult. Again, this is something for which the Guidelines and Best Practice will be very useful, as it can show students how to make best use of these activities.

Reflection upon learning success 

Students were asked in the E-diaries to reflect on their learning success. Comments also reflected on the materials themselves, how they used them and any difficulties they encountered during the practice session. Comments included: “I believe the way I practised was quite efficient and next time I will try to focus on fluency and avoid hesitations” *SUR student 5, E-diary session 1]. “it was a good practice regarding pronunciation” *SUR Student 15, session 4 E-diary] “In terms of active listening and comprehension I would say that I had no problems understanding the speaker, the speech was coherent, the sentences and ideas were well connected throughout the entire speech. *…+ Notes helped me remember the name of institutions, numbers. I also tried to write down key words not sentences which worked fine with small paragraphs but I can’t say that I managed to keep the same direction when dealing with larger paragraphs. I still have to work on rephrasing, I have to avoid writing down phrases that otherwise can be simplified.” *SUR Student 3, E-diary session 1] “at some points I was writing down complete sentences and was missing the following point, or even though I had come up with symbols for certain things I wasn’t using them”. *SUR Student 6, E-diary 4] “I found this session useful because I found certain terms which I didn’t know and it helped to practise interpreting as the majority of practice I had been doing recently was note-taking.” *SUR Student 2, session 1 E-diary] “I found *working with the general preparatory learning activities+ useful because it was helpful to go over the types of aspects that should be researched before doing an interpreting assignment.” *SUR Student 2, E-diary session 2]

These comments show that the students used the range of different understandings of ‘reflection’ as given in the guidance notes to the E-diary: some students reflected on what they had achieved in the session, where their problems lay, what specific skills the session had been useful for or what they had learned for future practice sessions. These different understandings will be picked up in the assessment of the E-diary as an evaluation tool and as a learning tool (see Section 4.5 below). Self-evaluation 

In terms of self-evaluation, some students showed a keen awareness of areas and skills which were progressing or which needed further practice: “In terms of evolution, I think now I begin to rely more on my memory again and things have slightly improved. Before, I was focusing too much on my note-taking, I knew that I had to pick-up concepts only not sentences but in practice I was doing the opposite of


34

Deliverable 6.3

that. While I was struggling to write down, I was no longer focused on the speech and at the end everything was lost. Now basically I am more aware that I have to listen mainly and write down only what is more likely to be forgotten. I have developed a glossary of symbols and I try to use them as much as I can and whenever I meet a new word that comes up repeatedly I invent a symbol for it. I am trying to eliminate the stress of notetaking as much as I can and I’m trying to find the right way to get the best out of notetaking.” *SUR Student 3, E-diary session 2] “This monologue made me aware of the fact that I need a better glossary for professions” *SUR Student 4, E-diary session 2].



Some students also developed their own methods of measuring how well they were progressing, such as using goals (e.g. by setting a target to increase number of sentences after which the student would interpret). Two students used digital recording devices to record their target text production, and then listened back to the output to evaluate their accuracy and delivery: “I noticed that a big chunk of the text had gone into English ‘unedited’ and sounded strange.”

The use of recording devices was advised during the tutorial sessions and two students took heed of this advice. It would be interesting to consider why none of the other students made recordings of their interpreting practice even though most students have personal access to recording devices (e.g. smart phones, laptops) and a number of recording facilities are available on-site (video equipment, Dictaphones, interpreting booth software). This perhaps raises the question whether students would be more likely to record their output if a recording capability were integrated into the IVY environment. Sense of presence and appreciation of 3D capabilities 

Prior research on learning in VLEs has highlighted the observation that 3D VLEs are capable of creating a sense of presence which is difficult or impossible to achieve with other online learning tools. One open question is how much ‘realism’ there needs to be in a 3D world to create an immersive experience. The comments of some students indicate that this is highly task-dependent: “I could say that my memory skills are improving by practice and maybe my stress management skills although I am not sure if the IVY environment can help in that domain because it is not easy to substitute physical presence with electronic means.” [SUR Student 15, E-diary session 1]

The comment above came from an early E-diary (session 1) and the student did not complete the questionnaire so it is impossible to know whether further work in the environment changed her view. It also raises questions for the design of further tasks in the IVY environment and to what extent it will be able to simulate conditions for stress management training. 

Other students made highly interesting comments about their priorities when working in the environment, e.g.


35

Deliverable 6.3

“I am more concerned about just having good audio control and play functionality instantly available rather than graphics and personalisation, environment, etc.” *SUR Student 15, E-diary 1] “IVY is still difficult to use so I focused on listening materials instead of the environment”. *SUR Student 4, session 2 E-diary]

Similarly, another student said that meeting others in the IVY environment led to wasting time and that he found this “distracting”. Such comments show that some students have a tendency to perceive the environment simply as an ‘audio player’. The issue of distraction was dealt with during the first tutorial and students were advised to use the interpreting practice mode rooms for individual work to avoid distraction from other users. Concerns about distraction were also raised in the tutorials where one student said that she was “so easily distracted in Second Life” and “can’t even stop looking at the screen” *SUR Student 17, tutorial session 3+. Rather than using the 3D capabilities to simulate realism and focus concentration, it is clear that some students find the environment too distracting. It is important to note, though, that the student who placed less importance on graphics, personalisation and the environment rated the IVY environment as “very realistic”. Given that this student was not able to access SL very often due to technical problems (see SUR case study 1, above), it is difficult to draw conclusions from this discrepancy. It seems that the student would simply have required more time in SL to appreciate some of the 3D features. Conversely, some students saw the benefits of augmented reality, with one student saying she teleported to the location in Second Life where the material was set so as to “simulate real interpreting practice between two speakers” *SUR student 11, session 3 tutorial+. Similarly, another student commented in the questionnaire that she “liked the scenarios (presence of people and multiple chairs, etc.)” but had problems with loading the environment and that was distracting. This shows that some learners can appreciate the 3D capabilities but that they really do need suitable hardware to benefit from what they can offer. Collaborative work 

Whilst some students met in SL, this was mainly to help each other in getting going and to talk. Four students reported using the live mode in the questionnaires but there was no further information about how they used this mode in their E-diaries, the tutorials or in the questionnaires. Only one student gives a reason why the live mode was not used: “Because of different arrangements, we *…+ failed to meet each other in the Second Life island online practice session in our spare time.” *SUR Student 7, E-diary session 2]



Some students asked about possibilities for working together. It was explained that they cannot work together in the Interpreting Practice mode at the moment, since the sound from the audio player is local to the user rather than being broadcast e.g. for all users in the same room. In the initial tutorials, students were advised that at present the Interpreting Practice mode is mainly for self-study (and that teleporting to any of the virtual interpreting scenarios in this mode would avoid distraction, since other users cannot access a scenario when it is ‘occupied’, unless they are teleported over there by the ‘occupier’).


36

Deliverable 6.3



The e-diaries, questionnaires and tutorials showed that a large amount of collaborative work was done regularly in language groups, which clearly is beneficial e.g. one student commented that “working in a group we were able to pick up new terms” *SUR Student 1, E-diary session 3]. This shows evidence of peer learning but this did not translate to peer learning in the virtual environment. It is also important to note that the evaluation period at this site included a four-week Christmas vacation during which time many of the students were away from campus and could have made good use of the Live mode while they were not able to meet in person. This opportunity was not taken up. One explanation is that the students worked on essay assignments during the Christmas period and did not do much interpreting practice. Informal comments by the students point in this direction. However, the lack of online collaboration also shows that students who have never been exposed to collaborative online tools like SL will need some time to get used to using this.



IVY resources were, however, used for face-to-face group work (e.g. use of transcripts, learning activities) in addition to and sometimes in conjunction with other resources.

4.4 Key outcomes of the evaluation with interpreting students The data we have gathered enables us to discuss the impact of the IVY solution (i.e. its effectiveness and suitability) for interpreter training. Our working definition of perceived learning success encompassed students’ comments about the development of interpreting-related skills, professional awareness, and selfassessment skills. Most students reported improvement in the area of interpreting-related and self-assessment skills. Clearly, the evaluation tools we used (the E-diaries in particular) encouraged students to reflect on what they were doing, how they were learning and how they were progressing in terms of preparation, SL comprehension, memory, and TL production (this was described in detail in Section 4.3). The ability to reflect critically on one’s own performance is crucial in the interpreting profession and it is not one that can be developed fully in times of reduced contact hours at universities. Therefore, environments such as IVY provide an important opportunity for students of interpreting to develop selfassessment abilities not only in autonomous practice, but also in live interactions with other students of interpreting and potential clients of interpreting services. As shown in Section 4.3.2 (‘Coping with task overload and need for guidance’, above), the majority of students would benefit from more tutor and peer support in evaluating the effectiveness of their practice sessions and their target language output. In addition, a number of students wrote that they developed an increased awareness of the interpreter’s role and responsibilities. For example, they realised the value of preparation or of using an appropriate register for delivering professional interpreting services. This points to an important impact of the IVY solution, i.e. instilling in interpreting trainees a sense of responsibility and an awareness of the code of professional conduct. This would have been difficult in traditional face-to face classes or learning environments solely confined to audioand video-materials. It is expected that in the future the IVY solution will provide interpreting trainees with opportunities to interact with clients in the Live mode, which will


37

Deliverable 6.3

surely lead to more students’ comments about the usefulness of the IVY tool in the development of professional awareness. Some evidence for the SL ‘steep learning curve’ was observed among students across the three evaluation sites. This emerged both in the quantitative and qualitative analysis, which seems to confirm the steep learning curve generally observed in similar environments (cf. Review of Best Practices, D3.1). The students at the Aristotle University of Thessaloniki who spent the shortest time in IVY found the tool least helpful in developing skills vital for interpreting. Students who used the environment at the Adam Mickiewicz University in Poznao and at the University of Surrey clearly appreciated the tool more, as they used the self-study period more efficiently and thus had a better chance to overcome difficulties that typically occur at the beginning of the work in SL. Those students who spent a longer time in IVY (AMU, SUR) reported that the tool was either helpful or quite helpful for them in the development of skills crucial in interpreting, such as memory, comprehension, selfmonitoring or note-taking. Students at the University of Surrey (the only group asked to rate the realism of the environment) considered IVY to be quite realistic. The qualitative analysis of the E-diaries also indicated that tutorials helped most students to overcome the difficulties with accessing and using SL tools. Another point is that the students who could use the environment without (technical/ access) problems often made a rather flat or superficial use of it. One possible explanation for some students’ lack of in-depth use of the opportunities offered by the environment might be that students at the beginning stages of interpreting training still perceive interpreting in terms of interlingual transfer of information and fail to notice the interactional, socio-cultural dimension of interpreting which the 3D environment can support. This points to the need to focus the initial stages of interpreter training on the socio-cultural aspects of interpreting and its dialogic character (e.g. varying length of turns, etc.), especially in view of comments which clearly indicate that this is a problem to some students: “the fact that I have to interpret several turns and not an entire text does not help me to keep concentrated and to follow the meaning of the text" [UCY Student 3]. Tutors should provide guidance to students to help them get acquainted with these particular facets of interpreting and develop adequate interpreting strategies. Insufficient personal hardware was one of the main reasons why some students were not able to benefit fully from the IVY environment in SL. However, in Surrey where the SL client was installed on-site, students who could not use their own computers or laptops did not make use of the SL availability on-site either. This could indicate that they were disillusioned with the software and did not want to try again, or felt that having to be on-site did not fit in with their self-study timetable. Regardless of the evaluation site, if a student encountered issues with incorrect SL settings early on in his or her student SL experience, it was found that the student was put off using it, even if the settings were adjusted quickly so that the student could use the environment. This means that demanding system requirements are likely to have a discouraging effect on the students. It is clear, therefore, that consideration should be given to ‘minimum hardware requirements’ (i.e. the graphics card that is needed, computer processing speed required, etc.) and a minimum set of computer skills needed to operate in SL. Student comments from all three evaluation sites indicate that training time


38

Deliverable 6.3

needs to be provided to the students lacking such skills in order for them to get acquainted and feel comfortable with the virtual environment before setting out to work in it on their own. The sense of presence and the environment’s 3D capabilities were appreciated by some students but it was not found to be essential to all students. For example, there were students at the University of Surrey who found it distracting and said that access to the audio was the priority. Another student wrote that although she understands that SL gives her “the possibility to observe the situation, the positioning of the speakers and interpreters in a given space (e.g. an office, or court setting)”, she nevertheless thinks that “SL is more of an addition (and sometimes a trouble-generating one)” and therefore she would prefer “watching a short film presenting the interpreting situation on a Backbone-like website” [AMU Student 5]. This suggests that much more time is needed to familiarise students with the many opportunities and ‘augmented capabilities’ that the 3D environment offers and that have been explored in the relevant educational literature, i.e. opportunities for simulation and collaboration, and ultimately opportunities for understanding interpreting as an activity that goes far beyond rendering a source text into a target text. Additionally, a closer monitoring of the students’ progress (tutorials) may help them to appreciate the 3D capabilities more fully. The range of IVY materials was considered adequate and students appreciated having the transcripts and learning activities. However, there were also students who wanted more materials and/or topics that were more specific to their interpreting courses (e.g. business, finance). Students also remarked that the closer to real-life the dialogues were (natural flowing speech, turn-taking, interruptions), the more exciting and engaging they were to interpret. For example, this student said: “the naturalness of the speakers was also very helpful” *UCY Student 1+. A similar opinion was expressed by this student: "It is very important that the monologue is carried out in a natural way [so that it includes various] elements of oral speech, [such as] pauses, the speaker correcting herself [etc]. This makes it very different from [simply listening to a] well-written and well-structured text read out and makes it a very useful tool when preparing for [an] interpreting [job] under real conditions” [UCY Student 2]. In addition, it emerged at the University of Thessaloniki that special attention should be paid to the information provided in the brief, particularly when using general descriptions (e.g. “a journalist” as opposed to “a journalist from Financial Times”), as it could help the students during the preparation stage. At the University of Surrey, linking the aim of the practice session to the choice and use of material/mode was regarded beneficial and could be encouraged further. Irrespective of the evaluation site, not all students used the preparatory and reflective learning activities available to them in IVY, or at least reported doing so in their E-diaries. This could suggest that those students did not appreciate the value of the learning activities in helping them improve their interpreting performance. However, it could also mean that those students simply need more guidance from tutors to encourage them to use the IVY learning materials to their advantage. If this is true, it underscores the value of using IVY in blended learning/teaching contexts. The students at AMU and UCY did not use the Live mode because it was not always fully operational at the time. At the University of Surrey a few students reported using the Live


39

Deliverable 6.3

mode in the questionnaires but there was no further information about how they used this mode in their diaries, the tutorials or in the questionnaires. Only one student gives a reason why the Live mode was not used: “Because of different arrangements, we *…+ failed to meet each other in the Second Life island online practice session in our spare time.” *SUR Student 7] However, instead of working collaboratively in SL, the E-diaries, questionnaires and tutorials at SUR showed that a large amount of collaborative work was done regularly in language groups using IVY resources (e.g. transcripts, learning activities), which clearly is beneficial e.g. one student commented that “working in a group we were able to pick up new terms” *SUR Student 2+. This shows that students’ practice in the Live mode clearly needs to be organised by the tutor, at least at the beginning of the training in IVY, to motivate the students and show them the most efficient way of using this mode. There were students who pointed out this need in the E-diary, suggesting that the following improvement be introduced in future versions of the IVY tool: “introducing a messenger (or a message board) to make appointments with native speakers of languages of your interest” [AMU Student 4]. Overall, the IVY project team thought that the level of guidance given to students in IVY was sufficient, but it has been shown that further guidance is in fact required in order to promote greater autonomy and systematicity of practice sessions and a more frequent use of the Live mode, and to encourage students to use their E-diary reflections. More advice also needs to be given in tutorials to help the students practise more efficiently and productively. As shown in Section 4.2.1 of this Report, the IVY environment is well-suited to blended learning/teaching contexts, particularly in the beginning stages of interpreter training. It offers the students a broad range of materials with appropriate levels of difficulty, which, however, have to be used in a manner adjusted to students’ skills and individual needs. It is the tutors’ responsibility to recognise those needs and set the hierarchy of skills for individual students’ practice accordingly. Therefore, tutors should suggest ways in which interpreting sessions in IVY should be structured and attempt to obtain feedback from the students to check that the guidelines s/he provided were in fact followed. Such guidance could be given in traditional face-to-face interpreting classes. We have learned from our analysis of students’ E-diaries that the lack of such guidance could result in the students using the IVY environment rather superficially, which prevented them from benefitting fully from the affordances of this environment. The issue of suitability of IVY for blended learning/teaching contexts should be taken up in future projects evaluating the effectiveness of virtual learning tools in interpreter training, such as EVIVA.

4.5 Effectiveness of evaluation methodology The mix of E-diaries and face-to-face tutorials, in which the students and tutors discussed the use of the IVY environment and the students’ approach to practising interpreting outside class, in general proved to be very useful. The addition of the questionnaire at the end of the evaluation period gave the students an opportunity to give an overall assessment of the IVY environment, its materials and its contribution to their learning and progress. The use of the E-diaries proved to be an effective method of eliciting feedback and, equally important, as a tool for encouraging self-reflection. However, the quality and content of the


40

Deliverable 6.3

E-diaries varied and as a result comparison between students at different stages of the evaluation period proved difficult. This is perhaps also due to students’ different learning rates, abilities at the start of the course, and use of the E-diary itself. Given that the main other tool was face-to-face tutorials (apart from the final questionnaire, which had a different aim), it seems that students used the diaries to make much more individualised and detailed comments than some of them would have made in class. Conversely, some issues were raised during the tutorials which were not recorded in the Ediaries, thereby justifying the complementarity of the evaluation tools used. Of course, the students did not necessarily write what we had expected or as much as we expected from them, but it seems that it was possible to elicit interesting points from this qualitative data; points that would be difficult to elicit otherwise. It might also be the case that students were carrying out more practice than was reported in the E-diaries and that not all aspects of all practice sessions were recorded fully. Because the E-diary is not a direct observation but a written account of the practice session, the student can select what is and is not recorded. Similarly, the students may have filled in the E-diary sometime after the practice session, which might have reduced accuracy or completeness, but on the other hand could have clarified the student’s reflection. It was sometimes difficult to determine from the E-diaries whether the students were referring to IVY materials and resources, whether those resources were being used online or offline/from the BACKBONE corpus or in Second Life. The additional space provided for background research/glossary work was not used as fully as might have been expected. It is possible that the students recorded this information elsewhere. Most probably, for practical reasons the students wrote down the results of terminological search and other important information relating to background knowledge in one place, i.e. in their notebooks, own glossaries, etc. It is perhaps worthwhile to reflect on and assess the design and structure of the E-diary. While it was important to have an open-ended structure in order to encourage the students to build a narrative of their interpreting practice and thus high-quality qualitative comment, it is also the case that the free structure elicited responses and outcomes that are not necessarily easily comparable. The E-diary was designed as a tool with a dual purpose: for evaluation of the students learning as part of the IVY project; and for the students to monitor and evaluate their own learning and progress. It is therefore essential that the Ediary is balanced in such a way as to accommodate these two purposes and this balance should be taken into consideration when evaluating the effectiveness of the E-diary as a tool.


41

Deliverable 6.3

5 Evaluation with users of interpreting services There has been an increasing recognition that success in interpreter-mediated communication is a shared responsibility (see e.g. Ozolins & Hale 2009 for public service interpreting contexts; Pistillo 2004 for business interpreting contexts). This is why the IVY project addressed not only the needs of interpreting students, but also potential clients of interpreting services. Two client groups were considered in the IVY project. The first group were students in higher education and vocational training contexts who were to evaluate the IVY tool using questionnaires. The second group, which was considered to be the most heterogeneous, were adult learners from different walks of life. This Chapter outlines the overall approach to this part of the evaluation (Section 5.1) and then presents the findings of the two major evaluation activities, i.e. the hands-on workshop with adult learners (Section 5.2) and the results of the online survey for all ‘client’ groups (Section 5.3).

5.1 Overall approach With regard to the client target groups, the IVY project has focused on 

creating the Exploration mode as part of the IVY 3D environment, where potential users of interpreting services can learn about interpreting and about how to communicate effectively through an interpreter;



developing learning activities for the Exploration mode to help potential users of interpreters assess and apply their knowledge;



shaping the Live mode so that it becomes an attractive collaborative environment for both interpreting students and potential or future users of interpreting services.

An evaluation of the 3D environment from a client’s point of view was conducted with the aim of providing initial insights into their opinions of such a novel learning opportunity. Following the consortium’s experiences with potential client groups, which confirmed the well-known low interest of these groups in how to work with an interpreter, the original evaluation plans were adapted to suit this situation. Initially it had been planned to conduct two evaluation workshops with adult learners, one in Bangor and one in Surrey, and to use surveys to cover students in Higher Education and vocational training. However, because of difficulties with recruiting workshop participants in Surrey (see Deliverable 6.2, the report accompanying the evaluation workshops), an online approach was chosen instead. Whilst the Bangor workshop was delivered to adult learners as planned, the Surrey workshop was replaced by an online module which was set up to demonstrate the IVY environment and which included a pre- and post-test as well as an online questionnaire survey at the end for users to rate the environment and to provide their views and opinions. The online module was used with participants from both ‘client’ groups, i.e. adult learners as well as students. Sections 5.2 and 5.3 present the activities and results of the workshop and the online survey. Each Section will begin by reporting the evaluation method. This will be followed by key outcomes and conclusions.


42

Deliverable 6.3

5.2 Hands-on Workshop The evaluation workshop with adult learners took place during the Virtual Learning Technologies 2012 Workshop, which was held at Bangor University on 31 October 2012. The Workshop was a one-day event that attracted some 25 participants, of which 12 took part in the hands-on evaluation session in the morning. Further details about the workshop itself are reported in Deliverable 6.2. This section focuses on the method and the outcomes of the evaluation with potential clients of interpreting services.

5.2.1 Method The workshop included twelve participants, nine male and three female, between 23 and 59 years of age. Seven reported to have some familiarity with interpreter-mediated communication; the remaining five reported no familiarity. Since the workshop was part of a larger event on virtual learning technologies, it had attracted mainly professionals specialising in the topic of virtual worlds and computer-aided technologies. The evaluation, therefore, combined an assessment of the IVY 3D environment from the point of view of professionals who would be potential clients of interpreting services and an evaluation in terms of usability, based on software developer standards with which this group was particularly familiar. At the beginning of the workshop, the participants were formally welcomed and introduced to the project by way of an overview presentation. Then a specific overview of the functionality of the environment and the HUD was provided, using a video presentation. Thirdly, the workshop participants were presented with written instructions for the evaluation tasks. To ensure that the activities carried out by the workshop participants during the evaluation phase covered all the functionalities of the IVY environment, four tasks were designed, the first involving the collection of the HUD from the IVY reception, and the remaining three relating to the different features of the IVY-VE HUD and environment. All tasks were carried out by each participant individually in the IVY SL environment, using laptops over a wireless connection at Bangor University. The laptops were equipped with Intel i5 CPUs, 8GN RAM and Nvidia NVS4200 GPUs. The official SL client (v.3.4.1 (266581)) was used. The participants used generic avatars and IVY HUD logins provided by the Bangor IVY team. They were allowed to ask questions if they needed any clarification. Such questions were recorded by the experiment facilitators. Once the tasks were completed the participants were asked to complete a questionnaire. The questionnaire (see Appendix 4) focused on assessing the fitness for purpose of the IVY environment, i.e. tried to gauge how the clients perceived the user-friendliness of the IVY environment and the pedagogical value of the IVY solution. The use of a structured questionnaire-based assessment was preferred to an interview-based assessment as it is a popular method for evaluating interactive systems, in that it produces comparable results. The System Usability Scale in particular has been used extensively in human-computer interactions research for evaluating interactive systems. It is widely cited and numerous studies have demonstrated its reliability. In addition, employing such an established


43

Deliverable 6.3

method, compared to an interview-based assessment, also allowed for the comparative placement of our system among other interactive tools (Bangor et al. 2008).

5.2.2 Outcomes of the Evaluation Workshop Participant profiles (demographic questions) As explained above, of the twelve participants, seven reported to have some familiarity with interpreting. However, the mean score of experience was relatively low (mean = 1.7 on a 5point Likert scale ranging from 1 = ‘no experience’ to 5 = ‘very experienced’; SD = .7). What was interesting was the variety of situations in which the participants had encountered interpreter-mediated communication. These situations ranged from seeing an interpreter on television [BNG client 1] to the following experiences: “Have booked and helped prepare interpreters for various events in the university. Have also relied on interpreters in order to communicate in bilingual situations” [BNG Client 2] “Mainly I have attended presentations that had a live translation service but also I am bi-lingual and have had to translate in the past for family members” [BNG Client 3] “In corporate world” [BNG Client 5] “A few encounters with non-professional interpreters” [BNG Client 6] “Working with conference interpreters in previous role, giving speeches at conferences, also friend works as interpreter for NGO in Europe” [BNG Client 10]

Given that the workshop took place in Wales, the wide range of situations is perhaps not too surprising. The North of Wales, where Bangor is situated, is in particular a region with a strong bilingual tradition, where interpreters are required on a daily basis in meetings, educational settings and many other situations. Wales is thus representative of many regions in Europe, where the work of interpreters is an integral part of everyday life, albeit without hardly ever being particularly noticed. Further in terms of demographics, it also comes as little surprise in this group of participants that their general computer experience was high, attaining a mean score of 4.4 on a 5-point Likert scale ranging from 1 = ‘no experience’ to 5 = ’expert’; SD = .8. The familiarity with Virtual Environments and games received mean scores of 3.1 (SD = 1.5) and 3 (SD = 1.2) respectively. Interestingly, the mean score regarding the familiarity with Second Life was only 2.4 (SD 1.6), reminding us that even among software developers the use of, or familiarity with, 3D virtual environments cannot necessarily be taken for granted. Positive overall comments The workshop participants had several positive comments about the IVY environment. They reported favourably on the “simplicity of controlling audio”, the “multiple different locations”, the ease of selecting modes, and the location aesthetics. In general they thought that the idea behind it was pedagogically sound, and found the immersive properties of the environment stimulating and appropriate for the simulation of interpreting situations: “Immersion in simulated interpreting contexts could be an aid to interpreting” *BNG Client 1]


44

Deliverable 6.3

Other positive opinions concerned the ease of navigation, round-the-clock accessibility from different locations, and the overall design of the 3D environment: “Easy to use, many options, accessible from anywhere. It would motivate people to use the system since it is mobile” *BNG Client 2+ “Well designed world, easy to use” *BNG Client 3+ “A nice way of 3D working environments, good simple audio control, mixture of different modes” *BNG Client 4+

What became clear is that the adult learner group not only worked in the Exploration mode and viewed the virtual interpreting scenarios as they would be used in the Live Mode, but that the group also explored extensively the Interpreting Practice mode. Whilst this was partially given by the set evaluation tasks and whilst it may have been fostered by the participants’ general curiosity, it is possible that there is a connection here between this group’s attraction to the Interpreting Practice mode and the specific design of this mode (i.e. of the virtual interpreting scenarios, the integration of the audio player and other functions). The adult learners rated the design of the available practice facilities positively overall, and this may have encouraged them to look around in the environment as a whole rather than just focusing on the Exploration mode. If this were the case, this would provide an optimistic perspective for the use of this learning environment with potential users of interpreting services. With regard to the Live mode, the adult learners also appreciated the different contexts in which they could learn about interpreting. Apart from the positive rating itself, the result is interesting in that there seems to have been sufficient awareness among this group of learners about the variety of settings in which interpreters work and hence the importance for interpreter clients to understand these settings. Concerns and critique There were also some concerns, of course. Most of the more critical comments centred around “the lack of visual cues on who is speaking”. In addition, two participants felt that the IVY-VE was not as interactive as modern games and that more training would be required in order to use it, mostly due to the interaction provided by SL. There were also clients who generally perceived the environment as promising but had some reservations regarding, for example, hardware and broadband requirements, the lack of game-like interactivity and the level of realism. One client suggested that the lack of link between the voice being played and the robots (lack of gestures) reduced the sense of realism because it meant that it would not be essential for the clients to be in the virtual location where the interpretation took place. Moreover, one client found the environment distracting and could not understand the benefits afforded by a 3D space to discover more about interpreting. Suggestions for improvement Participants reported, as possible future enhancements, the addition of visual cues regarding which interlocutor is speaking, a facility to record and replay one’s interpretation,


45

Deliverable 6.3

and directional sound as one way of addressing the problem of the missing link between the audio and the robots. Participants also felt there is much potential in systems like IVY, particularly for remote training and with scenarios that are difficult to recreate. Three participants pointed out that VLEs are becoming more accessible through web technologies, offering a high level of accessibility and interoperability and, thus, may gradually become more commonly used tools in education. Usefulness for interpreter clients The responses to this point demonstrate that most participants felt that the environment would “definitely” be a useful tool for users of interpreting services to learn about interpreting. One responded pointed out that “it clearly makes sense to involve all the senses in a virtual environment and some situational awareness, setting own pace etc” *BNG client 5+

Another participant drew the parallel with a game environment. He felt that the IVY environment is “much like games”, which are “popular”, and that the environment could therefore provide a useful basis for training. One other participant said that it would be useful but that it may need “more improvement to make a more convincing experience” and one was “unsure of the advantages”. An interesting comment was made by one respondent in relation to interpreter training and the development of the interpreting market. Whilst he felt that it would be unlikely for a 3D environment to be used for the delivery of interpreting services in real life in the near future (i.e. remote interpreting), as it would be “too drastic a technological shift”, and as it would be easier to use phone or videoconference for this, this environment would be “very useful as an education and training tool” which was very “promising, given more development, bandwidth and server capacity”. This participant also felt that 3D environments will “probably be the ‘norm’ in 5/10 years’ time when present youngest generation are more comfortable/familiar with 3D virtual environments” but that “a bang in adoption of 3D environments has to happen first”. This attitude was echoed in other statements which showed that the participants were of the opinion that there is much potential in systems like the IVY-VE, particularly for remote training and in settings that are difficult to recreate. Effectiveness of design model All participants felt that the use of web technologies offers a simple and intuitive way of managing the settings in the IVY-VE. Furthermore, the option to bypass SL regional availability problems by using a dedicated browser was also highly valued. The advantage of this also became apparent with the group of Greek interpreting students (as reported in section 3.2.3). Furthermore, most participants felt that if the aforementioned enhancements were introduced, the system would be a powerful tool for simulation and training.


46

Deliverable 6.3

System Usability Scale In terms of usability evaluation, the IVY environment attained an overall System Usability Scores of 65.5/100, with individual scores ranging from 32.5 and 87.5 (SD = 18.7). The overall score corresponds well with SUS scores of web-based applications, with a mean score of 60 (see Bangor et al. 2008). Along with the participants’ open comments reported above, this confirms the technological approach adopted for developing the IVY environment. It was, for example, our contention that by relying on web technologies for scenario management, text rendering and internationalisation support, the IVY architecture would perform much better than any solution based on scripting in SL. The open-ended answers given by the workshop participants show that this approach was indeed supported from a developmental point of view. The flexibility of the HUD, which was built with web technologies as well, was highlighted as well. It was felt by the workshop participants that this resulted in a flexible interface which proved intuitive and easy to use. The participants also commended the web infrastructure for being easily extendable and enabling additional features to be incorporated. What also needs to be pointed out, however, is that scalability remains a challenge, mostly due to the SL infrastructure. The evaluation results show that our solution works well but it is limited from the number of primitives that can be used on the IVY Island. This also affects actor management in terms of populating the scenarios (e.g. number of robots and avatars on the island). The evaluation with clients and developers thus confirmed once again that Second Life clearly offers a number of advantages when it comes to creating a complex, accessible and collaborative world. The quality of graphics available is improving, almost comparably to modern games, and the feeling of immersion is becoming stronger. However, SL imposes a series of limitations, mostly related to the lack of scalability mechanisms, unlike modern Massively Multiplayer Online Games (MMOGs).

5.3 Online Survey This Section describes the pedagogical evaluation carried out with clients of interpreting services. The evaluation addressed clients from the three client target groups, namely learners in higher education (HE), vocational training and adult learning settings. As described in the report to accompany Deliverable 6.2 (Pedagogical Evaluation Workshops), workshops were organised and advertised among relevant client groups by the Surrey IVY project team. The following workshops were planned:  A presentation and evaluation session with students from the Surrey Business School (learners in HE) on Wednesday 5 December 2012.  A presentation and evaluation session with students from the School of Law, University of Surrey (learners in HE) on Wednesday 5 December 2012.  A presentation and evaluation session with vocational training students from the Guildford College of Law (Surrey) on Wednesday 12 December 2012.


47

Deliverable 6.3



A one-day face-to-face workshop with trainers of legal professionals in February 2013. The presentation and evaluation of the IVY environment was embedded in a workshop focusing on the use of current and emerging technologies in the field of interpreting practice and training.

It did, however, prove difficult to attract these target groups and this could be attributed to reasons such as lack of time or a lack of awareness as to how learning to work with an interpreter could be relevant to their own professional practice and their career development. As a result, it was decided that the first three workshops listed above would be cancelled and an alternative method of targeting these clients would be sought. The fourth workshop mentioned above was postponed and will now take place in May 2013, forming part of the IVY exploitation strategy (see Deliverables 6.2 and 8.1).

5.3.1 Method It was assumed that two of the main reasons why the client groups were not reached by the face-to-face workshop approach were: (1) lack of time available to attend on-site face-to-face workshops; (2) lack of awareness as to how learning to work with an interpreter could be relevant to their own professional practice and their career development. It was therefore decided that a different evaluation strategy had to take these two factors into consideration and that an online module would be a suitable approach. Addressing the first point above, this would enable respondents to do the module at a time and location to suit them. To address the second point above, the module was designed in such a way as to foreground the relevance of training to work with an interpreter to the respondents’ professional context and to provide an introduction to interpreting before presenting the IVY environment and its learning opportunities for the client target groups. This aimed to contextualise the IVY environment so that clients with no prior experience of working with interpreters or training in interpreter-mediated communication could follow the module and be able to understand and evaluate the innovative approach to training offered by such an environment. This section describes the design and content of the module, testing of the module, the participants and respondents, and the duration of the module. The online module was designed using the SurveyMonkey online module software (www.surveymonkey.com) and blended content which could give information about interpreting and the IVY environment, and questions to elicit opinions about the environment. As a general approach, the content from the planned workshops was adapted and restructured to suit the requirements of the online module. The content created for the workshops consisted of information about interpreting as a profession, preparation for working with an interpreter and advice on how to speak through an interpreter. This was primarily based on the content of the IVY Exploration mode and was supplemented by additional videos showing interpreting pitfalls and videos created specifically for these workshops demonstrating the IVY environment in use. The original workshop content was tailored to the needs and perspectives of the specific target groups, as shown in Figures 1 and 2 (below).


48

Deliverable 6.3

Fig. 1: Slide from presentation to students at Surrey Business School, University of Surrey

Fig. 2: Slide from presentation to students at School of Law, University of Surrey

This was adapted to suit the requirements of the online module, namely that it had to address (potential) clients of interpreting services in any field (law, business, medicine, etc.). This meant that some of the customised content had to be made more ‘generic’ to be relevant to clients in different fields:

Fig. 3: Adaptation of presentation content to online module

The module consisted of the following sections:  Introduction  Demographic questions  Pre-test


49

Deliverable 6.3

     

What is interpreting? Interpreting pitfalls Introduction to the IVY solution Impressions of the IVY solution Post-test Close

Introduction: The introduction to the module comprised an overview of the module content and initial questions to encourage respondents to start thinking about their previous experience, if any, working with an interpreter. The purpose of these pages was to engage respondents with a topic which they might not have considered either at all or in any depth.

Fig. 4: Introduction page from online evaluation module

Fig. 5: Demographic questions from online evaluation module

Demographic questions: Key questions in this section related to finding out the area in which the respondent worked, to which target group the respondent belonged, and the respondent’s experience with computers, virtual worlds, virtual learning environments and computer games. These questions were designed to gauge their competence with computer technologies in general and with virtual worlds and VLEs more specifically. As has been shown from earlier research (including Deliverable 3.1 Review of Best Practices), the learning curve in Second Life can be steep— although this was not wholly supported by the IVY pedagogical evaluation with interpreting students—and the level of user competence could influence how the respondents react to and evaluate the IVY environment. Pre-test and post-test: The pre-test aimed to assess the respondents’ knowledge of interpreting before being given information about interpreting.

Fig. 6: Pre-test questions from online evaluation module


Fig. 7: Content from online evaluation module about interpreting and working with an interpreter

50

Deliverable 6.3

The questions focused particularly on the respondents’ knowledge of modes and types of interpreting and the challenges of interpreter-mediated communication. The same questions were then asked in the post-test at the end of the module as a way of seeing whether the respondents had learned anything about interpreting during the course of the module. The module was set up so that the respondents could not go back and change any answers in the pre-test once they had been given the information. What is interpreting: This section contained basic factual information about interpreting as a profession, the types and modes of interpreting and why it is important to work with a professional interpreter. This information was based on the content of the Exploration mode. Interpreting pitfalls: In this section videos of interpreter-mediated communication were used to illustrate the challenges of interpreting, such as dealing with cultural differences, and to show examples of good and bad practice for people working with interpreters (e.g. addressing the interlocutor in the second person and not in the third person). The respondents were asked to watch the videos and were given text boxes where they could record their observations. The videos were linked to the SurveyMonkey module from the YouTube website. The first video showed an informal business meeting with an interpreter working between Vietnamese and English. The second video showed a conversation with a social worker in which the interpreter was working between Spanish and English. The first video simply showed the dialogue and respondents were asked to record their observations. The second video was split into two parts: the first part showed incorrect interpreting techniques (e.g. use of the third person) and the second part showed correct interpreting techniques (e.g. interpreter addresses the interlocutor in the second person).

Fig. 8: Video clip to show interpreting pitfalls (Spanish-English conversation between a social worker and a client)

Introduction to the IVY solution: Having provided an overview of interpreting and shown respondents the issues involved in working with an interpreter, the IVY environment was then presented as a solution for providing client training. In the same way that respondents might not have been familiar with the practice of interpreting or working with an interpreter, they might also not have not had much or any experience with 3D virtual worlds


51

Deliverable 6.3

or virtual learning environments, and it was therefore important to introduce them to the IVY solution using images and videos of the environment in use.

Fig. 9: Introduction to IVY solution from online evaluation module

The IVY solution section also served to demonstrate the Exploration mode, showing how the content that had been shown earlier in the module could be presented in the IVY environment. It was hoped that this would enable respondents to see how the IVY environment allows for different methods of delivery.

Fig. 10: Video showing Prezi in IVY Exploration mode

Fig. 11: Screenshot of learning activities in IVY Exploration mode

In addition to the Exploration mode, the respondents were also shown a video of the IVY Live mode in use, to give them an idea of how they could make use of this mode either with professional colleagues or with (trainee) interpreters.

Fig. 12: Video of the IVY Live mode


52

Deliverable 6.3

Impressions of the IVY environment: This section invites respondents to consider the environment that they have just been introduced to and to evaluate it. These questions were based on questionnaires that were developed for the planned evaluation workshops but were adapted to reflect the fact that respondents could only evaluate perceived user experience and learning success (e.g. “I would need a lot of training to work in this environment”) as they had not actually had any hands-on experience of the IVY environment. The videos of the IVY environment shown in the ‘Introduction to the IVY solution’ section were designed to recreate the user experience as closely as possible so that the respondents would be given a realistic impression of the IVY environment in use and therefore be able to evaluate it on this basis.

Fig. 13: Questions from the “Impressions of the IVY Environment” section

Once created, the online module was tested by three users at the Surrey test site to ensure that the users could progress through the module as intended (e.g. respondents could select multiple answers for certain questions). Feedback from the test respondents was incorporated into the module before it was sent out to respondents. An email with the link to the module was sent to a number of contacts from potential target groups, including business contacts, contacts in the legal and medical fields, trainers of legal professionals, etc. The link to the module was also posted on social media sites such as Facebook and LinkedIn. Participants were advised on the first page of the module that it would take around 30 minutes to complete. The module was open for seven weeks, at the end of which period the responses from SurveyMonkey module were collected and collated. The module remained active, however, so that other users could still access and complete the module for learning purposes. Given the difficulty in attracting and engaging clients of interpreting services, it was considered beneficial to leave the module open as a way of continuing to engage clients and provide an opportunity for learning. This is in line with the IVY project’s exploitation strategy (see also Deliverable 8.1).

5.3.2 Module results A total of 26 respondents took part in the module. Of these, 11 completed the module (i.e. provided answers from the beginning through to the end of the post-test) and 15 left the module without finishing it. It may be interesting to reflect on why respondents did not complete the module. A possible explanation is that it took longer than anticipated to complete, as mentioned in informal feedback from one of the respondents. A second


53

Deliverable 6.3

possible explanation is that some respondents may have misunderstood the purpose of the post-test as five respondents did not provide any answers for the post-test and two respondents filled in “already answered” or “same as before”, perhaps suggesting that they did not see the reason for answering the same set of questions a second time. A third explanation could be that respondents had difficulty viewing the videos embedded in the module, as two respondents reported that they could not watch the videos on their computers. The module responses are collated according to the different sections of the module where respondents were asked to provide answers or record observations, namely an overview of respondent profiles from the demographic questions, responses to the pre-test, what is interpreting, interpreting pitfalls, impressions of the IVY solution and responses to the posttest and comparison between pre-test and post-test responses to gauge whether the respondents had acquired any (further) knowledge about interpreting during the course of the module. Respondent profiles (demographic questions) Table 3 (below) provides an overview of the respondent profiles given as answers to the demographic questions. It shows that of the 26 respondents, 17 were female and 9 male. The majority of respondents were British residing in the UK (22 respondents), although responses were also gathered from Germany (1), Switzerland (1) and the USA (2). Respondents were asked to indicate which target group they belonged (i.e. a professional working in a private company or public service; a university student, e.g. of business, IT, law, medicine; or a trainee, e.g. in business, IT, law, medicine) and the professional field in which they work. Expertise with computers

Expertise with virtual worlds

Expertise with computer games

Training to work with an interpreter? Yes No No

Advanced

Novice

No

Novice

No

Advanced Advanced Expert

Novice Novice No

No No No

Novice Novice No

No No No

Intermediate Intermediate

No No

No No

Novice Novice

No No

Medicine

Intermediate

Novice

No

Intermediate

No

Other: Arts Law

Advanced Intermediate

Novice No Intermediate Intermediate

Novice Novice

No No

Law

Advanced

Expert

No

Expert

No

Law

Intermediate

No

No

Intermediate

No

C1 C2 C3

M M F

50-59 60+ 40-49

English British British

England Wales UK

C4

F

30-39

British

UK

Professional

C5 C6 C7

F M M

0-29 0-29 50-59

British Irish English

England UK UK

Professional Professional Professional

C8 C9

F F

30-39 0-29

British British

UK GB

C10

F

30-39

USA

C11 C12

F F

30-39 0-29

Puerto Rican UK British

Professional University student Professional

C13

M

0-29

English

England

C14

F

0-29

Bulgarian

UK

UK England

Professional University student University student University student


Expertise with SL

Field worked in

Advanced Intermediate Novice

Country of residence

Intermediate No Advanced Expert Novice No

Nationality

Advanced Advanced Expert

Age bracket

Law Business Other: Interpreting agency – training Other: Government IT Medicine Other: Law & languages Business Law

Gender

Professional Professional Professional

Respondent ID

Type of learner

Table 3: Overview of respondent profiles from client online module

54

Deliverable 6.3

C15 C16 C17 C18

M F F F

30-39 30-39 50-59 40-49

German British Latina Polish

UK UK USA UK

Professional Professional Professional University student

C19 C20 C21 C22 C23

M F F F M

30-39 30-39 40-49 60+ 40-49

British British British British Romanian

Professional Professional Professional Professional Professional

C24

M

0-29

British

UK England England England Switzerland UK

C25 C26

F F

30-39 60+

British German

England Professional Germany Professional

Professional

Law Law Medicine Other: Languages & Translation Business Business Business Medicine IT

Advanced Intermediate Advanced Advanced

No No Intermediate No

No No Novice No

Intermediate No Intermediate No

No No Yes No

Expert Advanced Advanced Intermediate Expert

No Novice Novice Novice Novice

No No Novice No No

Intermediate Novice Novice No Novice

No No No No No

Advanced

No

Intermediate

No

Novice No

No No

Other: Advanced Architecture Law Intermediate Other: Teacher Advanced training

Novice No Intermediate Intermediate

Tables 4 and 5 (below) show the target groups to which the respondents belong and the fields in which they work: Table 4: Target group Professional, e.g. working in a private company or public service University student, e.g. business, IT, law, medicine Trainee, e.g. business, IT, law, medicine Other

21 5 0 0

Table 5: Professional field Law Business IT Medicine Other

8 5 2 4 7

Other fields in which respondents work were: interpreting (training); government; law and languages; arts; languages and translation; architecture; and teacher training. Although it is disappointing that no trainees took part in the module as learners in vocational training settings was one of the groups that were targeted in particular, it is by no means surprising when taken in the context of the difficulties encountered throughout the project to engage adult learners in the different educational settings. It does, however, point to the need for further reflection on how to target these user groups and to demonstrate the relevance and importance of training to work in multilingual and multicultural contexts in general. Respondents were then asked to assess their expertise with computers, 3D virtual worlds, Second Life and computer games: Table 6: Expertise with computers, 3D virtual worlds, Second Life and computer games Expertise with Expertise with 3D Expertise with computers virtual worlds/ Second Life environments Expert 4 1 1 Advanced 14 2 0 Intermediate 8 4 2 Novice 0 11 2 No experience 0 8 21


Expertise with computer games 1 2 7 11 5

55

Deliverable 6.3

Table 6 (above) shows that all of the respondents rated their expertise with computers as intermediate or higher, but their expertise with 3D virtual worlds, Second Life and computer games was significantly lower. There was, however, more expertise rated intermediate or above in computer games than that in 3D virtual worlds or in Second Life. The expertise in computers and the relative lack of expertise in 3D virtual worlds and in Second Life is perhaps unsurprising, given that, in comparison to the use of computers and computer games in everyday life, 3D virtual worlds and Second Life are not as widely used or as familiar. Respondents were then asked to assess their level of familiarity with the work of professional interpreters and to detail what experience (if any) they had had with interpreters. The answers provided in the open-ended comment fields were broadly categorised in order for the responses to be compared: Table 7: Familiarity with the work of professional interpreters Level of familiarity No or very little experience Quite familiar: Have seen interpreters in action (e.g. live, on television) Quite familiar: Have worked with/used interpreters Very familiar: Have worked with/used interpreters a lot Very familiar: Work in language-related professions or work as an interpreter

Number 6 8 7 2 2

Table 7 (above) shows that there was quite a broad spectrum of experience with interpreters, with 11 of the 26 respondents having worked with interpreters or used the services of an interpreter. It is interesting, however, that only 2 of the 26 respondents stated that they had had training to work with an interpreter. One of these respondents (C1 – a law professional) stated that he “worked with interpreters on a professional basis” and had been “trained to use interpreters to obtain evidence through statements or suspect interviews”. Another respondent who had “extensive experience working with interpreters” and had received training (C17, professional in medicine) indicated that this training took place at James Madison University and comprised 40 hours covering “oral and written medical interpreting”. The final section of the demographic questions section asked respondents to indicate the extent to which they agreed with two statements relating to training in intercultural communication in general and, more specifically, the need for training to work with an interpreter. Table 8: Respondents’ views on training in intercultural communication and working with an interpreter

Communication with people from other cultures requires training People who speak through an interpreter need training

Strongly agree

Slightly agree

Neither agree nor disagree

Slightly disagree

Strongly disagree

10

10

4

1

1

10

9

5

0

2

Of these responses, the majority considered that training was needed for communicating with people from other cultures and for working with an interpreter. One respondent (C14) strongly disagreed with both statements and it is perhaps unsurprising that this respondent © IVY – February 2013 511862-LLP-1-2010-1-UK-KA3-KA3MP

56

Deliverable 6.3

did not answer any more questions after this one. The other respondent who strongly disagreed with statement 2 (C18, a Polish student working in languages and translation) also slightly agreed that training is required for communicating with people from other cultures, perhaps indicating a slight inconsistency in response. This is also surprising given that this respondent works in the field of languages and translation. Pre-test The following section of the module sought to find out what and how much respondents already knew about interpreting and working with and interpreter. These questions were then repeated at the end of the module in the post-test section. The rationale behind including a pre-test and a post-test was twofold: (1) To evaluate whether the content could be used to learn about interpreting and working with an interpreter by comparing the answers given in the pre-test with those given in the post-test; (2) To make an initial investigation into whether this mode of delivery (i.e. an online module) is suitable for the target user groups to learn about interpreting and working with an interpreter. As a general observation, the responses to the pre-test showed that the respondents in the most part had a relatively good basic understanding of the principles of working with an interpreter, the modes of interpreting (e.g. consecutive, simultaneous), the areas of interpreting (e.g. conference, business, public service interpreting), and how to book and interpreter. For example, when asked to explain what “consecutive interpreting” is (question 16), 20 respondents answered the question and all of these understood the concept – to a greater or lesser extent – except for two respondents. Similarly, most respondents (15 out of the 20) showed an understanding that the interpretation is given as the speaker is talking, with some respondents mentioning headsets or earphones. Four did not know the concept and 1 gave an ambiguous answer (“Interpreter listens and speaks in a different 'place' at the same time”, which could be confused with remote interpreting). When asked about “whispered interpreting”, the 19 respondents to this question either knew or were able to make educated guesses, although only 8 knew that the interpreter provided whispered simultaneous interpretation in close proximity to a small number of people who do not know the source language. However, when asked to explain the concept of liaison interpreting, eight respondents said they did not know and, from the remaining 12 who answered this question, it was evident that the concept was not understood, with some respondents confusing it with relay interpreting, remote interpreting and interpreting to a wider audience. The next set of questions related to the modes of interpreting (e.g. consecutive, simultaneous) used in different interpreting settings (e.g. conference interpreting, business interpreting, public service interpreting). This can perhaps be seen as a question that requires a slightly deeper knowledge of interpreting and how the mode of interpreting is linked to the purpose of the communication. Generally speaking, this proved more difficult for respondents.


57

Deliverable 6.3

The final four questions were designed to elicit the respondents’ knowledge about working with an interpreter. The respondents gave informed responses which demonstrated a reasonable level of awareness regarding the challenges that the interpreter and the client encounter during an interpreter-mediated event. For example, when asked about booking an interpreter, respondents mentioned finding an “accredited” interpreter, using national registers and finding interpreters who have experience working in their professional field. Similarly, respondents showed an awareness of the information that an interpreter might need before the assignment, such as practical information (e.g. date, time, duration, location), who the speakers/participants will be, the topic and any related or specialised vocabulary, level of formality and objectives of the communication. Question 20 asked respondents what speakers should avoid doing when speaking through an interpreter and of the 21 who answered this question, 19 gave the correct answer (“All of the above”). Finally, when asked to consider aspects of a transcribed section of speech that an interpreter might find challenging, the respondents gave astute and considered answers, referring to aspects such as idiomatic expressions, colloquialisms, syntactic breaks and poor syntax, and cultural references such as the “Students’ Union” and “Bananas in Pyjamas”. Given that only two respondents indicated that they had received training about working with an interpreter, the respondents demonstrated a good basic knowledge of the challenges inherent in interpreter-mediated communication. What is interpreting? This section followed on from the pre-test in that it asked respondents to consider why interpreting is a challenging task. Instead of being questions sought to elicit information about what and how much the respondents learned during the module (as these questions were not asked again at the end of the module), this question was designed to maintain the interest of the respondents: it prevented the module from being a series of pages which provided information that the respondents had to read and allowed the respondents to consider for themselves why interpreting is a challenging task before being given some suggestions on the next page of the module. Although not part of the pre-test, the responses given to this question confirm the findings from the pre-test as regards the respondents’ understanding and knowledge about why professional interpreters should be used, as illustrated in the following examples: “the consequences are far reaching or critical to the users” *C2+ “Ideas can be very abstract or ethereal, which makes description difficult even in one language.” *C4+ “reality is cut up in different way in different languages and concepts do not always overlap between languages/cultures” *C18+ “Misinterpreting vital information” *C6+ “People may not be using a speech but speaking off the cuff.” *C9+ “syntax, grammar & metaphor are not the same across languages” *C11+ “Directly translated words and phrases can mean different things e.g. I am full/je suis pleine” *C16+ “*The speech+ may employ highly-specialised, industry-specific concepts.” *C24+


58

Deliverable 6.3

These responses show a clear awareness of some of the difficulties involved in transposing meaning from one language into another. Interpreting pitfalls The section on ‘Interpreting pitfalls’ then illustrated the challenges in interpreting by showing two videos, and respondents were asked to record their observations. This section aimed to show respondents how information about working with an interpreter can be presented, i.e. via video clips. There are three key findings from this section: 1. Two respondents (C4, C5) reported that they could not play the videos on their computers. This shows that the use of video is not fool proof as a way of delivering content to learners. 2. All of the respondents except two who had stayed with this module up to this point recorded their observations on the videos. This shows that the respondents were engaged with the material being presented, although it must be borne in mind that 10 of the 26 respondents had left the module by this point. 3. Respondents who did record their observations of these videos gave insightful comments which further supplemented their profiles as informed learners who had a good basic understanding of working with an interpreter. Given that video 2 showed the differences between good and practice technique and therefore highlighted the key problems with the interpreted communication, the respondents’ comments on video 1 regarding the cultural differences are of particular interest: “Lack of understanding of culture regarding year of birth. Lack of understanding regarding body size in relation to success.” *C1+ “Cultural issues cause offence when interpreted literally Gestures cause offence” *C2+ “cultural references re: year of rat etc. misunderstood & interpreter doesn't highlight it; referring to business man's size as sign of prosperity; pointing finger at business women when indicating he's hoping for good luck, crossing his fingers.” *C3+ “cultural references not understood” *C8+ “There were cultural differences which were not fully explained such as the Rat and Pig issue.” “You're a rat should have been explained as a zodiac. Same with pig and Mr Jack's answer. Also her comment about his waste [sic] line. His crossing fingers might be problematic as it might be insulting in Vietnam” *C15+ “The references to the Chinese calendar didn't go down so well...” *C18+

These videos therefore give further credence to the view that the respondents who took this module have an understanding of how cultural differences can lead to misunderstanding and/or offence. Using the videos in this module also showed that video is not a fully reliable method for online learning, as it may be perceived to be when compared to the use of a 3D virtual environment.


59

Deliverable 6.3

Impressions of the IVY environment Having been shown more ‘traditional’ ways of presenting information on training to work with an interpreter (text-based content, video content), the respondents were then introduced to the IVY environment and shown demonstration videos of the IVY Exploration and Live modes. The first question in this section asked respondents to state to what extent they agreed with a number of statements. The responses are summarised in Table 9: Table 9: The extent to which respondents agreed with the statements given Strongly agree

Slightly agree


Slightly disagree

Strongly disagree

8

6

0

1

1

7

5

2

0

2

10

4

1

0

1

8

6

1

0

1

(c) ...Should include role plays

6

6

2

1

1

(d) ...Can take place in a 3D world

6

4

5

0

1

4. A 3D world provides useful options for such training 5. Training should be conducted with interpreters

7

5

3

0

1

9

3

1

0

1

1. Communication with people from other cultures requires training 2. People who speak through an interpreter need training 3. If training to work with an interpreter is given, it ... (a) Can be delivered online (b) ...Should include video clips

Table 9 (above) shows that 87.5% of respondents either strongly or slightly agree that communication with people from other cultures requires training and that 75% of respondents agree that people who speak through an interpreter need training. This can be compared with the answers to the same question at the beginning of the module (at the end of the demographic section), in which 77% of respondents (20 out of 26) agreed with the first statement and 73% (19 out of 26) with the second statement. While it may look as though there has been a 10% increase in respondents understanding the importance of training to communicate with people from other cultures, the number of respondents who left the module must be borne in mind and that this might have led to this distortion. Nonetheless, the vast majority of respondents agree that training can take place online, that role plays should be included in the training provision, and that 3D worlds can be used for this kind of training, although there is a discrepancy between the responses to statements 3d and 4. What is perhaps of particular interest is that 12 out of the 14 respondents recognise the value of training being conducted with an interpreter, thus supporting the contention that joint training can be beneficial for all parties involved in the communication. The following set of statements sought to assess respondents’ impressions of the IVY environment. As mentioned in section 5.3.1 (above), it must be borne in mind that respondents had not actually used the IVY environment during the online module and therefore could only assess perceived user experience and learning success. However, the © IVY – February 2013 511862-LLP-1-2010-1-UK-KA3-KA3MP

60

Deliverable 6.3

videos of the IVY environment shown in the ‘Introduction to the IVY solution’ section were designed to recreate the user experience as closely as possible so that the respondents would be given a realistic impression of the IVY environment in use and therefore be able to evaluate it on this basis. Table 10: The extent to which respondents agreed with the statements given

I would like to use the IVY 3D environment. The environment looks unnecessarily complex. The environment looks easy to use. I feel I would need technical support to work with this environment. The various functions in this system look well integrated. The environment looks coherent. Most people would learn to use this environment very quickly. The environment looks very cumbersome to use. I would feel very content using this environment. I would need to learn a lot about using Second Life before I could start working in this environment.

Strongly agree

Slightly agree


Slightly disagree

Strongly disagree

5 4

3 3

4 0

1 5

2 4

4 1

5 5

3 2

1 5

2 3

6

6

3

0

1

5 3

7 8

3 2

0 2

1 1

2

1

3

6

4

5

2

5

2

2

2

3

3

3

5

Overall, it can be seen from Table 10 (above) that the responses to the IVY environment were positive. In terms of usability, the majority of users agreed that the environment looked easy to use, that most people would learn to use the environment very quickly and that the environment did not look cumbersome to use. The statements regarding the user’s knowledge about using Second Life and the level of technical support that the user would require are somewhat conflicting, however: in response to the statement “I feel I would need technical support to work with this environment”, 6 respondents agreed while 8 respondents disagreed; to the statement “I would need to learn a lot about using Second Life before I could start working in this environment”, 5 agreed, while 8 disagreed. This would appear to show that the respondents would not be deterred by the need to learn how to work in the 3D world – and this is supported by the responses to the statement “Most people would learn to use this environment very quickly”, to which 11 out of 16 respondents agreed – but that once in the 3D environment they felt they would need support to work with the environment. This would then suggest that users would need support for working and learning in the IVY environment itself rather than needing rigorous inductions or support when starting to use Second Life. This finding is perhaps in line with the profile of the respondents as professionals who are competent users of computers. The respondents were then asked to provide more descriptive responses to open-ended questions relating to the positive and negative aspects of the environment, the environment as a tool for clients, the use of immersive 3D virtual learning environments, and any additional features or improvements that they would like to see in either 3D virtual worlds


61

Deliverable 6.3

or the IVY environment. Each question was answered by a maximum of 13 respondents, two of whom did not directly answer the question with one saying “no particular comment” and the other stating that she “couldn't view demo *videos+ due to restricted computer access”. Question 27: What do you think are the positive aspects of the IVY environment? The main themes that came out of the 10 responses to this question were that the environment appeared to allow users to simulate real-life, access the environment from a remote location, interact with other users, and access information. Simulation: Of the 10 respondents that answered the question, 5 respondents mentioned simulation as being a positive aspect of the environment. The responses elicited, even just within the theme of ‘simulation’, were varied as they referenced different reasons why simulation may be beneficial for the user, such as it allowing for “situational role-play” (C1), “situations that it would be difficult to deliver as training opportunities” (C3), “people to actually ‘see’ the environments where they may work” (C9), “real life situations” (C11) and “a variety of different situations” (C24). The nature of these responses indicates that the respondents feel that the environment’s ability to recreate a diversity of real-life situations is beneficial to both interpreters and clients. Remote access: Of the 10 respondents that answered the question, 3 respondents mentioned remote access as being a positive aspect of the environment. These responses varied, with one respondent feeling that being able to access the environment “from anywhere” (C1) was particularly beneficial, whilst another pointed to the environment allowing “people to work together without having to physically meet up making things faster and cheaper” (C9). One respondent wrote that the environment would allow her to work “at my own time and pace, much more flexible” (C2). These responses suggest that not only is the remote access to the environment valuable in terms of it being accessible from anywhere, but also that users could also appreciate the benefits of this such as costeffectiveness, time-saving and independent learning. The idea of independent/autonomous learning was also mentioned by C17 who stated that “learning at users *sic+ own pace” was a positive aspect of the environment. Interaction: Of the 10 respondents that answered the question, 2 respondents mentioned interaction as being a positive aspect of the IVY environment. The first of these respondents (C2) mentioned that the 3D aspect to the environment presented “many more options to interact” and the second (C8) stated they thought it would be “great to actually interact with interpreters”. These responses, both from professionals working in the business field, indicate that the interactional aspect of the environment that allows clients and interpreters to meet was seen as positive. Accessing Information: Of the 10 respondents that answered the question, 2 respondents mentioned accessing information as being a positive aspect of the IVY environment. Both of these responses mentioned the way that the environment could be used to allow information, in various forms, to be displayed and accessed. One respondent (C20) mentioned “there are multiple ways of displaying the information” and another (C2) stated that “3D presents many more options to interact and pick up additional information”. Again, both of these respondents were professionals working in the business field, and their


62

Deliverable 6.3

responses could be an indication that the multiple ways of accessing and displaying information is attractive to others working in their field. Question 28: What do you think are the negative aspects of the IVY environment? The main themes that came out of the 11 responses to this question were that the environment appeared to be unrealistic, technologically complex, and difficult for new users to use. Lack of realism: Of the 11 respondents that answered the question, five respondents mentioned the lack of realism as being a negative aspect of the IVY environment. The responses ranged from comments on the environment looking unprofessional to comments comparing the IVY environment with other technologies. Respondent C20 stated that “the people figures looked stereotypical e.g. Woman with blonde hair and large chest, Man with broad shoulders....Couldn't see a mix of ages” and C24 stated “it looks too comical perhaps slightly unprofessional”. In terms of a comparison with other technologies, C2 stated that the environment was “not as photorealistic as video conference” whilst C9 identified that the environment “will be less real as you cannot see the other person in reality, and so might be slightly forced or awkward” but went on to add that “however *it+ is better than something like Skype because you can see the working environments”. This is a clear indication that the clients are not experiencing the sense of realism that could perhaps have been expected: this could possibly be due to an increase in use of online video technologies (e.g. Skype, Google Hangout) which have seen people become accustomed to seeing the person they are communicating with. Technologically Complex: Of the 11 respondents that answered the question, three respondents mentioned the complexity of the technology as being a negative aspect of the IVY environment. All of these responses highlighted concerns that the respondents had with the use of the IVY environment on users’ computers. One respondent (C4) remarked that “you may need to ensure that people with slightly older versions of web browsers are still able to access the environment”, another (C17) stated “technology problems with computer” and a third respondent (C11) mentioned that it “might be difficult to use with some environments, internet connections, computers”. These responses support the outcomes of the pedagogical evaluation with the interpreting students who experienced difficulties using Second Life on their laptops/home computers. Difficulty for new users: Of the 11 respondents that answered the question, three respondents mentioned the difficulty for new users as being a negative aspect of the IVY environment. All of these responses centred on the concern that the environment looked as if it could be difficult to use for someone who was not used to 3D worlds. One respondent (C1) mentioned “some less technically able may find the use of a 3D world daunting and very alien to them”, a second (C8) noted that it “could be tricky for a first time user of second life/virtual environment” and a third (C16) wrote that “*i+t looks like it might be more comprehensible to people who are familiar with computer games”. These responses support the responses above where six respondents agreed that they would need technical support to work with this environment.


63

Deliverable 6.3

Question 29: Explain whether you think the IVY environment will be a useful tool for interpreters’ clients. Eleven respondents wrote an answer to this question with eight respondents saying the IVY environment will be a useful tool for interpreters’ clients, two stating they felt that the environment would only benefit some clients, and one writing, “probably - but I need to know more”. Of the eight respondents who believed that the IVY environment would be a useful tool for interpreters’ clients, five expanded on their answers to give reasons why they thought this to be the case. Three of these respondents mentioned that the information about the role of the interpreter was particularly useful for the client as it helps put “the role of interpreters into context” (C17) and allows clients to have “a preference of which method they feel more comfortable working with” (C20), although the latter comment may be based on a misunderstanding that a client can choose the mode of interpreting. Respondent C2 noted that the IVY environment as an “extra level of support for learning is obviously very beneficial” and C9 commented that “it allows people to actually view an environment and see how it all fits together”. The two respondents who felt that the IVY environment would only benefit some clients both mentioned that clients could be “busy” (C8) or “stretched for time” (C16) but respondent C8 did go on to say that it could be useful for those using interpreters on a regular basis. Respondent C16, a law professional, observed that the IVY environment would be useful in terms of facilitating “role play to practise beforehand” but also felt that “the rest looks unnecessarily burdensome and would put busy practitioners off” as well as it “not being suitable for the Bar”. This would suggest that an increase in information regarding interpreting and support for learning about interpreting would be valued amongst clients of interpreters. However, it should also be noted that, like all professionals, clients of interpreters are busy and therefore any content and mode of delivery needs to be concise and easily accessible in order for clients to learn as efficiently as possible. Question 30: What is your opinion of using an immersive 3D Virtual Learning Environment, such as the IVY environment? This question was answered by 10 respondents, five of whom responded positively without any reservations and four responded positively but had reservations. Some of the comments that came out of the unreserved responses were that the environment “brings training to life” (C1) and is “in line with future technologies” (C2). The reservations that some respondents cited mostly focussed on the technical specifications being applicable to all users and whether or not the IVY environment was suitable for all clients. In terms of apprehension about the amount of technical knowledge or equipment the user may need to access the IVY environment, respondent C4 stated “I can see the benefits as long as there is technical support for people using it” and C24 noted “I think it’s good; although it must be intuitive to use... users should be learning about something other than how to use the learning environment! It must also be as accessible as possible and there may be hardware constraints to using this that other delivery methods © IVY – February 2013 511862-LLP-1-2010-1-UK-KA3-KA3MP

64

Deliverable 6.3

are less susceptible to.” This may show that, although clients might be willing to use the IVY environment to learn about interpreting and interact with interpreters, they would be hesitant about doing so without sufficient technical support. It could also be the case that, due to the perceived steep learning curve that is associated with 3D virtual worlds, clients could be put off using the IVY environment because of the time required to learn how to use the platform itself. This idea was also reflected in the response from respondent C8 who wrote that, “Given that I only use interpreters once or twice a year I think it could be too intensive. I do think training is necessary though - perhaps a quick presentation/one page”. This perhaps points to an existing lack of awareness of the intricacies of working with an interpreter, although it could be argued that some training – however minimal – is preferable to no training at all. Similarly, respondent C20 stated “I would get used to it however I may be reluctant at first until I saw what it could do and what I could get out of using it”, and these responses indicate that, although some clients would be happy to use the environment, they would have to be sure that their time was going to be spent wisely. Question 31: Do you have any other comments or can you think of any additional features or improvements that you would like to see in either 3D virtual learning environments or the IVY environment? Seven respondents wrote an answer to this question, with five giving clear features or improvements that they felt could improve the IVY environment. Interestingly, all of these in some way related back to the idea of the environment not giving a sense of realism. Respondents C1 and C20 both mentioned “linking video clips of real interpretation situations” (C20) with C1 suggesting that combining “actual video clips of interpreting within the 3D world thus bringing the real and 3D world together as a *single+ training unit”. The incorporation of video was also mentioned by C3 who added that “maybe have a live video stream between the users so that they can actually see each other’s face”. This would suggest that the respondents were perhaps disappointed by the realism of the avatars, something that was noted by respondent C2, who suggested “better avatar functionality, perhaps facial and gestures could be improved” and C20 who felt that “*t+he human figures shouldn’t be so stereotypical or look like Lara Croft!” Other ideas included “training facilities for 3D navigation and interaction” (C3), which, again, perhaps highlights client concerns about being able to use the environment without assistance, and “subtitles - I have no idea how accessible second life is to people who are disabled” (C11). It is interesting that this respondent (C11) mentioned in the demographic questions section that she had used British Sign Language interpreters “quite a bit” and her concern with issues of accessibility is likely to be informed by her prior experience. Post-test The aim of the post-test was to determine whether the respondents learned anything during the course of the module and, if so, what they could learn from such an online training module. The results were compared with the responses given in the pre-test. For the purposes of the post-test, the ‘correct’ answers were based on the content given in the module, e.g. even though it is conceivable that the simultaneous mode of interpreting could be used in certain specific business interpreting settings (e.g. a speech or presentation given at the beginning of a business meeting), the module stated that the simultaneous mode is © IVY – February 2013 511862-LLP-1-2010-1-UK-KA3-KA3MP

65

Deliverable 6.3

primarily used in conference interpreting, so this was the answer that was considered ‘correct’. Out of the 26 respondents who started the module, 12 respondents completed the posttest. The answers from one of these 12 respondents (C7) were eliminated from the comparison between the pre- and post-test because the respondent answered “already answered” for questions 33, 35, 36 and 38 (i.e. the open comment questions). This perhaps shows that the respondent did not understand the purpose of the post-test or did not want to spend the time answering the questions. What follows is a comparison of 11 respondents’ answers to the pre- and post-test questions. Question 15 (pre-test)/32 (post-test) Interpreters specialise in different areas, especially conference interpreting, business interpreting and public service interpreting. Which of these do you think normally relate to bilingual situations (i.e. interpreting between two languages) and which do you think are often multilingual (i.e. interpreting between many languages)? For this question, ‘multilingual’ should only have been selected for ‘conference interpreting’ and ‘bilingual’ selected for ‘business interpreting’ and ‘public service interpreting’. In the pre-test, 6 out of the 11 respondents selected multilingual for conference interpreting only and bilingual for both business interpreting and public service interpreting. This shows that just over half of the respondents knew (or were able to make an educated guess) about modes and areas of interpreting. Of the 5 respondents who made incorrect selections in the pre-test, 3 made the correct selections in the post-test. This could show that the respondents had learned the difference between the different modes and areas of interpreting during the course of the module. Question 16/33 There are four different modes of interpreting: consecutive; liaison; simultaneous and whispered. What do you think each of these mean? For each of the different modes of interpreting, the respondents were asked to answer in open field comment boxes. Ten respondents knew what consecutive interpreting entailed in the pre-test; one did not know (C20) in the pre-test (“Order of words”) but gave the response “Waiting for the person to finish speaking before relaying the message” in the post-test. In addition to learning during the course of the module, there is also potential for learning more during the course of the module. Of the 10 who knew what consecutive interpreting involved in the pre-test, some respondents showed a deeper knowledge in the post-test, e.g. C9 who added “either a sentence or a paragraph etc.” after “Interpreting after the speaker has finished speaking” whereas the response in the pre-test was “Interpreting after the speaker has finished”. Fewer respondents knew about liaison interpreting in the pre-test, with only 1 respondent having a vague understanding of (“Interpreting what is said between two or more people” [C9]). In the post-test, 9 respondents had a significantly improved understanding of liaison interpreting, including C9 who used the word “conversation” in his/her response, perhaps


66

Deliverable 6.3

showing that the importance of the “dialogue” in liaison interpreting had been acquired during the module. As with consecutive interpreting, the majority of respondents understood what simultaneous interpreting involved in the pre-test; only 2 did not know in the pre-test, but both respondents demonstrated a good understanding of simultaneous in the post-test. Two respondents did not know what whispered interpreting entailed in the pre-test and it was interesting that these were the same respondents who did not know what simultaneous interpreting was (C11 and C20). Both respondents demonstrated in the posttest that they had learned what whispered interpreting was in the module. Additionally, there was improvement in responses from those who showed an understanding of whispered interpreting in the pre-test e.g. their responses in the post-test included information about whispered interpreting as a mode which can target an individual recipient or small group requiring interpretation and that the interpreter is in close proximity to the recipient(s). Question 17/34 Which mode of interpreting do you think is more likely to occur in which area of interpreting? (You can select multiple answers per row.) Again, answers were marked based on the information that was provided in the module. While it is conceivable that simultaneous interpreting could take place in a business setting (e.g. presentation at a business meeting), consecutive interpreting is given in the module as the predominant type of interpreting used in a business setting. It was therefore expected that simultaneous interpreting would not be an answer for the business setting. For consecutive interpreting, five respondents selected this mode as a mode used in conference interpreting in the pre-test, with four of these five not selecting it (i.e. providing a correct answer) in the post-test. One respondent did, however, not select conference interpreting in the pre-test and then selected it in the post-test. Eleven respondents selected business interpreting in the pre-test (correct); however, only 10 selected business interpreting in the post-test. Six respondents selected PSI (correct) in the pre-test, with 8 selecting it in the post-test, although one respondent who selected PSI in the pre-test did not select it in the post-test. It was shown above that respondents knew less about liaison interpreting than about simultaneous or consecutive interpreting. For liaison interpreting, then, three respondents selected it as being used in conference interpreting (incorrect); of those three responses, two also selected conference interpreting in the post-test. One respondent who had not selected conference interpreting in the pre-test selected conference interpreting in the post-test. Three respondents selected business interpreting in the pre-test (correct) but 10 respondents selected business interpreting in the post-test, showing that 7 respondents had learned that liaison interpreting was used in business interpreting during the course of the module and hence and improved understanding. Three respondents selected PSI in the pretest (correct) and six selected it in the post-test, again showing an increased knowledge about liaison interpreting. One respondent did, however, select PSI in the pre-test and not in the post-test. © IVY – February 2013 511862-LLP-1-2010-1-UK-KA3-KA3MP

67

Deliverable 6.3

For simultaneous interpreting, respondents were only expected to select conference interpreting as the area in which simultaneous interpreting is used. Nine selected conference interpreting in the pre-test but all respondents selected it in the post-test. In the pre-test, five selected business interpreting and three of these selected it again in the posttest, with one who did not select business interpreting in the pre-test selecting it in the post-test. Nine respondents selected PSI in the pre-test yet only two of these selected PSI in the post-test. Based on the content of the module, it was expected that respondents would only select business interpreting and PSI as areas in which whispered interpreting is used. Eight selected conference interpreting (incorrect) in the pre-test; only 3 of these selected conference interpreting in the post-test. Six respondents selected business interpreting in the pre-test and six in the post-test, but of the six who selected business interpreting in the post-test, only four of these had selected it in the pre-test. This shows that two respondents had learned during the course of the module that whispered interpreting is used in business interpreting, while (a different) two respondents had selected it in the pre-test but not in the post-test. Six respondents selected PSI in the pre-test and 7 in the post-test: of these seven, three had selected it in post-test where they had not selected it in pre-test, whereas two had selected it in pre-test and not in post-test. Questions 18/35, 19/36 and 21/38 18/35: If you needed to book an interpreter, how do you think you could go about doing this (e.g. how would you find an interpreter, what information would you need to give the interpreter)? 19/36: Can you name three things that an interpreter may need to know before starting an interpreting assignment? 20/38: The following section of text comes from an interview with the Director of Trading at a British university. He is talking here about the concept of fair trade. Imagine that an interpreter is interpreting what he says into another language. Look at the section of speech below and try to think of three things an interpreter might find difficult about what he says or how he speaks. As shown above, all respondents gave sensible and practical answers to these questions in the pre-test so it is perhaps unsurprising that there was no real change between the answers given in the pre-test and those given in the post-test. It is therefore difficult to evaluate whether the content on booking, preparing and working with an interpreter was useful or not because the respondents had a good understanding of what an interpreter would need to know before starting the module. Question 20/37 What should you avoid doing when you are speaking through an interpreter? Telling impromptu jokes Speaking off the record Using wordplay All of the above


68

Deliverable 6.3

The answer expected from the respondents was “All of the above”. Three respondents selected one of the first three answers in the pre-test; all respondents selected “All of the above” in the post-test, except for one respondent (C9) who selected “telling impromptu jokes” in the post-test, which was not the answer selected in pre-test (“using wordplay”). The pre-test responses show that the respondents had a good level of knowledge and awareness relating to the challenges inherent in working with an interpreter before starting the module, so it is difficult to assess the usefulness and relevance of the content in the module dealing with booking, briefing and working with an interpreter. The respondents were less knowledgeable about the modes and areas of interpreting and the improvement in the respondents’ answers to these questions in the post-test show that the respondents learned about the modes of interpreting and the settings in which they are used. In addition, it was shown that those respondents who did already have an understanding of the issues involved in working with an interpreter were able to improve and refine their knowledge with the content in this module.

5.3.3 Key outcomes This section aims to draw together the findings from the pedagogical evaluation with (potential) clients of interpreting services. The first part looks at the outcomes relating to the respondents who took part in this module. There is then a summary of the findings in terms of the provision, content and delivery of training material for the target user groups. The final part draws together the respondents’ assessment of the IVY environment. Module respondents The profiles of the respondents given in the demographic questions showed that the majority of the respondents were professionals. Five respondents were university students. None of the respondents were trainees. It is also the case that most of the respondents who finished the module and completed the post-test (C1-4, C7-9, C11, C17, C20, C24) had stated that they were either quite or very familiar with the work of interpreters (except C20 and C24) and that this perhaps meant that a) they already had an understanding of the challenges involved in working with an interpreter and how interpreter-mediated communication takes place and b) that this experience meant that they were interested in completing the module. Although the respondents who completed the post-test showed an increase in understanding in the different modes and areas of interpreting, it remains the case that the respondents had an understanding of interpreting before starting the module and that these are perhaps not the people who require this kind of training. The professionals who completed the module and had had experience of working with an interpreter had perhaps already learned much of this content by actually working with an interpreter and perhaps having to learn how to work with interpreters on a trial-and-error basis in the course of their professional work. The aim of this kind of content, then, should perhaps aim to a) improve their understanding – which it has been shown here to do, and b) target rather trainees who have not yet experienced working with an interpreter so that they can be given all the information and/or opportunities for practice required for working with an interpreter to avoid having to learn on a trial-and-error, on-the-job fashion.


69

Deliverable 6.3

The difficulty in engaging clients of interpreting services with this module reinforces the findings that it is difficult to target clients in general. There was a relatively high module drop-out rate which could in some cases be attributed to the fact that the module did take longer to complete than suggested and that some respondents had problems accessing the videos. If online modules are to be used as a mode of delivering this kind of training it would perhaps be worth finding out why respondents leave the module before completing it and exploring how these reasons can be addressed. Provision, content and delivery of training material The online module was designed with the twin aim of 1) providing information about interpreting and working with an interpreter and 2) eliciting impressions of the IVY environment as a means of delivering training. The module therefore had to be designed in such a way as to balance pedagogical and evaluative objectives. Following demographic questions, the module was structured to:   

provide information about interpreting and working with an interpreter; show ‘traditional’ ways of how this information is provided, e.g. text-based and video training; introduce the IVY environment as one way of delivering the content and elicit feedback on this environment.

These sections were preceded and followed by a pre- and post-test respectively to assess what the respondents knew at the start of the module and what they had learned by the end of the module. Given that the answers to the pre-tests showed that respondents had a good awareness of the challenges of working with an interpreter, it was difficult to assess the relevance or success of the content about working with an interpreter. The comparison of the answers in the pre- and post-tests did show, however, that respondents learned about modes and areas of interpreting, or that those with an existing knowledge improved their knowledge. This is perhaps due to the fact that the majority of the respondents were professionals, many of whom were familiar with the work of interpreters. It would have been very interesting if trainees who had had no experience of working with an interpreter had completed the module, as it may be the case that they would have been less familiar with the work of interpreters in professional contexts and that they would have benefitted more from the content providing information about booking, preparing and working with an interpreter. Engaging trainees would therefore be a very productive next step in evaluating the learning potential of the module content, with a view to bringing together professionals with their (perhaps self-taught) experience with interpreters and trainees who may have minimal or no experience with interpreters. This would lead to a point where trainees and professionals have the same level of knowledge about working with an interpreter. Moreover, regardless of how this training is provided – either via the IVY Exploration mode, online modules, video chat services such as Skype or Google Hangout, face-to-face workshops, or a combination thereof – the IVY Live mode can still be used as a ‘safe’ environment for clients to put their knowledge into practice. A second main finding relates to the use of video in the IVY environment. Concerns were raised by respondents about the need for advanced/specific technical requirements needed © IVY – February 2013 511862-LLP-1-2010-1-UK-KA3-KA3MP

70

Deliverable 6.3

to run the Second Life client – which is true – but it is also true that viewing videos also requires a minimum specification (although these requirements are usually less restrictive). A number of respondents indicated that video clips embedded in the environment would increase the level of realism, but some respondents had difficulty viewing the video that was embedded in the online training module. This shows that there are hardware/software specifications for any mode of delivery and that the basic requirements must be specified to and met by individual users prior to the start of the training. In terms of training content, a large majority of respondents said that training can take place online, role plays should be included in the training provision and that 3D worlds can be used for this kind of training. In addition, respondents recognised the benefit of training taking place with an interpreter. These are aspects that will be taken up in the follow-up project to IVY, EVIVA, which will explore how these different training environments can best be used for different training tasks, including role play. A final point regarding the evaluation of the IVY environment rests on the distinction between Second Life as a platform and the IVY environment as a training site within that platform. In the evaluation, it is sometimes unclear whether respondents are referring to Second Life as a whole or to the IVY environment specifically. It is therefore important in future training/evaluation content that this distinction is drawn clearly and that the users understand how the two interrelate. Some observations and comments clearly relate more to Second Life (e.g. technical requirements), whereas others focus more on the IVY environment itself (e.g. diversity of interpreting scenarios). Assessment of the IVY environment It must be remembered here that the respondents who took part in this module did not have hands-on experience of the IVY environment but that they were shown the principle of delivering training in a 3D virtual world via demo videos. It is within this context that the outcomes are discussed. The two aspects that most respondents focused on were usability and the need for technical support and realism and real-life simulation. Usability and need for technical support: Generally speaking, respondents said that the environment would be easy to use, that they would learn to use it quickly and that it did not look cumbersome to use. However, when asked in the open-ended questions to indicate negative aspects of the IVY environment, they mentioned that the environment “might not work in older versions of web browsers” or that there might be “problems with computer”. Similarly, respondents indicated concerns that “some less technically able may find the use of a 3D world daunting and very alien to them” and that it “could be tricky for a first time user of Second Life/virtual environment”. It is interesting that the respondents feel that they would not have any difficulties using Second Life but that they suggest it might be other users that may have difficulty from a technical or usability perspective. While most respondents said they would learn to use the environment very quickly, just under half of respondents said that they would need technical support to work in this environment. Realism and real-life simulation: Respondents said that the idea of simulating real-life situations was a positive aspect of the IVY environment and reported the facility for interaction as an advantage of the IVY environment, and this was supported by their views © IVY – February 2013 511862-LLP-1-2010-1-UK-KA3-KA3MP

71

Deliverable 6.3

that role plays should form part of the training and that training should be conducted with interpreters. Whilst the simulation of authentic interpreting situations was considered valuable, the realism of the IVY environment was compared unfavourably with other online technologies. The number of respondents who referred to the realism of 3D game worlds and seeing interlocutors live on online video services is indicative of the level of realism users now expect. Issues of realism and the use of other online learning technologies (e.g. videoconferencing, video-based environments) will be addressed in the EVIVA project.


72

Deliverable 6.3

6 Conclusions In this section we highlight the main impacts of the IVY solution on the two target user groups. One general outcome is that the evaluation confirmed the anticipated effect of the steep learning curve reported in VLE pedagogy (e.g. Sanchez 2007, Carr 2010) only to a certain extent. The users whose experience in IVY we investigated, i.e. the interpreting students, differed with regard to their level of computer skills and with regard to the ease with which they grasped the main principles of the IVY environment. Similarly, the responses from interpreter clients indicated that many of them found the environment (or what they saw of it) easy to use and did not anticipate many difficulties. However, all participants in the evaluation indicated in one form or another that they would appreciate an induction to and guidance for different aspects. This need for individualised support calls for using IVY in blended teaching/learning contexts where an adequate amount of instruction can be provided in a continuous manner. Therefore, as set out in the Exploitation Strategy (D8.1), we decided to create a Visitor Centre rather than make the IVY platform fully available. In the Visitor Centre we will be able to give guided tours to tutors and other institutional representatives interested in using our tool for their training purposes. We intend to let visitors explore the opportunities afforded by this environment and in so doing they will acquire “mentor” status enabling them to co-ordinate the training of future users in blended contexts. The IVY team plan to discuss with the mentors their institutional needs in order to accommodate them on a caseby-case basis. The mentors will help us to spread and grow the IVY network by attracting new students of interpreting and prospective clients of interpreting services. We believe that this approach will also be useful for deciding on directions of further development which will be taken up further in the EVIVA project. It should also be noted that the range of materials available in IVY is easily extendable because, in addition to the immediate IVY solution, it is also possible to add monologues and dialogues in other languages/language combinations via the administration panel which was developed with content provision in mind. This enables the IVY team to increase the number of language combinations continuously, a feature which is bound to make the IVY environment more appealing to a wider audience of prospective users. As regards the technological aspects, the Second Life environment clearly offers a series of advantages when it comes to creating a complex, accessible and collaborative world. The evaluation participants mostly found the 3D environment intuitive and easy to use. They appreciated the flexibility of the HUD interface, which was built with web technologies. From a visual point of view, the environment was perceived by most evaluators to be fairly realistic, albeit with room for improvement. However, the evaluation also highlighted the fact that users who do not have sufficient hardware/access capabilities are likely to get discouraged from exploring the full potential of the IVY tool. In order to benefit fully from working in the IVY environment it is essential for users to meet a number of hardware requirements.


73

Deliverable 6.3

A final key outcome, which requires further examination, is that the IVY Live mode was only explored to a limited extent (by the interpreting students). At the same time, the Live mode provides unprecedented opportunities for collaborative learning and student (and teacher) mobility. The Live mode could be considered to be at the very heart of an environment that has as its declared intention the support of training in business and Public service Interpreting where dialogue interaction has an important role to play. It is the Live mode that takes IVY beyond CAIT tools which satisfy only cognitive constructivist principles of learning. The Live mode turns IVY into a tool that meets the principles of social constructivist learning, such as social interaction and social participation. The Live mode therefore enables students to conceptualise interpreting not only as a cognitive activity but also as a social experience. However, it appears that students may need even more guidance and tutor support in order to benefit fully from the advantages of the Live mode interaction.


74

Deliverable 6.3

7 References Angelelli, C.V. (2006) “Designing Curriculum for healthcare interpreting Education: A Principles Approach”. In Cynthia B. Roy (ed.) (2006). New Approaches to Interpreter Education Washington, Interpreter Education Series, Vol. 3. Washington, D.C.: Gallaudet University Press, 23-46. URL: http://gupress.gallaudet.edu/excerpts/NAIE.html [accessed 08.02.2009]. Brooke, J. (1996) “SUS: a “quick and dirty” usability scale”. In P.W. Jordan, Thomas, B. Weerdmeester, B. A. McClelland. A. L. Usability Evaluation in Industry. London: Taylor and Francis. Bangor, A. Kortum, Philip T. and Miller, J.T. (2008) “An Empirical Evaluation of the System Usability Scale”. International Journal of Human-Computer Interaction 24 (6): 574–594. Benson P. (2001) Teaching and researching autonomy in language learning. Harlow: Pearson Education. Carr, D., Oliver, M., Burn, A. (2010) “Learning, Teaching and Ambiguity in Virtual Worlds”. In A. Peachey, Gillen, J., Livingstone, D., Smith-Robbins, S. (eds.) Researching Learning in Virtual Worlds London: Springer, 17-30. Holec, H. (1981) Autonomy and foreign language learning. Oxford: Pergamon Press. Horvath, I. (2007) “Autonomous learning: what makes it work in postgraduate interpreter training?” Across Languages and Cultures 8(1), 103-122. Kiraly, D. (2005) “Project-Based Learning: A Case for Situated Translation” Meta, 50: 1098-1111. Kiraly, D. (2003) “From Teacher-Centred to Learning-Centred Classrooms in Translator Education: Control, Chaos or Collaboration?” In A. Pym, Fallada, C., Biau, J. R. Orenstein J. (eds): Innovation and E-learning in Translator Training. Universitat Rovirai Virgili Tarragona: Intercultural Studies Group. Kiraly, D. (2000) A Social Constructivist Approach to Translator Education. Empowerment from Theory to Practice. Manchester, UK/Northampton, MA: St. Jerome Publishing. Ritsos, P.D., Gittins, R., Braun, S., Slater, C., Roberts, J.C. (2013) “Training Interpreters using Virtual Worlds”, LNCS Transactions on Computational Science, Springer (accepted for publication) Samson, R. (2005) “Computer-assisted translation”. In M. Tennent (ed.) Training for the New Millennium: Pedagogies for Translation and Interpreting. Amsterdam: John Benjamins, 101126. Sanchez, J. (2007) “Breaking the Second Life High Learning Curve”. URL: http://www.educause.edu/blog/joesanchez/BreakingtheSecondLifehighLearn/167270 [accessed 13/04/2011]. Sandrelli, A. (2012) “A historical overview of CAIT (presentation delivered during IVY Research Seminar: Exploiting Emerging Technologies to Prepare Interpreters and their Clients for Professional Practice”. London 23 November 2012) Sobkowiak, W. (2012) “Five years in Second Life, or: Phonetically Augmented Virtuality in Second Life English as a Foreign Language”. URL: http://www.scribd.com/doc/108718699/Paving-EFL-inSL [accessed 13/04/2011]. Tymczyoska, M. (2009) “Integrating in-class and online learning activities in a healthcare interpreting course using Moodle”. JoSTrans - The Journal of Specialised Translation: 12 URL. http://www.jostrans.org/issue12/art_tymczynska.php [accessed 13/04/2011].


75

Deliverable 6.3

8 Appendix 1: Interpreting Practice E-diary 9 Appendix 2: IVY Evaluation Questionnaires – Interpreting students 10 Appendix 3: IVY Evaluation Questionnaire – Clients


76

Your name:

3. Space for your material collection, e.g. for notes on your background research, glossary work, usage notes, etc.

2. Space for your narrative/reflective comments on your progress

1. Guidance notes

This e-diary contains

practised real-time interpreting or repeated the turns of a monologue/dialogue several times recorded your interpretation and listened back to it to assess your performance worked with the transcript of a monologue/dialogue (say what you did) worked on your own, in a pair, or in a group (explain what you did) used the IVY learning activities (say what activities you used and how)

1

what you have learnt, e.g. in terms of source and target language terms and phrases, subject knowledge, communicative functions (i.e. did you achieve your aim, did you learn something else), what interpreting-related skills you think you have improved (i.e. listening and analysis, memory, production skills, stress management skills), etc. Assess the usefulness and level of difficulty of the IVY materials you practiced with today what problems you encountered, how you solved them and why you solved them in this way whether the way you practised was efficient, and what you would do differently next time.

Interpreting practice e-diary

• • •

•

Reflect on the usefulness of what you did. Comment, for example, on:

5. Learning success and problems

• • • • •

Explain how you chose to work. You might have, for example:

4. Method of practice

• • •

If you used the IVY monologues/dialogues, give the title; if you used the IVY live mode or worked outside the IVY environment, describe the materials you used State the language or language pair you worked in Say why you chose this specific material (e.g. interest in topic area, terminology, setting) Describe the specific aim of your practice session (e.g. to practise note-taking, to improve listening comprehension, to research the terminology, to improve your output; to continue from a previous session).

Give the following information about the material(s) you used for your interpreting practice, i.e.

The purpose of this diary is to enable you to record what interpreting practice you have done, either in the IVY Second Life environment or outside this environment. You are encouraged to write freely in order to reflect on your interpreting practice and the progress you are making. Your e-diary will form the basis of your progress reviews with your tutors and programme director. •

3. Materials used

Say whether you used the IVY monologues/dialogues or the IVY live mode, or whether you worked outside the IVY environment (and say why).

2. What you did

Write the date of your practice session on your e-diary form and write down how long you spent practising interpreting.

1. Date and time

These guidance notes are designed to help you write your e-diary. A set of reminders appears at the top of each page of your e-diary.

Guidance notes

Purpose of e-diary

2012-13

E-Diary

Interpreting Practice


Your text…

2


Your text…

Please use the space below for notes on your background research, glossary work etc.

Date of session:

Please use the space below to explain what you did and to comment on the materials you used, how you practised with them and how you would assess your learning success.

Notes on session 1

Practice session 1

Time spent:

Please copy this page for each session.

Please copy this page for each session.

3

Participant IVY login:

Sex: Male / Female

1

PART II. On a scale 1 to 5 rate how helpful IVY VLE practice has been in dealing with: 1. complex sentences (not helpful) 1 2 3 4 5 (very helpful) 2. incoherent utterances (not helpful) 1 2 3 4 5 (very helpful) 3. accents and dialects (not helpful) 1 2 3 4 5 (very helpful) 4. sloppy pronunciation (not helpful) 1 2 3 4 5 (very helpful) 5. fast speech (not helpful) 1 2 3 4 5 (very helpful) 6. repetitions (not helpful) 1 2 3 4 5 (very helpful) 7. numbers and proper names (not helpful) 1 2 3 4 5 (very helpful) 8. challenging terminology (not helpful) 1 2 3 4 5 (very helpful) 9. metaphors and idiomatic language (not helpful) 1 2 3 4 5 (very helpful) 10. abstract concepts (not helpful) 1 2 3 4 5 (very helpful) 11. ambiguities (not helpful) 1 2 3 4 5 (very helpful)

5. What interpreting exercises do you do on a regular basis (sight translation, reformulation, etc.)? in class: ………………………………………………………………………………………………. individual practice: ……………………………………………………………………………………………….

4. How many hours of individual interpreting practice do you think you should do per week? ……………………………………………………………………………………………….

3. What freely available materials do you use or could you use for interpreting practice? ………………………………………………………………………………………………. ……………………………………………………………………………………………….

2. Have you ever interpreted before starting this interpreting course? If so, describe it briefly. ………………………………………………………………………………………………. ……………………………………………………………………………………………….

1. What is your language combination? Indicate how long you have been learning your B and C languages. Language A: ………………; Language B: ……………… studied:… years; Language C: ……………… studied:… years;

PART I. Answer the following questions:

Name:

Pedagogical Evaluation of IVY by students of interpreting: QUESTIONNAIRE

5

4

5

4 5

5

4

4

5

4

(very helpful)

(very helpful)

(very helpful)

(very helpful)

(very helpful)

2

2. Have you used computer-based learning, online learning and virtual environments before? If so, please specify which ones and how you would rate your level of expertise: (very low) 1 2 3 4 5 (very high). …………………………………………………………………………………………………... …………………………………………………………………………………………………... …………………………………………………………………………………………………... …………………………………………………………………………………………………...

1. What would you change/improve/add in IVY VLE? …………………………………………………………………………………………………... …………………………………………………………………………………………………... …………………………………………………………………………………………………... …………………………………………………………………………………………………... …………………………………………………………………………………………………...

PART III. Answer the following questions:

On a scale 1 to 5 rate to what extent statements below reflect your interpreting experience in IVY VLE. 1. I felt stressed while interpreting in IVY VLE. (not at all true) 1 2 3 4 5 (very true) 2. Practice in IVY VLE enhanced my concentration skills. (not at all true) 1 2 3 4 5 (very true) 3. IVY VLE practice has not improved my comprehension skills. (not at all true) 1 2 3 4 5 (very true) 4. My memory skills have not improved after practice in IVY VLE. (not at all true) 1 2 3 4 5 (very true) 5. Practice in IVY VLE helped me to achieve a more fluent delivery. (not at all true) 1 2 3 4 5 (very true) 6. IVY VLE practice did not improve my stress management skills. (not at all true) 1 2 3 4 5 (very true) 7. Interpreting in IVY VLE felt more artificial than interpreting in-class. (not at all true) 1 2 3 4 5 (very true) 8. Practice in IVY VLE increased my self-confidence in interpreting situations. (not at all true) 1 2 3 4 5 (very true) 9. IVY VLE helped me to prepare efficiently for interpreting assignments. (not at all true) 1 2 3 4 5 (very true) 10. I feel more confident practicing interpretation in class than in IVY VLE. (not at all true) 1 2 3 4 5 (very true) 11. IVY VLE made me realize that my role as an interpreter is to help interlocutors communicate effectively. (not at all true) 1 2 3 4 5 (very true)

12. high information load (not helpful) 1 2 3 13. lack of target language equivalents (not helpful) 1 2 3 14. cultural references (not helpful) 1 2 3 15. irony and humor (not helpful) 1 2 3 16. forms of address (not helpful) 1 2 3

6.

5.

4.

3.

2.

1.

Gender:

Male □

Female □ mostly in groups □ mostly in groups □

mostly alone □

Dialogues:

…… hours per week on average …… hours per week on average

Using IVY materials (e.g. transcripts/printouts) Using other materials (e.g. given in class/ your own)

No □

Used since (date)

1

(= very little)

2

Level of experience 3 4

5 (= very much)

No □

Not sure □

…… hours (total)

…… hours

…… hours

…… hours

Specific preparatory learning activities

Specific reflective learning activities

Generic learning activities

Learning activities:


Dialogues


Live mode

Monologues

1

Please indicate the amount of time you have spent in the IVY Second Life environment on the following materials and using the different modes during the evaluation period.

Please provide any reasons why this was/was not sufficient in the box below.

Was this number of hours sufficient? Yes □

…………………… hours

How many hours of interpreting practice did you do on average per week during the evaluation period, i.e. since week 6 (including all practice outside class time, i.e. in and outside Second Life, on your own, in pairs and in groups)?

Name of tool used

If yes, please specify which ones and how you would describe your expertise with these tools.

Yes □

Have you ever used any other computer-based learning programmes, online learning platforms or virtual environments before (whether for interpreting practice or not)?

Very poor

Poor

Fair

Quite difficult

Neither difficult nor easy

Quite easy

Good

Quite unrealistic

Neither realistic nor unrealistic

Quite good

Quite realistic

Very good

Very realistic

Very easy

Very good

2

Please use the box below for any additional comments about the SL environment and the IVY content (e.g. realism, sense of presence in the interpreting situation, use of robots, content and quality of monologues/dialogues, etc.).

13. How would you rate the audio quality of the monologues and dialogues? Very poor Quite poor Neither poor nor good

Very unrealistic

12. How realistic did you find the different interpreting scenarios (e.g. meeting room, tourist office, etc.)?

Very difficult

11. How would you rate the IVY Second Life environment in terms of usability?

ACTIVITY Interpreting practice (monologues /dialogues) Live interactions

10. How would you rate the IVY Second Life environment as a place to practise interpreting and perform live interactions?

Part 3: The IVY Second Life environment

f.

e.

d.

c.

b.

……………………………………………………………….……………………………………………………

……………………………………………………………………………………………….……………………………………………………...………………………………

Grammar of your A and B language 1 2 3 4 5 Lexis and terminology in your A and b language 1 2 3 4 5 Features of spoken language (e.g. dealing with sloppy pronunciation, fast delivery, different accents, repetitions, incoherence in the source texts, etc.) 1 2 3 4 5 Interpreting skills (e.g. memory training, note-taking, etc.) 1 2 3 4 5 Target text production (e.g. accuracy, completeness and coherence of your target text) 1 2 3 4 5 Target text delivery (e.g. improving delivery and fluency of your target text, etc.) 1 2 3 4 5

Scale: 1 = very unhelpful; 2 = quite unhelpful; 3 = neither helpful nor unhelpful 4 = quite helpful; 5 = very helpful

How would you rate the helpfulness of the IVY Second Life environment in dealing with the following areas of interpreting practice?

a.

9.

both □

both □

both □

How much interpreting practice time per week did you spend on average outside the IVY Second Life environment,

Learning activities:

mostly in groups □

mostly alone □ mostly alone □

Monologues:

Please indicate briefly whether you worked with the different materials and in the different modes mostly alone or in pairs/groups or both.

Part 2: Specific areas of interpreting practice

8.

7.

Pedagogical evaluation: Interpreting students

What language pairs did you use in the IVY Second Life environment? Please indicate all language combinations and language directions you worked with.

……………………………………………………………….……………………………………………………

……………………………………………………………………………………………….……………………………………………………...………………………………

Have you ever interpreted (including in your first degree) before you started the MA? If so, please describe it briefly.

Language C: …………………… studied for ….. years

Language B: …………………… studied for ….. years

Language A/mother tongue: ……………………

What are your working languages? Please indicate how long you have been learning your B and C languages:

Part 1: General information

Name: ……………………………………………………………….

IVY EVALUATION QUESTIONNAIRE


i.

h.

g.

f.

e.

d.

c.

b.

a.

Scale:

Thank you for taking the time to complete this questionnaire.

I think that using the IVY SL environment helped me improve my interpreting performance. 1 2 3 4 5 I think that using the IVY environment enhanced my concentration skills. 1 2 3 4 5 I think that using the IVY environment helped to improve my memory skills. 1 2 3 4 5 I think that practising in the IVY environment helped to improve my comprehension skills. 1 2 3 4 5 I think that practising interpreting in the IVY environment helped me to achieve a more fluent delivery. 1 2 3 4 5 I think that practising in the IVY environment helped to increase my self-confidence in interpreting situations. 1 2 3 4 5 I think that the IVY environment helped me understand how to prepare efficiently for interpreting assignments. 1 2 3 4 5 I think that working in the IVY environment made me more confident interpreting in class. 1 2 3 4 5 I would recommend the IVY environment to other trainee interpreters. 1 2 3 4 5

1 = completely disagree; 2 = somewhat disagree; 3 = neither agree nor disagree 4 = somewhat agree; 5 = completely agree

3

14. To what extent do you agree with the following statements about your interpreting experience in the IVY Second Life environment?

Part 4: Your interpreting experience in the IVY Second Life environment


Bangor University

School of Computer Science

October 2012

IVY Virtual Environment Usability Evaluation

IVY - Interpreting in Virtual Reality

I understand that all data that is a result of my participation will remain strictly confidential. I understand that I may request a summary of the results of this study by contacting one of the researchers (Panagiotis D. Ritsos ([email protected]), Robert Gittins ([email protected]), Jonathan C. Roberts ([email protected]). I understand that I am free to withdraw my consent and terminate my participation at anytime without prejudice. My responsibility as a participant is to take part in the experiment actively, and willingly and if I choose to do so I will exercise my right to withdraw.

I understand that all data that is a result of my participation will remain strictly confidential. I understand that I may request a summary of the results of this study by contacting one of the researchers (Panagiotis D. Ritsos ([email protected]), Robert Gittins ([email protected]), Jonathan C. Roberts ([email protected]).

I understand that I am free to withdraw my consent and terminate my participation at anytime without prejudice.

My responsibility as a participant is to take part in the experiment actively, and willingly and if I choose to do so I will exercise my right to withdraw.

Experimenter’s Signature . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

This form will be produced in duplicate. One copy should be retained by the participant and the other by the researcher.

Date . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

I certify that I have fully explained the investigation to the above individual.

Experimenter’s Signature . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

This form will be produced in duplicate. One copy should be retained by the participant and the other by the researcher.

Date . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

I certify that I have fully explained the investigation to the above individual.

Participant’s Signature . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

The experiment, and my role in the investigation have been fully explained to me by Panagiotis D. Ritsos, and I fully understand his explanation. The procedures and risks have been explained to me fully and I have been able to have all questions answered to my satisfaction.

The experiment, and my role in the investigation have been fully explained to me by Panagiotis D. Ritsos, and I fully understand his explanation. The procedures and risks have been explained to me fully and I have been able to have all questions answered to my satisfaction.

Date . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

This is to certify that I hereby agree to participate as a volunteer in an experiment as an authorised part of the research undertakings within the School of Computer Science at Bangor University, under the supervision of Jonathan C. Roberts.

This is to certify that I hereby agree to participate as a volunteer in an experiment as an authorised part of the research undertakings within the School of Computer Science at Bangor University, under the supervision of Jonathan C. Roberts.

Participant’s Signature . . . . . . . . . . . . . . . . . . . . . . . . . . . . .



Date . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Consent Form

Consent Form

Evaluation Tasks

(b)

1

Once you log in you will see Mode Selection View, shown below. You will review the operation of each mode, starting from the Interpreting Mode. You will notice there are icons for all modes, a button to teleport you to the IVY reception and a log-out button. This is the only point you will be able to access the teleport to IVY reception button, which you will need between each task.

Next select ‘me’ on your SL toolbar (top left corner) and select ’inventory’ (alternatively press ctrl + i). In your inventory there should be a folder labeled ‘IVY HUD Advanced MK2’. Open this folder and right click on the red box labeled ‘HUD Advanced MK2’ and select ‘Attach to HUD ⇒ Bottom Left’. You will then see the view shown below. In addition to having your SL login and password, you will have received IVY credentials from the experiment facilitators. Using your IVY credentials (not your SL account) log in.

(a)

The first task requires you to obtain the Heads-up Display (HUD) from the IVY reception, where your avatar appears when you log-in, and wear it on your viewport. To obtain the HUD, you need to left click on the HUD dispenser box shown below (a). A small window appears in your top right corner (b). This window will offer you the choice of keeping or discarding the box. Select ’Keep’.

Task 1. Logging into the IVY Heads-up Display (HUD)

Please read each task description before you start each task. Feel free to refer to the task description though, when you do start using the HUD.

Evaluation Tasks

Project IVY

Evaluation Tasks

1

2

Interpreting is required for the Usability Evaluation on 31/10/2012.

However, there is chance that other users are currently playing with the system and have already made some dialogue and monologue selection. Other dialogues and monologues taking place in the same location in the IVY island will be unavailable, until those scenarios are not used any more. In such a case you will see those titles with a dimmed font and a lock icon in front of them as shown below.

Select a dialogue or monologue based on its title and see further details. Notice how the bread-crumb displays your choices. You can use the ‘yellow’ links to navigate back or use the top link ’Back one level’ to return to previous views. You will also find there are arrows like these C B on the toolbar above the HUD. However, these will take you back to the mode selection so should be avoided.

Select the language combination of the scenario you want – any choice apart from the entries with Hebrew and Russian will do. Notice that the list expands and a scrollbar at the side allows you to see the options further in the list.

In the second Task you will explore the interpreting mode, where you can select various dialogues and monologues and practice interpreting1 them in various different locations. You first select the form of interpreting scenario you want to play, choosing between dialogues and monologues. Notice at the bottom of the HUD a bread-crumb of your choices appears.

Task 2. Interpreting Mode

Project IVY

Evaluation Tasks

3

The view below shows the player and the teleport window. Pressing ‘Teleport’ will teleport you to the location of your chosen scenario. Teleport before you start playing the selected dialogue/monologue. Also, in some locations you may need to look around the environment to locate the scenario actors. Find a suitable sitting place by right-clicking on a chair and choosing ‘sit here’ from the drop menu.

Upon selecting a specific, unused title you will see the view below, displaying information about this particular scenario. Information includes keywords, the scene where the dialogue or monologue takes place and a brief, describing the scenario. Press ’Launch Player’ – the view will change to the player and initiate the native SL teleport window.

You can nonetheless inspect those dialogues/monologues to see if you want to play them in the future by selecting them normally. You will not be able to launch the player however, as show below.

Project IVY

Evaluation Tasks

4

Go to next page

Press the button ‘Modes’, to return to the mode selection view and continue with task 3.

Use the player to listen to the chosen dialogue and monologe. Once you are done with a scenario you can return to the Interpreting mode menu by pressing the Menu button.

returns you to the Interpreting Mode menu.

jumps to the previous turn. This is full recursive so at turn 1 it will take you back to the last turn of the dialogue/monologue

jumps to the next turn

pauses playback of the current turn. Pressing play continues.

takes you back to the start of this particular turn

plays the current audio track/turn

Controlling the player is done as such:

The player gives you an audio-player like interface for controlling the dialogue or monologue, displaying the title of your chosen scenario, the domain and a count of total turns, including both interlocutors. As you press play a progress bar shows which interlocutor is speaking at a given time.

Project IVY

Evaluation Tasks

5

A series of teleport buttons to those locations are available. The breadcrumb showing your selections is, once again, shown at the bottom of the HUD. Once you select a scene the SL teleport window will pop-up and you can teleport to the selected location. Choose any location, bearing in mind you mind end in the same room with other evaluation participants – so do not be alarmed!

Once you select the Live Interaction Mode you will see the view below. Access the locations by selecting ’Live Mode Locations’

In the third Task you will explore the Live Interaction Mode, where you can teleport to the rooms of the IVY island, invite other members of the IVY group and perform some role play/interpreting. To access the Live Interaction Mode locations return to the Mode Selection view and select Live Interaction Mode.

Task 3. Live Mode

Project IVY

Evaluation Tasks

6

Go to next page

Press the button ‘Modes’, to return to the mode selection view and continue with task 4.

You can return to the Live Location Menu, Mode Menu or Log-off with the relative buttons.

This will pop-up the IVY group window and you can offer teleports to other members of the IVY group to your location by finding the name of the member on the list and clicking on the (i) icon next to their name. A window will then appear and you should click on the cog in the bottom right corner and select ’teleport’. You do not need to offer any teleports for this task.

In this view you can also access the IVY Group interface in SL by double clicking on the icon:

Project IVY

Evaluation Tasks

(b)

(b) HTML Panels

7

Follow the instructions on panels and take a look at the information displayed, regarding Interpreters, their practice, what they offer to their clients.

(a) Powerpoint Panels

Roam around the Exhibition hall and see the various sections. There are two exhibitions, one with Powerpoint panels and one with panels with HTML attached (light blue arrows on the floor). You can focus on the HTML panels using the zoom button(⊕ ), which appears on the toolbar above the panel, when you hover your cursor on the latter (b).

(a)

Return to the mode selection view and choose Exhibition mode (a). Once you select the mode the SL teleport window will pop-up and you can teleport to the Exhibition Hall (b).

Task 4. Exploration Mode

Project IVY

30 ⇒ 39

Male

40 ⇒ 49

Novice

Intermediate

Advanced

50 ⇒ 59

Novice

Intermediate

Novice

Intermediate

Novice

Intermediate

Advanced

Advanced

Advanced

Slightly

Moderately

Very

Expert

Expert

Expert

1

.......................................................................................

.......................................................................................

.......................................................................................

.......................................................................................

8. Explain your experience with Interpreters:

Not at all

7. How familiar are you with the use of professional Interpreters:

No experience

6. How would you rate your expertise with Computer Games?

No experience

5. How would you rate your expertise with Second Life?

No experience

Expert

Extremely

59 +

4. How would you rate your expertise with Virtual Worlds/Environments?

No experience

3. How would you rate your expertise with computers?

- 29

2. Age Group

Female

1. Gender

Please mark your responses with an ‘X’ in the appropriate box.

Demographic Questionnaire

Questionnaire ID: [

]

18. I needed to learn a lot of things before I could get going with this system

17. I felt very confident using the system

16. I found the system very cumbersome to use

15. I would imagine that most people would learn to use this system very quickly

14. I thought there was too much inconsistency in this system

13. I found the various functions in this system were well integrated

12. I think that I would need the support of a technical person to be able to use this system

11. I thought the system was easy to use

10. I found the system unnecessarily complex

9. I think that I would like to use this system frequently

2

1

1

1

1

1

1

1

1

1

2

2

2

2

2

2

2

2

2

3

3

3

3

3

3

3

3

3

4

4

4

4

4

4

4

4

4

5

5

5

5

5

5

5

5

5

5

4

1

3

agree

disagree 2

Strongly

Strongly

Please mark your responses with an ‘X’ in the appropriate box.

Usability Assessment – System Usability Scale (SUS)

Questionnaire ID: [

]

]

3

.......................................................................................

.......................................................................................

.......................................................................................

.......................................................................................

22. Our design model is to fuse web technologies with Second Life, and to provide a content management system for users. Is this an effective approach and where could it be applied?

.......................................................................................

.......................................................................................

.......................................................................................

.......................................................................................

21. Briefly describe any type of enhancement that you would like to see in IVY Virtual Environment:

.......................................................................................

.......................................................................................

.......................................................................................

.......................................................................................

20. Describe some negative aspects of the IVY Virtual Environment:

.......................................................................................

.......................................................................................

.......................................................................................

.......................................................................................

19. Describe some positive aspects of the IVY Virtual Environment:

Post-Experiment Questionnaire

Questionnaire ID: [

4

.......................................................................................

.......................................................................................

.......................................................................................

.......................................................................................

25. What is your opinion of the future of Virtual Learning Environments?:

.......................................................................................

.......................................................................................

.......................................................................................

.......................................................................................

24. What is your opinion of using an immersive three-dimensional Virtual Learning Environment, such as the IVY Virtual Environment:

.......................................................................................

.......................................................................................

.......................................................................................

.......................................................................................

23. Explain whether you think IVY Virtual Environment will be a useful tool for interpreters and their clients?

Questionnaire ID: [

]

ivy – interpreting in virtual reality

ivy – interpreting in virtual reality

Suggest Documents

ivy â interpreting in virtual reality

Trust in Virtual Reality

Augmented Reality & Virtual Reality in the Library

Virtual Reality and Augmented Reality in Plastic

Virtual Reality

VIRTUAL REALITY, AUGMENTED REALITY AND ROBOTICS IN ...

VIRTUAL REALITY

The virtual bat: echolocation in virtual reality

CARTOGRAPHIC GENERALIZATION IN VIRTUAL REALITY

virtual setting for training in interpreting

Experimental Application of Virtual Reality - The Virtual Reality ...

Virtual-Reality Therapy

Virtual Reality - arXiv

Virtual Reality

Virtual reality camera

Towards Virtual Reality Games

Virtual reality - Alitalia

Immersive Virtual Reality

virtual reality - Ipsos

VIRTUAL REALITY SUMMER!

virtual reality? - IAB

VIRTUAL REALITY SUMMER!

Virtual reality camera

Virtual reality - Alitalia

ivy – interpreting in virtual reality