Designing and Evaluating Online Language Learning Materials - ijssh

International Journal of Social Science and Humanity, Vol. 1, No. 4, November 2011

Designing and Evaluating Online Language Learning Materials for Higher Education Ana Sevilla-Pavón, Antonio Martínez-Sáez, and Ana Gimeno-Sanz. 

all, designers should test the accessibility, user-friendliness and validity of the materials. Another important aspect that should be looked out relates to the multimedia features of the materials, which should be present and varied enough. As for the learning objects, they should be clear and well organised, while the contents should be relevant. Other aspects such as the clear presentation, the organization and order of all the learning objects should be borne in mind. In addition, the designer should reflect on the pedagogical use and on the orientation of the contents. Finally, the use of the materials in different training and teaching processes should be taken into consideration [4]. The aim of this paper is to describe the process of design and evaluation of the InGenio FCE Online Course and Tester. These are two language learning CALL materials for students of English, developed by the CAMILLE Research Group at the Universidad Politécnica de Valencia (UPV) in Spain. These materials belong to the InGenio system, a Learning Management System (LMS) which comprises a web-delivered language-independent authoring tool capable of managing databases on a remote server.

Abstract—Nowadays, finding, choosing and developing or adapting materials is a very important component of education as well as a key element for the success of the whole learning process. A language teacher is also a materials developer and the materials development process should aim at ensuring that local needs and wants do not dictate decisions and that important learning principles are not forgotten. In the case of materials for Computer-Assisted Language Learning (CALL), a very thorough and pedagogically-sound set of guidelines should be followed prior to the design of the materials. Following that, a careful evaluation should take place so as to guarantee that those guidelines have been followed and that the materials are therefore valid, efficient and effective. This is especially important in the case of in-house CALL materials development. This paper describes the basis for the evaluation and validation of the InGenio FCE Online Course and Tester. These are two language learning in-house CALL materials for students of English, developed by the CAMILLE Research Group at the Universidad Politécnica de Valencia (UPV) in Spain. A large part of the focus of this paper will be on the aspects concerning the evaluation of the two previously mentioned materials according to well-established frameworks and guidelines from the most relevant literature. Index Terms—CALL, in-house development, design, evaluation.

materials,

software

II. THE INGENIO ONLINE SYSTEM AND WORKING CONTEXT The InGenio system allows teachers from all over the world to design and publish materials to suit their students‟ particular needs. For the creation and implementation of the InGenio system, the template approach to software authoring [5] was taken as a basis, and the relevant aspects concerning material design and evaluation were borne in mind. The template approach allows designers and content authors to use and adapt the predefined templates provided, which integrate video, graphics, audio and text. To create the materials FCE Online Course and Tester, authors have used the InGenio 15 exercise templates and also two additional templates which have enabled the creation and introduction of reference sections and glossaries. These templates were designed bearing in mind that they must ensure variety, coherence and consistency when an author is engaged in the process of creating multimedia activities. The professionals who have developed InGenio have also followed the principle that the templates should be as open and flexible as possible in order to encourage authors to design creative tasks rather than restricting and confining their pedagogical input. The templates that are available allow courseware writers to design exercises ranging from learner observation to learner production in order to practice all four language skills (reading and listening comprehension, and writing and speaking production). These templates include traditional e-learning activities such as multiple choice exercises,

I. INTRODUCTION Nowadays, finding, choosing and developing or adapting materials is a very important component of education as well as a key element for the success of scientific and teaching activities. When it comes to language teaching and learning, according to Tomlinson [1], “every language teacher is a materials developer”, and materials development processes should be “based on universal principles” [2] so as to ensure that local needs and wants do not dictate decisions and that important learning principles are not forgotten. The process of Computer-Assisted Language Learning (CALL) materials design usually takes place in several steps in which it is important to make the right decisions on different task-related aspects such as topics and actions, participants and mode [3]. Once the materials have been created, different quality-related aspects should be borne in mind [4]. First of

 Manuscript received September 16, 2011; revised September 29, 2011. This work was supported in part by the Valencian Regional Government in Spain (Generalitat Valenciana) under Grant FPI (Formación del Personal Investigador). Authors are with the CAMILLE Research Group, Deparment of Applied Linguistics, Universidad Politécnica de Valencia, Camino de Vera, S/N, 46022 Valencia, Spain (e-mail: [email protected], [email protected], [email protected] ).

269

International Journal of Social Science and Humanity, Vol. 1, No. 4, November 2011

franca in the current globalised world and that learning is not only something optional but something students must do if they want to explore other working options out of their countries of origin. In the particular case of the Universitat Politècnica de València, higher education should not only be seen as a context for teaching specific contents of a particular discipline or degree; it should also be considered as an adequate space for teaching and learning languages as a complement to technical and scientific degrees. Bearing in mind this new European context, the UPV set a new linguistic requirement, i.e. all students must achieve an upper-intermediate level of a foreign language before graduating. The First Certificate in English exam (FCE) is a means to evaluate the linguistic skills of the students who wish to certify their upper-intermediate level of English. This official examination compiles the four basic skills; reading comprehension, written production, listening comprehension and oral production. These skills are distributed in five exam sections or „Papers‟; Reading, Writing, Use of English, Listening and Speaking. The „Use of English‟ section is the one devoted to evaluating specific grammar and vocabulary skills. Moreover, the FCE Exam includes the evaluation of oral skills and their weight is practically equivalent to that of the reading comprehension and writing sections. This fact shows the real need of learning a second language, which is to foster communication among learners and speakers of English, leaving behind the times in which learning a second language was mainly focused on developing the students‟ grammar and reading comprehension skills. A new version of the FCE exam is available since December 2008 and the Computer based version was launched in 2010. Its new structure, from 2008, consists of five „Papers‟.

gap-filling exercises, reordering exercises, open input exercises, association exercises and vocabulary building. The templates can be used to design exercises and activities deriving from a number of teaching methodologies. Some examples of exercise types and their corresponding template are the following: fact finding (observation), ordering and sorting (reordering), comparing (matching), problem-solving (open input without sound, open input with sound, monitoring comprehension, multiple-choice questions, gap-filling exercises, reinforcing new vocabulary, vocabulary building, clickable image, word search puzzles and hanged-man exercises).

Fig. 1. Listening exercise within the InGenio FCE Tester section

The “content manager” featured in InGenio enables subject specialists to create a database from which to share and select materials by organising the multimedia components and materials according to a number of specifications (e.g. language, level, skill, target group, etc.), thus creating a pool of multimedia exercises and resources. The authoring tool automatically converts the contents into learner-ready materials delivered via the InGenio web-based Learning Environment. The system also incorporates a student assessment utility that allows tutors to supervise student scores, written input and general progress [6]. Furthermore, InGenio also includes a “translation utility” enabling existing courseware to be adapted into any number of source languages and a utility to translate the graphical user interface into any number of languages. Perhaps more importantly, though, is the fact that the system allows teachers, from any given point in the world, to create, adapt and publish materials to suit their students‟ particular needs. The context in which the authors of the FCE Online Course and Tester work cannot be understood without bearing in mind the higher education areas in most European Union Member States, which have experienced a number of important changes due to their willingness to comply with the Bologna process. This process cannot only be seen as a reform of some of the most general aspects of higher education; it must also be seen as a process with particular and specific implications with regard to language learning, teaching and evaluation, the main goal being to develop a common and cohesive higher education area across the whole European territory. International mobility in the economic and education field alerts about the need to prepare European teachers and students for a context of disappearance of boundaries in the European Union; a context of dynamic marketplace and research collaboration projects among education institutions with many national languages and thus not sharing a common native language for communication. English as a lingua franca in the European territory is frequently seen as the alternative solution. This reflection should be added to the fact that English is also the lingua

III. THE DESIGN AND EVALUATION OF ONLINE LANGUAGE LEARNING MATERIALS Chapelle and Douglas [7] highlight the important role of student assessment, as students usually worry a lot about what is expected from them and try to fulfil the teacher‟s expectations as far as the tasks and activities they have to complete are concerned, especially if their partial or final grades depend on their performance. Because of this, it is very important to provide students with information on what would be assessed and how, as well as on the type of tasks they will be asked to do in such a way that they know how a correct or incorrect way of solving problems and completing activities will affect their grades. A good way of providing this information is through an interface specifically devoted to the students‟ grades and individual reports such as the ones shown below, taken from the InGenio FCE Course and Tester:

Fig. 2. InGenio student progress report.

270

International Journal of Social Science and Humanity, Vol. 1, No. 4, November 2011

specific decisions. 3) To design the contents included by bearing in mind a well-defined target group of users. 4) To obtain practical results. 5) To make sure that not only the evaluation process but also the evaluation product provide useful data and information. 6) To check whether something works or not as it had been planned. Another important procedure within the evaluative framework used is known as third-party evaluations, which present two important challenges to the evaluator [9]: the need to make right decisions as far as appropriate evaluation criteria are concerned and the need to know the programmes or materials to be evaluated as well as their different contents and resources, their options of use in different contexts and with different students, etc. Although the approaches can differ from author to author, according to Burston [13], there are some requirements that have to be fulfilled by any type of programme. These are: pedagogical validity, curricular adaptability, efficiency, effectiveness and pedagogical innovation. Once content writers or authors have checked that those requirements are fulfilled, they may look at the following categories [12]: 1) Technical characteristics: reliability, ease of use, etc. 2) Activities (procedure): design of activities and the different typologies. 3) Importance of suiting needs and methodologies followed by teachers (approach): linguistic and learning presuppositions. 4) Importance of suiting students‟ needs and demands (design).

According to Dowsing [8], designers should ensure that test management, which is usually very time-consuming, is kept to a minimum. He also advises that the interface should be general in such a way that, if necessary, it could be used by a large number of institutions in a greater variety of contexts. When using CALL materials, another aspect to be taken into account is whether the interface will be visible or not, which is a decision the designers have to make. This decision will depend on whether the fact that the interface is visible or invisible threatens to cause variations in the students‟ performance, which in turn would depend on the kind of tasks to be completed or in what is being assessed [7]. Although the processes of assessment and evaluation usually take place once the design process has been completed, both processes are often complementary and so interconnected that they may even overlap [9]. The main reasons why designers and academic staff rely on evaluation are that they need to “assess students‟ attitudes and perceptions in a learning environment where technology is used, make sure that technology is working the way they had predicted and test whether methodologies and strategies are viable and effective, with a view to improving teaching practices” [9]. There is a varied range of ways to collect data when conducting the evaluation process, the most basic ones being the checklists and surveys, whereas some more complex methodologies comprise multifaceted and longitudinal studies. In the case of the InGenio FCE Course and Tester, the designers have chosen to use multiple kinds of data in such a way that quantitative and qualitative approaches are combined. According to several authors, this combination can be a good means of increasing the validity of a CALL study [10] [11]. As for already-existing evaluation frameworks to be followed, Hubbard [12] could be a very good starting point, as this author provides some of the key principles that delimit formative and summative/accumulative evaluation. These principles are very useful when evaluating teaching projects and the materials developed as a result of them, such as in the case of the InGenio Online FCE Course and Tester. The formative evaluation has taken place during the process of development of the materials in an attempt to obtain as much information and improvements as possible from the process itself; whereas the summative evaluation will be put into practice once the whole project is finished and the materials which have been developed -the InGenio Online FCE Course and Tester- are fully implemented and used by students at the UPV. The questions arising from the formative modality of evaluation have been set out by content designers and writers in such a way that the feedback emitted and the results obtained have a positive impact on learning. The information gathered through the different evaluative approaches is very useful for content writers and designers, who make use of it during the process of adaptation and improvement of the materials. The general aims of the evaluation studies concerning CALL materials such as the InGenio materials are listed below [9]: 1) To prove that the materials which have been developed are useful and that they respond positively to their specific purposes. 2) To bring them about by taking into consideration

Fig. 3. Pre-questionnaire, integrated into the InGenio platform.

According to Hubbard [12], it is advisable to make use of a methodological framework for CALL evaluation based on previous methodologies and on other existing frameworks for the evaluation of the language teaching process. In this way, one of those older frameworks that are still valid and that could therefore be used is the one provided by Richards and Rodgers [13]. These authors proposed a flexible framework which presented an open structure which had the additional advantage of not being limited by a particular language or by a specific teaching or learning process. Chapelle [14] provided another important framework based on theory and focused on tasks, which provided a new view based on an interactionist perspective [9]. This framework comprises the following principles: 1) The evaluation of CALL is determined by a specific situation. 2) Two main perspectives when dealing with CALL evaluation are a critical analysis of software and planned 271

International Journal of Social Science and Humanity, Vol. 1, No. 4, November 2011

activities as well as an empirical analysis carried out by students. 3) Criteria on the quality of the activities developed should be based on theory and research on SLA (Second Language Acquisition). 4) Criteria should be applied in compliance with the aims of the activities. 5) Learning potential should be the main criterion when evaluating CALL. According to Levy and Stockwell [9], the first of the abovementioned principles is related to the fact that the results obtained from a study should not be seen as indicators of proven effectiveness, as the study -as well as the parameters which are involved in the whole process- is much more complex. Concerning the second principle, Chapelle [14] mentions two interrelated levels, starting from both levels of critical analysis: CALL software and CALL activities planned by a teacher. In this case, a programme is evaluated in a non-contextualised way, the abovementioned checklists being one of the possible ways to carry out this analysis and evaluation. The second level would be based on an empirical analysis phase by taking into consideration the students‟ performance while bearing in mind the context provided by the teacher who plans and organises the implementation of a particular language learning programme. A third level based on an empirical and contextualised analysis involves gathering data related to the use that students make of a specific programme while trying to obtain evidence of negotiation of meaning during the interaction process. Bearing this in mind, Chapelle [14] established a list of criteria to be followed when designing CALL tasks:

would not have taken part in the whole design and development process.

IV. CONCLUSION Given the great amount of online language learning multimedia materials that are nowadays available for both teachers and language learners and their increasingly demanding expectations and requirements with respect to educational technologies, the assessment of those learning objects and materials is a key factor in making the most out of them and in assuring variety and adaptation to the learners' characteristics. The main guidelines for the evaluation and assessment have been presented throughout this paper. The theory has already been put into practice for the design and development processes of the materials in the InGenio Online FCE Course and Tester after careful planning in the form of a framework to be used when conducting the final evaluation stages of these materials. Since the phases devoted to the design, edition and evaluation of contents have already been completed, and so have the activities and tasks, the materials are ready to be used by students at our institution. At this point, the evaluation and validation phases will take place bearing in mind all the parameters listed throughout this paper. REFERENCES [1]

[2]

TABLE I: CRITERIA FOR CALL APPROPRIATENESS (ADAPTED FROM CHAPELLE, 2001).

[3] Language Learning potential Learner fit

Meaning focus

Authenticity

Positive impact

Practicality

The degree of opportunity present for beneficial focus on form The amount of opportunities for engagement with language under appropriate conditions given learner characteristics The extent to which the learner‟s attention is directed toward the meaning of the language The degree of correspondence between the CALL activity and target-language activities of interest to learners out of the classroom The positive effects of the CALL activity on those who participate in it The adequacy of resources to support the use of the CALL activity

[4]

[5]

[6]

[7] [8]

[9]

[10]

The last option, based on a third-party evaluation, could be very positive if conducted by external evaluators with a wide experience in the design and use of CALL materials and programmes such as the FCE Online Course and Tester. The data gathered would provide content designers and writers with useful information about the strengths and limitations of the materials created and developed. However, there is also a risk that these external evaluators do not take into consideration the whole spectrum of possibilities, as they

[11]

[12]

[13]

272

B. Tomlinson. Introduction: Are Materials Developing? In: B. Tomlinson (Ed.) Developing Materials for Language Teaching. London and New York: Continuum. 2003. B. Tomlinson. Principles and procedures for self-access materials. In Studies in Self-Access Learning Journal, 2010, 1(2), pp. 72-86. C. A. Chapelle. English language learning and technology: Lectures on applied linguistics in the age of information and communication technology. Amsterdam: John Benjamins Publishing. 2003. E. Barberà, T. Mauri, and J. Onrubia (Eds.) Cómo valorar la calidad de la enseñanza basada en las TIC – Pautas e instrumentos de análisis. Barcelona: Graó. 2008. A. Gimeno-Sanz. CALL Software Design and Implementation: the template approach. Valencia: Servicio de Publicaciones de la Universidad Politécnica de Valencia. 2002. A. Gimeno-Sanz, A. Martí nez-Sáez, A. Sevilla-Pavón and J. M. de Siqueria Rocha. Student assessment in the InGenio online authoring system: results taken from Intermediate online English. In Durán Martí nez, R. and Sánchez-Reyes Peñamarí a, S. (Eds.), Linguistic tools for teachers of English: towards a bilingual education. Salamanca: Ediciones Universidad de Salamanca. 2010, pp. 51-78. C. A. Chapelle and D. Douglas. Assessing language through computer technology. Cambridge: Cambridge University Press. 2006. R. D. Dowsing. The computer-assisted assessment of practical IT skills. In S. Brown, P. Race y J. Bull (Eds.), Computer-Assisted Assessment in Higher Education. Kogan Page Ltd, London, 1999, pp. 131-138. M. Levy and G. Stockwell. CALL Dimensions: Options and Issues in Computer-Assisted Language Learning. New Jersey: Erlbaum Associates. 2006. W. Harless, M. Zier and R. Duncan. Virtual Dialogues with Native Speakers: The Evaluation of an Interactive Multimedia Method. In CALICO Journal, 1999, 16 (3), pp. 313-337. R. Trinder. Conceptualisation and development of multimedia courseware in a tertiary educational context: juxtaposing approach, content and technology considerations. In ReCALL Journal, 2003, 15, pp. 79-93. P. Hubbard. Elements of CALL Methodology: Development, Evaluation, and Implementation. In Pennington, M. (ed.) The Power of CALL. Houston: Athelstan. 1996. J. C. Richards and T. S. Rodgers. Approaches and Methods in Language Teaching. Cambridge: Cambridge University Press. 1986.

International Journal of Social Science and Humanity, Vol. 1, No. 4, November 2011 [14] C. A. Chapelle. Computer applications in second language acquisition: Foundations for teaching, testing, and research. Cambridge University Press. 2001. Ana Sevilla-Pavón, born in Albacete, Spain. PhD Candidate within the "Languages and Technology" PhD programme, Department of Applied Linguistics, Universitat Politècnica de València, Spain. 2011: Master of Research Degree (Diploma of Advanced Studies), Department of Applied Linguistics, Universitat Politècnica de València, Spain. 2007: Degree in Modern Languages (English and French), Faculty of Philology, Universidad de Valencia (Spain). She is holder of a four-year research grant awarded by the Valencian Regional Government (Generalitat Valenciana). She is part of the CAMILLE Research Group, led by Dr. Ana Gimeno at the Department of Applied Linguistics at the Universitat Politècnica de València where she is developing an online assessment program for the Cambridge First Certificate in English Examination (FCE). She has participated in a number of national and international conferences on applied linguistics and in research projects such as PAULEX, co-financed by the Spanish Ministry of Education, entitled to develop a computer system capable of creating, managing and delivering the foreign language exams which are compulsory in the official Spanish University Entry Examinations. She has also written, participated in, and published several articles and reviews. Among those, she publised the following book chapters: "Modalities of Assessment in Online Preparatory Materials", in Second Language Teaching and Learning with Technology (2011), Sylvie Thouësny and Linda Bradley (Eds.), Dublin: Research-publishing.net; “Student Assessment in the InGenio Online Authoring System”, in Linguistic tools for teachers of English - towards a bilingual education (2010), Sonsoles Sánchez-Reyes Peñamarí a and Ramiro Durán Martínez (Eds.), Salamanca: Aquilafuente. She also published the paper: “El uso de materiales didácticos online destinados a la preparación del nuevo examen informatizado CBFCE” (2010), in Revista Latinoamericana de Tecnología Educativa (RELATEC), 2, vol. 2. She is particularly interested in applied linguistics, educational technology and computer assisted language teaching and testing. Ms. Ana Sevilla Pavón is member of the following professional societies: the Computer Assisted Language Learning Consortium (CALICO), the European Association of Computer-Assisted Language Learning (EuroCALL), the Spanish Association of Applied Linguistics (AESLA) and the International Association of Computer Science and Information Technology (IACSIT). She obtained the "Best research article" award in the 1st International Workshop on Technological Innovation for Specialized Linguistic Domains (TISLID10) for the paper “Modalities of Assessment in Online Language Learning Materials” in October 2010 from the Universidad Nacional a Distancia (UNED), Madrid, Spain. She was member of the Organising Committee of the EuroCALL Conference 2009 and is currently member of the Scientific Committee of the "4ºColóquio Internacional Brasil x Espanha".

Master of Research Degree (Diploma of Advanced Studies), Department of Applied Linguistics, Universitat Politècnica de València, Spain. 2010: Masters in Secondary Education Teaching (Specialty: English) at the Universitat de València. He is currently developing an online preparatory course for the Cambridge First Certificate in English Examination (FCE). He has participated in a number of national and international conferences on applied linguistics and in research projects such as PAULEX, co-financed by the Spanish Ministry of Education, entitled to develop a computer system capable of creating, managing and delivering the foreign language tests which are compulsory in the official Spanish University Entry Examination. He has also written, participated in, and published several articles and reviews as well as several book chapters. Antonio Martí nez-Sáez is member of the following professional societies: the Computer Assisted Language Learning Consortium (CALICO), the European Association of Computer-Assisted Language Learning (EuroCALL), the Spanish Association of Applied Linguistics (AESLA) and the International Association of Computer Science and Information Technology (IACSIT). He obtained the "Best research article" award in the 1st International Workshop on Technological Innovation for Specialized Linguistic Domains (TISLID10) for the paper “Modalities of Assessment in Online Language Learning Materials” in October 2010 from the Universidad Nacional a Distancia (UNED), Madrid, Spain. He was member of the Organising Committee of the EuroCALL Conference 2009 and is currently member of the Scientific Committee of the "4ºColóquio Internacional Brasil x Espanha".

Dr. Ana Gimeno-Sanz, born in Leicestershire, United Kingdom. Member of the teaching and research staff at the Department of Applied Linguistics of the Universidad Universitat Politècnica de València, Spain. Senior Lecturer in English since 1985. Head of the CAMILLE Research group at the UPV. MA and PhD in English Philology (Universitat de València, Spain) and MA in Journalism (Universitat de València - Levante EMV, Spain). 2000-09: Director of the RandDandI Linguistic Assistance Office at UPV for nine years. Coordinator for UPV of the English language exam for the Spanish national university entry exams. Coordinator for UPV of the English language exam for the over-25-years-of-age university entry exams. Former President of the European Association for Computer Assisted Language Learning (EUROCALL). Areas of research: language learning and teaching, more specifically in the fields of English for Specific Purposes (ESP) and Computer Assisted Language Learning (CALL). She has published a number of research papers on language learning and teaching, more specifically in the fields of English for Specific Purposes (ESP) and Computer Assisted Language Learning (CALL). She has published CALL Software Design and Implementation: the template approach (SP-UPV, 2002), and is co-author of several books for learners of English for engineering purposes. Project Manager of several funded multimedia CALL RandD projects which have led to the publication of a number of language courses on CD-ROM. Participation in other EU-funded projects include: MultiConcord, Airline Talk, City Talk, ICT4LT, Hello Net, Welcome, TNP II and III in Languages; ENLU, LANQUA, etc. Project Manager of Proyecto InGenio, funded by UPV, consisting of an online dedicated CALL authoring tool, content manager and learning environment to enable the publication of interactive courseware for foreign language learning on the web. Courses published with this tool include: Intermediate Online English, Valencià Interactiu - Grau Mitjà. Co-ordinator of the LINGUA-funded CALL@CandS Project (2004-2007), aiming to design and deliver online A1 and A2 level courses for learners of Czech and Slovak. Co-ordinator of the EURODIS Project (2010-2011), aimed at disseminating EU-funded projects.

Antonio Martínez-Sáez, born in Albacete, Spain. Currently: PhD Candidate within the "Languages and Technology" PhD programme, Department of Applied Linguistics, Universitat Politècnica de València, Spain. From 2008, researcher - holder of a four year research grant awarded by the Valencian Regional Government (Generalitat Valenciana) under a competitive selection process. Member of the CAMILLE (Computer Assisted Multimedia Interactive Language Learning Environment) Research and Development Group, led by Dr. Ana Gimeno at the Department of Applied Linguistics at the Universidad Politécnica de Valencia (UPV). From 2010: University teacher at the UPV. 2007: Degree in English studies and in translation and interpreting studies (Spanish, English, and French), Universitat de València (Spain). 2010:

273