SITE 2016 - Savannah, GA, United States, March 21-26, 2016
Improving student performance and language assessment through ICT use Dr.Cristina Perez-Guillot Dr.Julia Zabala Delgado Universitat Politecnica de Valencia, Spain Spain
[email protected] [email protected] The goal of our study was to analyse the effects of the use of computer software for placement testing at the Language Centre of the Universitat Politècnica de València (UPV) Spain. We included the analysis of the opinions of all those involved in the process: students, teachers and administrative personnel. We started our research by analysing the uses of the specific software PARIS.in order to identify the main changes in the process of placement testing as a result of implementing IT tools, we compared the process, the results and the costs to those of previous paper based tests. We analyzed students opinions by means of a questionnaire administered to approximately 2500 students, and continued by collecting their views by means of a semi-structured interview. All data collected were quantitatively and qualitatively analysed. The uses of IT for testing in languge Centres clearly improve the process of placing students at the appropriate levels and optimize the resourses.
Introduction The creation of the European Higher Education Area (EHEA) has provided an evidence to the need for the development of language policies at universities. University-based language centres have made a great contribution to the development of language learning since their emergence and have played a major role in the development and implementation of language policies and language education as well as have been vital to meet the challenges of the Bologna Process and the emerging multilingual Europe. The Language Centre at the Universitat Politècnica de València (UPV) plays a key role in the selection and training process of students opting for mobility grants and has designed a computer-based language test that can be administered to a large number of students simultaneously, with immediate release of results, contributing to the internationalisation of the UPV and its integration into the EHEA. The goal of our study was to analyse the effects of the use of computer software for placement testing at the Language Centre of the Universitat Politècnica de València (UPV) Spain. Since computer based testing has been used at the university for some time and the software used had been analysed in several research papers, our goal was to focus on the changes in the process, advantages and disadvantages. Furthermore, our goal included the analysis of the opinions of all those involved in the process: students, teachers and administrative personnel. Context
-1617-
SITE 2016 - Savannah, GA, United States, March 21-26, 2016
The UPV comprises three campuses on three different locations: Valencia, Gandía on the coast at about one hour away from Valencia and Alcoy, inland, also South of Valencia and at a distance of about an hour and a half. There are 13 Schools and Faculties , 42 Departments, 30,000 Students ans 2,660 Faculty and research personnel. The Universitat Politècnica de València Mission is to provide integral education combined with ethical principles, to contribute to the economic, social and cultural development of the Spanish and Valencian society and to develop an institutional model characterised by its values of excellence, internationalization, solidarity and efficiency. Internationalization is one of the UPVs priorities. Our exchange figures in the academic year 2013/14 where 1,574 outgoing students and 2,198 incoming, most of them thought the Erasmus+ Programme. The Language Centre of the Universidad Politécnica de Valencia is a University Centre which one of its main lines of action is to provide supplementary training in foreign languages so as to facilitate the integration of its members in the European Higher Education Area, which is an essential part of the philosophy of internationalisation characterising the UPV,
The implementation of the Common European Framework of Reference for Languages (CEFR; Council of Europe 2001) has provided a common basis for the development of language programmes and exams, by identifying levels of language proficiency that allow the assessment of students’ progress and the measurement of language skills, thus enabling a comparison between systems. To implement this new system, language centres have gradually had to adapt the design and content of their courses to the new definition of levels of language proficiency according to the particular conditions of their learning environment. Despite the differences in the manner in which language centres have adapted the CEFR levels, a problem common to all language centres is how to quickly and effectively determine the students’ knowledge of the language before their enrolment on the courses, especially at the beginning of the academic term. At the Language Centre (CDL) of the Universidad Politécnica de Valencia (UPV) we have developed a computer-based placement test that allows the assessment of a large number of students in a reliable and efficient way. One of the most important issues in the development of a test is determining the purpose for which it is designed. “The most important consideration in designing a test is the use for which it is intended so the most important quality criterion is its usefulness” (Bachman and Palmer 1997: 17). In fact, it is a two-way process in the sense that the same type of test can be used for different objectives, but also its purpose may determine the type of test to be used. Depending on their purpose, tests can be used either as a source of information for decision-making in an educational
-1618-
SITE 2016 - Savannah, GA, United States, March 21-26, 2016
setting or as a tool for investigating specific aspects of language (Bachman 1990: 52). Placement and diagnostic tests fall within the first group and their aim is “identifying the appropriate instructional level or specific areas in which instruction is needed” (Bachman 1990: 70). The aspects taken into account when designing the placement test were that the test was not to exceed one hour and that the results should be almost immediately available, since most students are unwilling to spend an excessive amount of time undergoing testing or to return another day for their results. Also, from an organizational standpoint, it seemed more advantageous to design a flexible and effective test which foregoes the inclusion of more complex activities that extend the length of the test, since there is a higher administrative and management workload at the beginning of each academic year, and time and human resource management must be optimised accordingly. Therefore, the best solution, taking into account that there are more than 600 students each semester, is to use a computer-based test, given its flexibility of use and the rapid analysis of results.
Methodology This study was carried out in the Language Centre of the Universitat Politècnica de València, using the centre's resources to research and develop the alternatives proposed. In the first phase of the study the initial manner in which language competence tests were developed in the center was examined , we knew that the test had been developed on the one hand by a group of experts comprising teachers with wide experience in the CDL, who were familiar with the organization and contents of the courses and were also members of official certificate examination boards with an extensive background in testing; and on the other by technicians who have worked extensively with the CDL and are aware of its needs. Among the advantages of a custom-designed test are its greater flexibility, the immediate release of results, lower stress, the management of large volumes of students, the optimisation of time and human resources, lower costs and adaptation to the specific requirements of the centre (Cubeta 2008). Our main purpose was to detect the existing problems. Strategies for improvement were defined and a number of working hypotheses considered for computerized evaluation environments. The improvements and hypotheses are related to topics as diverse as time saved by teaching staff, system security, the reliability of the results, the increase in multimedia capabilities, the benefits of monitoring and the need for proper feedback.
-1619-
SITE 2016 - Savannah, GA, United States, March 21-26, 2016
We started our research by analysing the uses of the specific software PARIS, developed to meet the specified requirements for placement testing at the Language Centre of the UPV. In order to identify the main changes in the process of placement testing as a result of implementing IT tools, we compared the process, the results and the costs to those of previous paper based tests. The tests for which the software was developed are used to provide access to language courses offered by the center, scholarships for academic exchange students (ERASMUS), and high academic performance (ARA) subjects for which English is the medium of instruction.
The comparison was performed for placement testing in three different languages, English, German and French. The reason for choosing to analyse its use for three languages responded to the different needs of each of the language units, in terms of number of teachers, number of students and peculiarities of the placement test. Our next step, as users were requested to take a satisfaction survey , consisted in analyzing students opinions by means of a questionnaire administered to approximately 2500 students; of which only 20% completed the questions. The questionnaire was composed of 11 questions and designed to be administered online through a web page designed with asp (active server pages) technology. The number of students called for the use of a questionnaire to allow for relatively easy processing of data collected. Since our goals included not only the analysis of the process and the opinions of students, but also analyzing the experience of teachers responsible for the test and their opinion on its advantages and disadvantages, we continued by collecting their views by means of a semi-structured interview. A semi-structured interview was chosen since it allowed for the exploration of emergent paths of research unforeseen by the researchers and the number of teachers involved made the analysis of the data feasible. Lastly, we aimed to study the opinion of administrative personnel working for the Language Centre and in charge of administrative procedures during placement testing sessions. Again and due to the relatively limited number of people involved, a semistructure interview was used to compare the objective analysis of the procedures observed to their experience in managing their administration. All data collected, results of the questionnaire and results of the interviews were quantitatively (in the case of the questionnaire) and qualitatively analysed (for semistructured interviews) Conclusions The uses of IT for testing in a University language centre clearly improve the process of placing students at the appropriate levels in terms of practicality and optimization of resources. Furthermore, our study concluded that the changes were perceived as
-1620-
SITE 2016 - Savannah, GA, United States, March 21-26, 2016
positive not only by students but also by the teachers responsible for the language units, independently from their computer literacy since the results obtained were easy to read and reliable. The possibility of obtaining facility values, discrimination indexes and realibility coefficients in real time proved to be an asset in tests with objective marking, clearly reducing time for test analysis and facilitating the improvement of items. The functionalities implemented to avoid student malpractice reduce costs as less invigilators are needed. Future uses of the software are many, as many of its possibilities, such as using web links or embedded videos have not been explore to date. In any case, they certainly seem promising and as far as our results indicate, worth to explore. Keywords: Computerised Assessment Language Competence Level Tests Multimedia Capabilities
-1621-
