Information for Score Users, Teachers and Learners - ETS

8 downloads 78 Views 2MB Size Report
research on the TOEFL iBT available to all test score users in a user-friendly .... well as English learners who prepare for the TOEFL test. The primary use of ...
Insight TOEFL iBT ® Research

Series I, Volume 5

Information for Score Users, Teachers and Learners

ETS — Listening. Learning. Leading.®

TOEFL iBT ® Research • Series 1, Volume 5

Insight

Foreword

Preface

We are very excited to announce the TOEFL iBT® Research Insight Series, a bimonthly publication to make important research on the TOEFL iBT available to all test score users in a user-friendly format.

Since the 1970’s, the TOEFL test has had a rigorous, productive and far-ranging research program. But why should test score users care about the research base for a test? In short, because it is only through a rigorous program of research that a testing company can demonstrate its forward-looking vision and substantiate claims about what test takers know or can do based on their test scores. This is why ETS has made the establishment of a strong research base a consistent feature of the evolution of the TOEFL test.

The TOEFL iBT test is the most widely accepted English language assessment, used for admissions purposes in more than 130 countries including the United Kingdom, Canada, Australia, New Zealand and the United States. Since its initial launch in 1964, the TOEFL® test has undergone several major revisions motivated by advances in theories of language ability and changes in English teaching practices. The most recent revision, the TOEFL iBT test, was launched in 2005. It contains a number of innovative design features, including the use of integrated tasks that engage multiple language skills to simulate language use in academic settings, and the use of test materials that reflect the reading and listening demands of real-world academic environments. At ETS we understand that you use TOEFL iBT test scores to help make important decisions about your students, and we would like to keep you up to date about the research results that assure the quality of these scores. Through the TOEFL iBT Research Insight Series we wish to both communicate to the institutions and English teachers who use the TOEFL iBT test scores the strong research and development base that underlies the TOEFL iBT test, and demonstrate our strong, continued commitment to research. We hope you will find this series relevant, informative and useful. We welcome your comments and suggestions about how to make it a better resource for you. Ida Lawrence Senior Vice President, Research & Development Division, Educational Testing Service

The TOEFL test is developed and supported by a worldclass team of test developers, educational measurement specialists, statisticians and researchers. Our test developers have advanced degrees in such fields as English, language education and linguistics. They also possess extensive international experience, having taught English in Africa, Asia, Europe, North America and South America. Our research, measurement and statistics team includes some of the world’s most distinguished scientists and internationally recognized leaders in diverse areas such as test validity, language learning and testing, and educational measurement and statistics. To date, more than 150 peer-reviewed TOEFL research reports, technical reports and monographs have been published by ETS, many of which have also appeared in academic journals and book volumes. In addition to the 20-30 TOEFL-related research projects conducted by ETS Research & Development staff each year, the TOEFL Committee of Examiners (COE), comprised of language learning and testing experts from the academic community, funds an annual program of TOEFL research by external researchers from all over the world, including preeminent researchers from Australia, the UK, the US, Canada and Japan. In Series One of the TOEFL iBT Research Insight Series, we provide a comprehensive account of the essential concepts, procedures and research results that assure the quality of scores on the TOEFL iBT test. The six issues in this Series will cover the following topics:

oooooooo 1

Information for Score Users, Teachers and Learners

Issue 1: TOEFL® iBT Test Framework and Development The TOEFL iBT test is described along with the processes used to develop test questions and forms. These processes include rigorous review of test materials, with special attention to fairness concerns. Item pretesting, try outs and scoring procedures are also detailed.

Issue 2: TOEFL Research The TOEFL® Program has supported rigorous research to maintain and improve test quality. Over 150 reports and monographs are catalogued on the TOEFL website. A brief overview of some recent research on fairness and automated scoring is presented here.

Issue 3: Reliability and Comparability of Test Scores Given that hundreds of thousands of test takers take the TOEFL iBT test each year, many different test forms are developed and administered. Procedures to achieve score comparability on different forms are described in this section.

Issue 4: Validity Evidence Supporting Test Score Interpretation and Use The many types of evidence supporting the proposed interpretation and use of test scores as a measure of English-language proficiency in academic contexts are discussed.

TOEFL iBT ® Research • Series 1, Volume 5

Issue 6: TOEFL Program History A brief overview of the history and governance of the TOEFL Program is presented. The evolution of the TOEFL test constructs and contents from 1964 to the present is summarized. Future series will feature summaries of recent studies on topics of interest to our score users, such as “what TOEFL iBT test scores tell us about how examinees perform in academic settings,” and “how score users perceive and use TOEFL iBT test scores.” The close collaboration with TOEFL iBT score users, English language learning and teaching experts and university professors in the redesign of the TOEFL test has contributed to its great success. Therefore, through this publication, we hope to foster an ever stronger connection with our score users by sharing the rigorous measurement and research base and solid test development that continues to ensure the quality of TOEFL iBT scores meets the needs of score users. Xiaoming Xi Senior Research Scientist, Research & Development Division, Educational Testing Service

Contributors The primary author of this section is Terry Axe. The following individuals also contributed to this section by providing their careful review as well as editorial suggestions (in alphabetical order). Cris Breining Emilie Pooler Rosalie Szabo

Issue 5: Information for Score Users, Teachers and Learners Materials and guidelines are available to aid in the interpretation and appropriate use of test scores, as well as resources for teachers and learners that support English-language instruction and test preparation.

2 oooooooo

Mikyung Kim Wolf Xiaoming Xi

Information for Score Users, Teachers and Learners This volume of the TOEFL® Insight Series will discuss how TOEFL iBT scores and related services can be used by institutions to help recruit and select applicants with the English-language proficiency needed to succeed at their university. It will also discuss resources available to English-language teachers who want to help students improve their English-language communication skills as well as English learners who prepare for the TOEFL test. The primary use of TOEFL scores is to evaluate the English-language proficiency of applicants for graduate/ postgraduate and undergraduate programs where English is the language of instruction.

Insight For Score Users A valid and reliable assessment of a person’s Englishlanguage proficiency in an academic setting is critical in providing universities with the information they need to evaluate an applicant’s ability to succeed at their institution. TOEFL is the most widely available academic English-language assessment in the world, with 4,500 test centers in more than 165 countries. The test development, delivery, scoring and supporting research ensure that scores are valid and reliable for all test takers, providing a means of evaluating individuals worldwide on the same scale. This allows universities to evaluate a diverse pool of applicants.

ETS does not set passing or failing scores for the TOEFL test. Instead, individual score users set score requirements that will help them TOEFL iBT scores are reported on a score scale of The test development, delivery, scoring recruit and select 0-30 for each of the four language skills – reading, and supporting research ensure that applicants whose listening, speaking and writing. The total score is scores are valid and reliable for all test English-language reported on a scale of 0-120, which is the sum of takers, providing a means of evaluating proficiency meets the four section scores. The test taker score report individuals worldwide on the same scale. their standards. includes scores and performance feedback for each This allows universities to evaluate a Some score users skill. This allows test takers to focus on those skills diverse pool of applicants. will set a total score that need improvement. requirement only, while others may set requirements for one or more of the Because the TOEFL test focuses on an individual’s ability skill sections in addition to or in place of a total score. to communicate in English using all four language skills – reading, listening, speaking and writing – it can help score users identify applicants who have the English-language skills necessary to be successful in their programs. More than 8,000 institutions, agencies and organizations in more than 130 countries use TOEFL scores to evaluate English-language proficiency. A complete list of official TOEFL destinations is available on the TOEFL website at www.ets.org/toefl/institutionlist.

ETS encourages institutions to consider whether, in addition to a total score, a minimum requirement in one or more of the section scores would be appropriate for specific programs. For example, many MBA programs emphasize oral proficiency as students are required to interact frequently in class, engage in group work and make presentations. If speaking ability is important to success in a program, setting a minimum score requirement for that skill may be valuable. Two applicants with the same total score may have very different abilities in the four different language skills, which could impact their ability to succeed academically. Establishing section score requirements ensures that individuals have the profile of English-language abilities that matches the needs of the institution or program.

oooooooo 3

Information for Score Users, Teachers and Learners

Guidelines for Appropriate Score Use ETS encourages institutions to observe the following guidelines for appropriate score use: • Accept only official TOEFL scores from ETS that are no more than two years old. TOEFL scores measure English-language proficiency at a particular point in time. Proficiency may improve or decrease based on the amount of study and exposure to the language following the test date. For this reason, TOEFL scores are only valid for two years. • Consider all relevant information related to English proficiency, not only TOEFL scores.

TOEFL iBT ® Research • Series 1, Volume 5

Validity evidence should be gathered for specific uses of TOEFL total scores or section scores. For example, ETS researchers conducted a study about the use of TOEFL iBT Speaking scores for initial screening of International Teaching Assistants (ITAs). The study concluded that scores from the TOEFL iBT Speaking section can be used successfully as an initial screening to determine whether ITAs have the necessary English-language proficiency to assist in teaching students at the university level. A copy of the study is available in the Research section of the TOEFL website at www.ets.org/toefl/research08-02.

Resources ETS provides a variety of resources to help score users set and review their TOEFL score requirements:

• Consider TOEFL test scores as an aid in interpreting an applicant’s performance on other standardized tests such as the SAT® or GRE® tests.

A Standard-Setting CD has been developed by ETS for all four skills and made available to institutions for free. In a standard-setting session, a panel reviews test items and test taker responses at various score levels for • Avoid using rigid cut scores; ranges provide each skill to determine the minimally acceptable level of more flexibility in selecting qualified applicants English-language for admission. proficiency a ETS provides a variety of resources to first-year student help score users set and review their • Consider establishing requirements for TOEFL would need to TOEFL score requirements. section scores, not just total scores. be successful at the institution. Standards can be set for one or more of • Consider the kinds and levels of English-language the sections. Setting requirements for all four language proficiency required in various fields and levels skills takes two full days to complete. To receive a copy of of study. the CD-ROM with instructions and materials to help score users set admissions requirements for all four skills, contact • Evaluate the validity of your admissions criteria by [email protected]. performing a validity study. • Review requirements on a regular basis, and make adjustments as needed. • Avoid setting score requirements so high that you eliminate qualified applicants, or so low that your applicants will not be successful. • Consider that programs providing ESL support may have more flexibility in accepting applicants with lower proficiency.

4 oooooooo

TOEFL iBT® Score Comparison Tools and other ETS tools provide score comparisons that aim to give you the information you need to make the best admissions decisions for your institution. For instance, we have conducted research to develop these helpful tools that make it easier for you to compare TOEFL iBT® scores to the IELTS® academic module and compare TOEFL iBT scores to TOEFL CBT scores and TOEFL PBT scores. Note that the TOEFL CBT test was discontinued in September 2006 and scores became invalid as of August 2008. Score users should remove references to the TOEFL CBT test from their websites.

The score comparison tables and online interactive widget were based on recent research conducted between the TOEFL test and an alternative test, IELTS academic. For the research, ETS obtained a sample of 1,153 students who had both IELTS and TOEFL scores. Under this approach, TOEFL iBT score ranges are identified for test takers meeting a particular score level on the equivalent IELTS score band. The score comparison results for each section (Listening, Speaking, Reading and Writing) and the total test scores showed that most of the students in the sample scored in the middle to mid-high score ranges on both tests. The full report Linking TOEFL iBT® Scores to IELTS® Scores — A Research Report is available on our website at www.ets.org/toefl/comparescores. The website also contains the score comparison tables and interactive widget. The comparisons between TOEFL CBT scores and TOEFL iBT scores are based on a field test of 3,000 test takers in 30 countries conducted in 2003-2004 before the introduction of the TOEFL iBT test. The comparisons between TOEFL CBT scores and TOEFL PBT scores are based on a similar research study conducted between November 1997 and March 1998. Score comparisons can be useful for understanding relationships between similar sections on two tests, but differences in the tests make it difficult to establish exact comparisons, especially between total scores. For example, the TOEFL iBT test measures speaking ability, but the TOEFL CBT test and TOEFL PBT test did not. The TOEFL iBT Writing section score is based on responses to two writing tasks, one of which is based on listening to a short academic lecture and reading related text. On the TOEFL PBT test, the Structure and Written Expression section score is based on multiple-choice questions only. The TOEFL PBT essay score is reported separately from the total score. The TOEFL® Advanced Search Tool (www.toeflgoanywhere.org) lists an institution’s name, code number, website address and TOEFL score requirements. The directory allows students to use advanced criteria such as score requirements to search for institutions accepting TOEFL scores. Score users may wish to consider using the TOEFL Advanced Search tool to review score requirements set by peer institutions when establishing their own requirements. For section score requirements, please refer to the individual university website. As mentioned previously, some institutions set

Insight their score requirements for the total score only, whereas others set section score requirements according to the needs of their particular programs within the university, especially if the program requires strong language skills in a particular area, such as speaking for a graduate business program. The TOEFL Test and Score Data Summary is an annual report published by ETS that provides score means and percentile data for TOEFL test takers during the previous calendar year. Data are available for a variety of criteria, including total and section scores, reason for taking the test, gender, native language and native country. The Test and Score Data Summaries for more than 15 years of TOEFL testing are available in the Research section of the TOEFL website at www.ets.org/toefl/data_summaries. ETS encourages institutions to look at these data, as well as the performance of students at their institutions, when setting or reviewing TOEFL iBT score requirements. The Online Speech Sample Service was introduced in 2009. This service allows score users to listen to a 60-second portion of an applicant’s scored speaking response to one of the TOEFL iBT integrated speaking tasks. Integrated tasks require test takers to combine their English-language skills, as is typically done when communicating in an academic setting. They may listen to a short lecture and respond orally to a prompt based on what they have heard, or read a passage, listen to a short lecture and then give a spoken response based on what they have read and heard. It is important to note that the speech sample represents the typical performance of the test taker. Some of the test taker’s spoken responses may have been stronger than the sample, while others may have been weaker. In addition, the sample represents extemporaneous speaking and is not a prepared speech. The speech sample service is offered at no charge to any official TOEFL institution, once a confidentiality agreement has been signed and ETS has assigned user names and passwords to the institution. Samples are available for two years after the test date. Combined with test scores and other evidence of English-language proficiency, the speech sample provides a robust profile of an applicant’s ability to communicate effectively in English.

oooooooo 5

Information for Score Users, Teachers and Learners

For English-Language Teachers and Learners The goal of English-language instruction is to help learners use English to perform tasks and to communicate effectively. For university study or for other educational or professional endeavors, the ability to communicate in English is a key element of success. The TOEFL iBT test measures the Englishlanguage skills needed to succeed in higher education. English-language teachers are a primary resource to help students build a foundation and learn to perform tasks and communicate in English. There is no specific length of time or English-language course that will ensure an improvement in a person’s English proficiency. To help students improve their Englishlanguage skills, it is important for them to use English as much as possible. This can include such activities as listening to English-language programs on the radio, TV or internet; reading newspapers, magazine articles and books written in English and writing summaries of what has been read; and reading aloud in class and discussing topics in English. The more students practice their communication skills under the guidance of an English-language teacher, the more likely their English-language skills will improve.

Resources for Teachers

TOEFL iBT ® Research • Series 1, Volume 5

English-language teachers should also encourage students to share their score reports with them, so that they can review the section scores and performance feedback on the test taker’s TOEFL score report. It is very important for teachers to look not only at the total score, but at the four section scores to help identify areas of strength and weakness for each particular student. This is helpful in directing students to exercises that focus on the particular skills that need improvement. The following resource was designed specifically for teachers to help plan, design and implement strong skillbuilding curricula. It can also be used by teachers to create interactive lessons that engage students and reinforce core language skills. • Propell™ Workshop Manual for TOEFL iBT® Test: The Propell workshop is a one-day professional development program for English-language teachers. This manual is based on that workshop. It is an interactive, hands-on approach to developing a TOEFL iBT curriculum. It includes scoring explanations, learning strategies, activity books, a complete practice test and a CD-ROM. ETS has also created tools to measure student progress and identify areas for improvement.

• TOEFL® ITP Assessment Series: These tests provide a quick and reliable way to assess student proficiency, ETS has developed a number of resources to supplement conduct student placement or to evaluate the the curriculum for teachers of English as a second or a effectiveness foreign language. While some are aimed specifically of your EnglishETS has developed a number of at preparation for the TOEFL test, other products are language learning resources to supplement the curriculum available to help teachers develop an effective Englishprogram. TOEFL for teachers of English as a second or a language curriculum. foreign language. While some are aimed ITP uses previously administered specifically at preparation for the TOEFL To help students prepare for the TOEFL iBT test, we test, other products are available to help TOEFL PBT test recommend that teachers become familiar with the questions and teachers develop an effective Englishtest. There are sample questions from the TOEFL iBT contains reading, language curriculum. test on the TOEFL website listening, and (www.ets.org/toefl/samplequestions/). structure and written expression items. Institutions Teachers may also want to take TOEFL® Practice Online can choose who they want to test and when for an to get a feel for the actual testing experience. In addition affordable price. to looking at the sample questions, it can be helpful for teachers to become familiar with the scoring guidelines • Criterion® Online Writing Evaluation Service: This (rubrics) that are used by TOEFL raters to score the is a web-based instructional tool that helps students Speaking and Writing sections. The rubrics for the plan, write and revise essays, and provides instant Speaking and Writing tasks can be viewed at the ETS scoring and annotated diagnostic feedback. website (www.ets.org/toefl/english_programs).

6 oooooooo

•E  nglish Skills Series: There are three easy-to-use, skill-building products from AmEnglish® designed to help students communicate better in English: Pronunciation in English, Idioms in English and Writing in English. Students can improve English communications skills and improve listening, comprehension, vocabulary, speech, pronunciation and writing.

Resources for Learners ETS has products that provide hands-on, targeted instruction to familiarize students with the format and types of questions on the TOEFL iBT test. • TOEFL iBT Tips is a free guide created to help students as they prepare for test day, with practical tips for each of the four skills. • The TOEFL® Test Prep Planner is a free 8-week prep planner that helps students set weekly practice goals and stay on track with tips and tasks for the weeks leading up to test day. The planner helps students understand how to prepare for the test effectively and to build the English skills needed for success. The TOEFL Test Prep Planner is an exclusive item available through institutions and agencies that assist students with TOEFL Services. The Planner can be ordered through this link www.ets.org/toefl/materials.

Insight • TOEFL® Practice Online has the only official practice tests that give students the experience of taking the real TOEFL iBT test. Students will be able to review and answer previously administered TOEFL iBT test questions and get instant scores and feedback on all four skills measured on the test — reading, listening, speaking and writing. • The Official Guide to the TOEFL® Test is a best-selling guide, developed by ETS and published by McGraw-Hill, that offers more than 30 hours of lessons and practice activities that students can use in a classroom language lab or at home. It includes 2 full-length practice tests, real sample responses and actual score raters’ comments. This product is available for purchase in either a paper or an eBook format. •T  OEFL® TV Channel on YouTube®: The TOEFL program has created an educational channel on YouTube® (www.youtube.com/toefltv), where teachers and students share videos of their best English-language learning tips and lessons. Most of the above TOEFL materials (except the TOEFL Test Prep Planner) can be purchased from ETS at www.ets.org/store.

oooooooo 7

Information for Score Users, Teachers and Learners

Notes

8 oooooooo

Contact Us [email protected]

Insight

TOEFL iBT® Research • Series 1, Volume 5

Copyright © 2011 by Educational Testing Service. All rights reserved. ETS, the ETS logo, LISTENING. LEARNING. LEADING., CRITERION, GRE, PROPELL, TOEFL and TOEFL iBT are registered trademarks of Educational Testing Service (ETS) in the US and other countries. *SAT is a registered trademark of the College Board. YouTube is a registered trademark of Google, Inc. ETS10283

Listening. Learning. Leading.® www.ets.org

760652