for one or more subjects or may be used by many HEIs for the same subject. ...
TSA Oxford (PPE) 3000+ candidates. TSA trial .... Section 1 – Aptitude and Skills.
Pre-admissions testing - an overview
Tim Oates Group Director Assessment Research & Development
Context for admissions tests in the UK - stretch & challenge, A* in A levels, 15% gaining three or more A grades; contrasting needs of ‘recruiting’ and ‘selecting’ universities - fluctuating demand for a generic admissions test, lobbies and special reports (Schwartz 2004; Sykes 2010); failure of SAT trial - widening participation agenda - contingent and enforced reduction in HE participation (50% target in UK; 29% Germany; 70% Finland) - embarrassing realities re gender, school type - opposition to admissions tests re ‘school organisation’, cost and assessment load - high proportion of departmental tests; low level of validation work (by contrast, over 20 validation reports associated with Cambridge Assessment tests)
The world of admissions tests Janet Graham – SPA Definition of ‘admissions test’ The tests can be in the summer/autumn in the academic year before admission or at interview, normally from November onwards in the academic year before admission. A test can be an HEI's own devised test or a test devised by an HEI or group or consortium of HEIs with one or more testing/awarding bodies. A test may be used by one HEI for one or more subjects or may be used by many HEIs for the same subject.
1
Tests developed for certain subjects/courses by testing bodies, normally with one or more HEI UKCAT
GAMSAT
9
9
Dentistry
HPAT
LNAT
STEP
BMAT
MSAT
9
9
9
Dietetics
9
Law
9
Mathematics 9
Medicine
9
Occupational Therapy
9
Physiological Sciences
9
Physiotherapy
9
Podiatry
9
Radiography
9
Speech & Language Therapy
9
Veterinary Medicine
9
UKCAT – UK Clinical Aptitude Test HPAT – Health Professions Admission Test STEP – Sixth Term Examination Papers MSAT – Medical School Admissions Test
GAMSAT – Graduate Australian Medical School Admissions Test LNAT – National Admissions Test for Law BMAT – BioMedical Admissions Test
Institutions’ own tests Course
Institution
Course
Institution
Anglo-Saxon, Norse & Celtic
Cambridge
European & Middle Eastern Lang.
Oxford
Applied Biomedical Science
Nottingham Trent
Experimental Psychology
Oxford
Archaeology & Anthropology
Cambridge
Fashion Merchandise Management
Westminster
Architecture
Cambridge
Fine Art
Oxford
Business Management
City of Bath College
Geography
Cambridge
Classics
Cambridge
History
Cambridge
Classics
Oxford
History of Art
Cambridge
Computer Science
Oxford
Journalism
Bell College
Computer Visualisation & Animation
Bournemouth
Mathematics
Oxford
Economics & Management
Oxford
Mathematics & Computer Sc.
Oxford
Education
St Martins College
Mathematics & Philosophy
Oxford
Education Studies
Cambridge
Mathematics & Statistics
Oxford
English
Cambridge
Modern Lang. & Linguistics
Oxford
English & Modern Languages
Oxford
Music
Cambridge
Institutions’ own tests Course
Institution
Course
Institution
Music
Oxford
Psych., Phil. & Physiology
St Martins College
Nursing
St Martins College
Diagnostic Radiography
St Martins College
Oriental Studies
Cambridge
Social Work
Birmingham
Philosophy
Cambridge
Social Work
De Montfort
Philosophy & Modern Lang.
Oxford
Social Work
New College Durham
Philosophy, Politics & Econ.
Oxford
Social Work
N.E. Worcestershire College
Philosophy & Theology
Oxford
Theology & Religious Studies
Cambridge
Physics
Oxford
Veterinary Medicine
Nottingham
Physics & Philosophy
Oxford
2
Cambridge Assessment university admissions tests Cambridge ‘family’ of tests: BioMedical Admissions Test (BMAT) 6500 candidates in 1300 centres (86% UK-based) Thinking Skills Assessment (TSA) 2500 candidates in Cambs (85% UK-Based) TSA Oxford (PPE) 3000+ candidates TSA trial Imperial College London 2000 TSA University College London 150-200 English Literature Admissions (ELAT) 1500 candidates in Oxford STEP Maths test 1500 (Cambridge and Warwick) Westminster and University College School 200
(reserved/mothballed: Admissions Test unitTEST (trial) 1800 candidates in 25 centres)
Admissions test numbers and national context 7 tests developed by testing bodies (UKCAT; GAMSAT: HPAT; LNAT; STEP; BMAT; MSAT) 60 tests developed by institutions (increase from 47 in 2006 attributed by SPA to better reporting rather than development of new tests) SAT1 and uniTEST trialled in the UK 116 Universities in UK 250,000 courses at Sussex (modular provision) 47 major ‘routes’ at Cambridge (eg natural sciences with specialisms eg physics) 343 courses at UWE (ugrad, pgrad excluding short courses) Used by 0.7% of the 48,237 courses currently in the UCAS scheme (stable figure; source SPA) - around 16% of the 309 HE providers in the UCAS scheme - and are a portion of the evidence used in admissions to those courses although some do use them in de-selection
Admissions test numbers and national context 67 tests in total 34 of these used by Oxford and Cambridge
5 ‘national’ tests UKCAT BMAT STEP GAMSAT LNAT
24 universities 5 universities 2 universities 5 universities 10 universities
20,185 candidates 6,050 candidates 1,500 candidates 1,288 candidates 6,749 candidates
Total number of applicants in 2007-08 = 582,657 Total taking four ‘main’ tests (ie excluding STEP) = less than 6%
3
SPA criteria Admissions tests should: - have rigorous validation and reliability testing - be supported by statistical and research evidence - ensure the minimum of bias in the test questions so the test is valid for all applicants from all backgrounds in a UK context - be readily available and accessible to those with evidence of specific requirements in a timely way - provide exemplar materials and tests with answers - and many of the tests do have such material freely available for students to practice the test - be fairly and professionally administered - be able to demonstrate it is fit for purpose and add value as part of holistic decision-making - be approved for use through the institution’s relevant structures and processes
Admissions in the US - myths ‘…it’s all by SAT…’ - each college has its own criteria for admission - college grades (grade point average) - High School Transcript - letter of recommendation - admissions essay or essays - extracurricular activities - increased use of AP (curriculum based assessment - analogue of A level) - SAT 1 or ACT score - SAT 2 (subject based) - again we see multiple sources of evidence
Our own criteria We develop assessments in close conjunction with HE, usually on request Any assessment must make a unique contribution to the admissions process It should not unnecessarily replicate previous and concurrent assessments Any assessment must yield valid information in respect of suitability for specific HE provision
4
Issues Cost Burden Validity Utility
Cost Reasonable cost for each assessment Travel to centre Preparation courses and materials
Burden Number of assessments in 16-19 Logistics of sitting Duplication - (eg most obviously - BMAT and UKCAT) More stress and ‘critical gateway’ assessments
5
Issues No magic bullet Incremental validity The challenge of developmental validation TSA uneveness Predicting what exactly? End attainment, avoiding drop out The definitely in, the definitely out, and the large population in the middle Interest in non-cognitive dimensions STEP - a very interesting example of a norm-referenced test The very peculiar attacks on A level - it prepares for HE, it predicts using UMS The tariff - an interesting case of throwing information away Recent work at Cambridge - state versus independent, UMS, and GCSE A level choice and widening participation
Issues and questions for all Can an assessment be coached-for? Does the assessment duplicate anything else – prior or concurrent? Is repeat assessment for ‘retention by candidates’ necessary – deep learning? What’s the evidence – validation data? Are candidates adequately supported? Does it contribute unreasonably to candidates’ pressure and stress? Are costs reasonable? Does it really open up access? Does it help with non-traditional entry and overseas candidates? What are the educational backwash effects? Is there adequate monitoring of the effects? What does admissions testing tell us about confidence in other assessments? What does it tell us about the utility of information from other sources? What does it tell us about standards over time?
BioMedical Admissions Test The BMAT is a subject-specific admissions test, currently taken by applicants to Medicine, Veterinary Medicine and related courses at: • Cambridge • Oxford • University College London • Royal Veterinary College • Imperial College London
6
Purpose of the BMAT: • to assess whether candidates possess the generic skills often used in undergraduate study; • to ensure that candidates’ scientific understanding is adequate for the study of medical and veterinary sciences; • to provide additional information for admissions staff when selecting from within a group of very able candidates; • to provide a ‘common yardstick’ for comparison of candidates from a variety of educational backgrounds; • to introduce a degree of transparency into the selection process.
BMAT – structure of the test Section 1 – Aptitude and Skills 1 hour – 35 multiple choice or short answer questions Section 2 – Scientific Knowledge and Applications 30 minutes – 27 multiple choice or short answer questions Section 3 – Writing Task 30 minutes – 1 from a choice of 3 short essay questions.
BMAT 2006 – student performance
7
1.0
1.0 Probability of obtaining Class I
Probability of obtaining at least Class II
BMAT – predictive value
0.8 0.6 0.4 0.2 0.0 5.0
7.6
10.2 12.8 BMAT score
15.4
18.0
0.8 0.6 0.4 0.2 0.0 5.0
7.6
10.2 12.8 BMAT score
15.4
18.0
TSA (Thinking Skills Assessment) Cambridge • • • •
Natural Sciences Engineering Computer Sciences Economics
Oxford • Philosophy, Politics and Economics
UCL • European Social and Political Studies
TSA – structure of the test 90 minutes – 50 five-option multiple choice questions including: - Critical Thinking; - Problem Solving. The Oxford PPE test will include an essay task. The test is still in the pilot phase. Results are used with caution, being considered in addition to, not instead of, traditional selection criteria.
8
TSA – online results
Correlations between TSA 2003 and 1st year exam performance N
Problem Solving
Critical Thinking
TSA Total Score
Computer Science
65
0.42(**)
0.33(**)
0.45(**)
Economics
25
0.31
0.27
0.32
Engineering
106
0.23(**)
0.12
0.20(*)
Natural Sciences
106
0.19(*)
0.21(*)
0.23(**)
(*) p
