Methods for Assessing Twenty-First Century Skills

7/19/2011

Methods for Assessing Twenty-First Century Skills

Projects Background variables: PISA, NAEP GRE: PPI (and High School downward extension) Elementary schools research collaborative Experimental studies of time management Drop‐out assessment and prevention (IES grant) Faking (detection and prevention; OUP book) “Out‐of‐box”: Cognitive fatigue, bullying, creativity,  health attitudes, intercultural sensitivity, team problem  solving, character strengths, emotional abilities – Workforce and college readiness conference (AERA,  College Board, ETS) and book

– – – – – – –

2

Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.

7/19/2011

1

7/19/2011

Assessments Catalogue Learning Skills • • • • •

Openness Learning & Ability Learning Strategies Engagement Test-Taking Strategies

Psychological Wellbeing • • • • • • •

Conscientiousness • • • • • •

3

Conscientiousness Time Management Self-Discipline Persistence Focus Achievement Striving

Resilience Test Anxiety Life Satisfaction Bullying Life at School Math Anxiety Coping

Social and People Skills • • • •

Agreeableness Extroversion Health and Wellness Emotional Management

Motivation • • • •

Interests Career Interests Self-Efficacy Motivational Strategies

Process Skills • Perceptual Speed • Working Memory

Knowledge Skills • • • • •

Crystallized ability Fluid Ability Critical Thinking Fluency Quantitative Ability

Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.

7/19/2011

Time Management 1. Setting Goals I am driven to achieve my goals I focus on what really matters

4. Staying Focused on the Present I can’t cope with change I live in the past

2. Meeting Deadlines 5. Planning Ahead I leave things to the last minute* I like to “live on the edge” I set realistic time estimates on each I don’t plan things I just do them task 3. Organizing Time and Tasks I write a weekly to-do list Without my organizer I am lost

6. Having a Workspace I organize my desk so I know exactly where things are I have ordered files

Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.

2

7/19/2011

Time Management 时间管理 • • • • • •

5

I focus on what really matters. 我专注于重要的事情. I leave things to the last minute*. 我把事情留到最后一 刻再做。 I set realistic time estimates on each task. 我给每个任务 制定合理的时间。 I write a weekly to‐do list. 我会列个日程表。 Without my organizer I am lost. 没有了记事簿，我就不 知道该干什么。 I organize my desk so I know exactly where things are. 我 会整理书桌以便知道物品的确切位置。

Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.

7/19/2011

Work Ethic (Effort to Pass) • Intentions – “I intend to try as hard as I can to pass my classes this year.” – “Pienso hacer todo lo que pueda para aprobar mis clases este año.”

• Attitudes – “It is beneficial to me to try as hard as I can to pass my classes this year.” – “Me conviene hacer todo lo que pueda para aprobar mis clases este año.”

• Subjective norms – “My parent/guardian believes I should try as hard as I can to pass my classes this  year.” – “Mis padres o tutor legal creen que debo hacer todo lo que pueda para aprobar mis clases este año.”

• Perceived Control – “It is up to me how hard I try to pass my classes this year.”  – “Es decisión mía cuánto haga para aprobar mis clases este año” 6

Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.

7/19/2011

3

7/19/2011

7

8

Confidential and Proprietary. Copyright © 2010 by Educational Testing Service. All rights reserved. ETS, the ETS logo, LISTENING. LEARNING. LEADING., Criterion, Focus on Standards, GRE, Keeping Learning on Track, SLEP, TOEFL and TOEIC are registered trademarks of Educational Testing Service (ETS) in the United States and other countries. EXADEP, Examen de Admisión a Estudios de Posgrado, iCritical Thinking, ProofWriter, Propell, Secondary Level English Proficiency, SIR II, Test de français international, TFI, The Praxis Series, TOEFL IBT, and TOEIC Bridge are trademarks of ETS. All other trademarks are property of their respective owners. 11096

7/19/2011

Confidential and Proprietary. Copyright © 2010 by Educational Testing Service. All rights reserved. ETS, the ETS logo, LISTENING. LEARNING. LEADING., Criterion, Focus on Standards, GRE, Keeping Learning on Track, SLEP, TOEFL and TOEIC are registered trademarks of Educational Testing Service (ETS) in the United States and other countries. EXADEP, Examen de Admisión a Estudios de Posgrado, iCritical Thinking, ProofWriter, Propell, Secondary Level English Proficiency, SIR II, Test de français international, TFI, The Praxis Series, TOEFL IBT, and TOEIC Bridge are trademarks of ETS. All other trademarks are property of their respective owners. 11096

7/19/2011

4

7/19/2011

9

Confidential and Proprietary. Copyright © 2010 by Educational Testing Service. All rights reserved. ETS, the ETS logo, LISTENING. LEARNING. LEADING., Criterion, Focus on Standards, GRE, Keeping Learning on Track, SLEP, TOEFL and TOEIC are registered trademarks of Educational Testing Service (ETS) in the United States and other countries. EXADEP, Examen de Admisión a Estudios de Posgrado, iCritical Thinking, ProofWriter, Propell, Secondary Level English Proficiency, SIR II, Test de français international, TFI, The Praxis Series, TOEFL IBT, and TOEIC Bridge are trademarks of ETS. All other trademarks are property of their respective owners. 11096

7/19/2011

Cooperative vs. Competitive Learning Forced Choice – I would like to be the best in my class  in mathematics.   – I try very hard in mathematics  because I want to do better in the  exams than the others.  – I make a real effort in mathematics  because I want to be one of the best.  – In mathematics I always try to do  better than the other students in my  class.  – I do my best work in mathematics  when I try to do better than others.  10

– In mathematics I enjoy working with  other students in groups.  – When we work on a project in  mathematics, I think that it is a good  idea to combine the ideas of all the  students in a group.  – I do my best work in mathematics  when I work with other students.  – In mathematics, I enjoy helping  others to work well in a group.  – In mathematics I learn most when I  work with other students in my  class.

Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.

7/19/2011

5

7/19/2011

7/1 9/2 011

12

11

Confidential and Proprietary. Copyright © 2010 by Educational Testing Service. All rights reserved. ETS, the ETS logo, LISTENING. LEARNING. LEADING., Criterion, Focus on Standards, GRE, Keeping Learning on Track, SLEP, TOEFL and TOEIC are registered trademarks of Educational Testing Service (ETS) in the United States and other countries. EXADEP, Examen de Admisión a Estudios de Posgrado, iCritical Thinking, ProofWriter, Propell, Secondary Level English Proficiency, SIR II, Test de français international, TFI, The Praxis Series, TOEFL IBT, and TOEIC Bridge are trademarks of ETS. All other trademarks are property of their respective owners. 11096

7/19/2011

6

7/19/2011

PPI Scales and Items Knowledge and Creativity

Has a broad perspective on the field Is among the brightest persons I know Produces novel ideas Is intensely curious about the field

Resilience

Accepts feedback without getting defensive Works well under stress Can overcome challenges and setbacks Works extremely hard

Communication Skills

Speaks in a clear, organized, and logical manner Writes with precision and style Speaks in a way that is interesting Organizes writing well

Teamwork

Supports the efforts of others Behaves in an open and friendly manner Works well in group settings Gives criticism/feedback to others in a helpful way

Planning and Organization

Sets realistic goals Organizes work and time effectively Meets deadlines Makes plans and sticks to them

Ethics and Integrity

13

Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.

14

Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.

Is among the most honest persons I know Maintains high ethical standards Is worthy of trust from others Demonstrates sincerity 7/19/2011

7

7/19/2011

Situational Judgment Test You always work hard and get good grades. On a group project, the other kids on your team are goofing off and letting you do all the work. What would you do in this situation?

1. Confront the team and tell them this is a group effort 2. Tell the teacher about the situation 3. Just continue on and do the best you can to finish the project 4. Make a list of all the tasks that need to be done and ask the others to choose an assignment

®

Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).

Text-Based SJT: Stem You are part of a study group that has been assigned a large presentation for class. As you are all dividing up the workload, it becomes clear that both you and another member of the group are interested in researching the same aspect of the topic. Your colleague already has a great deal of experience in this area, but you have been extremely excited about working on this part of the project for several months. Which of the following is the best approach to dealing with this situation? ®

8

7/19/2011

Text-Based SJT: Response (A) Flip a coin to determine who gets to work on that particular aspect of the project. (B) Insist that, for the good of the group, you should work on that aspect of the project because your interest in the area means you will do a particularly good job. (C) Compromise your preferences for the good of the group and allow the other person to work on that aspect of the project. (D) Choose a different group member to work on that aspect of the project. ®

Copyright © 2011 Educational Testing Service. All rights reserved.

1 8

9

7/19/2011

Copyright © 2011 Educational Testing Service. All rights reserved.

Copyright © 2011 Educational Testing Service. All rights reserved.

2 0

10

7/19/2011

Copyright © 2011 Educational Testing Service. All rights reserved.

Copyright © 2011 Educational Testing Service. All rights reserved.

2 1

2 2

11

7/19/2011

23

Copyright © 2011 Educational Testing Service. All rights reserved.

12