Problem / Question Project Overview Variables ...

A Proposed Credibility Index (CI) in Peer Assessment Yao Xiong, Deborah Goins, Hoi K. Suen, Wik-Hung Pun, and Xiaojiao Zang | The Pennsylvania State University

Is the proposed Credibility Index (CI) a reliable and valid method in assessing peer rater credibility for peer assessment?

Hypotheses • Based on the assumption that a credible rater ought to rate each dimension consistently credibly, we expect a high correlation among the CI values calculated for each dimension. • A rater will be more credible if he or she is rating a project within his or her area of expertise or interest.

Project Overview • CI is a refinement of CPR method to facilitate computer-assisted peer assessment to capture levels of accuracy of raters.

• Empirical examination of CI for peer assessment in Creativity, Innovation, and Change MOOC (PSU, September, 2013). • The proposed CI combines three dimensions: • Accuracy: Consistency with an instructor rating during a training/calibration session • Consistency: Stability of rating scores within one peer rater for a given assignment • Transferability: Maintenance of the same level of accuracy across assignments

Procedures Step 1 Creation of projects and grading criteria for rating by research team and instructors • Goal • Starting Point • Supplementary Materials

Step 2 Within the first few weeks of the course, students were asked to participate in rating at least two of the six projects and indicate if project is of an areas of expertise or interest

Step 3 At the completion of the course, course instructor rated the six course relevant projects (“ground truth”)

Results

Materials

Problem / Question

Step 4 For each student, three dimensional CI values and overall CI value calculated based on two “groundtruth” projects being rated

“Groundtruth” projects

Scores for the Projects • Student overall rating (So) • Student maximum/minimum rating (Smax/Smin) • Instructor rating (I)

𝐼𝑛𝑎𝑐𝑐𝑢𝑟𝑎𝑐𝑦(𝐼𝑁𝐴) = (𝑆𝑜𝑖 − 𝐼𝑖 )

𝐼𝑛𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦(𝐼𝑁𝐶) = (𝑆max_𝑗 − 𝑆min_𝑗 ) 𝑁𝑜𝑛𝑡𝑟𝑎𝑛𝑠𝑓𝑒𝑟𝑎𝑏𝑖𝑙𝑖𝑡𝑦(𝑁𝑇𝑅) =

Type III SS

Project

14

0.472

2

0.030

0.015

1.50 0.224

28

0.1815

0.006

0.64 0.925

Project* Area

𝑖 = 1, 2 2

DF

Area

Variables / Equations 2

Source

𝑗 = 1, 2

𝑆𝑜1 − 𝐼1 − (𝑆𝑜2 − 𝐼2 )

𝐼𝑁𝐴1 𝐼𝑁𝐴2 𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦(𝐴𝐶𝐶) = 1 − 0.5 + 𝐼𝑁𝐴1𝑀𝐴𝑋 𝐼𝑁𝐴2𝑀𝐴𝑋 0.5 𝐼𝑁𝐶1 + 𝐼𝑁𝐶2 𝐶𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦(𝐶𝑂𝑁) = 1 − 𝐼𝑁𝐶𝑀𝐴𝑋 𝑁𝑇𝑅 𝑇𝑟𝑎𝑛𝑠𝑓𝑒𝑟𝑎𝑏𝑖𝑙𝑖𝑡𝑦(𝑇𝑅𝐴) = 1 − 𝑁𝑇𝑅𝑀𝐴𝑋 𝐴𝐶𝐶 + 𝐶𝑂𝑁 + 𝑇𝑅𝐴 𝐶𝐼 = 3

Description of Data • Total of 984 students graded at least two of the six projects • Included only the first and second projects that student graded due to a very small number of a third project being graded • ACC, CON, TRA, and overall CI are all negatively skewed

2

Mean F Value Pr > F Square 0.034 3.34