The answer key as a source of error in performance assessment

✍ Scribed by John Norcini; Rosemary Reshetar; Rebecca Lipner

• Publisher: Springer Netherlands
• Year: 1997
• Tongue: English
• Weight: 476 KB
• Volume: 1
• Category: Article
• ISSN: 1382-4996
• DOI: 10.1007/bf00159274

✦ Synopsis

Objective: To determine the impact of variability in answer key construction (i.e., option weighting) on total errors of measurement and to compare several designs for reducing this effect.

Methods: A video-based format that assesses ability to interpret arteriograms is studied because it reproduces with high fidelity an important task faced by cardiologists. Responses given by highly qualified examinees are used to develop answer keys by the aggregate scoring method, and generalizability theory is applied to estimate the error variance in answer key construction.

Sample: Two hundred and two examinees who volunteered to participate in the study after taking their certifying examination in cardiovascular diseases.

Results: Results show less scorer variability than case variability; however, several scorers make a sizeable reduction in measurement error.

Conclusion: Although there is some error in the answer key construction process, its size is relatively small and it should not be a major impediment to the use of performance-based item formats.