Inter-Coder Agreement for Computational Linguistics

✍ Scribed by Artstein, Ron; Poesio, Massimo

• Book ID: 120314214
• Publisher: MIT Press
• Year: 2008
• Tongue: English
• Weight: 329 KB
• Volume: 34
• Category: Article
• ISSN: 0891-2017
• DOI: 10.1162/coli.07-034-R2

✦ Synopsis

This article is a survey of methods for measuring agreement among corpus annotators. It exposes the mathematics and underlying assumptions of agreement coefficients, covering Krippendorff's alpha as well as Scott's pi and Cohen's kappa; discusses the use of coefficients in several annotation tasks; and argues that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in computational linguistics, may be more appropriate for many corpus annotation tasks—but that their use makes the interpretation of the value of the coefficient even harder.