A user study of relevance judgments for E-Discovery
β Scribed by Jianqiang Wang; Dagobert Soergel
- Publisher
- Wiley (John Wiley & Sons)
- Year
- 2010
- Tongue
- English
- Weight
- 506 KB
- Volume
- 47
- Category
- Article
- ISSN
- 0044-7870
No coin nor oath required. For personal study only.
β¦ Synopsis
This paper presents a comparative user study that investigates the relevance judgments made by assessors with a law background and assessors without. Four law students and four library and information studies (LIS) students were recruited to judge independently the relevance of 100 documents for each of four requests for production of responsive documents for litigation purposes, two requests for the tobacco Master Settlement Agreement collection and two for the Enron corporate emails collection. Both quantitative and qualitative methods are used to analyze data collected through the relevance judgment task, an entry questionnaire, and an exit interview. Being given the same task guidelines, the LIS student assessors judged relevant documents just as accurately as the law student assessors, while they judged nonrelevant documents slightly less accurately than the law student assessors. In addition, participants achieved moderate to substantial agreement on their relevance judgments. Relevance judgment speed varied significantly among participants, although on average it was about the same for the two groups. Factors influencing the accuracy and the speed of participants" relevance judgments are discussed based on preliminary analysis of qualitative data collected through the exit interviews.
π SIMILAR VOLUMES
## Abstract Relevance judgment has traditionally been considered a personal and subjective matter. A user's search and the search result are treated as an isolated event. To consider the collaborative nature of information retrieval (IR) in a group/organization or even societal context, this articl
Studies concerned with the evaluation of information systems have typically relied on judgments of relevance as the fundamental measure in determining system performance. In most cases, subjects are asked to assign a relevance score using some category rating scale (l-4, l-11, or simply relevant/non
## Abstract The purpose of this study is to bring together empirical studies on userβbased relevance criteria and to provide a conceptual framework for the organization of factors affecting relevance judgments and information selection. Through systematic analysis applying a metaβethnographic appro
The concept of relevance has played a major role in the evaluation of information retrieval and its research without much consideration of users. While methodological deficiencies have been noted and criticized, the study of relevance is still suffering from much discussion of alternative methods. T
This paper is a study of the problem of relevance in inductive concept learning. It gives definitions of irrelevant literals and irrelevant examples and presents ecient algorithms that enable their elimination. The proposed approach is directly applicable in propositional learning and in relation le