𝔖 Bobbio Scriptorium
✦   LIBER   ✦

A user study of relevance judgments for E-Discovery

✍ Scribed by Jianqiang Wang; Dagobert Soergel


Publisher
Wiley (John Wiley & Sons)
Year
2010
Tongue
English
Weight
506 KB
Volume
47
Category
Article
ISSN
0044-7870

No coin nor oath required. For personal study only.

✦ Synopsis


This paper presents a comparative user study that investigates the relevance judgments made by assessors with a law background and assessors without. Four law students and four library and information studies (LIS) students were recruited to judge independently the relevance of 100 documents for each of four requests for production of responsive documents for litigation purposes, two requests for the tobacco Master Settlement Agreement collection and two for the Enron corporate emails collection. Both quantitative and qualitative methods are used to analyze data collected through the relevance judgment task, an entry questionnaire, and an exit interview. Being given the same task guidelines, the LIS student assessors judged relevant documents just as accurately as the law student assessors, while they judged nonrelevant documents slightly less accurately than the law student assessors. In addition, participants achieved moderate to substantial agreement on their relevance judgments. Relevance judgment speed varied significantly among participants, although on average it was about the same for the two groups. Factors influencing the accuracy and the speed of participants" relevance judgments are discussed based on preliminary analysis of qualitative data collected through the exit interviews.


πŸ“œ SIMILAR VOLUMES


Collaborative relevance judgment: A grou
✍ Xiangmin Zhang πŸ“‚ Article πŸ“… 2002 πŸ› John Wiley and Sons 🌐 English βš– 161 KB

## Abstract Relevance judgment has traditionally been considered a personal and subjective matter. A user's search and the search result are treated as an isolated event. To consider the collaborative nature of information retrieval (IR) in a group/organization or even societal context, this articl

Order effects: A study of the possible i
✍ Eisenberg, Michael ;Barry, Carol πŸ“‚ Article πŸ“… 1988 πŸ› John Wiley and Sons 🌐 English βš– 909 KB

Studies concerned with the evaluation of information systems have typically relied on judgments of relevance as the fundamental measure in determining system performance. In most cases, subjects are asked to assign a relevance score using some category rating scale (l-4, l-11, or simply relevant/non

Consolidating user relevance criteria: A
✍ Stephen Bales; Peiling Wang πŸ“‚ Article πŸ“… 2006 πŸ› Wiley (John Wiley & Sons) 🌐 English βš– 275 KB

## Abstract The purpose of this study is to bring together empirical studies on user‐based relevance criteria and to provide a conceptual framework for the organization of factors affecting relevance judgments and information selection. Through systematic analysis applying a meta‐ethnographic appro

Toward a theory of user-based relevance:
✍ Park, Taemin Kim πŸ“‚ Article πŸ“… 1994 πŸ› John Wiley and Sons 🌐 English βš– 866 KB

The concept of relevance has played a major role in the evaluation of information retrieval and its research without much consideration of users. While methodological deficiencies have been noted and criticized, the study of relevance is still suffering from much discussion of alternative methods. T

A study of relevance for learning in ded
✍ Nada Lavrač; Dragan Gamberger; Viktor Jovanoski πŸ“‚ Article πŸ“… 1999 πŸ› Elsevier Science 🌐 English βš– 288 KB

This paper is a study of the problem of relevance in inductive concept learning. It gives definitions of irrelevant literals and irrelevant examples and presents ecient algorithms that enable their elimination. The proposed approach is directly applicable in propositional learning and in relation le