This paper presents a comparative user study that investigates the relevance judgments made by assessors with a law background and assessors without. Four law students and four library and information studies (LIS) students were recruited to judge independently the relevance of 100 documents for eac
Fuzzy versus probabilistic models for user relevance judgments
โ Scribed by Koll, Mathew ;Srinivasan, Padmini
- Publisher
- John Wiley and Sons
- Year
- 1990
- Tongue
- English
- Weight
- 908 KB
- Volume
- 41
- Category
- Article
- ISSN
- 0002-8231
No coin nor oath required. For personal study only.
โฆ Synopsis
A crucial aspect of information retrieval is the process of making relevance judgments. Although this highly complex decision making procedure still eludes researchers, it is evident that a number of mental models are involved: models of the information need, retrieval system, database, user's knowledge in the subject area etc. In general, It is accepted that relevance judgments are made by evaluating documents for an overall conceptual match with the informatlon need. In this study, we take the view that, given a document and a query, users flrst judge the document against the individual concepts in the query and then use some inferenclng process to derive from these "atomic" decisions a "compound" judgment for the entire query. In other words we adopt a bottom up approach to this decision making process. In this context we examine strategies that may be used to infer compound relevance judgments made from judgments against smaller units of the information need. The probabilistic and fuzzy models are used as two points of reference against which to analyze the user's decision process in making compound judgments. Each model has a different way of approaching the problem. The objective was to empirically examine the relative effectiveness of the models at predicting compound judgments made by users, from their atomic judgments. The results show that the fuzzy model Is sometimes a better predictor than the probabilistic model. The conclusions are interesting when comparing performances across AND and OR queries. No differences were observed between the two models for the OR queries. Both models tend to underestimate relevance for the AND queries. However, the probabilistic model regularly underestlmated relevance more than the fuzzy model. The conclusion made Is that the user seldom employs the AND operator as rigidly as assumed by these models and perhaps by most IR systems. Also, matching functions that better approximate the user's decision process are required to lead to more effectlve systems.
๐ SIMILAR VOLUMES
## Abstract We applied policy capturing and bootstrapping methods to investigate the relevance judgment process, with a particular focus on understanding how judges summarize an overall relevance judgment from five specific aspects of relevance. Our data come from relevance judgments made in the de
## Abstract Relevance judgment has traditionally been considered a personal and subjective matter. A user's search and the search result are treated as an isolated event. To consider the collaborative nature of information retrieval (IR) in a group/organization or even societal context, this articl