๐”– Bobbio Scriptorium
โœฆ   LIBER   โœฆ

Fuzzy versus probabilistic models for user relevance judgments

โœ Scribed by Koll, Mathew ;Srinivasan, Padmini


Publisher
John Wiley and Sons
Year
1990
Tongue
English
Weight
908 KB
Volume
41
Category
Article
ISSN
0002-8231

No coin nor oath required. For personal study only.

โœฆ Synopsis


A crucial aspect of information retrieval is the process of making relevance judgments. Although this highly complex decision making procedure still eludes researchers, it is evident that a number of mental models are involved: models of the information need, retrieval system, database, user's knowledge in the subject area etc. In general, It is accepted that relevance judgments are made by evaluating documents for an overall conceptual match with the informatlon need. In this study, we take the view that, given a document and a query, users flrst judge the document against the individual concepts in the query and then use some inferenclng process to derive from these "atomic" decisions a "compound" judgment for the entire query. In other words we adopt a bottom up approach to this decision making process. In this context we examine strategies that may be used to infer compound relevance judgments made from judgments against smaller units of the information need. The probabilistic and fuzzy models are used as two points of reference against which to analyze the user's decision process in making compound judgments. Each model has a different way of approaching the problem. The objective was to empirically examine the relative effectiveness of the models at predicting compound judgments made by users, from their atomic judgments. The results show that the fuzzy model Is sometimes a better predictor than the probabilistic model. The conclusions are interesting when comparing performances across AND and OR queries. No differences were observed between the two models for the OR queries. Both models tend to underestimate relevance for the AND queries. However, the probabilistic model regularly underestlmated relevance more than the fuzzy model. The conclusion made Is that the user seldom employs the AND operator as rigidly as assumed by these models and perhaps by most IR systems. Also, matching functions that better approximate the user's decision process are required to lead to more effectlve systems.


๐Ÿ“œ SIMILAR VOLUMES


A user study of relevance judgments for
โœ Jianqiang Wang; Dagobert Soergel ๐Ÿ“‚ Article ๐Ÿ“… 2010 ๐Ÿ› Wiley (John Wiley & Sons) ๐ŸŒ English โš– 506 KB

This paper presents a comparative user study that investigates the relevance judgments made by assessors with a law background and assessors without. Four law students and four library and information studies (LIS) students were recruited to judge independently the relevance of 100 documents for eac

Policy capturing models for multi-facete
โœ Xiaoli Huang; Ryen W. White ๐Ÿ“‚ Article ๐Ÿ“… 2006 ๐Ÿ› Wiley (John Wiley & Sons) ๐ŸŒ English โš– 163 KB

## Abstract We applied policy capturing and bootstrapping methods to investigate the relevance judgment process, with a particular focus on understanding how judges summarize an overall relevance judgment from five specific aspects of relevance. Our data come from relevance judgments made in the de

Collaborative relevance judgment: A grou
โœ Xiangmin Zhang ๐Ÿ“‚ Article ๐Ÿ“… 2002 ๐Ÿ› John Wiley and Sons ๐ŸŒ English โš– 161 KB

## Abstract Relevance judgment has traditionally been considered a personal and subjective matter. A user's search and the search result are treated as an isolated event. To consider the collaborative nature of information retrieval (IR) in a group/organization or even societal context, this articl