A comparison of group and individual performance among subject experts and untrained workers at the document retrieval task
✍ Scribed by Wilbur, W. John
- Publisher
- John Wiley and Sons
- Year
- 1998
- Tongue
- English
- Weight
- 269 KB
- Volume
- 49
- Category
- Article
- ISSN
- 0002-8231
No coin nor oath required. For personal study only.
✦ Synopsis
Useful retrieval depends on the ability to predict which the judgments by which all methods of retrieval must be documents a user will find helpful in answer to a query. rated and also hold the key to improved retrieval perfor-Our interest is the common case when no information mance. Over the past several years and as a by-product is provided about the user other than the query and the of work on a retrieval system that we use for MEDLINE query is in natural language. In this setting it is well acrecord processing and retrieval, we have accumulated two cepted that a human can make useful predictions in the form of judgments about what will likely prove useful to test sets of MEDLINE documents in the area of molecular another human. We present data showing that when the biology with multiple human judgments of relevance predictions of a group of humans are averaged, the remade by people knowledgeable in the field. In a recent sult is a better predictor. If performance is measured as study of this data (Wilbur, 1996) we found that a panel a precision, the group performance increases with the of judges whose votes are weighted equally is able to size of the group and approaches a limit of approximately 50% improvement over average individual perfor-predict better than an individual, what are useful documance on our data. Superior performance by groups ments in answer to a query. In that work it was argued raises the question of how. The groups we studied were that because the panel out-performed the individuals, the subject experts and a natural question was whether the basis for the panel's superiority could not be common superior performance resulted from the pooling of their sense or common linguistic abilities which we virtually subject knowledge. In order to answer this question we studied also a group of untrained individuals. To our sur-all possess uniformly. Such could not place the group prise we found that while untrained individuals had a significantly ahead of the individual. We hypothesized somewhat inferior performance compared to trained inthat the group's superior performance must be a consedividuals, the group of untrained individuals together quence of detailed subject knowledge in the area of the performed better than any single trained individual and documents involved, which is possessed by the members almost at the level of the trained group.