There have been a number of major evaluations of the performance of retrieval systems against large full text and surrogate (bibliographic) databases. These evaluations have concentrated on the experimental determination of the Precision Ratio, the fraction of retrieved items that are relevant to an information request, and the Flecall Ratio, the fraction of the total number of relevant items that were actually retrieved. While these measures have met with general acceptance, they have also generated much controversy.
The purpose of this article is to review the results of several of the largest evaluations and to propose a simple model for the performance of such systems that may help explain the relationship between these measures and user behavior.
Recall and Precision Summary
Data
Recall and Precision summary data from several studies are displayed in Table . These will be discussed in more detail in the "Evaluation Studies" section of this article, but it is apparent that recall results are quite different among studies. Various factors may influence study results, including the type and size of database, whether a controlled vocabulary was used, and which of the many database and search variables that might affect recall were studied.
Evaluation Studies
The two "Cranfield" evaluation studies by Cyril Cleverdon in 1962 and 1966 pioneered the area of information systems evaluation. A similar study was done on a large