A model of calibration for subjective probability distributions and its application to aggregation of judgements

✍ Scribed by Paula E. Curtis; William R. Ferrell; Ira Solomon

• Publisher: John Wiley and Sons
• Year: 1985
• Tongue: English
• Weight: 803 KB
• Volume: 4
• Category: Article
• ISSN: 0277-6693
• DOI: 10.1002/for.3980040404

✦ Synopsis

A simple model is proposed of the statistical structure underlying the calibration of auditors' subjective probability distributions for account balances, and potentially for other unknown quantities. The model relates calibration curve shape to two parameters which represent over-or underconfidence and over-or underestimation. It is fitted to data from expert auditors. Different types of account appear to have different calibration characteristics. The model helps predict approximately the effects on calibration of aggregating individual subject distributions. Aggregation improves accuracy, but produces a strong tendency towards underconfidence. One aggregation method, predicting the best judgement in the group and using it as the group judgement, is found to be quite effective, much better than averaging the fractiles of individual distributions.

KI:Y WORIX Calibration Subjective probability assessment Expert-use Combination of forecasts Auditing

A calibration curve shows one aspect of the quality of probability judgements. For discrete events, it is a plot of p against r, where p is the observed proportion of occurrences of events and r is the subjective probability of those events. For continuous variables, a subjective probability density (or distribution) function (s.p.d.f.) is assessed and p is the proportion of times the true value o f a quantity is observed to be below the r fractile of the s.p.d.f. o f that quantity. In either case calibration is perfect when p = r (see Lichtenstein, Fischhoff & Phillips (1 98 1) for a review).

If one wishes to take subjective judgements of probability at face value in the sense of expected * This paper is based on a graduate report written by the first author, and supervised by the second. The third author is responsible for the experimental work.