was followed by discussion which often required intervention by the moderator, Bill Brown.
DerSimonian introduced the topic with an example of a meta-analysis of several trials on calcium supplementation for the prevention of pre-eclampsia. Discussing the wide differences in study design and quality of these trials, she illustrated that there may well be reasons to undertake a new trial, even though the meta-analysis implied an overall effect. In her example, not all of the trials were randomized and the data from some were suspect. Using a Bayesian framework, she showed how the meta-analysis results could be used to suggest a set of prior distributions which could be incorporated into the design and monitoring of a new trial. She suggested that a similar approach might be taken when results are available from only a single prior trial or when findings emerge while the new trial is in progress.
In the ensuing discussion, the questions which we would examine for the rest of the day emerged. Does a meta-analysis help us decide whether a trial should be done in a given population? As new results emerge, should the data monitoring board consider them? If so, should they be formally or informally used? It was pointed out that meta-analysis is a process rather than a formal methodology and the exact relevance of meta-analysis will vary, depending on the trial and the individuals making the decision.
Olkin began by noting that the information explosion of the past 50 years has led to a need for conclusions to be drawn from diverse, imperfect studies. The subject area of meta-analysis is not new, going back to a 1904 paper by Karl Pearson. Single experiments have diversity and statistics does its best with some diversity and moderate sample size. A good meta-analysis requires intimate knowledge of the substantive field plus statistical skills. While meta-analyses based on individual patient data are more valuable than those based on literature review, it is prohibitive for investigators to get individual patient data for every meta-analysis.
Certain diagnostics are helpful in establishing the robustness of a meta-analysis. If quality scores are used, analysis with and without these scores should be reported. Procedures similar to those used in regression analysis, such as omitting one study at a time and looking at the effect, are helpful. Discovery of differences between studies will suggest further analyses to account for them. The statistical theory for omitting the most or least significant studies is presently incomplete.