Mayer Brezis wrote:
> Many thanks for your prompt and interesting responses on the more
> general question of discrepancies between large RCT's and meta-analysis.
>
> My question is more specifically for a meta-analysis that /includes/ a
> large RCT with results contradictory to that of the meta-analysis.
>
> A current example I have in mind is a RCT on corticosteroids for sepsis
> (CORTICUS, NEJM 2008) showing no effect on mortality while a subsequent
> meta-analysis (/including/ this RCT) concluded corticosteroids reduce
> mortality in sepsis (JAMA 2009).
>
> I'm looking for another example to better understand the problem.
You might want to think about the definition of the word
"contradictory". If the RCT has a p-value of 0.06 and the meta-analysis
has a p-value of 0.04, are the results really contradictory? A better
definition might be that the results are contradictory if the confidence
interval of the meta-analysis does not contain the point estimate of the
large RCT. That's not perfect either. Another possibility might be: the
confidence interval of the large RCT lies entirely inside the range of
clinical indifference and the confidence interval of the meta-analysis
lies entirely outside the range of clinical indifference (or vice versa).
The reference is a bit dated, but there was an article that asked the
question "Why do so many RCTs produce contradictory results" and
outlined some of the reasons (e.g., different population, different
length of follow-up). Many of the factors listed here are also factors
associated with heterogeneity problems in a systematic overview and can
also serve as a possible explanation why a large RCT can produce
contradictory results from a systematic overview that includes that RCT.
Horwitz, R. I. (1987). "Complexity and contradiction in clinical trial
research." Am J Med 82(3): 498-510. Abstract: "Randomized clinical
trials have become the accepted scientific standard for evaluating
therapeutic efficacy. Contradictory results from multiple randomized
clinical trials on the same topic have been attributed either to
methodologic deficiencies in the design of one of the trials or to small
sample sizes that did not provide assurance that a meaningful
therapeutic difference would be detected. When 36 topics with
conflicting results that included over 200 randomized clinical trials in
cardiology and gastroenterology were reviewed, it was discovered that
results of randomized clinical trials often disagree because the
complexity of the randomized clinical trial design and the clinical
setting creates inconsistencies and variation in the therapeutic
evaluation. Nine methodologic sources of this variation were identified,
including six items concerned with the design of the trials, and three
items concerned with interpretation. The design issues include
eligibility criteria and the selection of study groups, baseline
differences in the available population, variability in indications for
the principal and concomitant therapies, protocol requirements of the
randomized clinical trial, and management of intermediate outcomes. The
issues in interpreting the trials include the regulatory effects of
treatments, the frailty of double-blinding, and the occurrence of
unexpected trial outcomes. The results of this review suggest that
pooled analyses of conflicting results of randomized clinical trials
(meta-analyses) may be misleading by obscuring important distinctions
among trials, and that enhanced flexibility in strategies for data
analysis will be needed to ensure the clinical applicability of
randomized clinical trial results."
Unfortunately, I do not think that the full free text of this article is
available on the web.
--
Steve Simon, Standard Disclaimer
"What do all these numbers mean? Sensitivity,
specificity, and likelihood ratios"
Wednesday, February 17, 11am-noon, CST
Free to all! www.pmean.com/webinars
|