I agree with Steve Simon that when there is an apparent clumping, it is best
to try to explain that, and to treat the clumps separately.
I also agree with him that for very homogeneous study results, the fixed and
random effects methods give similar results. The more heterogeneity there
is, the further appart the results will be with the two methods (the point
estimates will differ somewhat, but mainly, the confidence interval will be
wider for the random effects method. Therefore, the most conservative thing
is to always use the random effects method. When there is truly no
heterogeneity, the random method will give the same result as the fixed, and
the more heterogeneity is present, the more the random effets method will be
Philosophically, the way to decide which method applies is to ask this
question. If all of the studies (or centers) expanded their sample sizes
many-fold, would one expect all of the results to converge to the same point
estimate? If so, then one believes all of the variability between study
results comes from scatter because of small study sizes. Then weighting
should be solely on the basis of study size (or variance), and the fixed
effects model is appropriate. If on the other hand one would not expect the
different study results to converge to the same point estimate, because of
real differences in the populations being sampled, then the random effects
model is appropriate.
As you can see, this is a sort of philosophical expectation that one is
guessing at. The only emipirical way to address the question is with
heterogeneity analysis. Unfortunately, with small numbers of studies (or
centers) heterogeneity analysis is underpowered, so that "non-significant"
heterogeneity may be found in a situation in which there is some real
heterogeneity. For this reason it may be wise to use the random effects
model even when no "significant" heterogeneity is found. As mentioned
above, the results will be very similar, but the random effects model will
give a more conservative, wider confidence interval.
However, there is a potential problem with any random effects analysis. If
the different studies (or centers) have different populations, but the
studies are not a random sample of the different population centers, then
the result will be biased to the extent that the sample is biased. Other
than using many studies (or centers), there is no remedy for this problem.
However, the larger confidence interval of the random effects model again
seems to be more appropriate and conservative than the fixed effects
approach. Actually, presentation of the simple range of study (or center)
results may be the most appropriate way to deal with this sampling problem.
That way having more samples toward one end of the range does not bias the
result. But of course, an extreme outlier can shift the range. Thus,
outliers must be examined carefully for explanations of the heterogeneity.
The bottom line is that, even with the more conservative random effects
method, the results of a meta-analysis (or a multi-center study) will be
biased to the extent that the studies (or centers) are not representative.
David L. Doggett, Ph.D.
Senior Medical Research Analyst
Health Technology Assessment and Information Services
ECRI, a non-profit health services research organization
5200 Butler Pike
Plymouth Meeting, Pennsylvania 19462, U.S.A.
Phone: (610) 825-6000 x5509
FAX: (610) 834-1275
e-mail: [log in to unmask]
From: Simon, Steve, PhD [mailto:[log in to unmask]]
Sent: Monday, August 20, 2001 7:57 PM
To: [log in to unmask]
Subject: Re: Fixed effects versus random effects models - idiot's guide?
Andrew Booth writes:
>Does anyone have a simple explanation or "rule of thumb" for explaining to
>when a fixed effects or a random effects method should be used for a
There is a fair amount of controversy about this. I like to think of a
meta-analysis as a multi-center trial where each center uses a different
protocol. Since the multi-center trial requires random effects, so should a
The controversy occurs because many times there will be a sharp disagreement
between studies where some of them will cluster at one point and others will
cluster at a different point. This violates the assumption of normality for
the random effects model.
A new trend is to look for trends that might explain the underlying
heterogeneity (e.g., baseline risk) and incorporate these trends into a
model. This sometimes goes by the name of meta-regression.
Some people test for heterogeneity and then choose. I don't like this
approach--if there is little heterogeneity, the random and fixed models will
be close anyway, so why not always choose the random effects model?
I am not an expert in meta-analysis--just an informed consumer. So take my
comments with a grain of salt.
Steve Simon, [log in to unmask], Standard Disclaimer.
STATS: STeve's Attempt to Teach Statistics. http://www.cmh.edu/stats
Watch for a change in servers. On or around June 2001, this page will
move to http://www.childrens-mercy.org/stats