On 2/25/2017 5:10 AM, Jon Brassey wrote:
> I'm helping a student with a piece of work and he described
systematic reviews
> as being reliable. I'm troubled by this on a number of fronts.
Your student, I suspect, is using the term "reliable" in a generic sense
meaning "trustworthy" or "persuasive." The real question, I suspect, is
not whether systematic reviews are reliable, but rather are they valid?
There are a few empirical studies that could possibly help resolve the
question of validity.
Jacques LeLorier, Genevieve Gregoire, Abdeltif Benhaddad, Julie
Lapierre, Francois Derderian. Discrepancies between Meta-Analyses and
Subsequent Large Randomized, Controlled Trials. N Engl J Med.
1997;337(8):536-542.
What these authors did was compare a large randomized trial to an
earlier published meta-analysis. The results were so-so, but read the
article yourself and decide.
Ian Shrier, Jean-Francois Boivin, Robert Platt, et al. The
interpretation of systematic reviews with meta-analyses: an objective or
subjective process? BMC Medical Informatics and Decision Making.
2008;8(1):19.
This study presented a series of meta-analyses and found that different
people had different interpretations of identical results.
Laura Rosen, Michal Ben Noach, Elliot Rosenberg. Missing the forest
(plot) for the trees? A critique of the systematic review in tobacco
control. BMC Medical Research Methodology. 2010;10(1):34.
This study looked at how much inconsistency there was in how different
researchers conducted their meta-analyses and found that there were a
lot of problems.
These empirical studies are not such a great endorsement, perhaps, but
you have to put things in perspective. Compared to what preceded it, the
expert editorial, a systematic overview is a vast improvement. Nothing
makes this point better than the following satirical article.
A Caveman. The invited review - or, my field, from my standpoint,
written by me using only my data and my ideas, and citing only my
publications. J Cell Sci. 2000;113(18):3125-3126.
Does the systematic overview deserve a high spot in the hierarchy of
evidence? You could argue that a systematic overview is no better than
an observational study. One analogy that I cite is that a systematic
overview is like a multi-center study where every center gets to choose
its own protocol.
But another argument is that a systematic overview, by synthesizing
multiple clinical trials, has to be better than any single clinical
trial. If you have multiple clinical trials with potentially conflicting
interpretations, how do you handle this in an evidence based fashion
without a systematic overview? Do you chose the single study that has
the best quality score or the one with the largest sample size and
ignore everything else? That seems like a recipe for disaster.
Systematic overviews also should have better generalizability. If you
find ten studies, each conducted in different geographical location,
and they all say more or less the same thing, then you have better proof
that the research will generalize to your geogrpahic setting than you
would with any single study.
Another big advantage of systematic overviews is that they represent a
repeatable and reproducible form of research. If you lay out your
methods carefully, anyone else could repeat your work and get the same
results. They can tweak your analysis if they don't like your search
strategy or your inclusion/exclusion criteria, or your method of pooling
results.
Systematic overviews are also self-critical. If there is a limitation
due to publication bias, heterogeneity, or lack of good quality studies,
the tools of a systematic overview will identify those limitations.
Finally, most of the empirical evidence on the value of blinding,
concealed allocation, and other aspects of clinical trials comes from
systematic overviews. If you say that you believe in blinding but you
don't believe in systematic overviews, then your belief in blinding must
be faith based and not evidence based.
Most statisticians that I know put very little faith in systematic
overviews, but I am generally positive on them, especially on the
systematic overviews conducted by the Cochrane Collaboration. They're
not perfect, but nothing in evidence based medicine is perfect.
--
Steve Simon, P.Mean Consulting
I'm blogging now! blog.pmean.com
|