Continuing the theme from the recent academic article (Hurlbert,
S.H., Lombardi, C.M. (2009) Final collapse of the Neyman-Pearson
decision-theoretic framework and the rise of the neoFisherian. Annales
Zoologici Fennici (http://www.sekj.org/PDF/anz46-free/anz46-311.pdf - no
printing/editing allowed), 46, 5, 311-349.) …
Take a look at an essay (it’s free access) entitled: Odds
are: It’s wrong. Science fails to face the shortcomings of statistics.
http://www.sciencenews.org/view/feature/id/57091/title/Odds_Are%2C_Its_Wrong
in the weekly magazine – Science News.
But, look also at this bit on meta analysis in the same article:
“More recently, epidemiologist Charles Hennekens and
biostatistician David DeMets have pointed out that combining small studies in a
meta-analysis is not a good substitute for a single trial sufficiently large to
test a given question. “Meta-analyses can reduce the role of chance in
the interpretation but may introduce bias and confounding,” Hennekens and
DeMets write in the Dec. 2 Journal of the American Medical Association.
“Such results should be considered more as hypothesis formulating than as
hypothesis testing.”
This was a nice 2-pager outlining, again, the disparity between
meta-analytic and large-sample evidence – and a series of short
commentaries with a reply from the author:
Hennekens, C.H., & DeMets, D. (2009) The need for
large-scale randomized evidence without undue emphasis on small trials,
meta-analyses, or subgroup analyses. Journal of the American Medical Association,
302, 21, 2361-2362.
Commentaries:
Hennekens, C.H., DeMets, D., Bolland, M.J., Grey, A., Read, I.,
Vosk, A. &, Sacristan, J.A. (2010) Commentaries and reply
to the Hennekens and DeMets Commentary on Meta-Analysis. Journal of the
American Medical Association, 303, 13, 1253-1255.
The question for the users and digesters of meta-analysis in
psychology is whether these results apply to studies of psychological
attributes.
An older paper by Ioannidis, J.P.A., Cappelleri, J.C. &, Lau,
J. (1998) Issues in comparisons between meta-analyses and large trials. Journal
of the American Medical Association, 279, 14, 1089-1093 is perhaps worth
reading ..
Abstract
Context.—The extent of concordance between
meta-analyses and large trials on the same topic has been investigated with
different protocols. Inconsistent conclusions created confusion regarding the
validity of these major tools of clinical evidence.
Objective.—To evaluate protocols comparing meta-analyses and large
trials in order to understand if and why they disagree on the concordance of
these 2 clinical research methods.
Design.—Systematic comparison of protocol designs, study
selection, definitions of agreement, analysis methods, and reported
discrepancies between large trials and meta-analyses. Results.—More
discrepancies were claimed when large trials were selected from influential
journals (which may prefer trials disagreeing with prior evidence) than from
already performed meta-analyses (which may target homogeneous trials) and when
both primary and secondary (rather than only primary) end points were
considered. Depending on how agreement was defined, kappa coefficients varied
from 0.22 (low agreement) to 0.72 (excellent agreement). The correlation of
treatment effects between large trials and meta-analyses varied from -0.12 to
0.76, but was more similar (0.50-0.76) when only primary end points were
considered. When both the magnitude and uncertainty of treatment effects were
considered, large trials disagreed with meta-analyses 10% to 23% of the time.
Discrepancies were attributed to different disease risks, variable protocols,
quality, and publication bias.
Conclusions.—Comparisons of large trials with meta-analyses may reach
different conclusions depending on how trials and meta-analyses are selected
and how end points and agreement are defined. Scrutiny of these 2 major
research methods can enhance our appreciation of both for guiding medical
practice.
I have copies of these if anyone is interested.
Regards .. Paul
M: +64-(0)21-415625