Benson, Teresa wrote:
> From those of you who review/appraise articles regularly, I would like
> to hear what kinds or errors you find most often in the statistical
> methodology. For example, do you find researchers:
> -using parametric tests with skewed data distributions,
> -using one-tailed tests when they should have done two-tailed,
> -using multiple t-tests instead of an ANOVA, or
> -assigning effects as fixed when they should be random effects; and/or
> What other errors have you seen in real-world articles?
>
> I will be training nurses to critically appraise the statistical
> methodology sections of articles, and since their time and knowledge of
> statistics are limited, I hope to focus mainly on errors they are likely
> to find in real articles. (They do already know to check p-values and
> confidence intervals when significant findings are reported, and power
> in the case of nonsignificant findings.) Your feedback would help me be
> most efficient and effective, and would be greatly appreciated.
> Additionally, if you know of any articles that directly address the
> error rate of these things in published studies, I’d find that useful to
> motivate the nurses. Thanks in advance,
With all due respect, I would suggest that you discourage the critical
appraisal of statistical analysis choices. There are several reasons for
this:
1. It takes a lot of experience to spot problems with a statistical
analysis. Even things that look simple superficially, such as when to
use a mean and when to use a median, can have a lot of subtle distinctions.
2. There's a lot of people out there who know one approach to data
analysis very well, and they tend to apply that approach to everything
they see. One person, for example, was an expert in structural equations
modeling (SEM), and he told me (quite sincerely) that he didn't
understand why every paper didn't use this approach. SEM can account for
measurement error in your independent variables and didn't every
research problem suffer from measurement error? It's like the old
saying, when your only tool is a hammer, everything looks like a nail.
3. Nine times out of ten (I apologize that this is a subjective
estimate, but it is one that I believe in, and I think that I've seen
others cite this rate), if there is a problem with a paper, it is with
how the data was collected and not with how it was analyzed.
That being said, I would encourage your students to look at things like,
1. was there a good control group,
2. how were dropouts, exclusions, and non-compliers handled,
3. did they measure the right outcome variable.
These are aspects of the statistical DESIGN and not the statistical
ANALYSIS.
But if you want to talk about problems with statistical analysis, the
two big problems are
1. inadequately (sometimes grossly inadequate) sample sizes,
There are lots of references about this, here are a few in my files:
Why Have Recent Trials of Neuroprotective Agents in Head Injury Failed
to Show Convincing Efficacy? A Pragmatic Analysis and Theoretical
Considerations. Andrew I.R. Maas, Ewout W. Steyerberg, Gordon D. Murray,
Ross Bullock, Alexander Baethmann, Lawrence F. Marshall, Graham M.
Teasdale. Neurosurgery 1999: 44(6); 1286-1298.
Ethics and sample size P. Bacchetti, L. E. Wolf, M. R. Segal, C. E.
McCulloch. Am J Epidemiol 2005: 161(2); 105-10.
http://aje.oxfordjournals.org/cgi/content/full/161/2/105
2. failure to include confidence intervals and a discussion of clinical
importance.
Here are a few references:
How well is the clinical importance of study results reported? An
assessment of randomized controlled trials. Chan KB, Man-Son-Hing M,
Molnar FJ, Laupacis A. Cmaj 2001: 165(9); 1197-202.
Is 3-mm Less Drowsiness Important? Portnoy JM, Simon SD. Annals of
Allergy, Asthma and Immunology 2003: 91(4); 324-325.
Clinical vs Statistical Significance. Hopkins WG, Sportscience.
www.sportsci.org/jour/0103/inbrief.htm
--
Steve Simon, Standard Disclaimer.
My new website, www.pmean.com, is up and running!
Sign up for my brand new newsletter,
The Monthly Mean, at www.pmean.com/news
|