Hello Teresa:
I continue to see many issues with statistical analysis and reporting in the urology subspecialty literature and can relate to your query. Please see two papers on the topic attached that directly address your question I believe.
Ph*
Philipp Dahm, MD, MHSc, FACS
Associate Professor of Urology, Associate Program Director and Director of Clinical Research
Department of Urology
University of Florida, College of Medicine
Box 100227, Room N2-15
Gainesville, FL 32610-0247
Phone: (352) 273-7647
Fax: (352) 392-8846
Email: [log in to unmask]
-----Original Message-----
From: Evidence based health (EBH) [mailto:[log in to unmask]] On Behalf Of Benson, Teresa
Sent: Tuesday, October 14, 2008 5:29 PM
To: [log in to unmask]
Subject: Re: What are common errors in a study's stats?
My apologies, I did not intend to imply that all studies have
statistical errors-- in fact, I assume the vast majority do not. I
should also have clarified that this is but one segment of a larger EBM
training that has been going on for almost three years-- and for these
three years, we have completely avoided statistical topics because of
the nurses' discomfort. Instead, we've told them, "When you get to the
article's statistical methodology, just trust that they've done it
correctly." However, I think it's time to get past this avoidance.
After three years of trainings from external EBM experts, refresher
trainings, and periodically reviewing articles in teams, the nurses are
pretty good at comparing a study's stated a priori objectives to the
results to look for post-hoc "fishing," as well as spotting threats to
internal bias-- randomization, blinding, intervention/performance bias,
confounders, choice of outcome measure, attrition (including
questionable assumptions for ITT analysis), etc. I definitely wouldn't
start on statistical topics without first getting them comfortable with
issues of external validity and clinical relevance, internal validity,
and some basics about stated results: p-values, confidence intervals,
power, and measures of association & outcome (ARR, relative risk, RRR,
NNT, odds ratios, sensitivity/sensitivity, likelihood ratios, predictive
value, etc.) Now that our staff has demonstrated comfort with these
concepts for a couple of years, appraising a study's statistical
methodology is our next logical step.
If you have any further thoughts, I'm all ears. Thanks again,
Teresa Benson
-----Original Message-----
From: Evidence based health (EBH)
[mailto:[log in to unmask]] On Behalf Of William Grant
Sent: Tuesday, October 14, 2008 2:53 PM
To: [log in to unmask]
Subject: Re: What are common errors in a study's stats?
Thanks Dr. Conroy:
Also, Ms. Benson:
With all due respect, not every study is statistically flawed and I hope
you are not suggesting that. All studies have limitations, some of which
are
statistical and some of which are not.
Some studies attempt to address findings that were never a part of the
original intent or the original design. Even statistics won't save the
day
there.
Power deserves discussion in all outcomes: Positive, negative,
non-significant.
You note that your students know how to check p-values and confidence
intervals but do they understand that these two measures are based upon
different assumptions about the underlying data - - frequencies vs
probabilities?
Interpreting the medical literature is not easy. Analysis of
assumptions,
appropriateness of design to the intended question, application of the
protocol, follow-up loss, identification of appropriate outcome
measures,
all have a role. Focusing only on the statistical aspects does not IMHO
do
service to students.
Bill
>>> Ronan Conroy <[log in to unmask]> 10/14/2008 2:40 PM >>>
On 14 Oct 2008, at 18:44, Benson, Teresa wrote:
> From those of you who review/appraise articles regularly, I would
> like to hear what kinds or errors you find most often in the
> statistical methodology. For example, do you find researchers:
> -using parametric tests with skewed data distributions,
Skewed distributions can be analysed using parametric tests.
Generalised linear models, for example, allow this, as do negative
binomial regression and quantile regression. There are plenty of
parametric approaches to such data. They are usually more powerful
than rank-based procedures, and allow the calculation of useful
measures of effect size. Even tests that we think of as nonparametric
calculate parameters, of which the most useful is the Wilcoxon rank
sum test. The parameter behind the test is directly equivalent to the
area under the ROC curve.
>
> -using one-tailed tests when they should have done two-tailed,
I have seen virtually no one-tailed tests ever.
>
> -using multiple t-tests instead of an ANOVA, or
ANOVA does not test any useful hypothesis. You will find that most
people in the area of EBH favour regression models where each term has
an interpretation. We don't want to know that 'there is a
relationship' - we want to know what that relationship is. Shotgun t-
tests have almost died out, at least in my corner of the woods.
>
> -assigning effects as fixed when they should be random effects; and/or
I cannot think of a single erroneous decision in medical practice that
has been based on such a confusion.
Statistical errors are usually not of this sort. They are usually
undetectable in publication, and consist in the selection of which
data to present, or which models to build. Type III errors, in a word.
Ronan Conroy
=================================
[log in to unmask]
Royal College of Surgeons in Ireland
Epidemiology Department,
Beaux Lane House, Dublin 2, Ireland
+353 (0)1 402 2431
+353 (0)87 799 97 95
+353 (0)1 402 2764 (Fax - remember them?)
http://www.flickr.com/photos/ronanconroy/sets/72157601895416740/
P Before printing, think about the environment
|