John,
There may be no neat statistical solution to your multiple outcomes problem.
However, this may not be as big a deal as you might think. As you pointed
out, your three outcomes (symptom reduction, re-hospitalisation, quality of
life) are not independent. That is, if your intervention causes a "health
gain" then all three outcomes would likely change for the better. Multiple
outcome corrections such as the Bonferroni are only appropriate for
independent outcomes. Any statistician who tells you to use such a
correction here, doesn't understand your situation (just tell them your
multiple outcomes are not independent and they will change their advice).
The simplest thing for you to do is to calculate your outcomes in some way
that is easily interpreted clinically, such as the straight difference, or
the risk ratio (or its inverse, the number-needed-to-treat); in other words
the size of the effect. The log odds ratio is a choice statisticians like,
but this is difficult to interpret clinically. Also, calculate the 95%
confidence interval for each outcome. Now you know more than you would have
known by calculating p values. You can judge clinical significance by the
size of the effects, and statistical significance by the size of the
confidence intervals.
A multivariate analysis is more complicated, but will tell you the relative
dependence of each outcome on your intervention. Here your effect size will
be the correlation coefficient r (which will also have a confidence
interval). This method will also allow you to adjust for differences in
patient characteristics between your intervention groups. The p values for
these r's are essentially irrelevant. You are interested in variables
(outcomes or patient characteristics) with the biggest r values, not
necessarily significant p values. The p values will get smaller with no
limit as you increase the study size. The effect sizes will not change much,
but instead will converge to a stable value as you increase the study size.
The most elegant method of all would be a Bayesian analysis, which (unlike p
value hypothesis testing) can calculate overall probabilities for multiple
dependent variables. Unfortunately, you would need to find a Bayesian
statistician to do this analysis for you, as it can be computationally
intensive, and there is no cook-book software available for this.
Don't be a slave to p values and hypothesis testing. These were invented for
other areas of science, and only statisticians really know their hidden
assumptions and limitations. For interpreting the meaning of medical
studies, the size of the effect (showing the clinical significance) is more
important than the statistical significance, provided the confidence limits
are not ridiculous.
David L. Doggett, Ph.D.
Senior Medical Research Analyst
Health Technology Assessment and Information Services
ECRI, a non-profit health services research organization
5200 Butler Pike
Plymouth Meeting, Pennsylvania 19462, U.S.A.
Phone: (610) 825-6000 x5509
FAX: (610) 834-1275
e-mail: [log in to unmask]
-----Original Message-----
From: John Done [mailto:[log in to unmask]]
Sent: Monday, January 29, 2001 1:23 PM
To: [log in to unmask]
Subject: Power problem with multiple outcome measures
Can anyone provide advice/ reference on dealing with the following issue:
In a community based mental health care intervention RCT study there are 3
equally important primary(?) outcomes namely :
symptom reduction, re-hospitalisation, quality of life.
Problem is being able to recruit to the study given limited resources (
financial and proximity to organising centre). If we take one outcome then
we can recruit a sample size which ensures adequate power. However,
statisticians tell us that increasing the number of outcomes requires
increased sample size ( the arguments are similar to those for Bonferroni
correction for multiple comparisons during data analysis, I understand) .
Reducing the number of outcomes meets the needs of the statisticians but
will ensure that research questions important to others remain unanswered.
One possible solution that has occurred to me is that we assume a
multivariate design/analysis ie these are all manifest measures of the
latent variable ' health gain ' . An additional benefit might be that
small, statistically insignificant changes across a range of measures might
hide a significant change on 'health gain'. Surely this is a problem faced
by anyone evaluating complex ( ie multidimensional) healthcare
interventions isn't it ?
Advice on this would be welcome from you methodological gurus out there.
Thanks
Dr. D. John Done,
Dept Psychology,
University of Hertfordshire,
Hatfield,
Herts
AL10 9AB
Tel: 01707 284638
Fax 01707 285073
"Its not about reductionism or antireductionism its about what works" Lewis
Wolpert 1997
|