Thanks to Martin Bland, Rebecca Walwyn, Graham Wetherill, and John Hughes
for their thoughtfulness and helpful input. My supervisor's question
concerned underpowered phase II studies (question immediately below). I
received reference recommendations through other channels. I would like to
share these at the very bottom.
Brett
P.S. Additional thought or references would be very welcome.
* * *
Brett Larive [log in to unmask]
Dept of Biostatistics/Wb4 phone: 216-444-9925
Cleveland Clinic Foundation fax: 216-445-2781
9500 Euclid Avenue
Cleveland, OH 44195
QUESTION:
1. Frequently we confront clinical investigators who wish to
perform randomized clinical trials using a clinical endpoint as
the primary outcome, but due to funding or logistical constraints
can employ limited sample sizes which are sufficient to detect
only very large effects. The detectable effect size is typically
regarded as not totally outside the realm of possibility, but
larger than is expected and much larger than the minimum clinically
imortant effect. In an attempt to lend respectability to
the trial the investigators tend to use the term "Pilot Study"
to refer to it, as a way of acknowleging the limited power. However,
from our statistical persecptive such studies are not true
pilot studies because there are no concrete plans to follow them
with definative Phase 3 trials with adequate power.
The clinical investigators' logic typically is that if their
study shows "interesting trends", then there may be a reasonable
chance of convincing funding agencies to support a definitive
trial in the future.
There is an extensive statistical literature on Phase II clincial
trials conducted for the explicit purpose of screening interventions
to determine which interventions should be evaluated in Phase III
trials. There is also a literature dealing with preliminary
trials for screening interventions based on surrogate endpoints,
with the idea that interventions shown to effect the surrogate
endpoints would then be investigated in full scale trials. There is
also a literature on the conduct of true pilot studies, defined as
preliminary trials to evaluate logistical issues regarding the conduct
of a planned full-scale phase III trial. But the scenario we are
concerned with does not appear to fall into any of these 3 categories
because there is no clear linkage between the conduct of the
proposed trial with a future adquately powered phase 3 trial.
To our knowlege, the statistical literature refers to such trials
as underpowered Phase 3 studies and strongly discourages
their conduct. But given the high frequency of such "low-powered"
Phase 3 trials in spite of the best efforts of statisticians, we
would like to investigate what statistical approaches may have
been developed that may be appropriate to guide their design and
analysis. Clearly, bayesian approaches might be considered. We
would be interested in the perspective of others regarding this
issue.
To: Brett Larive <[log in to unmask]>
Subject: Re: QUERY: alternatives to underpowered Phase 3 studies
From: [log in to unmask]
Date: Fri, 20 Feb 2004 15:50:47 +0000
Dear Brett,
I would be very interested in any responses you receive as it is a topic
that ties in very closely with my own work. I have not been working in
this area long but my current impression is that psychiatry, or mental
health more generally, is one of the hot spots for underpowered phase 3
trials. A few discussions have led me to wonder whether part of the reason
for this is the lack of a well accepted route to conducting definitive
randomised trials in this area. Many of the interventions are complex and
I am beginning to become aware of the literature there. Some of this is
starting to impose a structure on this aspect of the problem but I am not
sure that opinion has progressed as far as it needs to to influence how
these trials are funded. In a large proportion of the trials I come across
the interventions are not drug treatments or they involve a combination of
drug and non-drug treatments. It seems to me that investigators are trying
to attach terminology found elsewhere to describe where they think they
are on route to a definitive trial. This seems to have led to different
usages of terminology and some confusion (on my part at least!). My
feeling is that there is a more accepted route to funding a phase III
trial for a drug intervention and that this includes funding for phase I
and phase II trials. In contrast the route to funding a phase III trial of
non-drug interventions seems to be more fuzzy and the Medical Research
Council, UK has recently conducted a review ('Clinical Trials for
Tomorrow') which has highlighted this. I am hoping that when this is
resolved the definitive trials in this area will be a more distinct
category and will be appropriately powered for the purpose. In the
meantime I think investigators see possibilities for funding
non-definitive trials by dressing them up as if they were definitive.
Best wishes
Rebecca
Rebecca Walwyn
Statistician
Clinical Trials Unit
Institute of Psychiatry
103 Denmark Hill
London
SE5 8AZ
England
From: "Graham Wetherill" <[log in to unmask]>
To: "Brett Larive" <[log in to unmask]>
Subject: RE: QUERY: alternatives to underpowered Phase 3 studies
Date: Fri, 20 Feb 2004 17:38:23 -0000
Brett,
I have no magic solutions.
I do not know if sequential trials could solve your problem - perhaps only
if the treatment time is short (i.e. results are known very quickly after
recruitment. Potentially, a 2-stage design could limit expenditure if
there were no hint of interesting results. - This might require a change
in attitude from funding authorities.
Sometimes trials are designed with the "gold standard" as primary, with
less chance of success, where a good primary stands a much better chance
of success.
It would be interesting to calculate the power, average sample size, etc
from the scenario described below (i.e. average total sample size and
average total cost from pilot plus main) - it would require some
assumptions about what would be "interesting trends".
Graham
Date: Sun, 22 Feb 2004 16:11:25 +0000
From: jmhughes <[log in to unmask]>
To: Brett Larive <[log in to unmask]>
Subject: QUERY: alternatives to underpowered Phase 3 studies
Dear Brett,
I must declare an interest. I am a member of an ethical committee but
the views expressed are entirely my own.
It seems to me that the prosposed study is unethical because the
detectable effect is "much larger than the minimum clinically important
effect". Consequently patients are put at risk apparently only for the
purpose of raising "a reasonable chance of convincing funding agencies"
for support. This is not an ethical aim of a clinical trial.
Furthermore if the trial is so low powered any significant difference
found would presumably have arisen by chance.
If there is any value in the proposed treatment its clinical benefit
should be assessed by an adequately powered clinical trial.
John Hughes
Date: Thu, 26 Feb 2004 11:17:41 +0000
From: "Bland, M." <[log in to unmask]>
To: Brett Larive <[log in to unmask]>
Subject: Re: QUERY: alternatives to underpowered Phase 3 studies
I do not think that Bayesian methods will really help. If the study is
not big enough, it is not big enough however you analyse it. I think that
such studies are best termed "exploratory". The aim is not to produce a
definitive answer to a question, but to explore whether answering the
question would be worthwhile. Conventional sample size calculations are
therefore irrelevant. They are designed for the definitive study. I
agree entirely that "pilot" is the wrong word. Pilot studies are small
studies to test data collection etc., not to produce findings of interest
in their own right.
Martin
REFERENCES RECOMMENDED FROM OTHER SOURCES:
Estey, Thall et al. Blood, 99:4343-4349, 2002
Inoue, Thall and Berry, Biometrics 58:823-831, 2002
Thall et al. Biometrika 75:303-310, 1988
Schaid et al. Biometrika 77:5057-513, 1990
Thall, Cook. "Dose-Finding Based on Efficacy-Toxicity Trade-Offs"
Biometrics (to appear)
Martin Posch, Peter Bauer, Werner Brannath (2003), Issues in designing
flexible trials. Statistics in Medicine, Volume 22, Issue 6, Date: 30
March 2003,
Pages: 953-969
P.Y. Liu. "Phase II Selection Designs" from Handbook of Statistics in
Clinical Oncology, John Crowley editor, Marcel Dekker 2001, 082479025
P.Y. Liu, M LeBlanc, M Desai. "Fals positive rates of randomized phase II
designs." Controlled Clinical Trials 1999; 20:343-352
Berry DA (2003). Statistical Innovations in Cancer Research. In
Cancer Medicine e.6. Ch 33, pp 465-478. London: BC Decker. (Ed:
Holland J, Frei T et al.)
Berry DA (2004). Bayesian statistics and the efficiency and ethics of
clinical trials. Statistical Science 18. (To appear.)
Coffey CS and Muller KE. (2003) Properties of Internal Pilots with the
Univariate Approach to Repeated Measures, Statistics in Medicine, 22:
2469-2485.
