I just finished reading Alvan Feinstein's great article about this
called:
The problem of cogent subgroups: A clinicostatistical tragedy. J Clin
Epidemiol, Vol 51, No 4, pp 297-99, 1998
He makes the excellent point that there is a difference between the
clinical interpretation study results of a priori subgroups and the
statistical interpretation the results of post hoc subgroups. This
really get to the crux of the issue.
Best wishes,
Dan
****************************************************************************
Dan Mayer, MD
Professor of Emergency Medicine
Albany Medical College
47 New Scotland Ave.
Albany, NY, 12208
Ph; 518-262-6180
FAX; 518-262-5029
E-mail; [log in to unmask]
****************************************************************************
>>> Stephanie Chan <[log in to unmask]> 11/24/08 6:02 PM >>>
Hello all,
I am giving a talk to my housestaff tomorrow on the pitfalls of subgroup
analysis. Although I could find many examples of flawed subgroup
analysis,
I would like to give a good example of a subgroup analysis that showed
benefit in a subgroup, while there was no benefit in the overall
population,
and this subgroup analysis was subsequently verified by other studies.
Does
anyone have a good example I could use?
Thanks!
Stephanie Chan, M.D.
On 11/21/08, McCormack James <[log in to unmask]> wrote:
>
> Dear JUPITER readers:
>
> I just thought I would add my two cents/pence - insert your own
currency if
> you would like - to the JUPITER evaluation.
>
>
> 1) Two studies have shown CRP does NOT improve our ability to make an
> estimate of CVD risk
> NEJM 2006;355:2631-9 Arch Intern Med 2006;166:1368-73
>
>
> 2) If you go to the CRP calculator website developed by people who
> "invented" CRP you can see how small a change knowing CRP makes on
risk
> assessment. This is important because a number of clinicians are
saying if
> one is at intermediate risk (10-20% risk over 10 years) - a very
arbitrary
> breakpoint - then measure CRP and if it is elevated then you should be
> considered "high risk". Plug in factors that would give a patient a
15% risk
> - 60 y/o male, SBP = 160,
> Total chol = 5 (200), HDL =1(40), CRP 2 mg/L. Then change the CRP to
> "high" - say 8 mg/dl - then the risk estimate goes to 17% - then make
it low
> - say 0.5 mg/L "low" then the risk estimate is 13%. So not much of
change
> especially considering the initial confidence intervals around the 15%
are
> at least +/- 2-3%. Even if it was completely accurate would this
difference
> change your decision to use a statin?
>
>
> 3) I would like to suggest an "emotional", grounded in evidence,
approach
> to statin use for primary prevention - comments would be appreciated
>
>
> a) I would tell patients, if you are around 65 and don't have a lot of
risk
> factors (normal BP, good family history etc) then if you took a medium
dose
> statin you would get roughly a 0.5 to 1.5% absolute reduction in CVD
over
> around 5 years- if you would like to take a statin great, if not great
>
>
> b) If you have a number of risk factors then the benefit is roughly
around
> 1.5% to 3% over around 5 years - if you would like to take a statin
great,
> if not great
>
>
> c) then, and most importantly, never measure or think about your
> cholesterol or CRP again
>
>
> If anyone is interested, I and a family medicine colleague have
recorded a
> podcast on this - you can get it through the iTunes store (type in
> Therapeutics Initiative) or at http://ti.ubc.ca/en/blog/2514
>
> Thanks.
>
>
>
> On 21-Nov, at 2:25 PM, Brian Alper MD wrote:
>
> Thanks. This explanation is helpful and helps us feel better about
the
> validity of this approach in this specific trial.
>
> The context of use of this question in part is what "critical
appraisal
> rules" to apply to early trial termination when assessing the validity
of a
> trial.
>
> Our systematic protocol for DynaMed has us check a number of criteria
> before giving a "level 1 (likely reliable) evidence" label to a
randomized
> trial. Specifically:
>
>
> - *DynaMed** criteria for level 1 (likely reliable) evidence for a
> randomized trial*
> 1. Full-text report available in English (or language well
> understood by participating editor)
> 2. Clinical outcome (also called patient-oriented outcomes)
> 3. Random allocation method (i.e. not assigned by date of birth,
day
> of presentation, "every other")
> 4. Allocation concealed
> 5. Blinding of all persons (patient, treating clinician, outcome
> assessor) if possible
> 6. Intention-to-treat analysis comparing groups according to
> randomization
> 7. Follow-up (endpoint assessment) of at least 80% of study
entrants
> AND adequate such that losses to follow-up could not materially
change
> results
> 8. Adequate statistical power
> 9. No other factors contributing substantial bias, such as
> 1. Differences in management between groups other than the
> intervention being studied
> 2. Differential loss to follow-up
> 3. Posthoc analysis
> 4. Subgroup analysis
> 5. Baseline differences between groups
> 6. Unclear how missing data is accounted for
>
> Criterion #9 allows us to down-grade a trial for unusual factors that
> appear to threaten validity or introduce bias.
>
> But early termination of trials appears to be increasing in frequency.
We
> are considering adding criteria for these situations.
>
> A draft set of additional criteria for level 1 evidence for a
randomized
> trial with early trial termination are:
>
> 1. decision by independent monitoring board without competing
interests
> 2. interim analysis is preplanned
> 3. statistical stopping rule accounts for multiple assessments
(lower p
> value threshold) for early termination for benefit
> 4. outcome driving stopping decision is sufficiently important to
> warrant early termination
>
>
> We'd be interested in your thoughts for critical appraisal of early
> termination decisions.
>
> Brian
> --------------------------------------
> Brian S. Alper, MD, MSPH
> Editor-in-Chief, DynaMed
(www.DynamicMedical.com<http://www.dynamicmedical.com/>
> )
> Medical Director, EBSCO Publishing
> 10 Estes St.
> Ipswich, MA 01938
> office (978) 356-6500 extension 2749
> cell (978) 804-8719
> fax (978) 356-6565
> home (978) 356-3266
> "It only takes a pebble to start an avalanche."
>
> This e-mail and any attached files transmitted are confidential and
solely
> for the use of the intended recipient. It may contain information
which is
> covered by professional or other privilege. If you are neither the
intended
> recipient of this e-mail nor the person responsible for delivering it
to the
> intended recipient, be advised that you have received this e-mail in
error
> and that any use of it is strictly prohibited. Please notify the
sender
> immediately by reply e-mail and then delete from your system. EBSCO
> Industries, Inc., its subsidiaries and divisions, accept no
responsibility
> for any loss or damage suffered by any person arising from the use of
this
> e-mail.
> -----Original Message-----
> From: Evidence based health (EBH) [
>
mailto:[log in to unmask]<[log in to unmask]>]
> On Behalf Of Steve Simon
> Sent: Friday, November 21, 2008 5:12 PM
> To: [log in to unmask]
> Subject: Re: O'Brien-Fleming stopping boundaries in JUPITER
>
> Maskrey Neal wrote:
> >
> > Brian Alper (Ipswich, MA; Dynamed Editor-in Chief and fellow group
> > member) and I have been debating the JUPITER study off line.
> >
> > [material deleted]
> >
> > However, there remain potential concerns for validity because of the
> > early termination. The trial report states "prespecified monitoring
> > plan" with "O'Brien-Fleming stopping boundaries determined by means
of
> > the Lan-DeMets approach". Does anyone you know anything about this
> > statistical technique to determine if this is a valid approach?
>
> It might help to review the O'Brien-Fleming approach first and
contrast
> it with a competing approach by Pocock. Then I'll tackle Lan-DeMets.
>
> You're already aware that if you take multiple looks at the data,
there
> is an increase in the risk of Type I error. It's not much different
than
> examining multiple outcome measures or multiple subgroups, but you
> shouldn't use Bonferroni because the test statistics that are examined
> 1/3 of the way through, 2/3 of the way through and at the end are so
> highly correlated. Bonferroni is inefficient when there is a high
degree
> of correlation.
>
> Stuart Pocock came up with an approach that effectively compared the
> p-value against an adjusted alpha level where the adjusted alpha level
> was constant across each interim evaluation. With two interim analyses
> conducted when 1/3 and 2/3 of the subjects completed the study, the
> adjusted alpha level would be 0.022 rather than 0.05. By means of
> comparison, Bonferroni would use an adjusted alpha level of 0.017.
>
> An alternative approach, proposed by Peter O'Brien and Thomas Fleming,
> uses a smaller alpha level at first and loosens up at further interim
> evaluations. In the same study, O'Brien-Fleming would effectively use
> p-values of 0.0005 at the first interim look, 0.014 at the second
> interim look, and 0.045 at the end of the study.
>
> There are some messy formulas involving the square root of the
fraction
> of patients enrolled at each interim look which I would just as soon
not
> comment on.
>
> Pocock and O'Brien-Fleming are the old-timers in the interim analysis
> world, and represent two mathematical extremes. In time, statisticians
> looking for further prestige and glory generalized these approaches in
> two ways. First, they examined families of interim analysis approaches
> that offered compromises between the Pocock and O'Brien-Fleming
> extremes. Second, they examined what would happen if the interim
> analysis were not evenly spaced with respect to the number of subjects
> completing the study.
>
> Kuang-Kuo Gordan Lan and David DeMets came up with a very simple
> generalization that achieved both of these objectives using an
approach
> involving an alpha spending function.
>
> You can use an linear alpha spending function that behaves much like
> Pocock and a cubic alpha spending function that behaves much like
> O'Brien-Fleming. Powers somewhere between 1 and 3 offer compromises
> between the two approaches.
>
> Now that's way to much detail, and I should have just said that the
> Lan-DeMets alpha spending function provides a common and well accepted
> approach for controlling the Type I error rate when one or more
interim
> analyses are conducted. But I'm hoping that some of you will be
> interested in the historical details.
>
> Disclaimer: I'm not an expert on interim analyses and I relied heavily
> on a classic textbook in this area, Group Sequential Methods with
> Applications to Clinical Trials by Jennison and Turnbull.
> --
> Steve Simon, Standard Disclaimer.
> Sign up for my brand new newsletter,
> The Monthly Mean, at www.pmean.com/news
>
>
> James McCormack, BSc(Pharm), Pharm D
> Professor
> Faculty of Pharmaceutical Sciences
> UBC, Vancouver, Canada
> 604-603-7898
>
>
>
>
>
>
-----------------------------------------
CONFIDENTIALITY NOTICE: This email and any attachments may contain
confidential information that is protected by law and is for the
sole use of the individuals or entities to which it is addressed.
If you are not the intended recipient, please notify the sender by
replying to this email and destroying all copies of the
communication and attachments. Further use, disclosure, copying,
distribution of, or reliance upon the contents of this email and
attachments is strictly prohibited. To contact Albany Medical
Center, or for a copy of our privacy practices, please visit us on
the Internet at www.amc.edu.
|