Your "double randomness" point is interesting. I don't know if I fully
understand it, but it seems to be related to publication bias. This
bias is caused by people only submitting and publishing positive and
statistically significant results. Thus, for a non-existant phenomenon,
we might only see in print the 1-in-20 result that was by chance a false
positive result. In subgroup analysis, we might also see the 1-in-20
subgroup result that is by chance a false positive result. But by
cherrypicking that one subgroup to hold up and test for significance, we
don't know what other subgroup results could have been tested.
Therefore, we cannot know what the Bonferroni correction should be.
This cherrypicking is then not a systematic testing of all results
available, but is a selective testing that is driven by the chance
occurance of an apparent positive finding for that subgroup (that may in
reality be a false positive finding). This selection by chance I
believe is the second level of randomness that your discussion refers
David L. Doggett, Ph.D.
Senior Medical Research Analyst
Health Technology Assessment and Information Services
ECRI, a non-profit health services research organization
5200 Butler Pike
Plymouth Meeting, Pennsylvania 19462, U.S.A.
Phone: (610) 825-6000 x5509
FAX: (610) 834-1275
e-mail: [log in to unmask]
From: Montori, Victor M., M.D. [mailto:[log in to unmask]]
Sent: Wednesday, November 07, 2001 11:35 AM
To: [log in to unmask]
Subject: Re: post hoc subgroup analysis
At one point I read a discussion in a cardiology journal about this that
went at it a different way. The statement there was that results of
clinical trials are the subject of treatment effects plus some random
component. The statistical tests we use take that into account. If you
an intriguing result from the study (a random event) and that motivates
post hoc subgroup analysis, your application of a statistical test to
random event (and that subgroup analysis) is in itself random (in other
words, if conducting the study again you do not find the intriguing
you would not perform the subgroup analysis - thus the subgroup analysis
a random event in itself). Our usual tests cannot acccount for this
"double-randomness". Can other people out there (more versed in stats
I am) make any sense of this? It sticked in my mind for some reason...
Bonferroni discussion, the Bayesian approach, and the qualitative
in the very useful classic paper by Guyatt and Oxman on this topic in
of Internal Medicine are also good ways of going at it.