Andy,

The answer to your question takes some explaining, but it is not very
complicated.  The statistical reason for avoiding "post hoc" subject
analysis (which is actually a posteriori analysis; "post hoc" comes from
another Latin expression meaning a non-sequitur; but nobody uses it this
way anymore) has to do with the probabilistic basis of p values.  If one
only accepts positive findings with a p value of 0.05 or smaller (the
common criterion), then one will be accepting a false postive finding 5%
of the time.  This does not necessarily mean a particular statistically
significant finding has a 95% probability of being true (that
probability is called the positive predictive value, or the post-test
probability, and requires a more complicated calculation called Bayes'
theorem).  Rather it means that on average one out of twenty
statistically significant trial results will be a false positive finding
by chance, because of the random dispersion of the data; and your
particular result has a 5% or less chance of being one of those
one-in-twenty false positive results.

But the p value criterion is based on the assumption that only one
statistical significance test (called a hypothesis test) is being
carried out. If you now carry out a second stat. sig. test, you have
doubled your chances that one of your two significance test results will
be one of the 1-in-20 false positive stat. sig. results.  To counter
that, you should change your p value criterion by the same factor; so
your criterion for your main sig. test, and your subgroup test should
now be 0.025.  This is called a Bonferroni correction.  That might not
be such a big problem; but usually people don't just do one subgroup
test.  They go wild and test every subgroup and every outcome measure
they can think of, to see if they can come up with something with stat.
sig.  With a Bonferroni correction this quickly reduces your p value
criterion to the vanishing point, so that nothing comes up with stat.
sig., including your primary outcome.  But in publications, people
rarely tell about all the sig. tests they carried out.  They just hold
up the one or few that were significant.  But if they haven't carried
out a Bonferroni correction for all their p value criteria, that is a
fraud, albeit frequently unwitting.

There is a further complication.  Statistical significance tests have
another assumption, that all the tests are on independent (separate)
samples of data.  But your subgroup analysis is on a part of the same
sample of data as your primary outcome.  This might still be okay, if
your subgroup outcome (or even another outcome measure on the whole
primary group) is independent of the primary outcome.  For example, the
primary outcome might be whether the patient has a coronary infarct, and
your secondary outcome might be whether the patient has a hang nail.
This is okay, as long as the Bonferroni correction is carried out.  But
if your secondary outcome is whether the patient has high blood
pressure, then a Bonferroni correction is inappropriate, because these
outcomes are not independent - it is expected that they would tend to
occur together.  Finding stat. sig. for both these outcomes merely
further confirms that the patient has cardiovascular disease.  A
Bonferroni correction would tend to hide that result, and would be
unnecessary and innapropriate (this fact is sometimes unrecognized by
critics of subgroup analysis).  The big problem comes when one does not
know whether two outcomes are independent or not.  If you don't use the
Bonferroni, your sig. test results are misleading.  But if you
inappropriately use the Bonferroni, you are making it too difficult to
detect the underlying phenomenon that is driving both of your outcomes.

The compromise that is generally proposed is to choose a primary outcome
measure or group before any results are known, and to do the
significance test on that data without a Bonferroni correction.  Then
you are free to run secondary significance tests on any other outcomes
or subgroups you desire a posteriori, also without Bonferroni
correction.  But you don't draw firm conclusions on these secondary
tests.  They are used to propose future research.  That is, they give
one clues; but they need to be repeated as a primary hypothesis on a
separate set of data.  This is a bit of a waste of good data.  Also,
many people think it is silly that how the results are interpreted must
be held captive to a decision made on paper in the past.

The above methods are based on what is known as "frequentist" methods of
calculating probabilities.  There is another older method called
Bayesian statistics.  It not only eliminates the need for a Bonferroni
correction for secondary results; but also has the advantage that it
directly gives what everyone wants, which is the probability that a
positive finding is a true positive finding.  The drawback is that the
calculations are complex and reiterative.  Some statisticians use these
methods, but few laymen.

David L. Doggett, Ph.D.
Senior Medical Research Analyst
Health Technology Assessment and Information Services
ECRI, a non-profit health services research organization
5200 Butler Pike
Plymouth Meeting, Pennsylvania 19462, U.S.A.
Phone: (610) 825-6000 x5509
FAX: (610) 834-1275
http://www.ecri.org
e-mail: [log in to unmask]



-----Original Message-----
From: Andy Smith [mailto:[log in to unmask]]
Sent: Wednesday, November 07, 2001 7:09 AM
To: [log in to unmask]
Subject: post hoc subgroup analysis


Hi

Can anyone tell me if there is a statistical reason why post-hoc
subgroup
analysis of subgroup data is less valid?
I can understand the logical point that if you find something you
weren't
expecting it may be less reliable but is there a quantitative expression
of
this idea?

(In simple terms !!)

Keep up the good work

Andy