Ted Harding discusses a critique of p-values that was published in a "Sunday
Telegraph" Review.
It's important to separate several issues here. Some people criticize
p-values because they fail to adequately characterize the magnitude of the
effect seen. This group advocates the use of confidence intervals instead of
p-values, but they are not Bayesians. The American Journal of Public Health
falls into this camp.
A second group criticizes the mindless use of 5% alpha levels. Sometimes a
1% or a 10% alpha level would be more appropriate. I've even seen a well
reasoned argument that for pediatric cancers, where there is such a sharp
limitation on the number of patients that can be studied, an alpha level of
50% (!) works best. The critics of the arbitrary choice of a 5% alpha level
are also not Bayesians.
Apparently the Sunday Telegraph review uses criticisms of the above two
groups to support the philosophy of the Bayesian group, described below.
This is an intellectually dishonest argument in my opinion.
A third group criticizes the entire framework upon which both confidence
intervals and p-values are based. They are the Bayesians and they talk about
things like the posterior probability that the null hypothesis is true. If
you believe that such a probability makes sense, then the p-value is a bad
estimate of this posterior probability.
The Bayesian argument requires a different way of thinking. It is something
that always gives me a headache when I try to understand it, but that is
probably just a limitation of mine. I'm afraid that I'm stuck with the old
paradigm thinking. A lot of people who are smarter than I am are
enthusiastic proponents of the Bayesian philosophy.
Those of us, like myself, who routinely use p-values or confidence intervals
are referred to as "Frequentists". If you hear that term pronounced with a
sneer, you'll know that you are talking about to a Bayesian. <grin>
Ted Harding asks:
>A: Is there any real evidence that false-positive rates exceed those
> implied by the P-values?
First of all, the p-value is not a false positive rate. It is
misinterpretations like this that the Bayesians criticize. The p-value is a
measure of evidence, and the smaller the p-value the more evidence that we
have against the null hypothesis. But to interpret the p-value as a
probability or a rate is generally dangerous. It's a probability conditioned
on the null hypothesis, yes. But too many people reverse the condition and
think that it represents a probability statement ABOUT the null hypothesis.
Second, the only way to estimate a false positive rate is to know what is
true and what is false. This is very tricky and quite often impossible. You
might as well try to answer the question: "How many ineffective drugs in the
United States have been approved by the FDA?" I think that the answer to
this question is unknowable.
Should proponents of EBM be concerned about understanding the Bayesian
philosophy? In my opinion, no.
I think we'll gradually see Bayesian philosophy creep in to the design and
analysis of clinical trials. For example, there are good Bayesian solutions,
I understand, to the tricky issue of early stopping of clinical trials. But
I doubt that we will see a wholesale rejection of both p-values AND
confidence intervals in my lifetime. Too many people like me fail to fully
understand the Bayesian paradigm for this to happen. So from a practical
viewpoint, most of the medical research for the foreseeable future will be
analyzed using the Frequentist paradigm.
Steve Simon, [log in to unmask], Standard Disclaimer.
STATS - Steve's Attempt to Teach Statistics: http://www.cmh.edu/stats
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|