Greetings, Folks
Some of you may have read, in yesterday's "Sunday Telegraph" Review
section, the article by Robert Matthews about "The great health hoax".
"Many scientific 'breakthroughs' are nothing but mirages based on flawed
research. They result in wasted taxes, false claims for drugs and
damaging health scares. Robert Matthews investigates a global scandal."
"Just why has the scientific community failed to act? The answer lies in
its squeamishness."
"Squeamishness", it turns out, about subjectivity. The article is a plea
for a switch to Bayesian approaches allowing "plausibility" to be
incorprated into the analysis, turning away from P-values and
"frequentist" confidence intervals. The article refers to a number
of studies, in which allegedly effects found to be statistically
significant in trials disappear or are apparently much reduced in routine
clinical practice.
Matthews claims that this is a result of the propensity for P-values and
confidence intervals to produce false results.
"When used to analyse clinical trials, significance testing can easily
double the apparent effectiveness of a new drug, and turn a borderline
result into a 'significant' breakthrough. It can throw up convincing
-- yet utterly spurious -- evidence for links between diseases and any
number of supposed causes."
"Fisher ... chose [the P-value] 0.05 because, he said, it was
'convenient'."
"... vital scientific questions ... are being decided by an entirely
arbitrary standard."
"Bayes's Theorem revealed that before any new finding can be deemed
'significant', a crucial factor must be included: its plausibility."
The American Journal of Public Health, it is claimed, banned P-values over
a two-year period with "a dramatic effect".
"... if scientists abandon significance tests like P-values, many of
their claims would be seen for what they really are: meaningless
aberrations on which taxpayers' money should never have been spent."
"The plain fact is that 70 years ago Ronald Fisher gave scientists a
mathematical machine for turning baloney into breakthroughs, and
flukes into funding. It is time to pull the plug."
These quotes, to give an impression of the main points in the argument.
>From our point of view, two issues arise.
One is the extent of reported degradation, in practice, of treatments
supposedly highly effective in clinical trials. This is a straight EBH
issue.
The second is that (as Matthews does not quite come clean about) a
P-value is the probability of a "false positive", and a confidence
interval has a computable reliability, subject in both cases to
the model being right and provided that the investigator has not gone
on a hunt for SOME test which will yield a significant P-value from the
data. On that basis, 1 in 20 of 0.05 P-values will be false positives,
and 1 in 20 of 95% confidence intervals will mislead. This is not new.
So, on this issue, two questions:
A: Is there any real evidence that false-positive rates exceed those
implied by the P-values?
B: Are the various instances (including those Matthews quotes, for which
you'll have to read the original) of treatments significantly
successful in trials but failures in practice, inflated by what one
may call "publication bias" (a compound of "submission bias" by
investigators and "acceptance bias" by editors)?
This is another straight EBH issue.
(Bayesian approaches are of course no more immune than the others to
randomisation flukes and extreme sampling variations, never mind the
issue of the reliability of the assumed prior distribution; and the
mechanisms of "publication bias" will not go away either.)
Any comments?
Best wishes to all,
Ted.
--------------------------------------------------------------------
E-Mail: (Ted Harding) <[log in to unmask]>
Date: 14-Sep-98 Time: 12:29:28
--------------------------------------------------------------------
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|