Both Ted Harding and Rod Jackson have denied the existence of a
'fundamental flaw' in statistics, and gone on to describe such a flaw
beautifully (or in Ted's case, longitudinally). I agree with Rod, that
statistics offer the best approaches we have at the moment (though
I am worried about the expression 'appropriately weighted', I expect
that would be used by some as an excuse for weighting with
prejudice which I am sure is not what Rod would want). I am not
saying that the flaw in statistics is fatal, merely a weakness. This
weakness leads to Ted's 'paradoxical dilemma', which I hope we
are all working to mitigate.
The problem is exactly as Ted described. Analyses are wide open
to different interpretation. Any technique is likely to be somewhat
ambiguous, but I think we need to get closer to answering
questions for individual patients before we can realise Lynda
Jackson's dream (can I paraphrase that as 'treating patients
according to their gene map, with supporting evidence' ?). The best
we could manage at the moment would be to apply statistical
analyses to results for similar patients. We would need to focus on
a very small sample of patients if we are really going to look at
(small groups of) genes. Using conventional statistics that would
lead to great difficulty achieving sufficient sample size to provide
reliable evidence - even if all the studies are properly designed,
executed and analysed !
We are far enough behind in amassing the evidence for the present
generation of medical technology. Do we really want commercial
interests to thrust unrealistic, unsupportable versions of Linda's
dream (i.e. lacking the evidence) at patients while more responsible
researchers cannot produce reliable evidence to refute
unreasonable claims ? I know history repeats itself but we should
also strive to learn from history, rather than sit back and let those
commercial interests _make sure_ history repeats itself.
In short, we are going to need better techniques for evaluating
evidence. The sooner we can develop them, the better.
Mike McDowall.
P.S. I am particularly glad that Ted's alternative model of the law on
drink driving is not used -
> 11 o'clock on a Saturday night in the town centre, with inebriated
> companions, and you get pushed up.
Many of us take turns to act as taxis for our friends (and what
about taxi drivers generally ?).
> On 22-Jun-00 Mike McDowall wrote:
> > Toby Lipman wrote :
> >> we have
> >> not adequately addressed the translation of population effects into
> >> individual applicability.
> >
> > This is an issue I queried the list about some time ago (to
> > deafening silence). We have developed models for prediction of
> > outcome after stroke but are not comfortable that they can easily be
> > mis-applied to individuals, where they are only really applicable to
> > populations. I am afraid this is a fundamental flaw of statistics in
> > general. When using Evidence from statistical analysis, you only
> > have a probability (often not defined) that the evidence will be
> > applicable in that particular case.
> >
> > So how will we fulfil Lynda Jackson's dream ?
> >
> > Yours sincerely,
> > Mike McDowall (Mr.).
>
> This is an important issue, but I would dispute that it represents "a
> fundamental flaw of statistics in general", at any rate where the
> principles are concerned.
>
> There is almost always a difference between the characteristics
> of a population and those of an individual, and in what is known
> about these, and it leads to a paradoxical dilemma.
>
> I have preached the following on various occasions, and I would like
> to give it an airing now.
>
> The Administrator and the Administratee have different (or potentially
> different) "populations" in mind in questions of chance and
> statistics. Their priorities (OK, utilities if you like) differ, and
> their decisions and their beliefs (if any) can legitimately differ
> radically in the face of the same apparent facts.
>
> A "medical" example (quotes used because simplistic and using
> round-number pseudo-risks) concerns breast cancer screening.
>
> Suppose a test (e.g. mammography) has a 90% chance of giving the
> correct answer (+ or -) if applied to any given subject (this
> chance arising through random events in the operation of the
> test procedure).
>
> Suppose that, amongst the population of women referred for
> mammography, 1 in 20 have the disease (could be even less if GPs get
> paranoid about failing to take proper care).
>
> Then, of those referred, the proportions with a positive test result
> will be 9/200 who have the disease, and 19/200 who do not.
>
> The proportion of those with a positive result who have the disease is
> 9/28, ~= 1/3. So a woman with a positive result probably does not have
> the disease -- says the Administrator.
>
> The woman who gets her own personal positive result may well view the
> matter differently: "They did a test which has 90% chance of getting
> it right, and this test says I have the disease. So I probably have
> it. The numbers of other women who have/don't have it shouldn't have
> anything to do with whether I have it."
>
> Well, you can dispute the last sentence, since she did have her own
> prior (historical) chance of getting it, if only one knew what it was.
> One estimate of that could be the cross-sectional population
> proportion (stratified appropriately by identifiable categories such
> as family history, personal life factors, etc.), to the extent that
> her history is "typical" -- or, in that splendidly deceptive word --
> "representative" of the population in her stratum..
>
> Her own true chance mechanism is the historical line of her own
> exposure to risk factors (including both ancestral events and
> exposures during her own lifetime); and, if one knew these and the
> risk mechanisms, then an estimate of her personal prior probability
> could in principle be made. This one could call a "longitudinal risk
> estimate" personalised for her.
>
> As far as she is aware, she has the _possibility_ of being one
> of those for whom the longitudinal prior risk is high, and who
> is in a position to tell her different? (I think Bayesians might
> wish to intervene here and impose a higher-level prior on this
> question, but that opens another dimension of discussion).
>
> So her last sentence could be interpreted as "why should my
> longitudinal risk be the same as the cross-sectional risk of
> the population of women 'like' me?"
>
> This might be the case if one could make an assumption (what
> in mathematical terms might be called an "ergodic hypothesis")
> that "long-term longitudinal risk equals large-population cross-
> sectional risk". Whether that is legitimate or not is a matter
> of fact according to the situation being considered, and the
> known facts about the particular patient: does her personal
> history "sample the risks" in the same proportions as the
> instantaneous population "samples the risks"?
>
> Otherwise, we have a situation where the Administratee has
> a different reference population for her chances than the
> Administrator (the doctor). In that case, there is no reason
> why they should agree about the "probability" that she has the
> disease, since they are considering different things. (Of course,
> doctors try to incorporate what they happen to know of the patient's
> personal history, and thereby redress the balance between individual
> and collective risk; but even then they may lack the information which
> would help them to interpret what they know in terms of objective
> risk).
>
> They also have different utilities. For the woman, having
> the disease would be a personal disaster and very little else
> would count: if it's at all likely, let's go all the way.
>
> The doctor falls between two stools: one the one hand, having
> concern for the patient and to that extent sharing her
> view; also, having his/her own back to cover and not wanting
> to get a reputation for unduly overlooking real cases.
> On the other hand, the doctor is the decision-maker for whether
> to refer, so is the front-line custodian of NHS and Practice
> resources, and will also feel that responsibility.
>
> And this gets worse as the Administrator gets more elevated:
> a real NHS Administrator may say "we're wasting our resources
> on 2/3 of the cases that this test shows as positive, not to
> mention the 190/200 costly tests with negative results. Let's
> tone down this screening program." This Administrator officially
> can only look at the cross-sectional probability, otherwise
> his job gets out of hand, since it is only cross-sectional
> data that he can at all readily consider.
>
> However, the Administrator is also entitled to take "humane utilities"
> into account: The function of the NHS is supposed to be to relieve
> distress of medical origin, and it could be that ignoring a 1/3 chance
> (cross-sectional) of breast cancer is politically unacceptable. But
> that is a matter of negotiation.
>
> There is an analogous situation in the administration of criminal law.
> There are very few crimes in the English statute book where the
> criminality of others is proof of of your own (the only one that
> springs to mind is Riotous Assembly -- mere presence at a riotous
> assembly makes you guilty by definition). Otherwise, association with
> criminals is not direct evidence of guilt, which has to be proved (in
> criminal law) "beyond reasonable doubt" by evidence directly bearing
> on the individual accused.
>
> Nevertheless, a particular individual who associates with criminals
> has a personal ("longitudinal") probability of becoming involved with
> them in a process leading up to a crime. If this probability could be
> reliably evaluated (and came out to a near-certainty) then it could
> legitimately lead to a conviction, even though there was no objective
> (e.g. forensic, witness) evidence showing that X was indeed there and
> indeed did this and that. Usually, the law of evidence inhibits this
> line of proof because of the uncertainties usually associated with it
> and in order to isolate the jury from prejudicial influence.
>
> The principle of "proof beyond reasonable doubt" is an administrative
> filter whose purpose is to protect the innocent, even if this means
> that a fair few guilty get off. The consequence is that people who are
> "probably guilty" often do not get convicted.
>
> A good example of legal cross-sectional versus longitudinal evidence
> would be the case of drink-driving. Merely being on the road at pub
> closing-time on a Saturday night raises your cross-sectional
> probability of being over the limit. In certain places at certain
> times this could even rise to "much more probable than not". Yet your
> longitudinal probability depends on your own drinking habits (from
> zero for teetotallers to pretty high for some people) as well as on
> what you happened to be doing that evening.
>
> Little of this gets into the law as it stands: You provide a sample,
> breath or blood or urine, which is scientifically analysed and,
> depending on the result, you are charged (and most likely convicted)
> or not. (People do give evidence that they "only had one glass of
> whisky so the breathalyser was probably wrong", but it doesn't usually
> cut much ice).
>
> Yet there is enough error in the laboratory analysis to lead to
> a fairly wide "grey area". This is allowed for, at present, by
> subtracting a margin of error such that a positive outcome is
> very unlikely unless the person really is guilty (in statistical
> terms, this is a pure classical hypothesis test; and as it
> happens this is the same logic as the mammography woman applied
> to herself).
>
> However, could the circumstances of the case (e.g. when and where
> driving and with whom) be used to resolve some cases in the grey area?
> 11 o'clock on a Saturday night in the town centre, with inebriated
> companions, and you get pushed up. 10 am on a Monday morning, alone
> and miles from anywhere, and you get pushed down, maybe. But the law
> ignores this. Nevertheless, the frequency of correct decision (viewed
> cross-sectionally) could be improved by taking it into account. This
> may lead to persistent injustice to one individual whose circumstances
> are regularly unfortunate. The principle of "proof beyond reasonable
> doubt" protects him. But it also allows others to escape justice.
>
> Other legal administrations (no names, no pack-drill) may take
> the view that "we will get all the criminals we can, and never mind
> the sufferings of the innocent". In the limit this can lead to
> prevalent "guilt by association"; in the case of the sample taken from
> the driver, this could mean that you _add_ the margin of error to the
> lab result, just in case by chance it had come out low.
>
> Whichever way it's done, taking circumstances into account amounts to
> adopting a Bayesian philosophy (with or without a numerical
> calculation); and this is also basically the Administrator's approach
> (though, as it happens, not that of administrators of the Criminal Law
> who are Hypothesis testers; in Civil cases, however, it tends to get
> much more "Bayesian" because of the "balance of probabilities"
> approach).
>
> Well, this has gone on long enough, I dare say.
>
> In summary: there are probabilities ("cross-sectional") which apply to
> populations, and probabilities ("longitudinal") which apply to
> individuals. These may be similar or different, in any case depend on
> different "reference sets", and one (usually the "cross-sectional")
> may be known much better than the other (which may not be known at
> all).
>
> Individuals and administrators (at different levels) may legitimately
> vary in how they evaluate probabilities (indeed, they are in fact
> looking at different information even if the data may look the same on
> paper), and may (almost certainly will) have different criteria
> (utilities) when it comes to determining what should be done.
>
> There is a distinction between a "hypothesis testing" philosophy
> (which avoids incorporation of prior probability) and a "Bayesian"
> philosophy (which tries to take it into account).
>
> In the case of the Administrator, who is looking at populations,
> there is a reference "cross-sectional" population within which
> "Bayesian prior frequencies" can be identified. Administrators
> are [closet] Bayesians.
>
> In the case of the individual, the prior probability that should
> really be taken into account depends on their "longitudinal"
> history, and may be unknown: In that case, it can be argued
> (and in fact I believe it is appropriate) that the prior-free
> "hypothesis-testing" philosophy is appropriate.
>
> I believe this is an inevitable dilemma. To some extent it can
> be alleviated if improved knowledge can lead to the two
> sides having a wider basis of common ground in the information
> they use. In any case, it is resolved in practice not "scientifically"
> but by negotiation -- i.e. political means.
>
> Best wishes to all,
> Ted.
>
> --------------------------------------------------------------------
> E-Mail: (Ted Harding) <[log in to unmask]> Fax-to-email: +44
> (0)870 284 7749 Date: 22-Jun-00
> Time: 13:45:52 ------------------------------ XFMail
> ------------------------------
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
