Thank for raising these interesting points about the problems associated with estimating post-test probabilities from pre-test probabilities. Instead of reasoning using simple Bayes rule with likelihood ratios based on those ‘with a diagnosis’ and ‘without a diagnosis’, they reason with lists of differential diagnosis based on the extended form of Bayes rule. For example, instead of ‘appendicitis’ or ‘no appendicitis’, they consider appendicitis or cholecystitis, salpingitis, mesenteric adenitis, ‘non-specific abdominal pain’ etc and use ratios of their ‘sensitivities’ (e.g. “guarding is common in appendicitis but less common NSAP”) to estimate probabilities. I explain this in the Oxford Handbook of Clinical Diagnosis. In contrast, Bayes rule and multiplying pre-test probabilities several times with likelihood ratios often gives wildly over-confident probabilities (e.g. 0.999, when 75% are correct). Perhaps the real answer is that it is Bayes rule with the independence assumption that is no good at estimating disease or event probabilities (not physicians)! The mistake may be assuming that the calculated probabilities are 'correct' and that any probabilities that differ from these are 'incorrect'. I would be grateful therefore if you could point out in your references a comparison of the calibration curves to assess the accuracy of probabilities generated using pre / post test probabilities using multiple products of likelihood ratios compared to the curves of physicians’ estimates of probabilities.
Huw Llewelyn MD FRCP
Consultant Physician in endocrinology, acute and internal medicine
Honorary Fellow in Mathematics, Aberystwyth University
Thank you very much, Roy, for your excellent comment!
Yes, I'm interested by a few references!
Best wishes,The simple answer is that physicians are not good at estimating disease or event probabilities. There is a large literature on this, going back to the 1970s.
This is the Achilles heel of the attempt to promote rational decision making based on simple mathematical models. It is not that there is doubt about Bayes Therorem. There should be lots of doubt about the data plugged into it, though.
Cognitive psychologists have been studying human limitations in making judgments such as probability estimates for even longer, and most of what they have found probably applies to physicians.
It is not clear how physicians actually make such estimates in paticular cases. It could be anything from pure intuition, to pattern recognition, to multivariate processes (one point for this, two for that, etc), to formal Bayesian calculation, use of prediction/ diagnostic rules, etc. (But keep in mind that many such rules do not perform well when applied to new populations.
There are quite a few studies, some of which I did a long time ago, to show that physicians' probabilistic diagnostic or prognostic judgments are not very accurate, and physicians have been shown to be subject to judgment biases, to misuse judgment heuristics, and to rely on non-diagnostic or non-predictive variables and/or fail to take into account predictive or diagnostic variables in specific cases.
If anyone is really interested, I could drag out a host of references, many not so new.
On Fri, Feb 19, 2016 at 10:02 AM, Brown Michael <[log in to unmask]> wrote:
Whether physicians are aware of it or not, they use a Bayesian approach in their daily practice when they estimate the patient's probability of having condition X based on elements of the history and physical (i.e., pretest probability) before ordering any diagnostic tests. If available for condition X, a clinical prediction rule may be used. Although this process is very far from an exact science, it is often good enough to move the clinician's suspicion above the treatment threshold or below the diagnostic threshold (alternative diagnoses considered). Although most of us would like to see things fit a more more exact mathematical formula, it is rare (at least in emergency medicine) to be able to make very precise probability estimates at the individual patient-level.
Mike
Michael Brown, MD, MSc
Professor and Chair, Emergency Medicine
Michigan State University College of Human Medicine
[log in to unmask]
cell: 616-490-0920
On Feb 19, 2016, at 5:47 AM, Kevin Galbraith <[log in to unmask]> wrote:
> Hi there
>
> Can anyone advise: when calculating post-test probability of a diagnosis using the likelihood ratio for a diagnostic test, how do we make our best estimate of pre-test probability?
>
> I understand that prevalence is often taken as a pragmatic estimate of pre-test probability. But I assume a patient who presents with symptoms of the condition has, by definition, a pre-test probability that is greater than the prevalence in the wider (or preferably age/sex specific) population.
>
> To estimate pre-test probability, are we reliant on finding an estimate from an epidemiological study whose subjects most closely reflect the characteristics of our individual patient? This would seem a serious limitation to the utility of the Bayesian approach.
>
> Thanks
>
> Kevin Galbraith