Dear Kim,
Usually, whenever there's discussion about optimal cut-offs for ROC curves, the following question is not asked, but should be:
"What 'cost' should I assign to a false positive relative to a false negative?" I.e. are both possible misclassifications equally bad for me in terms of outcome, or is one way of being wrong worse than the other way?
The criterion suggested below would be the optimal cut-off only in the case where both the costs of FPs and FNs are equal. In practice, from a decision science perspective, I think this is rarely the case. (Pascal's Wager being the most extreme example, in which cost(FN) / cost(FP) approaches infinity.)
The geometric solution to differential costing of errors would, I think, effectively involve rescaling the axes of the ROC curve plot by the ratio of costs, and then applying the rule below, which is graphically equivalent to moving the tangent line at 45 degrees from top right to bottom left (or top left to bottom right, depending on what's being plotted), then finding the first point of intersection between the tangent line and the ROC curve.
In general, I think trying to elicit answers to the question of relatively cost of error is a useful exercise for making more effective decisions, and is not asked often enough.
Best wishes,
Jon Minton
University of Glasgow
-----Original Message-----
From: A UK-based worldwide e-mail broadcast system mailing list [mailto:[log in to unmask]] On Behalf Of Kim Pearce
Sent: 11 September 2017 09:55
To: [log in to unmask]
Subject: Receiver Operating Characteristic (ROC) Curves and their uses
Hi everyone,
Can I ask two (hopefully) quick questions which relate to the receiver operating characteristic (ROC) curve method?
1)My first question is in regard to the derivation of the 'optimal' cut off for a clinical test via ROC curves...
Papers and documents which are essentially mathematically based (e.g. McNeil et al. 1975 and SigmaPlot ROC Curve Analysis documentation) focus on derivation of the 'optimal operating position' on the ROC curve 'where the slope of the tangent to the ROC curve' (m) equals
m = (FPC/FNC) x (P(D-))/(P(D+))
Where
P(D+) is the prior/pre-test probability of disease (i.e. prevalence of disease) FPC= average cost associated with false positive diagnosis.
FNC= average cost associated with false negative diagnosis.
The 'cost' can be a financial cost or a health cost.
Now more clinically based papers e.g. Copay et al. (2007), Clarke et al. (2007) [see below for web links] establish a cut off point that "provides equal sensitivity and specificity".
I would appreciate your views on this issue.
2) Say N patients have a quality of life (QoL) score recorded before and after surgery and a score change (QoL post surgery - QoL pre surgery) is subsequently calculated. After surgery, the patients are also asked the following (anchor) question : "has you condition changed?" with the following response options:
Completely recovered
Much improved
Slightly improved
unchanged
Slightly worse
Much worse
Say we wanted to find the change score which differentiates between the 'slightly improved' and 'unchanged' patients using ROC curve analysis. In this case, I assume that we would only use data for the patients in the categories 'slightly improved' and 'unchanged' even though the number of patients would be very much smaller than N?
Alternatively, if we wanted to establish the 5 score changes (cut offs) which differentiate between the following 5 dichotomous groups I would assume that we would use the data for all N patients in each case. Do you agree?
a) (Completely recovered) vs (Much improved, Slightly improved, unchanged, Slightly worse, Much worse)
b) (Completely recovered, Much improved) vs (Slightly improved, unchanged, Slightly worse, Much worse)
c) (Completely recovered, Much improved, Slightly improved) vs (unchanged, Slightly worse, Much worse)
d) (Completely recovered, Much improved, Slightly improved, unchanged) vs (Slightly worse, Much worse)
e) (Completely recovered, Much improved, Slightly improved, unchanged, Slightly worse) vs (Much worse)
Finally, say if we had a three level 'gold standard' measure (hypothetical example):
Definite Vit D deficiency
Probable Vit D deficiency
No Vit D deficiency
and for each of N patients we had one of these measures recorded together with a diagnostic test result.
If we wanted to use a ROC curve to determine which test score can be used as a lower bound for definite Vit D deficiency I would think that we would use only those patients in the 'definite Vit D deficiency' and 'probable Vit D deficiency' categories in our ROC analysis. Similarly, if we wanted to use a ROC curve to determine which test score can be used as a lower bound for probable Vit D deficiency, I would think that we would use only those patients in the 'probable Vit D deficiency' and 'No Vit D deficiency' categories in our ROC analysis. Do you agree?
Many thanks in advance for your views on these queries.
Best Wishes,
Kim
PS References are below
https://www.researchgate.net/profile/Barbara_Mcneil/publication/22346698_Primer_on_Certain_Elements_of_Medical_Decision_Making/links/56df105108aec4b3333b663e/Primer-on-Certain-Elements-of-Medical-Decision-Making.pdf
http://www.sigmaplot.co.uk/splot/products/sigmaplot/productuses/prod-uses42.php
https://www.researchgate.net/publication/6379627_Understanding_the_Minimum_Clinically_Important_Difference_A_Review_of_Concepts_and_Methods
http://clinchem.aaccjnls.org/content/clinchem/53/5/963.full.pdf
Dr Kim Pearce PhD, CStat, Fellow HEA
Senior Statistician
Haematological Sciences
Room MG261
Institute of Cellular Medicine
William Leech Building
Medical School
Newcastle University
Framlington Place
Newcastle upon Tyne
NE2 4HH
Tel: (0044) (0)191 208 8142
You may leave the list at any time by sending the command
SIGNOFF allstat
to [log in to unmask], leaving the subject line blank.
You may leave the list at any time by sending the command
SIGNOFF allstat
to [log in to unmask], leaving the subject line blank.
|
