Print

Print


The predictive validity of a test has to be assessed for a number of different situations:

1. For population screening.
2. For differential diagnosis
3. For diagnostic confirmation
4. For diagnostic exclusion
5. For predicting outcome with or without treatment (usually when a diagnosis has been confirmed or is probable).

The paper to which Teresa was referring addresses #5 but the discussion seems to be about the issues in general about #1, hence the confusion. I explain some of these principles in the Oxford Handbook of Clinical Diagnosis, especially the final chapter.

Huw
________________________________
From: OWEN DEMPSEY <[log in to unmask]>
Sender: "Evidence based health (EBH)" <[log in to unmask]>
Date: Sun, 8 Feb 2015 22:52:37 +0000
To: <[log in to unmask]>
ReplyTo: OWEN DEMPSEY <[log in to unmask]>
Subject: Re: Genetic tests and Predictive validity

I remain quite intrigued by Teresa's question, and now also by the apparent paucity of response. I would be interested to know if my thinking on this holds water or whether it is disastrously leaky!
Just to summarise:
Teresa said:

"Oh dear!  such a complicated topic.  Going back to the RASTER study on MammaPrint, can you speak to the following clinician rationales and comment?

·         “I’m going to order a MammaPrint test for my patient, because if it shows Low Risk, we can be 97% sure my patient won’t develop metastasis and I can confidently recommend no chemotherapy.”

        “I don’t think MammaPrint would be a good test for me to order for my patient, because even if she is likely to develop metastasis, the MammaPrint test would have a 30% chance of missing that and showing a false-negative Low Risk result instead—potentially misleading.”"

My response was:

"The shift of the low-high risk cut off point to favour low risk is odd.  It seems to favour (as in promote) specificity rather than sensitivity.  This is odd because usually cut-offs have tended to promote medicalisation, as in most screening tests.  But to decide on any particular cut off might be presupposing that there is a piece of knowledge, here called 'risk', that the test can fully characterise or obtain. Maybe it can't. But maybe the testers don't acknowledge that some things aren't necessarily measurable and perhaps never will be.  It would be interesting to know the confidence intervals for the findings of the study, it sounds as though chance may play a role. “

My further thoughts are:

Given a population with a significant (possibly) high prevalence of the target condition (in this case so called high risk of metastases) , then, a high specificity at the expense of sensitivity can be misleading;  whereas specificity would protect the already low risk low prevalence population from false negatives. But in  the face of an important high prevalence diagnosis, (high risk of metastases here), then the low sensitivity will miss, in this example, 30% of those that might benefit from chemotherapy.  This suggests that the cut off between low and high risk , should depend upon what is known of the prevalence of the risks, low and high, in the test population.  The more the prevalence of metastases, then the more sensitive the test should be, and the less specific, the rationale being that the more the prevalence of metastases then the more important it is not to miss them, simply because there are increasing numbers of them and the more forgivable it is to be less specific and to over diagnose those that would not benefit from chemotherapy as there are proportionately less of them.  This is complicated.  The example given suggests that the cut off inordinately favours specificity and I wonder how the authors justified their choice of cut off.

Of course for an individual the risk might be an unknown, if only because studies have not been done to measure risks of having or not having metastases without treatments, i.e. without a non-treatment arm.  In such a case there can be no rationale for the cut off.  However, in the example given it looks as if the population levels of risks have been estimated (but how accurately?) and a decision made for a cut off between low and high risk, but the cut off has for some reason been inordinately skewed towards specificity.  This could be because the tests cut offs were worked out on a low prevalence population but then inappropriately applied to a higher prevalence population. Does this sound possible?
best wishes
Owen
,

On 2 February 2015 at 16:34, OWEN DEMPSEY <[log in to unmask]<mailto:[log in to unmask]>> wrote:
The shift of the low-high risk cut off point to favour low risk is odd.  It seems to favour (as in promote) specificity rather than sensitivity.  This is odd because usually cut-offs have tended to promote medicalisation, as in most screening tests.  But to decide on any particular cut off might be presupposing that there is a piece of knowledge, here called 'risk', that the test can fully characterise or obtain. Maybe it can't. But maybe the testers don't acknowledge that some things aren't necessarily measurable and perhaps never will be.  It would be interesting to know the confidence intervals for the findings of the study, it sounds as though chance may play a role.


On Monday, 2 February 2015, Benson, Teresa <[log in to unmask]<mailto:[log in to unmask]>> wrote:
Oh dear!  such a complicated topic.  Going back to the RASTER study on MammaPrint, can you speak to the following clinician rationales and comment?

•         “I’m going to order a MammaPrint test for my patient, because if it shows Low Risk, we can be 97% sure my patient won’t develop metastasis and I can confidently recommend no chemotherapy.”

•         “I don’t think MammaPrint would be a good test for me to order for my patient, because even if she is likely to develop metastasis, the MammaPrint test would have a 30% chance of missing that and showing a false-negative Low Risk result instead—potentially misleading.”

From: Evidence based health (EBH) [mailto:[log in to unmask]] On Behalf Of Juan Acuna
Sent: Monday, February 02, 2015 9:01 AM
To: [log in to unmask]
Subject: Re: Genetic tests and Predictive validity

All,
Allow me first to apologize for the long email (for those of you who like short emails with more “simple” answers).

Going back to the original question, I think most of the answer lies in different considerations for two different areas: first, “genetic testing” is way too broad to be able to attempt a single answer. “Genetic” is way too broad so multiple considerations must apply but not so relevant to the discussion, I think; Second, as for Dx tests themselves, whether for genetic testing or not, the same basic considerations apply, I think, based on the basic definitions of [and the rationale used when calculating] the different operative characteristics of a given test.

Sensitivity and specificity are the perspectives of the test itself (not really that of the patient): one needs to know who is diseased or not (or to have the gold standard known) to calculate them. They define the test in isolation and are not a very useful tool for clinicians that are struggling with the opposite: uncertainty about the patient’s condition. Whether they change or not with prevalence is a discussion that is irrelevant. What is relevant is that one may use test with known Sn/Sp only after consideration of the possibility of having a disease in the patient.

The discussion about Sn/Sp changing with prevalence has been there forever. Due to the way they are calculated, the study to definitely prove if Sn/Sp change with prevalence (beyond mock calculations and simulations) is probably undoable (or at least I have never seen it). However, if the patient belongs to the same risk population than the one used in study that reports Sn/Sp for a test, the situation is good for us. For some tests (such as ultrasound for the Dx of birth defects) unmeasurable factors dealing with threatened reproducibility (such as the expertise of the operator) make generalization of Sn/Sp from a given study very difficult. Each operator must know his/her own Sn/Sp. For tests with results reported in numerical values that are associated with different probabilities of the disease, cutoff points might be avoided and the use of ROC reports might be far preferable.

PPV/NPV are the perspective of the clinician: patient that needs to be tested (whether the patient is symptomatic or not is irrelevant) under uncertainty of the presence of disease. I am in strong disagreement with Dan: the fact that they change with prevalence is not a good reason to suggest not to reporting them. Much less in the EBM context where the clinician is supposed to actually input the prevalence in the consideration of both the prescription of a test and the application of the results (“to what population does my patient belong?”). Furthermore, changing PPV/NPV values with changing prevalence dictate why some tests, albeit good (from the perspective of Sn/Sp) and used in some populations, should not be used (and turn into very ineffective tests) in some others of different risk.

LASt, P/N LR are derived from Sn/Sp. Thus share the same limitations. However, if I understand that a test is indicated given my patient’s risk for a disease, understanding how the pretest probability changes after a positive result is of great value. Nevertheless, if I used a wrong test (given the inadequate matching of the probability of disease and Sn/Sp of the test) the use of LRs will prove extremely disappointing.

In summary, each indicator, as always, has it use, with benefits and limitations. Not understanding in full the reach of each of those operative characteristics for a test and the diverse nature of my patients is the problem. All this framed in the need we sometimes have as physicians to oversimplify concepts (almost demanding “coking recipes” for everything) when concepts cannot be simple. This is a great problem and what true EBM addresses.

JA

Juan M. Acuña M.D., MSc., FACOG.
Chair, Department of Medical and Health Sciences Research
Associate Professor of OBGYN, Human and Molecular Genetics and Clinical Epidemiology,
Herbert Wertheim College  of Medicine
Florida International University

11200 SW 8th Street
AHC1-445
Miami, FL 33199
Ph (305) 348-0676



From: Evidence based health (EBH) [mailto:[log in to unmask]] On Behalf Of Raatz Heike
Sent: Monday, February 02, 2015 5:08 AM
To: [log in to unmask]
Subject: Re: Genetic tests and Predictive validity

Dear all,

Question is though: is the genetic test you want to evaluate actually used as a diagnostic test? From what I understand from the case mentioned by Teresa she is interested not in whether a genetic test accurately recognizes whether you have a certain genotype but whether you will in the future develop a certain phenotype. So you are not trying to find out whether the patient currently suffers from a condition but the risk of developing a condition in the future. Now unless you have a dominant gene that will always lead to the expression of a certain phenotype (like Huntingtons) you need to consider whether that genotype is not just one of many factors that can lead to a certain condition. For the examples mentioned like Mammaprint prognostic modelling seems much more appropriate to me than diagnostic accuracy though ultimately you need RCTs to prove that they improve patient reported outcomes and from what I saw last those don’t exist.

Best wishes, Heike


Heike Raatz, MD, MSc
Basel Institute for Clinical Epidemiology and Biostatistics
Hebelstr. 10
4031 Basel
Tel.: +41 61 265 31 07
Fax: +41 61 265 31 09
E-Mail: [log in to unmask]

From: Evidence based health (EBH) [mailto:[log in to unmask]] On Behalf Of Majid Artus
Sent: Monday, February 02, 2015 10:04 AM
To: [log in to unmask]
Subject: Re: Genetic tests and Predictive validity

Dear All,
Is it, or is it not, correct that one should follow the classic teaching that (loosely and notwithstanding the false partition here): from patient's perspective, sensitivity/specificity are what is relevant; and from clinician's perspective, PPV/NPPV are what is relevant?

Also, it is, isn't it, crucial to consider the media take on outcome of research and how careful researchers need to be in selecting the way they present the outcome of their research? MRI (NMR) in diagnosing autism is one example that springs to mind - the high sensitivity was jumped on by the media presenting it as a very accurate test missing the role of the varying prevalence in certain settings.
I find this discussion trail hugely thought provoking!
Best Regards
Majid

On 2 February 2015 at 00:19, Mark V Johnston <[log in to unmask]> wrote:
At the same time, don’t we need to know whether  the  patient  probably  has or does not have the  condition  of  interest?   Yes, prevalence and  other  factors  affect PPV and  NPV, but in my  opinion we  need to move away from the  oversimplified notion that test interpretation depends on  a single factor.


From: Evidence based health (EBH) [mailto:[log in to unmask]] On Behalf Of Mayer, Dan
Sent: Friday, January 30, 2015 8:08 PM
To: [log in to unmask]
Subject: Re: Genetic tests and Predictive validity


Hi Teresa,



You are absolutely correct.  This is why we should demand that diagnostic studies ONLY present the results of Sensitivity, Specificity and Likelihood ratios.



This issue has been a serious problem for many years and it is about time that more people spoke up about it.  Also, journal editors and peer reviewers should be up in arms against the practice of reporting PPV and NPV.



Best wishes

Dan

________________________________
From: Evidence based health (EBH) [[log in to unmask]] on behalf of Benson, Teresa [[log in to unmask]]
Sent: Friday, January 30, 2015 11:59 AM
To: [log in to unmask]
Subject: Genetic tests and Predictive validity
I’ve just started reading the literature on genetic tests, and noticing how many of them tend to focus on predictive value—that is, if a certain test accurately predicts whether a patient will or won’t get a particular phenotype (condition), the authors suggest the test should be used.  But if we’re deciding whether to order the test in the first place, shouldn’t we be focused on sensitivity and specificity instead, not PPV and NPV?  Predictive value is so heavily dependent on disease prevalence.  For example, if I want to get tested for a disease with a 2% prevalence in people like me, I could just flip a coin and regardless of the outcome, my “Coin Flip Test” would show an NPV of 98%!  So what does NPV alone really tell me, if I’m not also factoring out prevalence—which would be easier done by simply looking at sensitivity and specificity?  Someone please tell me where my thinking has gone awry!
For a concrete example, look at MammaPrint, a test which reports binary results.  In addition to hazard ratios, study authors often tout statistically significant differences between the probabilities of recurrence-free survival in the MammaPrint-High Risk vs. MammaPrint-Low Risk groups (essentially the test’s predictive values).  In the RASTER study (N = 427), 97% of the patients with a “Low Risk” test result did not experience metastasis in the next 5 years.  Sounds great, right?  But when you look at Sensitivity, you see that of the 33 patients in the study who did experience metastasis, only 23 of them were classified as “High Risk” by MammaPrint, for a 70% sensitivity.  If patients and clinicians are looking for a test to inform their decision about adjuvant chemotherapy for early stage breast cancer, wouldn’t the fact that the test missed 10 out of 33 cases be more important than the 97% NPV, an artifact of the extremely low 5-year prevalence of metastasis in this cohort (only 33 out of 427, or  0.7%)?
Drukker et al. A prospective evaluation of a breast cancer prognosis signature in the observational RASTER study. Int J Cancer 2013. 133(4):929-36. http://www.ncbi.nlm.nih.gov/pubmed/23371464<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ncbi.nlm.nih.gov_pubmed_23371464&d=AwMF-g&c=1QsCMERiq7JOmEnKpsSyjg&r=Ip5wwmlafmSkip30kemE5Q&m=GwNY8srJ3CEjlNbLqoSFCeH_n8m0cC6puP4CE3XW1VM&s=2chGNOKM9TfFH_1Hzl6g8am4NJO3LHdB0CIDAH8DaTM&e=>
Retel et al. Prospective cost-effectiveness analysis of genomic profiling in breast cancer. Eur J Cancer 2013. 49:3773-9. http://www.ncbi.nlm.nih.gov/pubmed/23992641<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ncbi.nlm.nih.gov_pubmed_23992641&d=AwMF-g&c=1QsCMERiq7JOmEnKpsSyjg&r=Ip5wwmlafmSkip30kemE5Q&m=GwNY8srJ3CEjlNbLqoSFCeH_n8m0cC6puP4CE3XW1VM&s=dA6wTdKzugoGqf6Fx5fjSEz4CFxLPDhMbyV5Mt-j2ak&e=>  (Provides actual true/false positive/negative results)

Thanks so much!

Teresa Benson, MA, LP
Clinical Lead, Evidence-Based Medicine
McKesson Health Solutions
18211 Yorkshire Ave
Prior Lake, MN  55372
[log in to unmask]
Phone: 1-952-226-4033<tel:1-952-226-4033>

Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.


________________________________
----------------------------------------- CONFIDENTIALITY NOTICE: This email and any attachments may contain confidential information that is protected by law and is for the sole use of the individuals or entities to which it is addressed. If you are not the intended recipient, please notify the sender by replying to this email and destroying all copies of the communication and attachments. Further use, disclosure, copying, distribution of, or reliance upon the contents of this email and attachments is strictly prohibited. To contact Albany Medical Center, or for a copy of our privacy practices, please visit us on the Internet at www.amc.edu<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.amc.edu&d=AwMF-g&c=1QsCMERiq7JOmEnKpsSyjg&r=Ip5wwmlafmSkip30kemE5Q&m=GwNY8srJ3CEjlNbLqoSFCeH_n8m0cC6puP4CE3XW1VM&s=__vs2OCqLIDvEtFDEkRu8CoWSznyDNAcsL389Go27lQ&e=>.



--
Please note my new email address: [log in to unmask]


Dr Majid Artus PhD
NIHR Clinical Lecturer in General Practice
Arthritis Research UK Primary Care
Centre
Research Institute for Primary Care & Health Sciences
Keele University
Staffordshire, ST5 5BG
Tel: 01782 734826
Fax: 01782 733911
http://www.keele.ac.uk/pchs/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.keele.ac.uk_pchs_&d=AwMF-g&c=1QsCMERiq7JOmEnKpsSyjg&r=Ip5wwmlafmSkip30kemE5Q&m=GwNY8srJ3CEjlNbLqoSFCeH_n8m0cC6puP4CE3XW1VM&s=1tHDI4xDX-49yHN21UNpB2IjZeCZqO9J1qExDBWgv4s&e=>


Please consider the environment before printing this email.
This email and its attachments are intended for the above named only and may be confidential. If it has come to you in error you should not copy or show it to anyone; nor should you take any action based on it, other than to notify the sender of the error by replying to the sender. Keele University staff and students are required to abide by the University's conditions of use when sending email. Keele University email is hosted by a cloud provider and may be stored outside of the UK.



--
Owen Dempsey MBBS MSc MRCGP RCGP cert II

07760 164420

GPwsi Substance Misuse Locala and Kirklees Lifeline Addiction Service




--
Owen Dempsey MBBS MSc MRCGP RCGP cert II

07760 164420

GPwsi Substance Misuse Locala and Kirklees Lifeline Addiction Service