Print

Print


All,
Allow me first to apologize for the long email (for those of you who like short emails with more "simple" answers).

Going back to the original question, I think most of the answer lies in different considerations for two different areas: first, "genetic testing" is way too broad to be able to attempt a single answer. "Genetic" is way too broad so multiple considerations must apply but not so relevant to the discussion, I think; Second, as for Dx tests themselves, whether for genetic testing or not, the same basic considerations apply, I think, based on the basic definitions of [and the rationale used when calculating] the different operative characteristics of a given test.

Sensitivity and specificity are the perspectives of the test itself (not really that of the patient): one needs to know who is diseased or not (or to have the gold standard known) to calculate them. They define the test in isolation and are not a very useful tool for clinicians that are struggling with the opposite: uncertainty about the patient's condition. Whether they change or not with prevalence is a discussion that is irrelevant. What is relevant is that one may use test with known Sn/Sp only after consideration of the possibility of having a disease in the patient.

The discussion about Sn/Sp changing with prevalence has been there forever. Due to the way they are calculated, the study to definitely prove if Sn/Sp change with prevalence (beyond mock calculations and simulations) is probably undoable (or at least I have never seen it). However, if the patient belongs to the same risk population than the one used in study that reports Sn/Sp for a test, the situation is good for us. For some tests (such as ultrasound for the Dx of birth defects) unmeasurable factors dealing with threatened reproducibility (such as the expertise of the operator) make generalization of Sn/Sp from a given study very difficult. Each operator must know his/her own Sn/Sp. For tests with results reported in numerical values that are associated with different probabilities of the disease, cutoff points might be avoided and the use of ROC reports might be far preferable.

PPV/NPV are the perspective of the clinician: patient that needs to be tested (whether the patient is symptomatic or not is irrelevant) under uncertainty of the presence of disease. I am in strong disagreement with Dan: the fact that they change with prevalence is not a good reason to suggest not to reporting them. Much less in the EBM context where the clinician is supposed to actually input the prevalence in the consideration of both the prescription of a test and the application of the results ("to what population does my patient belong?"). Furthermore, changing PPV/NPV values with changing prevalence dictate why some tests, albeit good (from the perspective of Sn/Sp) and used in some populations, should not be used (and turn into very ineffective tests) in some others of different risk.

LASt, P/N LR are derived from Sn/Sp. Thus share the same limitations. However, if I understand that a test is indicated given my patient's risk for a disease, understanding how the pretest probability changes after a positive result is of great value. Nevertheless, if I used a wrong test (given the inadequate matching of the probability of disease and Sn/Sp of the test) the use of LRs will prove extremely disappointing.

In summary, each indicator, as always, has it use, with benefits and limitations. Not understanding in full the reach of each of those operative characteristics for a test and the diverse nature of my patients is the problem. All this framed in the need we sometimes have as physicians to oversimplify concepts (almost demanding "coking recipes" for everything) when concepts cannot be simple. This is a great problem and what true EBM addresses.

JA

Juan M. Acuņa M.D., MSc., FACOG.
Chair, Department of Medical and Health Sciences Research
Associate Professor of OBGYN, Human and Molecular Genetics and Clinical Epidemiology,
Herbert Wertheim College  of Medicine
Florida International University

11200 SW 8th Street
AHC1-445
Miami, FL 33199
Ph (305) 348-0676



From: Evidence based health (EBH) [mailto:[log in to unmask]] On Behalf Of Raatz Heike
Sent: Monday, February 02, 2015 5:08 AM
To: [log in to unmask]
Subject: Re: Genetic tests and Predictive validity

Dear all,

Question is though: is the genetic test you want to evaluate actually used as a diagnostic test? From what I understand from the case mentioned by Teresa she is interested not in whether a genetic test accurately recognizes whether you have a certain genotype but whether you will in the future develop a certain phenotype. So you are not trying to find out whether the patient currently suffers from a condition but the risk of developing a condition in the future. Now unless you have a dominant gene that will always lead to the expression of a certain phenotype (like Huntingtons) you need to consider whether that genotype is not just one of many factors that can lead to a certain condition. For the examples mentioned like Mammaprint prognostic modelling seems much more appropriate to me than diagnostic accuracy though ultimately you need RCTs to prove that they improve patient reported outcomes and from what I saw last those don't exist.

Best wishes, Heike


Heike Raatz, MD, MSc
Basel Institute for Clinical Epidemiology and Biostatistics
Hebelstr. 10
4031 Basel
Tel.: +41 61 265 31 07
Fax: +41 61 265 31 09
E-Mail: [log in to unmask]<mailto:[log in to unmask]>

From: Evidence based health (EBH) [mailto:[log in to unmask]] On Behalf Of Majid Artus
Sent: Monday, February 02, 2015 10:04 AM
To: [log in to unmask]<mailto:[log in to unmask]>
Subject: Re: Genetic tests and Predictive validity

Dear All,
Is it, or is it not, correct that one should follow the classic teaching that (loosely and notwithstanding the false partition here): from patient's perspective, sensitivity/specificity are what is relevant; and from clinician's perspective, PPV/NPPV are what is relevant?

Also, it is, isn't it, crucial to consider the media take on outcome of research and how careful researchers need to be in selecting the way they present the outcome of their research? MRI (NMR) in diagnosing autism is one example that springs to mind - the high sensitivity was jumped on by the media presenting it as a very accurate test missing the role of the varying prevalence in certain settings.
I find this discussion trail hugely thought provoking!
Best Regards
Majid

On 2 February 2015 at 00:19, Mark V Johnston <[log in to unmask]<mailto:[log in to unmask]>> wrote:
At the same time, don't we need to know whether  the  patient  probably  has or does not have the  condition  of  interest?   Yes, prevalence and  other  factors  affect PPV and  NPV, but in my  opinion we  need to move away from the  oversimplified notion that test interpretation depends on  a single factor.


From: Evidence based health (EBH) [mailto:[log in to unmask]<mailto:[log in to unmask]>] On Behalf Of Mayer, Dan
Sent: Friday, January 30, 2015 8:08 PM
To: [log in to unmask]<mailto:[log in to unmask]>
Subject: Re: Genetic tests and Predictive validity


Hi Teresa,



You are absolutely correct.  This is why we should demand that diagnostic studies ONLY present the results of Sensitivity, Specificity and Likelihood ratios.



This issue has been a serious problem for many years and it is about time that more people spoke up about it.  Also, journal editors and peer reviewers should be up in arms against the practice of reporting PPV and NPV.



Best wishes

Dan

________________________________
From: Evidence based health (EBH) [[log in to unmask]<mailto:[log in to unmask]>] on behalf of Benson, Teresa [[log in to unmask]<mailto:[log in to unmask]>]
Sent: Friday, January 30, 2015 11:59 AM
To: [log in to unmask]<mailto:[log in to unmask]>
Subject: Genetic tests and Predictive validity
I've just started reading the literature on genetic tests, and noticing how many of them tend to focus on predictive value-that is, if a certain test accurately predicts whether a patient will or won't get a particular phenotype (condition), the authors suggest the test should be used.  But if we're deciding whether to order the test in the first place, shouldn't we be focused on sensitivity and specificity instead, not PPV and NPV?  Predictive value is so heavily dependent on disease prevalence.  For example, if I want to get tested for a disease with a 2% prevalence in people like me, I could just flip a coin and regardless of the outcome, my "Coin Flip Test" would show an NPV of 98%!  So what does NPV alone really tell me, if I'm not also factoring out prevalence-which would be easier done by simply looking at sensitivity and specificity?  Someone please tell me where my thinking has gone awry!
For a concrete example, look at MammaPrint, a test which reports binary results.  In addition to hazard ratios, study authors often tout statistically significant differences between the probabilities of recurrence-free survival in the MammaPrint-High Risk vs. MammaPrint-Low Risk groups (essentially the test's predictive values).  In the RASTER study (N = 427), 97% of the patients with a "Low Risk" test result did not experience metastasis in the next 5 years.  Sounds great, right?  But when you look at Sensitivity, you see that of the 33 patients in the study who did experience metastasis, only 23 of them were classified as "High Risk" by MammaPrint, for a 70% sensitivity.  If patients and clinicians are looking for a test to inform their decision about adjuvant chemotherapy for early stage breast cancer, wouldn't the fact that the test missed 10 out of 33 cases be more important than the 97% NPV, an artifact of the extremely low 5-year prevalence of metastasis in this cohort (only 33 out of 427, or  0.7%)?
Drukker et al. A prospective evaluation of a breast cancer prognosis signature in the observational RASTER study. Int J Cancer 2013. 133(4):929-36. http://www.ncbi.nlm.nih.gov/pubmed/23371464<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ncbi.nlm.nih.gov_pubmed_23371464&d=AwMF-g&c=1QsCMERiq7JOmEnKpsSyjg&r=Ip5wwmlafmSkip30kemE5Q&m=GwNY8srJ3CEjlNbLqoSFCeH_n8m0cC6puP4CE3XW1VM&s=2chGNOKM9TfFH_1Hzl6g8am4NJO3LHdB0CIDAH8DaTM&e=>
Retel et al. Prospective cost-effectiveness analysis of genomic profiling in breast cancer. Eur J Cancer 2013. 49:3773-9. http://www.ncbi.nlm.nih.gov/pubmed/23992641<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ncbi.nlm.nih.gov_pubmed_23992641&d=AwMF-g&c=1QsCMERiq7JOmEnKpsSyjg&r=Ip5wwmlafmSkip30kemE5Q&m=GwNY8srJ3CEjlNbLqoSFCeH_n8m0cC6puP4CE3XW1VM&s=dA6wTdKzugoGqf6Fx5fjSEz4CFxLPDhMbyV5Mt-j2ak&e=>  (Provides actual true/false positive/negative results)

Thanks so much!

Teresa Benson, MA, LP
Clinical Lead, Evidence-Based Medicine
McKesson Health Solutions
18211 Yorkshire Ave
Prior Lake, MN  55372
[log in to unmask]<mailto:[log in to unmask]>
Phone: 1-952-226-4033<tel:1-952-226-4033>

Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.


________________________________
----------------------------------------- CONFIDENTIALITY NOTICE: This email and any attachments may contain confidential information that is protected by law and is for the sole use of the individuals or entities to which it is addressed. If you are not the intended recipient, please notify the sender by replying to this email and destroying all copies of the communication and attachments. Further use, disclosure, copying, distribution of, or reliance upon the contents of this email and attachments is strictly prohibited. To contact Albany Medical Center, or for a copy of our privacy practices, please visit us on the Internet at www.amc.edu<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.amc.edu&d=AwMF-g&c=1QsCMERiq7JOmEnKpsSyjg&r=Ip5wwmlafmSkip30kemE5Q&m=GwNY8srJ3CEjlNbLqoSFCeH_n8m0cC6puP4CE3XW1VM&s=__vs2OCqLIDvEtFDEkRu8CoWSznyDNAcsL389Go27lQ&e=>.



--
Please note my new email address: [log in to unmask]<mailto:[log in to unmask]>


Dr Majid Artus PhD
NIHR Clinical Lecturer in General Practice
Arthritis Research UK Primary Care
Centre
Research Institute for Primary Care & Health Sciences
Keele University
Staffordshire, ST5 5BG
Tel: 01782 734826
Fax: 01782 733911
http://www.keele.ac.uk/pchs/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.keele.ac.uk_pchs_&d=AwMF-g&c=1QsCMERiq7JOmEnKpsSyjg&r=Ip5wwmlafmSkip30kemE5Q&m=GwNY8srJ3CEjlNbLqoSFCeH_n8m0cC6puP4CE3XW1VM&s=1tHDI4xDX-49yHN21UNpB2IjZeCZqO9J1qExDBWgv4s&e=>


Please consider the environment before printing this email.
This email and its attachments are intended for the above named only and may be confidential. If it has come to you in error you should not copy or show it to anyone; nor should you take any action based on it, other than to notify the sender of the error by replying to the sender. Keele University staff and students are required to abide by the University's conditions of use when sending email. Keele University email is hosted by a cloud provider and may be stored outside of the UK.