Print

Print


I remain quite intrigued by Teresa's question, and now also by the apparent
paucity of response. I would be interested to know if my thinking on this
holds water or whether it is disastrously leaky!
Just to summarise:
Teresa said:

"Oh dear!  such a complicated topic.  Going back to the RASTER study on
MammaPrint, can you speak to the following clinician rationales and comment?

·         “I’m going to order a MammaPrint test for my patient, because if
it shows Low Risk, we can be 97% sure my patient won’t develop metastasis
and I can confidently recommend no chemotherapy.”

        “I don’t think MammaPrint would be a good test for me to order for
my patient, because even if she is likely to develop metastasis, the
MammaPrint test would have a 30% chance of missing that and showing a
false-negative Low Risk result instead—potentially misleading.”"

My response was:

"The shift of the low-high risk cut off point to favour low risk is odd.
It seems to favour (as in promote) specificity rather than sensitivity.
This is odd because usually cut-offs have tended to promote medicalisation,
as in most screening tests.  But to decide on any particular cut off might
be presupposing that there is a piece of knowledge, here called 'risk',
that the test can fully characterise or obtain. Maybe it can't. But maybe
the testers don't acknowledge that some things aren't necessarily
measurable and perhaps never will be.  It would be interesting to know the
confidence intervals for the findings of the study, it sounds as though
chance may play a role. “


My further thoughts are:

Given a population with a significant (possibly) high prevalence of the
target condition (in this case so called high risk of metastases) , then, a
high specificity at the expense of sensitivity can be misleading;  whereas
specificity would protect the already low risk low prevalence population
from false negatives. But in  the face of an important high prevalence
diagnosis, (high risk of metastases here), then the low sensitivity will
miss, in this example, 30% of those that might benefit from chemotherapy.
This suggests that the cut off between low and high risk , should depend
upon what is known of the prevalence of the risks, low and high, in the
test population.  The more the prevalence of metastases, then the more
sensitive the test should be, and the less specific, the rationale being
that the more the prevalence of metastases then the more important it is
not to miss them, simply because there are increasing numbers of them and
the more forgivable it is to be less specific and to over diagnose those
that would not benefit from chemotherapy as there are proportionately less
of them.  This is complicated.  The example given suggests that the cut off
inordinately favours specificity and I wonder how the authors justified
their choice of cut off.

Of course for an individual the risk might be an unknown, if only because
studies have not been done to measure risks of having or not having
metastases without treatments, i.e. without a non-treatment arm.  In such a
case there can be no rationale for the cut off.  However, in the example
given it looks as if the population levels of risks have been estimated
(but how accurately?) and a decision made for a cut off between low and
high risk, but the cut off has for some reason been inordinately
skewed towards specificity.  This could be because the tests cut offs were
worked out on a low prevalence population but then inappropriately
applied to a higher prevalence population. Does this sound possible?
best wishes
Owen
,

On 2 February 2015 at 16:34, OWEN DEMPSEY <[log in to unmask]> wrote:

> The shift of the low-high risk cut off point to favour low risk is odd.
> It seems to favour (as in promote) specificity rather than sensitivity.
> This is odd because usually cut-offs have tended to promote medicalisation,
> as in most screening tests.  But to decide on any particular cut off might
> be presupposing that there is a piece of knowledge, here called 'risk',
> that the test can fully characterise or obtain. Maybe it can't. But maybe
> the testers don't acknowledge that some things aren't necessarily
> measurable and perhaps never will be.  It would be interesting to know the
> confidence intervals for the findings of the study, it sounds as though
> chance may play a role.
>
>
> On Monday, 2 February 2015, Benson, Teresa <[log in to unmask]>
> wrote:
>
>>  Oh dear!  such a complicated topic.  Going back to the RASTER study on
>> MammaPrint, can you speak to the following clinician rationales and comment?
>>
>> ·         “I’m going to order a MammaPrint test for my patient, because
>> if it shows Low Risk, we can be 97% sure my patient won’t develop
>> metastasis and I can confidently recommend no chemotherapy.”
>>
>> ·         “I don’t think MammaPrint would be a good test for me to order
>> for my patient, because even if she is likely to develop metastasis, the
>> MammaPrint test would have a 30% chance of missing that and showing a
>> false-negative Low Risk result instead—potentially misleading.”
>>
>>
>>
>> *From:* Evidence based health (EBH) [mailto:
>> [log in to unmask]] *On Behalf Of *Juan Acuna
>> *Sent:* Monday, February 02, 2015 9:01 AM
>> *To:* [log in to unmask]
>> *Subject:* Re: Genetic tests and Predictive validity
>>
>>
>>
>> All,
>>
>> Allow me first to apologize for the long email (for those of you who like
>> short emails with more “simple” answers).
>>
>>
>>
>> Going back to the original question, I think most of the answer lies in
>> different considerations for two different areas: first, “genetic testing”
>> is way too broad to be able to attempt a single answer. “Genetic” is way
>> too broad so multiple considerations must apply but not so relevant to the
>> discussion, I think; Second, as for Dx tests themselves, whether for
>> genetic testing or not, the same basic considerations apply, I think, based
>> on the basic definitions of [and the rationale used when calculating] the
>> different operative characteristics of a given test.
>>
>>
>>
>> *Sensitivity and specificity* are the perspectives of the test itself
>> (not really that of the patient): one needs to know who is diseased or not
>> (or to have the gold standard known) to calculate them. They define the
>> test in isolation and are not a very useful tool for clinicians that are
>> struggling with the opposite: uncertainty about the patient’s condition.
>> Whether they change or not with prevalence is a discussion that is
>> irrelevant. What is relevant is that one may use test with known Sn/Sp only
>> after consideration of the possibility of having a disease in the patient.
>>
>>
>>
>> The discussion about Sn/Sp changing with prevalence has been there
>> forever. Due to the way they are calculated, the study to definitely prove
>> if Sn/Sp change with prevalence (beyond mock calculations and simulations)
>> is probably undoable (or at least I have never seen it). However, if the
>> patient belongs to the same risk population than the one used in study that
>> reports Sn/Sp for a test, the situation is good for us. For some tests
>> (such as ultrasound for the Dx of birth defects) unmeasurable factors
>> dealing with threatened reproducibility (such as the expertise of the
>> operator) make generalization of Sn/Sp from a given study very difficult.
>> Each operator must know his/her own Sn/Sp. For tests with results reported
>> in numerical values that are associated with different probabilities of the
>> disease, cutoff points might be avoided and the use of ROC reports might be
>> far preferable.
>>
>>
>>
>> *PPV/NPV* are the perspective of the clinician: patient that needs to be
>> tested (whether the patient is symptomatic or not is irrelevant) under
>> uncertainty of the presence of disease. I am in strong disagreement with
>> Dan: the fact that they change with prevalence is not a good reason to
>> suggest not to reporting them. Much less in the EBM context where the
>> clinician is supposed to actually input the prevalence in the consideration
>> of both the prescription of a test and the application of the results (“to
>> what population does my patient belong?”). Furthermore, changing PPV/NPV
>> values with changing prevalence dictate why some tests, albeit good (from
>> the perspective of Sn/Sp) and used in some populations, should not be used
>> (and turn into very ineffective tests) in some others of different risk.
>>
>>
>>
>> LASt, P/N LR are derived from Sn/Sp. Thus share the same limitations.
>> However, if I understand that a test is indicated given my patient’s risk
>> for a disease, understanding how the pretest probability changes after a
>> positive result is of great value. Nevertheless, if I used a wrong test
>> (given the inadequate matching of the probability of disease and Sn/Sp of
>> the test) the use of LRs will prove extremely disappointing.
>>
>>
>>
>> In summary, each indicator, as always, has it use, with benefits and
>> limitations. Not understanding in full the reach of each of those operative
>> characteristics for a test and the diverse nature of my patients is the
>> problem. All this framed in the need we sometimes have as physicians to
>> oversimplify concepts (almost demanding “coking recipes” for everything)
>> when concepts cannot be simple. This is a great problem and what true EBM
>> addresses.
>>
>>
>>
>> JA
>>
>>
>>
>> *Juan M. Acuña M.D., MSc., FACOG.*
>>
>> Chair, Department of Medical and Health Sciences Research
>>
>> Associate Professor of OBGYN, Human and Molecular Genetics and Clinical
>> Epidemiology,
>>
>> Herbert Wertheim College  of Medicine
>>
>> Florida International University
>>
>>
>>
>> 11200 SW 8th Street
>>
>> AHC1-445
>>
>> Miami, FL 33199
>>
>> Ph (305) 348-0676
>>
>>
>>
>>
>>
>>
>>
>> *From:* Evidence based health (EBH) [
>> mailto:[log in to unmask]] *On Behalf Of *Raatz Heike
>> *Sent:* Monday, February 02, 2015 5:08 AM
>> *To:* [log in to unmask]
>> *Subject:* Re: Genetic tests and Predictive validity
>>
>>
>>
>> Dear all,
>>
>>
>>
>> Question is though: is the genetic test you want to evaluate actually
>> used as a *diagnostic* test? From what I understand from the case
>> mentioned by Teresa she is interested not in whether a genetic test
>> accurately recognizes whether you have a certain genotype but whether you
>> will in the future develop a certain phenotype*. *So you are not trying
>> to find out whether the patient currently suffers from a condition but the
>> risk of developing a condition in the future. Now unless you have a
>> dominant gene that will always lead to the expression of a certain
>> phenotype (like Huntingtons) you need to consider whether that genotype is
>> not just one of many factors that can lead to a certain condition. For the
>> examples mentioned like Mammaprint prognostic modelling seems much more
>> appropriate to me than diagnostic accuracy though ultimately you need RCTs
>> to prove that they improve patient reported outcomes and from what I saw
>> last those don’t exist.
>>
>>
>>
>> Best wishes, Heike
>>
>>
>>
>>
>>
>> Heike Raatz, MD, MSc
>>
>> Basel Institute for Clinical Epidemiology and Biostatistics
>>
>> Hebelstr. 10
>>
>> 4031 Basel
>>
>> Tel.: +41 61 265 31 07
>>
>> Fax: +41 61 265 31 09
>>
>> E-Mail: [log in to unmask]
>>
>>
>>
>> *From:* Evidence based health (EBH) [
>> mailto:[log in to unmask]] *On Behalf Of *Majid Artus
>> *Sent:* Monday, February 02, 2015 10:04 AM
>> *To:* [log in to unmask]
>> *Subject:* Re: Genetic tests and Predictive validity
>>
>>
>>
>> Dear All,
>>
>> Is it, or is it not, correct that one should follow the classic teaching
>> that (loosely and notwithstanding the false partition here): from patient's
>> perspective, sensitivity/specificity are what is relevant; and from
>> clinician's perspective, PPV/NPPV are what is relevant?
>>
>>
>>
>> Also, it is, isn't it, crucial to consider the media take on outcome of
>> research and how careful researchers need to be in selecting the way they
>> present the outcome of their research? MRI (NMR) in diagnosing autism is
>> one example that springs to mind - the high sensitivity was jumped on by
>> the media presenting it as a very accurate test missing the role of the
>> varying prevalence in certain settings.
>>
>> I find this discussion trail hugely thought provoking!
>>
>> Best Regards
>>
>> Majid
>>
>>
>>
>> On 2 February 2015 at 00:19, Mark V Johnston <[log in to unmask]> wrote:
>>
>> At the same time, don’t we need to know whether  the  patient  probably
>> has or does not have the  condition  of  interest?   Yes, prevalence and
>> other  factors  affect PPV and  NPV, but in my  opinion we  need to move
>> away from the  oversimplified notion that test interpretation depends on  a
>> single factor.
>>
>>
>>
>>
>>
>> *From:* Evidence based health (EBH) [mailto:
>> [log in to unmask]] *On Behalf Of *Mayer, Dan
>> *Sent:* Friday, January 30, 2015 8:08 PM
>> *To:* [log in to unmask]
>> *Subject:* Re: Genetic tests and Predictive validity
>>
>>
>>
>> Hi Teresa,
>>
>>
>>
>> You are absolutely correct.  This is why we should demand that diagnostic
>> studies ONLY present the results of Sensitivity, Specificity and Likelihood
>> ratios.
>>
>>
>>
>> This issue has been a serious problem for many years and it is about time
>> that more people spoke up about it.  Also, journal editors and peer
>> reviewers should be up in arms against the practice of reporting PPV and
>> NPV.
>>
>>
>>
>> Best wishes
>>
>>
>> Dan
>>  ------------------------------
>>
>> *From:* Evidence based health (EBH) [[log in to unmask]]
>> on behalf of Benson, Teresa [[log in to unmask]]
>> *Sent:* Friday, January 30, 2015 11:59 AM
>> *To:* [log in to unmask]
>> *Subject:* Genetic tests and Predictive validity
>>
>> I’ve just started reading the literature on genetic tests, and noticing
>> how many of them tend to focus on predictive value—that is, if a certain
>> test accurately predicts whether a patient will or won’t get a particular
>> phenotype (condition), the authors suggest the test should be used.  But if
>> we’re deciding whether to order the test in the first place, shouldn’t we
>> be focused on *sensitivity* and *specificity* instead, not PPV and NPV?
>> Predictive value is so heavily dependent on disease prevalence.  For
>> example, if I want to get tested for a disease with a 2% prevalence in
>> people like me, I could just flip a coin and regardless of the outcome, my
>> “Coin Flip Test” would show an NPV of 98%!  So what does NPV alone really
>> tell me, if I’m not also factoring out prevalence—which would be easier
>> done by simply looking at sensitivity and specificity?  Someone please tell
>> me where my thinking has gone awry!
>>
>> For a concrete example, look at MammaPrint, a test which reports binary
>> results.  In addition to hazard ratios, study authors often tout
>> statistically significant differences between the probabilities of
>> recurrence-free survival in the MammaPrint-High Risk vs. MammaPrint-Low
>> Risk groups (essentially the test’s predictive values).  In the RASTER
>> study (N = 427), 97% of the patients with a “Low Risk” test result did not
>> experience metastasis in the next 5 years.  Sounds great, right?  But when
>> you look at Sensitivity, you see that of the 33 patients in the study who
>> *did* experience metastasis, only 23 of them were classified as “High
>> Risk” by MammaPrint, for a 70% sensitivity.  If patients and clinicians are
>> looking for a test to inform their decision about adjuvant chemotherapy for
>> early stage breast cancer, wouldn’t the fact that the test missed 10 out of
>> 33 cases be more important than the 97% NPV, an artifact of the extremely
>> low 5-year prevalence of metastasis in this cohort (only 33 out of 427, or
>>  0.7%)?
>>
>> Drukker et al. A prospective evaluation of a breast cancer prognosis
>> signature in the observational RASTER study. Int J Cancer 2013.
>> 133(4):929-36. http://www.ncbi.nlm.nih.gov/pubmed/23371464
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ncbi.nlm.nih.gov_pubmed_23371464&d=AwMF-g&c=1QsCMERiq7JOmEnKpsSyjg&r=Ip5wwmlafmSkip30kemE5Q&m=GwNY8srJ3CEjlNbLqoSFCeH_n8m0cC6puP4CE3XW1VM&s=2chGNOKM9TfFH_1Hzl6g8am4NJO3LHdB0CIDAH8DaTM&e=>
>>
>> Retel et al. Prospective cost-effectiveness analysis of genomic profiling
>> in breast cancer. Eur J Cancer 2013. 49:3773-9.
>> http://www.ncbi.nlm.nih.gov/pubmed/23992641
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ncbi.nlm.nih.gov_pubmed_23992641&d=AwMF-g&c=1QsCMERiq7JOmEnKpsSyjg&r=Ip5wwmlafmSkip30kemE5Q&m=GwNY8srJ3CEjlNbLqoSFCeH_n8m0cC6puP4CE3XW1VM&s=dA6wTdKzugoGqf6Fx5fjSEz4CFxLPDhMbyV5Mt-j2ak&e=>
>>  (Provides actual true/false positive/negative results)
>>
>>
>>
>> Thanks so much!
>>
>>
>>
>> *Teresa Benson, MA, LP*
>>
>> Clinical Lead, Evidence-Based Medicine
>>
>> McKesson Health Solutions
>> 18211 Yorkshire Ave
>>
>> Prior Lake, MN  55372
>>
>> [log in to unmask]
>> Phone: 1-952-226-4033
>>
>>
>>
>> *Confidentiality Notice: This e-mail message, including any attachments,
>> is for the sole use of the intended recipient(s) and may contain
>> confidential and privileged information. Any unauthorized review, use,
>> disclosure or distribution is prohibited. If you are not the intended
>> recipient, please contact the sender by reply e-mail and destroy all copies
>> of the original message.*
>>
>>
>>
>>
>>    ------------------------------
>>
>> ----------------------------------------- CONFIDENTIALITY NOTICE: This
>> email and any attachments may contain confidential information that is
>> protected by law and is for the sole use of the individuals or entities to
>> which it is addressed. If you are not the intended recipient, please notify
>> the sender by replying to this email and destroying all copies of the
>> communication and attachments. Further use, disclosure, copying,
>> distribution of, or reliance upon the contents of this email and
>> attachments is strictly prohibited. To contact Albany Medical Center, or
>> for a copy of our privacy practices, please visit us on the Internet at
>> www.amc.edu
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.amc.edu&d=AwMF-g&c=1QsCMERiq7JOmEnKpsSyjg&r=Ip5wwmlafmSkip30kemE5Q&m=GwNY8srJ3CEjlNbLqoSFCeH_n8m0cC6puP4CE3XW1VM&s=__vs2OCqLIDvEtFDEkRu8CoWSznyDNAcsL389Go27lQ&e=>.
>>
>>
>>
>>
>>
>> --
>>
>> Please note my new email address: [log in to unmask]
>>
>>
>> Dr Majid Artus PhD
>> NIHR Clinical Lecturer in General Practice
>> Arthritis Research UK Primary Care
>>
>> Centre
>>
>> Research Institute for Primary Care & Health Sciences
>>
>> Keele University
>> Staffordshire, ST5 5BG
>> Tel: 01782 734826
>> Fax: 01782 733911
>> http://www.keele.ac.uk/pchs/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.keele.ac.uk_pchs_&d=AwMF-g&c=1QsCMERiq7JOmEnKpsSyjg&r=Ip5wwmlafmSkip30kemE5Q&m=GwNY8srJ3CEjlNbLqoSFCeH_n8m0cC6puP4CE3XW1VM&s=1tHDI4xDX-49yHN21UNpB2IjZeCZqO9J1qExDBWgv4s&e=>
>>
>>
>>
>> Please consider the environment before printing this email.
>> This email and its attachments are intended for the above named only and
>> may be confidential. If it has come to you in error you should not copy or
>> show it to anyone; nor should you take any action based on it, other than
>> to notify the sender of the error by replying to the sender. Keele
>> University staff and students are required to abide by the University's
>> conditions of use when sending email. Keele University email is hosted by a
>> cloud provider and may be stored outside of the UK.
>>
>>
>>
>
>
> --
> Owen Dempsey MBBS MSc MRCGP RCGP cert II
>
> 07760 164420
>
> GPwsi Substance Misuse Locala and Kirklees Lifeline Addiction Service
>
>


-- 
Owen Dempsey MBBS MSc MRCGP RCGP cert II

07760 164420

GPwsi Substance Misuse Locala and Kirklees Lifeline Addiction Service