Dear Abdelhamid,
Excellent suggestions! We do somehow need a separation of these two
approaches to evaluating diagnostic tests. The tricky part is the link
between accuracy and performance: when when accuracy is enough, and when
we can't we rely on accuracy to tell us about performance?
A small step towards this is identifying the role of the test, as one of:
1. Replacement (of an old test)
2. Triage (before the pivotal test) and
3. Add-on (to current testing)
These are described in a paper in this weeks BMJ:
Patrick M Bossuyt, Les Irwig, Jonathan Craig, Paul Glasziou Diagnosis:
Comparative accuracy: assessing new tests against existing diagnostic
pathways. BMJ 2006;332:1089-1092
http://bmj.bmjjournals.com/cgi/content/full/332/7549/1089
Role 3 is the most likely to raise questions about performance.
Best wishes,
Paul Glasziou
Dr. Abdelhamid Attia wrote:
> Dear Bob, Paul, and Arri
>
> Thank you very much for your valuable inputs and clarification to this
> matter and sorry for the late reply as Thursday & Friday are the week end in
> this part of the world and I was in a short vacation.
>
> So we can assume that a cross sectional blind comparison with a reference
> standard tests the test ACCURACY in the optimal conditions "for the test"
> and that RCTs test the test PERFORMANCE "in a clinical setting". Something
> analogous to the efficacy and effectiveness of therapy.
>
> If the above is convincing to you, I would like to add that as both aspects
> are important from the clinician point of view this should be clarified
> somehow in the hierarchy of searching for evidence for diagnosis as what is
> interpreted from this hierarchy is that RCTs can not be used in the
> evaluation of diagnostic tests at all rather than can be used in certain
> situations for a diagnostic test.
>
> Perhaps we might improve the hierarchy by splitting it into two: one to test
> for "test accuracy" in ideal situations (the present one that can be
> accordingly re-named) and to add another hierarchy for "diagnosis" in
> general that can start with RCTs on the very top of the hierarchy.
>
> We might also consider to design a special appraisal sheet/tool for RCTs
> dealing with diagnosis.
>
> Best of wishes and thank you very much,
> Abdelhamid
>
> Prof. Dr. Abdelhamid Attia
> Prof. of Ob & Gyn, Cairo University
> President; Arab Federation of EBM
>
>
> ----- Original Message -----
> From: "Arri Coomarasamy" <[log in to unmask]>
> To: <[log in to unmask]>
> Sent: Thursday, May 04, 2006 10:22 PM
> Subject: Re: Evidence for diagnostic tests from RCTs?
>
>
> Dear Abdelhamid,
>
> RCTs of tests are fraught with difficulties of their own.
> For a good analysis of this, please see:
>
> Bossuyt PM, Lijmer JG, Mol BW.
> Randomised comparisons of medical tests: sometimes invalid, not always
> efficient.
> Lancet. 2000 Nov 25;356(9244):1844-7.
>
> Some problems are:
> 1. a test may be perfect in diagnosing a condition (perfect accuracy), but
> if there is no effective treatment, then the testing will not improve
> outcomes despite its accuracy, as testing itself doesn't generally change
> clinical outcomes, but it is the treatment that follows from it.
>
> 2. even if a test+treatment strategy is shown to improve clinical outcomes,
> it does not necessarily mean the test has ANY value - the reason is best
> illustrated with an example:
>
> A large Cochrane review (with >30 RCTs) has shown aspirin reduces the risk
> of pre-eclampsia in moderate or high risk women. These are simple
> interventional trials comparing aspirin to placebo or no treatment in all
> sorts of patient groups.
>
> Now there are also some trials that have evaluated uterine artery Doppler (a
> test to predict pre-eclampsia)+ aspirin treatment versus standard treatment.
> If these trials show better outcome in the Doppler+asprin group, is that
> proof that Doppler is a useful thing to do? No! Not necessarily. This is
> because MORE women in the Doppler arm are likely to have received aspirin
> compared to the no Doppler arm ANYWAY (even if the Doppler test had no
> accuracy and indiscriminately labelled a proportion of women as "test
> positive" & and if aspirin were to be generally effective in reducing
> pre-eclampsia, then it is possible that Doppler arm would have shown benefit
> regardless of whether Doppler test was a good predictor of pre-eclampsia or
> not, or indeed, whether the Doppler test was done or not. So the better
> outcome could be due to more receiving aspirin in the Doppler group than
> there being anything special about Doppler!
>
> I think if a test itself can modify the outcome (eg amnio or CVS can cause
> miscarriage), then RCT has a role in their evaluation. I also think RCTs
> have a role in screening test. In other circumstances, generally, i think
> RCT has no role in the evaluation of a test - and the approach should be to
> use a test to work out the probability of disease, and then estimate the
> absolute effect that can be expected by treating with intereventions that
> have been tested in RCTs, by following the concepts given in one of paul's
> work:
>
> Glasziou PP, Irwig LM.
> An evidence based approach to individualising treatment.
> BMJ. 1995 Nov 18;311(7016):1356-9.
>
>
> Arri Coomarasamy,
> Guy's Hospital
> London
>
>
>
> -----Original Message-----
> From: Evidence based health (EBH)
> [mailto:[log in to unmask]] On Behalf Of Paul Glasziou
> Sent: 03 May 2006 22:14
> To: [log in to unmask]
> Subject: Re: Evidence for diagnostic tests from RCTs?
>
> Dear Abdelhamid,
> Can I expand a little on Bob Phillips point? The right study and
> hierarchy depends on the questions you want to answer.
> For diagnostic ACCURACY a cross sectional study with full verification
> with a blinded adequate gold standard is fine.
> That works even if we can't or don't want to treat, e.g, an untreatable
> tumour or Osgood-Schlatter's disease.
> But we may also want to know whether having the test available as part
> of a complex management strategy improves outcomes. But that is a
> complex brew of:
> 1. the test accuracy
> 2. the alternative tests
> 3. the available treatments
> 4. how clinicians interpret the test and behave as a result
> So the RCT tests all this brew together (and probably more), not the
> test accuracy. That is of greater interest, but usually less generalisable.
> However, I agree that some tests need this, e.,g. cardiotocography in
> labour. But others clearly don't - I am happy that the whispered voice
> test* is accurate for diagnosing deafness without need an RCT to know
> that it changes my or the patient's behaviour, or that the jolt
> accentuation test can rule out meningitis (though better cross sectional
> studies are needed to verify this).
> Cheers
> Paul Glasziou
> * Pirozzo, et al BMJ 2003.
>
>> Dear listers,
>>
>> I have been discussing with a colleague the best evidence
>> about diagnostic tests. In our discussion, I found that some tutors
>> take the oxford hierarchy of evidence at
>> http://www.cebm.net/levels_of_evidence.asp
>> at its face value while I see that RCTs can give also better evidence
>> than cross-sectional studies if they are feasible to perform. One of
>> the pitfalls of the hierarchy is that we may miss a better piece of
>> evidence as it doesn't mention RCTs or systematic reviews of RCTs.
>>
>> Teaching EBM, I always concentrate on the concept of bias in different
>> study designs and their implications on the process of evidence
>> generation rather than restricting my students to a "rigid" scheme and
>> I always ask them to start with searching on SR of RCTs, RCTs, etc..
>> down the hierarchy of evidence.
>>
>> For a live example:
>> In the Cochrane Database of SRs there is a systematic review for RCTs
>> comparing the accuracy and safety of chorionic villous biopsy and
>> amniocentesis in the diagnosis of genetic abnormalities.
>>
>>
> http://www.mrw.interscience.wiley.com/cochrane/clsysrev/articles/CD003252/fr
> ame.html
>
>> This piece of evidence about diagnosis is of course better than 10s of
>> cross-sectional studies. I think that we should add RCTs and their
>> systematic reviews at the very top of the evidence pyramid for
>> diagnosis as we do for therapy and harm?
>>
>> Any thoughts?
>>
>> Best of wishes,
>> Abdelhamid Attia
>>
>
>
>
--
Paul Glasziou
Director, Centre for Evidence-Based Medicine,
Department of Primary Health Care,
University of Oxford www.cebm.net
ph +44-1865-227055 fax +44-1865-227036
|