Roy, I agree that this is a real problem with complex interventions,
and it goes deeper than reviewers.
Details of complex interventions are often not reported in sufficient
detail or are not reported systematically to allow meaningful
synthesis and comparison across trials
(http://ajph.aphapublications.org/cgi/content/full/97/4/630).
Standardised outcome sets will help reviewers synthesise the results
of trials. Similar initiatives might help readers understand their
design, delivery, uptake and context.
We (Paul Montgomery and Sean Grant, both at Oxford) are currently
working on a CONSORT extension for psycho-social trials and hope that
participants will help us identify ways to improve reporting of these
details. You can find out more here:
http://www.spi.ox.ac.uk/research/centre-for-evidence-based-intervention/consort-study.html.
Feel free to email any of us - all suggestions and feedback received
with thanks.
Evan
Evan Mayo-Wilson, MPA, DPhil
Centre for Outcomes Research and Effectiveness
Research Department of Clinical, Educational & Health Psychology
University College London
1-19 Torrington Place
London WC1E 7HB
mobile: +44 (0) 7932 116535
[log in to unmask]
2011/9/19 Marsh Roy <[log in to unmask]>:
> I spend my working life finding and (critically I hppe) summarising evidence
> from systematic reviews, for public health use. I get very suspicious of
> meta-analyses in complex (often social) areas since I feel that the
> assumptions and simplificatioins involved in grouping studies together often
> do not do justice to the reality of the intervention. For instance, reviews
> seem to overlook implementational factors that are crucially important to
> decision-makers, such as the context in which the jntervention is carried
> out, resource requirements, timescale of outcomes, competency needs of the
> people carrying out the intervention, etc. Anything involving psychological
> / sociological / organisational theory also seems beyond the reach of
> systematic reviews. Example: should health promotion interventions with the
> outcome of behaviour change follow some psychological theory? If so, which?
> The relevant Cochrane reviews are distinctly ambivalent, and seem to end up
> advocating 'some sort of theory' and 'tailoring to local circumstances' (I
> paraphrase), thereby passing the responsibility on to the person who thought
> they would find the answer in the review. I only refer to Cochrane here
> because they are (rightly, a lot of work goes into them) my first port of
> call.
>
> And in the back of my mind, I always have Pawson's mantra 'What works for
> whom in what circumstances?' Most of the reviews I come across only go part
> way to answering that question.
>
> Roy Marsh
> ________________________________
> From: Evidence based health (EBH)
> [mailto:[log in to unmask]] On Behalf Of Susan Fowler
> Sent: 19 September 2011 01:56
> To: [log in to unmask]
> Subject: Re: Interpreting seemingly odd evidence from systematic reviews
>
> Whenever I teach about systematic reviews I use the example of a review done
> by Cochrane and another published in the New England Journal of Medicine.
> Both were on the effectiveness of echinacea on the common cold but came up
> with two results. Cochrane concluded it was not effective and the review in
> NEJM concluded that it was. The difference? One study was included
> in the NEJM article that was not in the Cochrane. The lesson - one must
> always read, critically appraise, and in the end decide for themselves.
> Duke has some excellent guidance on how to appraise all types of
> articles including systematic reviews here: http://www.mclibrary.duke.edu/subject/ebm?tab=appraising&extra=worksheets
>
> --
> Susan Fowler, MLIS
> Medical Librarian
>
> Evidence at Becker:
> http://beckerguides.wustl.edu/ebm
>
> Mobile Resources Guide:
> http://beckerguides.wustl.edu/mobileresources
>
> Becker Medical Library, Washington University in St. Louis
> 314-362-8092
> [log in to unmask]
>
>
>
>
>
> 2011/9/18 Brian Alper MD <[log in to unmask]>
>>
>> Systematic reviews are not automatically “level 1 evidence”. First the
>> systematic review has to be well conducted and appropriately combine
>> content. Second, the evidence reviewed has to have sufficient quality for
>> the conclusions to support “level 1 evidence”. Equating systematic reviews
>> (or randomized trials) with level 1 evidence without checking the details is
>> problematic.
>>
>>
>>
>> On the other hand “many previous epidemiologic studies, animal studies, …”
>> does not establish high-quality evidence and could be prone to biases
>> related to confounding factors.
>>
>>
>>
>> In this case the Cochrane review conducted a systematic review of
>> randomized trials (seeking unconfounded data) and really found insufficient
>> evidence to make a “level 1 evidence” claim of benefit or to make a “level 1
>> evidence” claim of no benefit. Despite many studies and “time-honored
>> practice” (which just means many have accepted this to be true without
>> evidence for a long time) we have never had evidence clearly establishing
>> benefit or no benefit for salt restriction. This Cochrane review basically
>> looks at the totality of evidence today (in case it changed since the last
>> time this was systematically evaluated) and still finds we do not have a
>> highly reliable answer.
>>
>>
>>
>> But interpreting the Cochrane review should still involve critical
>> analysis. For example when evaluating this Cochrane review for DynaMed we
>> considered that the meta-analyses combining all the trials were not the
>> ideal representation of the data because combining normotensive patients,
>> hypertensive patients, and patients with heart failure does not represent a
>> typical patient or general population. So instead we reported the detailed
>> results separately (with numbers of trials, numbers of patients, and
>> confidence intervals for differences or trends) for:
>>
>> · normotensive patients (non-significant trend toward lower
>> mortality and lower systolic blood pressure at end of trial, but no
>> significant differences in mortality or cardiovascular morbidity at longest
>> follow-up)
>>
>> · hypertensive patients (no significant differences in all-cause
>> mortality or cardiovascular morbidity, but trend toward reduced
>> cardiovascular mortality)
>>
>> · patients with heart failure (significant decrease in systolic
>> blood pressure but significant INCREASE in mortality in 1 trial)
>>
>>
>>
>> Ultimately we gave the review a conclusion of:
>>
>>
>>
>> · unknown whether reduced salt intake reduces cardiovascular
>> morbidity or mortality (level 2 [mid-level] evidence)
>>
>> based on Cochrane review of trials with insufficient power to exclude
>> clinically important differences
>>
>>
>>
>> Brian S. Alper, MD, MSPH
>>
>> Editor-in-Chief, DynaMed (www.ebscohost.com/dynamed)
>>
>>
>>
>> From: Evidence based health (EBH)
>> [mailto:[log in to unmask]] On Behalf Of Dr. Abdelhamid
>> Attia
>> Sent: Sunday, September 18, 2011 8:06 PM
>> To: [log in to unmask]
>> Subject: Interpreting seemingly odd evidence from systematic reviews
>>
>>
>>
>> Dear EBMers,
>>
>>
>>
>> I have been asked by a colleague about a systematic review that has been
>> heavily criticized “Reduced dietary salt for the prevention of
>> cardiovascular disease”.
>>
>> Being away from my specialty, I did not follow the reactions to its
>> publication and what followed.
>>
>> Anyhow, here is a quote of the questions posed by this colleague.
>>
>> =================
>>
>> a new cochrane review has concluded there is no clear evidence of
>> benefit—in terms of preventing CV events and deaths—from advising people to
>> reduce dietary salt intake[1,2]. This type I evidence came against many
>> previous epidemiological studies, animal studies, randomized trials and sure
>> against a time-honored practice. This caused many physicians to shout "FOUL"
>> over it [3].
>>
>>
>>
>> This situation raises many questions like;
>>
>> a) How many systematic reviews show conflicting results?
>>
>> b) How will the practicing physician interpret conflicting evidences?
>>
>> c) Do we need expert opinions of respected authorities to evaluate
>> level I evidence (Systematic reviews)?
>>
>> d) If so, will this threaten the whole concept of evidence based
>> medicine? Or at least turn the balance again towards medicine as a
>> profession?
>>
>> 1. Taylor RS, Ashton KE, Moxham T, et al. Reduced dietary salt for the
>> prevention of cardiovascular disease. Cochrane Database of Systematic
>> Reviews 2011; 7:CD009217.
>>
>> 2. Taylor RS, Ashton KE, Moxham T, et al. Reduced dietary salt for the
>> prevention of cardiovascular disease: a meta-analysis of randomized
>> controlled trials (Cochrane Review). Am J Hypertension 2011; 8:843-853.
>>
>> 3. He FJ, MacGregor GA. Salt reduction lowers cardiovascular risk:
>> meta-analysis of outcome trials. Lancet 2011; 378:380-382.
>>
>> ==================================
>>
>> I will appreciate any insight about this case and specifically the answer
>> to question b) when an ordinary clinician finds a strong reaction against a
>> systematic review as the one published in the lancet.
>>
>>
>>
>> Best of wishes,
>>
>> Abdelhamid Attia
>>
>>
>>
>> Prof. Dr. Abdelhamid Attia
>>
>> Prof. Of Obstetrics & Gynecology; Cairo University
>>
>> Assistant Secretary General Of The Egyptian Board (Fellowship)
>>
>> President Of The Arab Federation of Evidence-Based Medicine
>>
>> Secretary General of the Scientific Council of OBGYN, The Arab Board
>>
>> Editor-in-Chief of Kasr Al-Aini Journal of Obstetrics & Gynecology
>>
>> Editor Of Kasr Al-Aini Medical Journal
>>
>> Associate Editor of The Middle East Fertility Society journal
>>
>> __________ Information from ESET NOD32 Antivirus, version of virus
>> signature database 6474 (20110918) __________
>>
>> The message was checked by ESET NOD32 Antivirus.
>>
>> http://www.eset.com
>
>
>
>
>
>
>
|