Craig makes the very important statement that: 'A singular approach is always going to have more limits than a comprehensive approach.' I'm a great fan of SRs but they are only part of the knowledge base on a topic and they can miss a lot of very relevant evidence. Craig mentions qualitative evidence but they can also miss a lot of important quantitative evidence that doesn't fit the SR's eligibility criteria yet they can still be based on poor quality studies. For example, it's impossible to do a good longterm RCT of high vs low saturated fat with CHD events as  the outcome, because it's impossible to get people to stay on their allocated diet for the years of follow-up required. Yet there are multiple SRs and meta-analyses of these poor quality studies, including Cochrane reviews. Not surprisingly none of these reviews show strong effects because of poor compliance and removing one or two studies turns a significant protective effect of lowering SF consumption into a non significant result. This is currently causing a lot of unnecessary controversy worldwide as some people unthinkingly equate SRs (of poor  RCTs) as the only acceptable evidence. SRs are just one part of the evidence base. Sometimes they are the most important part, but sometimes they are not.  

Cheers Rod Jackson
University of Auckland, New Zealand

* * * * * * * *
sent from my phone


On 4/12/2015, at 15:16, Craig Lockwood <[log in to unmask]> wrote:

Dear Ian, Jon et al,

Qualitative methods are what i sometimes (jokingly) refer to as the 'dark side' - and although there are some well established, sophisticated methods for qualitative synthesis (see meta-aggregation or realist synthesis) the search for a rationale seems limited to 'things that can be counted' (depending on what 'worth' means).  A singular approach is always going to have more limits than a comprehensive approach. Understanding the effectiveness is very useful, but we also need to consider the meaningfulness, feasibility and applicability of evidence - perhaps then the measure of its 'worth' will be established.

The FAME scale developed by the Joanna Briggs Institute goes some way to responding to this.  Interesting timing on the post Jon, I was just watching some youtube interviews today where you were asking similar questions to this, the question is one of those 'wicked' problems!

cheers,
Craig


From: Evidence based health (EBH) [[log in to unmask]] on behalf of Johnson, Ian (MAN-DHX) [[log in to unmask]]
Sent: Friday, 4 December 2015 8:09 PM
To: [log in to unmask]
Subject: Re: Why do we do systematic reviews?

Hi Jon

 

I have been involved with EBM for much longer, but mainly for the Dark Side, to help the pharmaceutical industry demonstrate value of their new products. It is important to be able to judge the true ‘worth’ of a new asset, specifically in comparison to what is currently available. Sadly, regulatory requirements are such, particularly in the US, that is it not difficult to construct an RCT (please excuse my habitual use of TLAs) that will show a superior benefit:risk outcome. Whether or not this is real, or in comparison with the best alternative, is fraught with uncertainties. Very few companies venture into H2H comparisons, so the accumulation of evidence around all potential comparators through an SLR is necessary, and has, of course spawned all types of ITCs and NMAs. Some Journals now demand an SLR before publishing a ‘pivotal’ RCT that claims to offer an incremental benefit. Unfortunately for EBH, these demands still rest largely on published evidence, and are subject to publication bias, as we know that not many studies with negative results ever enter the public domain. The underlying principle of EBM demands a rigorous collation and interpretation of the evidence. Whether SLRs fulfil that requirement is debatable, but until RWD collection, possibly involving EMRs, is mandatory, physicians will have no alternative but to continue to use their own clinical judgment and personal experience. We use SLRs to answer a research question that has not been addressed before. Not all of these reviews are published as they often serve to direct clinical development plans. Moreover, they are, unavoidably, just a review of the evidence at a point in time and, over time, new studies may change the interpretation. So, I agree with points 1-4, but not necessarily your tongue-in-cheek point 5, as primary research is probably more valuable on a CV than secondary.

 

Ian

 

K IAN JOHNSON | PRINCIPAL CONSULTANT, EVIDENCE RESEARCH

DOUBLE HELIX CONSULTING

19-21 King Edward Street, Macclesfield, SK10 1AQ, UK

P: +44 (0)1625 624048 |M: +44 (0)7725 335342

E: [log in to unmask]: www.dhelix.com

T: @dhelixgroup |@dhelixconsult

 

 

From: Evidence based health (EBH) [mailto:[log in to unmask]] On Behalf Of Jon Brassey
Sent: 04 December 2015 06:41
To: [log in to unmask]
Subject: Why do we do systematic reviews?

 

Hi All,

 

I've been involved in 'EBM' for nearly twenty years and one question that I struggle with is the reason for doing a systematic review.  Over the years I've collected the following reasons:

 

1.To know if an intervention has any ‘worth’.
2.To quantify, quite tightly, how good the intervention is.
3.To understand the adverse events associated with the intervention.
4.To see what has been done before, to see if new research is needed.
5.To improve the CV.

 

I'd be really interested to hear if I've missed anything obvious or if you feel some of the reasons are more valid than others.  I've created a poll at http://rapid-reviews.info/2015/12/03/why-do-we-do-systematic-reviews/ so I'd really appreciate people's input on this.

 

Best wishes

 

jon


--

Jon Brassey

Trip Database

Find evidence fast

 

 

 

This email may contain confidential or legally privileged information, intended only for the addressee. If you have received this email in error, you are hereby notified that any disclosure, copying, distribution or reliance upon the contents of this email is strictly prohibited. Please contact the sender to arrange for correct delivery, and then delete this email. Any views or opinions presented in this email are those of the author. Please check the email for viruses, as the sender accepts no liability for damage caused by a virus transmitted via this email. Registered Office: McCann Complete Medical Ltd. Complete House, 19-21 King Edward Street, Macclesfield, Cheshire SK10 1AQ, UK Tel:+44 (0)1625 624000, Fax: +44 (0)1625 619812, Email: [log in to unmask], Web: www.complete-grp.com Registered in England No. 2503062