I think an adjective "reliable" meant to refer if (SR) are more trustworthy ( than isolated pieces of evidence, or non-SR methods of evidence synthesis), which in turn begs a difficult question "what is truth"? There are many definitions of the "truth" ( and I hope some of our colleagues who are philosophers and have interest in EBM can help out here), but two popular theories of truth are: correspondence and coherence theory.
The correspondence theory asses if the arguments are empirically correct i.e, do they correspond with reality ("gold standard") ( as for, example, whether predictive model accurately described the outcomes of interest such as mortality , which we can easily empirically verify by simply counting how many people actually died vs. how many people were predicted to die ).
The coherence theory assess if the arguments are internally consistent within the accumulated body of knowledge and accepted inferential system of the day (e.g., basing our inferences on the totality of evidence will give us the estimate that should be more trustworthy than the answer based on a selected piece of evidence, or if the criteria outlined by Mike are not met, we typically consider a SR less trustworthy, etc)
So, "reliable" in this context is measured according to the coherence criteria of the truth as defining the "gold standard" is difficult if not impossible, as Jon's message alluded. This also means that the accumulated body of knowledge indicate that our estimates will be closer to the "truth" if we adhere to the accepted methodological standards of analysis and synthesizing evidence ( and currently most of us believe is to conduct a SR on the topic of interest).
An interesting discussion...
Ben Djulbegovic
Sent from my iPad (please excuse typos )
On Feb 25, 2017, at 8:52 AM, Brown, Michael <[log in to unmask]<mailto:[log in to unmask]>> wrote:
Jon: Although I don’t have any “evidence” to back this up, my opinion after reviewing hundreds of systematic reviews is that they are NOT reliable. Unfortunately, low quality systematic reviews appear to be as common as low quality research publications of any other form of retrospective analysis. This is why rigorous critical appraisal of systematic reviews is essential before placing them at the top of a hierarchy of evidence.
There are many critical appraisal constructs out there, but at a minimum I believe they should meet the following criteria:
* Report a comprehensive search
* Describe the inclusion criteria for selecting individual studies
* Report on the methodological quality (risk of bias) of the individual studies
* Describe the heterogeneity among studies and methods used for combining results (if applicable)
* Describe approach to subgroup and sensitivity analyses
* If the included studies are not uniformly at low risk of bias, report a sensitivity analysis based on study quality or describe the potential effect of including low quality studies
Differences in the approach to any of the above bullet points may produce different results, thus the inconsistent conclusions of meta-analyses on the same topic.
Unfortunately, the last bullet is often ignored, which only provides fodder for the critics of meta-analysis (ie, garbage in = garbage out).
In other words, the methods of a systematic review may be pristine (eg, all elements of PRISMA statement reported), yet still produce erroneous conclusions if the quality of the underlying studies is poor.
Mike
Michael Brown, MD, MSc
Professor and Chair of Emergency Medicine
Michigan State University College of Human Medicine
[log in to unmask]<mailto:[log in to unmask]>
(cell) 616-490-0920
From: "Evidence based health (EBH)" <[log in to unmask]<mailto:[log in to unmask]>> on behalf of Jon Brassey <[log in to unmask]<mailto:[log in to unmask]>>
Reply-To: Jon Brassey <[log in to unmask]<mailto:[log in to unmask]>>
Date: Saturday, February 25, 2017 at 6:10 AM
To: "[log in to unmask]<mailto:[log in to unmask]>" <[log in to unmask]<mailto:[log in to unmask]>>
Subject: Query about systematic reviews
Hi All,
I'm helping a student with a piece of work and he described systematic reviews as being reliable. I'm troubled by this on a number of fronts:
* Is there any evidence that they are reliable?
* If they are reliable, in relation to what? A reliable car might not break down very often, so that outcome is clear. But I'm not sure what it means in relation to SRs
* If SRs aren't properly described as 'reliable' what are they best described as?
An underpinning assumption of SRs appears to be that if you identify all the published trials that bestows upon the SR a special status. This appears to be faith-based, not evidence-based.
Any help out there?
Best wishes
jon
--
Jon Brassey
Director, Trip Database<http://www.tripdatabase.com>
Honorary Fellow at CEBM<http://www.cebm.net>, University of Oxford
Creator, Rapid-Reviews.info<http://Rapid-Reviews.info>
