It should be noted that the entire statistical machinery of inference - so often discussed by the EBMers- is in fact based on the coherence theory of truth....

Sent from my iPhone
(Please excuse typos & brevity)

On Feb 25, 2017, at 16:53, Mohammed T. Ansari <[log in to unmask]> wrote:

Indeed....but someone was smart enough to point out that studying new users is a better way to understand reality than studying prevalent users -- before that, we had still had reproducibility and coherence. My point is that truth will remain elusive but we could come closer and closer to it. Reproducibility and coherence are in some ways circular arguments. 

On Sat, Feb 25, 2017 at 3:16 PM, Klim McPherson <[log in to unmask]> wrote:
Dear Mohammed,

That HRT reduces the risk of CHD/CVD is, in my opinion, the opposite of the truth, if double blind RCT's have any important role in determining that truth.

It is perfectly true that chronic HRT users show better CVD outcomes that non users, but that is probably because they are not typical of post menopausal women generally.

See attached

All best

Klim McPherson PhD FFPM FMedSci HonFRCP
Past Chair: UK Health Forum
Visiting Professor of Public Health Epidemiology
Nuffield Department of Primary Care Health Sciences
New College
Oxford University
Mobile 07711335993

Do not use obs-gyn email, instead




From: "EVIDENCE-BASED-HEALTH@JISCMAIL.AC.UK" <EVIDENCE-BASED-HEALTH@JISCMAIL.AC.UK> on behalf of "Mohammed T. Ansari" <[log in to unmask]>
Reply-To: "Mohammed T. Ansari" <[log in to unmask]>
Date: Saturday, 25 February 2017 16:35
To: "EVIDENCE-BASED-HEALTH@JISCMAIL.AC.UK" <EVIDENCE-BASED-HEALTH@JISCMAIL.AC.UK>
Subject: Re: Query about systematic reviews

Interesting indeed Ben, But isn't this kind of circular reasoning? 

If our vision of reality were skewed, we'd calibrate our tools accordingly. In a world that has some pattern to it, we could still achieve predictability...though gnosis of truth would be skewed.   

To me truth is gnosis with clarity of vision, not reproducibility of data. It is not about what the data is, but rather what it means. 

Chronic HRT post menopausal users will repeatedly show better CHD outcomes than non-user post menopausals. But that HRT reduces CHD may not be the truth about the intervention in any absolute sense of word.  

On Sat, Feb 25, 2017 at 10:00 AM, Djulbegovic, Benjamin <[log in to unmask]> wrote:

I think an adjective "reliable" meant to refer if (SR) are more trustworthy ( than isolated pieces of evidence, or non-SR methods of evidence synthesis), which in turn begs a difficult question "what is truth"? There are many definitions of the "truth" ( and I hope some of our colleagues who are philosophers and have interest in EBM can help out here), but two popular theories of truth are: correspondence and coherence theory.

The correspondence theory asses if the arguments are empirically correct i.e, do they correspond with reality ("gold standard") ( as for, example, whether predictive model accurately described the outcomes of interest such as mortality , which we can easily empirically verify by simply counting how many people actually died vs. how many people were predicted to die  ).

The coherence theory assess if the arguments are internally consistent within the accumulated body of knowledge and accepted inferential system of the day (e.g., basing our inferences on the totality of evidence will give us the estimate that should be more trustworthy than the answer based on a selected piece of evidence, or if the criteria outlined by Mike are not met, we typically consider a SR less trustworthy, etc)

So, "reliable" in this context is measured according to the coherence criteria of the truth as defining the "gold standard" is difficult if not impossible, as Jon's message alluded. This also means that the accumulated body of knowledge indicate that our estimates will be closer to the "truth" if we adhere to the accepted methodological standards of analysis and synthesizing evidence ( and currently most of us believe is to conduct a SR on the topic of interest).

An interesting discussion...


Ben Djulbegovic



Sent from my iPad (please excuse typos )

On Feb 25, 2017, at 8:52 AM, Brown, Michael <[log in to unmask]> wrote:

Jon: Although I don’t have any “evidence” to back this up, my opinion after reviewing hundreds of systematic reviews is that they are NOT reliable. Unfortunately, low quality systematic reviews appear to be as common as low quality research publications of any other form of retrospective analysis. This is why rigorous critical appraisal of systematic reviews is essential before placing them at the top of a hierarchy of evidence.

There are many critical appraisal constructs out there, but at a minimum I believe they should meet the following criteria:

  • Report a comprehensive search
  • Describe the inclusion criteria for selecting individual studies
  • Report on the methodological quality (risk of bias) of the individual studies
  • Describe the heterogeneity among studies and methods used for combining results (if applicable)
  • Describe approach to subgroup and sensitivity analyses
  • If the included studies are not uniformly at low risk of bias, report a sensitivity analysis based on study quality or describe the potential effect of including low quality studies

 

Differences in the approach to any of the above bullet points may produce different results, thus the inconsistent conclusions of meta-analyses on the same topic.

Unfortunately, the last bullet is often ignored, which only provides fodder for the critics of meta-analysis (ie, garbage in = garbage out).

In other words, the methods of a systematic review may be pristine (eg, all elements of PRISMA statement reported), yet still produce erroneous conclusions if the quality of the underlying studies is poor.

 

Mike

 

Michael Brown, MD, MSc

Professor and Chair of Emergency Medicine

Michigan State University College of Human Medicine

[log in to unmask]

(cell) 616-490-0920

 

 

 

From: "Evidence based health (EBH)" <EVIDENCE-BASED-HEALTH@JISCMAIL.AC.UK> on behalf of Jon Brassey <[log in to unmask]>
Reply-To: Jon Brassey <[log in to unmask]>
Date: Saturday, February 25, 2017 at 6:10 AM
To: "EVIDENCE-BASED-HEALTH@JISCMAIL.AC.UK" <EVIDENCE-BASED-HEALTH@JISCMAIL.AC.UK>
Subject: Query about systematic reviews

 

Hi All,

 

I'm helping a student with a piece of work and he described systematic reviews as being reliable.  I'm troubled by this on a number of fronts:

 

  • Is there any evidence that they are reliable?
  • If they are reliable, in relation to what?  A reliable car might not break down very often, so that outcome is clear.  But I'm not sure what it means in relation to SRs
  • If SRs aren't properly described as 'reliable' what are they best described as?

 

An underpinning assumption of SRs appears to be that if you identify all the published trials that bestows upon the SR a special status.  This appears to be faith-based, not evidence-based.  

 

Any help out there?

 

Best wishes

 

jon

 

--

Jon Brassey

Director, Trip Database

Honorary Fellow at CEBM, University of Oxford