Hi Benjamin,
Just from taking a quick look at the paper, I just felt like you could have had one more category as 'unclear' than just two categories:Adequate or Inadequate. I think the problem with just 'inadequate' is that you can lump studies with clearly no blinding with studies that could have had blinding/not clear from the paper. So two ES's (small & large) probably gets blended. Do you think it could have made a difference in ES comparisons?
Thank youAnoop
On Sun, Sep 25, 2016 at 8:28 AM, Djulbegovic, Benjamin <[log in to unmask]> wrote:
The problem with critical appraisal is that relies on the quality of reporting, but as we showed in these two papers, the quality of reporting is not necessary equal to the quality of actual conduct:
Fair, or not fair, the underlying premise behind critical appraisal is indeed "if it was not reported, it was not done". The insistence on the quality of reporting is of obvious societal benefit ( i.e. increase trust in clinical research, enables/improves conduct of meta-analyses, etc) but it can certainly unfairly characterized some, otherwise, well-done trials.Ben
Ps While reporting clinical research is distinctly different from assessing the practice of medicine, it should be noted that the premise that if "it was not reported, it was not done", has also been widely accepted in the quality of improvement arena, which directly impacts the way how health care is financed. Not surprising, many physicians consider this unfair way to characterize their performance.
Hi Helen,
The point I'm making is that journal articles are a summary of the activity undertaken in a trial. So, when we critically appraise a journal article, the starting point appears to be that this is a fair reflection of the trial.
As I suggested, and Brian reinforced, this is a problematic assumption. As he said, critical appraisal "can only identify reasons for uncertainty" - indicating that critical appraisal can't assert an article is fair.
BW
jon
On 24 September 2016 at 19:09, Helen Pullen <[log in to unmask]> wrote:
Does your critical appraisal include looking at the statistics in a paper? It appears that statistical literacy (I'm quoting from the book 'Better doctors, better patients, better decisions') is a required skill of all those appraising the evidence. It's a very interesting topic area. I'm a medical librarian teaching critical appraisal and basic medical stats to our staff (large teaching hospital).
Kind regards
Helen Pullen BSc MCLIP
A while back we published “Evolution of evidence-based medicine to detect evidence mutations.” http://www.ncbi.nlm.nih.gov/pu
bmed/25572989 or http://jrs.sagepub.com/content/108/1/8.long
This reported on a handful of high-profile or high-impact studies where critical appraisal as commonly applied was insufficient to capture the most concerning criticisms. The reasons varied – fraud in what was published, failure to cover what was not published, patterns that technically met “critical appraisal criteria” yet managed to require critical thinking to see the loopholes – but critical appraisal cannot provide absolute certainty – it can only identify reasons for uncertainty.
Brian S. Alper, MD, MSPH, FAAFP
Founder of DynaMed
Vice President of Innovations and EBM Development, EBSCO Healthwww.dynamed.com, health.ebsco.com
Twitter: @BrianAlperMD
Be a TRANSFORMER (RECYCLE your STRIVING and THRIVING) … https://www.linkedin.com/pulse
/transformer-brian-alper
From: Evidence based health (EBH) [mailto:EVIDENCE-BASED-HEALTH@
JISCMAIL.AC.UK ] On Behalf Of Jon Brassey
Sent: Saturday, September 24, 2016 7:09 AM
To: EVIDENCE-BASED-HEALTH@JISCMAIL.AC.UK
Subject: Limitations of critical appraisal
Hi All,
I've just been reading about the controversy around a potentially 'dodgy' trial and it got me thinking about the limitations of critical appraisal. This trial was deemed to have been fair and influenced numerous guidelines around the globe.
Critical appraisal of articles relies on the notion that the article is a fair reflection of the trial. This appears to be an act of faith.
While I'm not saying critical appraisal is not a worthwhile activity my concern is that people see a paper, appraise it and therefore assume that it means the study is fair/good. The reality is that after an appraisal what you can say is that the trial, as reported/represented in the article, appears to be fair/good.
Does critical appraisal need a health warning?
Best wishes
jon
Jon Brassey
Director, Trip Database
Honorary Fellow at CEBM, University of Oxford
Creator, Rapid-Reviews.info
--
Jon BrasseyDirector, Trip DatabaseHonorary Fellow at CEBM, University of OxfordCreator, Rapid-Reviews.info