Dear Ben, Jon and everyone
My aim when reading a paper is "If I were to repeat this study in my own setting, what is the probability that I would get a similar result within a range acceptable to me that would prompt me to make the decision recommended by the authors (eg implementing a treatment or test)".
In order that the probability of replication of the result within some range acceptable to me is high, the probability of non-replication due to chance must be low (ie N must be high enough), the probability of hidden or obvious bias must be low, the probability of poor or dishonest reporting must be low, the probability of a major effect of differences in the methods available to me and my type of patients must also be low. The probability of some confounding factor that I have not thought about yet must also be low!
Note that if my estimated probability of replication was 0.9 and I did repeat the study and could not replicate it, this might be because my repeat study was one of the 10% of my repeat studies with a 90% frequency of replication that failed to replicate a sound study in this instance. It might well be replicated by others in the long run.
This is the sort of thing that I teach in the Oxford Handbook of Clinical Diagnosis! Please advise.
Huw
Dr Huw Llewelyn MD FRCPAberystwyth UniversitySent from my iPhone
The problem with critical appraisal is that relies on the quality of reporting, but as we showed in these two papers, the quality of reporting is not necessary equal to the quality of actual conduct:
Fair, or not fair, the underlying premise behind critical appraisal is indeed "if it was not reported, it was not done". The insistence on the quality of reporting is of obvious societal benefit ( i.e. increase trust in clinical research, enables/improves conduct of meta-analyses, etc) but it can certainly unfairly characterized some, otherwise, well-done trials.Ben
Ps While reporting clinical research is distinctly different from assessing the practice of medicine, it should be noted that the premise that if "it was not reported, it was not done", has also been widely accepted in the quality of improvement arena, which directly impacts the way how health care is financed. Not surprising, many physicians consider this unfair way to characterize their performance.
Hi Helen,
The point I'm making is that journal articles are a summary of the activity undertaken in a trial. So, when we critically appraise a journal article, the starting point appears to be that this is a fair reflection of the trial.
As I suggested, and Brian reinforced, this is a problematic assumption. As he said, critical appraisal "can only identify reasons for uncertainty" - indicating that critical appraisal can't assert an article is fair.
BW
jon
On 24 September 2016 at 19:09, Helen Pullen <[log in to unmask]> wrote:
Does your critical appraisal include looking at the statistics in a paper? It appears that statistical literacy (I'm quoting from the book 'Better doctors, better patients, better decisions') is a required skill of all those appraising the evidence. It's a very interesting topic area. I'm a medical librarian teaching critical appraisal and basic medical stats to our staff (large teaching hospital).
Kind regards
Helen Pullen BSc MCLIP
A while back we published “Evolution of evidence-based medicine to detect evidence mutations.” http://www.ncbi.nlm.nih.gov/
pubmed/25572989 or http://jrs.sagepub.com/content/108/1/8.long
This reported on a handful of high-profile or high-impact studies where critical appraisal as commonly applied was insufficient to capture the most concerning criticisms. The reasons varied – fraud in what was published, failure to cover what was not published, patterns that technically met “critical appraisal criteria” yet managed to require critical thinking to see the loopholes – but critical appraisal cannot provide absolute certainty – it can only identify reasons for uncertainty.
Brian S. Alper, MD, MSPH, FAAFP
Founder of DynaMed
Vice President of Innovations and EBM Development, EBSCO Healthwww.dynamed.com, health.ebsco.com
Twitter: @BrianAlperMD
Be a TRANSFORMER (RECYCLE your STRIVING and THRIVING) … https://www.linkedin.com/
pulse/transformer-brian-alper
From: Evidence based health (EBH) [mailto:EVIDENCE-BASED-HEALTH@
JISCMAIL.AC.UK ] On Behalf Of Jon Brassey
Sent: Saturday, September 24, 2016 7:09 AM
To: EVIDENCE-BASED-HEALTH@JISCMAIL.AC.UK
Subject: Limitations of critical appraisal
Hi All,
I've just been reading about the controversy around a potentially 'dodgy' trial and it got me thinking about the limitations of critical appraisal. This trial was deemed to have been fair and influenced numerous guidelines around the globe.
Critical appraisal of articles relies on the notion that the article is a fair reflection of the trial. This appears to be an act of faith.
While I'm not saying critical appraisal is not a worthwhile activity my concern is that people see a paper, appraise it and therefore assume that it means the study is fair/good. The reality is that after an appraisal what you can say is that the trial, as reported/represented in the article, appears to be fair/good.
Does critical appraisal need a health warning?
Best wishes
jon
Jon Brassey
Director, Trip Database
Honorary Fellow at CEBM, University of Oxford
Creator, Rapid-Reviews.info
--
Jon BrasseyDirector, Trip DatabaseHonorary Fellow at CEBM, University of OxfordCreator, Rapid-Reviews.info
--------------------------------------------------------------------
Un o’r 4 prifysgol uchaf yn y DU a’r gorau yng Nghymru am fodlonrwydd myfyrwyr
(Arolwg Cenedlaethol y Myfyrwyr 2016)
www.aber.ac.uk
Top 4 UK university and best in Wales for student satisfaction
(National Student Survey 2016)
www.aber.ac.uk