Print

---

Just for transparency, there is a discussion on the Cochrane Collaboration Group's Bulletin Board in LinkedIn around this review. One of the commentators suggested that this review is "another testimony to the necessity for proper implementation of randomization in testing treatments". This was responded to by a second commentator stating that the "EBM model... needs re-evaluation. While in theory the EBM model has merit there is much bias in the "evidence" that is used and often raises questionable quality when there is a profit motive involved in those doing the research." and they make a reference to the book by Ben Goldacre, "Bad Pharma."

Below is my response:

Before we starting jumping to conclusions on whether or not the EBM model is a success or failure based on a single snap-shot in time, we should take a closer look at the evidence. Just in the abstract alone, we can get a few glimpses of issues that can affect the external validity (generalization) of the results of this review.

(1) The question being asked is very specific: "What is the likelihood that new treatments being compared to established treatments in randomized trials will be shown to be superior?" Therefore only 'superiority' trials should be looked at, while non-inferiority and underpowered trials should all be excluded from the analyses.

(2) Only four cohorts of consecutive, publicly funded, randomized trials (743 trials) were included in the analyses. With over 25,000 reports of RCTs being published every year in Medline alone, well I'll let you do the math :) How confident then are we that this truly represents RCTs in general.

(3) The authors report "We found that, on average, new treatments were very slightly more likely to have favorable results than established treatments, both in terms of the primary outcomes targeted and overall survival." OK, what about safety. That whole purpose of the RCT might have been to prove one drug has a more preferable side-effects profile than the competitor.

(4) And of course, we are forgetting one of the main reasons to undertake a RCT... just to prove using an experimental design that one drug is equivalent to another in efficacy and safety. If we use the rationale that drugs/interventions should only be used if they are 'better' than the competitor then there would be only ONE treatment option for any ONE disease.

(5) There is a misunderstanding that the authors' statement "Random allocation is only ethical, however, if there is genuine uncertainty about which of the treatment options is preferable. If a patient or their healthcare provider is certain which of the treatments being compared is preferable they should not agree to random allocation, because this would involve the risk that they would be assigned to a treatment they believed to be inferior." Off the bat this statement sounds like it only reflects efficacy, while in fact it should encompass other aspects of treatment including effectiveness, availability, compliance, preferences, moral/religious beliefs, safety, cost, etc. If we boil the whole argument down to statistically significant efficacy the we lose all the shades of grey that is allowed in the EBM model.

Since Ben is an active member on this listserv I didn't want it to look like I was criticizing his work behind his back and so I am posting my opinion here also.

Ahmed