Print

---

Hi Samia,

Hope all is well. I'll try to be brief since a lot of this has been covered by others. You have raised several points that we come across a lot when teaching systematic reviews.

First part relates to how many studies you need in order to conduct a meta-analysis. In principle, pooling (meta-analysis) can be conducted with at least two sets of analyzable data. If you have data from 2 RCTs then you can pool them statistically. Whether or you should or should not is another story. First we always tell our students that an assumption of reasonable clinical homogeneity is needed. If the studies are deemed to be from completely different populations to the point that we would not expect the effect to be similar then you should not be pooling. If you are confident in the clinical homogeneity of the studies providing the data then you can look at evidence of statistical heterogeneity. Most common tool today is the I-squared test and I won't go into this because there are numerous scientific articles and book chapters describing this strengths/ weaknesses and controversies for each approach.

Second part relates to the source of your data. You are contemplating pooling data from prospective randomized trials with prospective  and retrospective cohorts. Without going into too many details of why this inappropriate, I will note that it is very prevalent in the literature. There are many reasons why you shouldn't pool data from different study designs (experimental and observational as broad categories) as biases inherent to each study design make the pooled results unpredictable and may not represent the actual true effect estimate. The reason most people pool data from different study designs is out of necessity because of publication bias. Journals don't like to publish meta-analyses with two or three included trials. Throw in an additional 10 - 15 cohort studies and all of a sudden the meta-analysis looks much more robust. In fact, in many cases, you have just done the opposite.

A simple analogy would be to say that you stated a hypothetical RCT with the expectation of recruiting 1000 patients in each arm. After a few months, you realize that you will not recruit this number of patients in a realistic time frame. Therefore you decide to go to your hospital database and search for patient records that fulfill the inclusion criteria for your study (retrospective study). Even so, you still don't have the required number of patients' data. Therefore you start asking patients who fulfilled the inclusion criteria but refused randomization to still be monitored on the treatment choice they chose with their doctor to see the effect of treatment and side effects (prospective cohort). Here's where it all goes wrong... you then decide to combine the data from the randomized patients, retrospective cohort and prospective cohorts and report on the 'total patient population'. I don't know of any clinical researchers that would say this is appropriate, but we see this regularly in published systematic reviews. Like the old saying goes... garbage in... garbage out (GIGO).

Furthermore, what would the strength of the evidence be in this situation? How can you GRADE this evidence and predict effects in a specific patient population? Therefore the only reason to do this to increase statistical power and the cost is everything else. Simply put... don't do it.

Hope this helps.

Ahmed

P.S. Sorry this was longer than originally anticipated.