On 8/29/2013 3:37 PM, Allan Stubbe Christensen wrote:

> When making national guidelines is it okay to exclude all
> observational studies and only include rct, meta-analysis and
> systematic reviews. In many cases rct are clearly better than
> observational studies. But in many areas there aren't performed any
> good quality rct, but there might be some evidence from large cohort
> studies. Should these just be discarded.
>
> Well, I strongly believe that this is an wrong approach, but what is
> your take on this? And which arguments could best be used against
> this approach?
>
> GRADE is being used for this guideline, but isn´t that a misuse not
> including all the evidence?

There's a motivational statement at my son's school and normally I hate 
those things, but this one was rather clever. It said "Hard work will 
beat talent when talent doesn't work hard." I would argue that an 
observational study will beat a randomized study when the randomized 
study was done poorly. Several others have already voiced this sentiment.

That being said, setting a quality threshold in a systematic review, and 
looking at only randomized studies is often a very reasonable approach. 
You could be even stricter and insist on looking only at randomized and 
blinded studies. You could be even stricter and insist on looking only 
at randomized and blinded studies with concealed allocation.

The problem is not drawing a line in the sand, but insisting on using 
the same line in the sand no matter what the context. But you always 
need to draw the line somewhere. So you might include some observational 
studies but exclude those observational studies that had historical 
controls.

In a perfect world, you would include all the available studies and then 
perform a sensitivity check by excluding those studies that were of 
lower quality. But if there is a wealth of data from randomized studies, 
I can't fault someone for excluding non-randomized studies. There are 
only so many hours in a day.

There's also a risk in dividing out studies by quality. I don't have the 
citation in front of me, but there was a systematic overview involving 
mammography as a screening tool. When you looked at the best seven 
studies, it showed that screening was helpful. But when you excluded 
five of those studies that failed to meet a quality threshold, the 
remaining two studies showed that screening was worthless. What do you 
do in such a circumstance? I see the take-home message as that our 
belief in the objectivity of certain methods, such as meta-analysis, 
fails to recognize that many of the subjective decisions made during the 
protocol write-up can have a strong influence on the outcome.

Here are several peer-reviewed publications that support your 
perspective that you can't afford to ignore observational studies.

Journal article: Oded Yitschaky, Michael Yitschaky, Yehuda Zadik. Case 
report on trial: Do you, Doctor, swear to tell the truth, the whole 
truth and nothing but the truth? Journal of Medical Case Reports. 
2011;5(1):179. Abstract: "We are in the era of "evidence based medicine" 
in which our knowledge is stratified from top to bottom in a hierarchy 
of evidence. Many in the medical and dental communities highly value 
randomized clinical trials as the gold standard of care and undervalue 
clinical reports. The aim of this editorial is to emphasize the benefits 
of case reports in dental and oral medicine, and encourage those of us 
who write and read them." [Accessed on May 17, 2011]. Available at:
http://www.jmedicalcasereports.com/content/5/1/179

Journal article: Bonnie Kaplan, Gerald Giesbrecht, Scott Shannon, Kevin 
McLeod. Evaluating treatments in health care: The instability of a 
one-legged stool BMC Medical Research Methodology. 2011;11(1):65. 
Abstract: "BACKGROUND: Both scientists and the public routinely refer to 
randomized controlled trials (RCTs) as being the "gold standard" of 
scientific evidence. Although there is no question that 
placebo-controlled RCTs play a significant role in the evaluation of new 
pharmaceutical treatments, especially when it is important to rule out 
placebo effects, they have many inherent limitations which constrain 
their ability to inform medical decision making. The purpose of this 
paper is to raise questions about over-reliance on RCTs and to point out 
an additional perspective for evaluating healthcare evidence, as 
embodied in the Hill criteria. The arguments presented here are 
generally relevant to all areas of health care, though mental health 
applications provide the primary context for this essay. DISCUSSION: 
This article first traces the history of RCTs, and then evaluates five 
of their major limitations: they often lack external validity, they have 
the potential for increasing health risk in the general population, they 
are no less likely to overestimate treatment effects than many other 
methods, they make a relatively weak contribution to clinical practice, 
and they are excessively expensive (leading to several additional 
vulnerabilities in the quality of evidence produced). Next, the nine 
Hill criteria are presented and discussed as a richer approach to the 
evaluation of health care treatments. Reliance on these multi-faceted 
criteria requires more analytical thinking than simply examining RCT 
data, but will also enhance confidence in the evaluation of novel 
treatments. SUMMARY: Excessive reliance on RCTs tends to stifle funding 
of other types of research, and publication of other forms of evidence. 
We call upon our research and clinical colleagues to consider additional 
methods of evaluating data, such as the Hill criteria. Over-reliance on 
RCTs is similar to resting all of health care evidence on a one-legged 
stool. [Accessed on May 24, 2011].
http://www.biomedcentral.com/1471-2288/11/65.

GA Wells, B Shea, D O'Connell, J Peterson, V Welch, M Losos, P Tugwell. 
The Newcastle-Ottawa Scale (NOS) for assessing the quality of 
nonrandomised studies in meta-analyses. Description: If you are 
conducting a systematic overview of nonrandomized studies, you need an 
objective method for evaluating the quality of these studies. The 
Newcastle-Ottawa scale provides a numeric score that you can use for 
excluding low quality studies, giving greater weight to higher quality 
studies, or for sensitivity analysis. This website was last verified on 
August 7, 2007. URL:
www.ohri.ca/programs/clinical_epidemiology/oxford.htm

Journal article: Paul Glasziou, Iain Chalmers, Michael Rawlins, Peter 
McCulloch. When are randomised trials unnecessary? Picking signal from 
noise BMJ. 2007;334(7589):349 -351. Abstract: "Although randomised 
trials are widely accepted as the ideal way of obtaining unbiased 
estimates of treatment effects, some treatments have dramatic effects 
that are highly unlikely to reflect inadequately controlled biases. We 
compiled a list of historical examples of such effects and identified 
the features of convincing inferences about treatment effects from 
sources other than randomised trials. A unifying principle is the size 
of the treatment effect (signal) relative to the expected prognosis 
(noise) of the condition. A treatment effect is inferred most 
confidently when the signal to noise ratio is large and its timing is 
rapid compared with the natural course of the condition. For the 
examples we considered in detail the rate ratio often exceeds 10 and 
thus is highly unlikely to reflect bias or factors other than a 
treatment effect. This model may help to reduce controversy about 
evidence for treatments whose effects are so dramatic that randomised 
trials are unnecessary." [Accessed on April 4, 2011]. See Critical 
Appraisal for related links and pages.
http://www.bmj.com/content/334/7589/349.abstract

Nick Black. Why we need observational studies to evaluate the 
effectiveness of health care. BMJ. 1996;312(7040):1215 -1218. Excerpt: 
"The view is widely held that experimental methods (randomised 
controlled trials) are the “gold standard” for evaluation and that 
observational methods (cohort and case control studies) have little or 
no value. This ignores the limitations of randomised trials, which may 
prove unnecessary, inappropriate, impossible, or inadequate. Many of the 
problems of conducting randomised trials could often, in theory, be 
overcome, but the practical implications for researchers and funding 
bodies mean that this is often not possible. The false conflict between 
those who advocate randomised trials in all situations and those who 
believe observational data provide sufficient evidence needs to be 
replaced with mutual recognition of the complementary roles of the two 
approaches. Researchers should be united in their quest for scientific 
rigour in evaluation, regardless of the method used." [Accessed November 
9, 2010]. Available at:
http://www.bmj.com/content/312/7040/1215.short.

Steve Simon, [log in to unmask], Standard Disclaimer.
Sign up for the Monthly Mean, the newsletter that
dares to call itself average at www.pmean.com/news