I agree that the validity of an obsevational study is dependent upon how
well confounders are identified and accounted for.
The relative benefits of observational v RCT studies were also debated a few
years ago in the NEJM.
Two comparative studies concluded that RCT and observational studies yield
similar results and the usual criticism that unrecognised confounders
distort observational studies may not be entirely valid.
NEJM 2000;342:1878-1886 [This takes you to a Pub Med Abstract]
NEJM 2000; 342:1887-1892 [This takes you to a Pub MEd Abstract]
A vigorous debate followed: A number of authors were critical of the way in
which the studies were compared and pointed out that evidence from
observational studies overestimates treatment effects. However, supporters
of observational methods questioned the external validity of RCT results and
pointed out that restriction to RCT evidence leads to a biased view of
research (mainly pharmaceutical based and funded). One author posed the
question, which results should be believed when observational & RCT evidence
Of particular note was the response to the correspondence by Concato
"We sympathise with all who find intellectual security in randomisation as a
method of ensuring the validity of study results."
"Surely, however, other methods (matching, stratification, adjustment and
restriction) are available to ensure validity when randomisation is absent"
Institute of Child Health
----- Original Message -----
From: "k.hopayian" <[log in to unmask]>
To: <[log in to unmask]>
Sent: Monday, August 18, 2003 9:44 PM
Subject: Re: the Million Women Study. Lancet 2003; 362:419-27 - is it
Thanks for drawing our attention to the these serious flaws in this
particular observational study. However, this should not make us write off
all observational studies to assess interventions, whether HRT or not.
Surely, the validity depends on how well confounding has been looked for
and account taken of it. An editorial in the BMJ a few years ago drew
attention to studies comparing RCTs and observational studies of the same
treatments, showing how accurately the latter could perform against the
BMJ 2001;322:879-880 ( 14 April )
Any casualties in the clash of randomised and observational evidence?
-------on 17/8/03 8:59 am, Rod Jackson at [log in to unmask] wrote:
> Dear list - the Lancet has just published a paper on a large non
> randomised study attempting to answer a series of therapy-related
> questions about the risks of breast cancer associated with use of
> different types of HRT and for different periods of exposure to HRT.
> The investigators asked over a million British women about their
> current and past use of HRT at the time they were invited for breast
> cancer screening (by mammography) as part of the national screening
> programme. The study has documented breast cancer incidence over an
> average of 2.6 years and breast cancer mortality over 4.1 years
> following the questionnaire/mammogram. However most of the analyses
> reported in the paper are based mainly on retrospectively-collected
> information on use of HRT as the follow-up is still very short.
> I had assumed we had learned from the Nurses Health Study analyses of
> HRT and disease (another non randomised study of HRT which at least
> was based on a prospective follow-up published in the NEJM
> 1991;325:756-62) that you cannot use non randomised studies to assess
> the effect of HRT on disease because of the substantial confounding
> - for example a halving of CHD risk associated with HRT uses was
> observed in the Nurses Health Study whereas RCTs show up to a
> doubling of CHD risk.
> Yet we now have the Million Women Study Collaborators making rather
> emphatic statements about the risk of breast cancer associated with
> use of HRT, different types and HRT and duration of use. My own
> critical appraisal of this study suggests that the flaws in this
> study are so significant (non-randomised and primarily retrospective)
> that one can have no confidence in the findings. They may or may not
> be correct but one cannot tell from this study. One response I have
> had already is that the RCTs don't answer some questions and this is
> the only data we have. My anwer is simple - false data is worse than
> no data as we have already discovered with the Nurses Health Study
> which probably caused significant morbidity and mortality among the
> women prescribed HRT for coronary prevention as the basis of this
> Over and above the potential for confounding in this study, are the
> added potential for bias due to retrospective analyses (not a problem
> in the Nurses Health Stud)y. A couple of illustrations of the
> problems of the retrospective data collection are given below.
> The authors report that past users were not at increased risk.
> However if HRT increases the risk of breast cancer perhaps many of
> those who were going to get HRT associated breast cancer after
> stopping, had their events before the cohort was assembled - a well
> known bias in retrospective studies.
> In an different purely hypothetical scenario lets assume HRT delayed
> breast cancer for a period of time but the delay only lasted for the
> first 5-15 years of treatment. In this scenario, the more
> appropriate cohort for the Million Women Study (ie all women who
> would have been eligible for mammogram about 15 years before the
> study questionnaire was completed) would still include most of the
> women on HRT who had their cancer delayed but would not include the
> women not taking HRT who had their cancer diagnosed during this 15
> year period prior to screening as the later group would not be
> eligible for the screening programme but the women on HRT with
> delayed cancer would.
> I would be keen to hear some debate on the validity of this study
> Rod Jackson
> Dr Rod Jackson MBChB PhD FAFPHM
> Professor of Epidemiology
> Head of Section of Epidemiology and Biostatistics
> & Director of EPIQ (Effective Practice, Informatics & Quality Improvement)
> School of Population Health
> Faculty of Medical & Health Sciences
> University of Auckland
> (Grafton Mews, 52-54 Grafton Rd)
> Private Bag 92019
> Auckland, New Zealand
> Phone: +64 (0)9-3737599 ext 86343
> Fax: +64 (0)9-3737494
> e-mail: [log in to unmask]
> EPIQ website: http://www.health.auckland.ac.nz/comhealth/epiq/epiq.htm