David L. Doggett, Ph.D.
Senior Medical Research Analyst
Health
Technology Assessment and Information Services
ECRI, a non-profit
health services research organization
5200
Butler Pike
Plymouth Meeting, Pennsylvania
19462, U.S.A.
Phone: (610) 825-6000
x5509
FAX: (610) 834-1275
http://www.ecri.org
e-mail:
[log in to unmask]
-----Original Message-----
From: Simon, Steve, PhD [mailto:[log in to unmask]]
Sent: Wednesday, October 17, 2001 7:47 PM
To: [log in to unmask]
Subject: Re: theory or evidence?Arash Rashidian writes:
>I came across a paper by Vandenbroucke and de Craen in Ann
>Intern Med. 2001; 135:507-513. I thought it might be of interest of
>the group.>In their paper they have used the long lasting arguments on
>'homeopathy' as an excuse to provide "a mirror image for
>scientific reasoning in conventional medicine'. I feel it is a
>very readable piece of work.I did enjoy it, and I agree with the authors basic premise. But the authors also left out an important issue which needs greater discussion, I believe, in the Evidence Based Medicine literature.
This is the problem of biased assimilation effect. This is the tendency to look harder for flaws in papers that we disagree with and to overlook flaws in papers that we agree with. This has been demonstrated empirically (see MacCoun 1998 for several good examples). I see it most often when dealing with research that is highly emotional and where opinions are very strongly held. Gun control proponents are highly critical of John Lott's research that claims that concealed carry laws reduce crime and at the same time will cite much of the public health literature that claims that having a gun in the home increases your risk of injury and death. Opponents of gun control praise Lott's research and criticize the public health literature. Both sets of research are based on weak observational designs and have similar flaws and shortcomings. There is similar polarization about the relationship between IQ and heredity. One side claims that half the research is bad and cites the other half as proof. The opposite side will do the same, but will reverse the two halves.
Is it possible that we are more critical of homeopathy research not because it is bad research, but because it supports a viewpoint that we disagree with? Are proponents of homeopathy too ready to overlook the gaps in the research?
I don't want to sound like one of the post-modernist thinkers, but I do believe that it is very hard to be truly objective in evaluating research. I've noticed disagreement even on something as basic as whether a given article provides supportive evidence of or refutes homeopathy (I don't have the citation handy). And I've noticed all too often the following request: "the conclusions of this paper are all wrong--help me find the flaw in their reasoning."
What do others think about this?
MacCoun, R. (1998). "Biases in the interpretation and use of research results." Annu Rev Psychol 49: 259-87.
Steve Simon, [log in to unmask], Standard Disclaimer.
The STATS web page has moved to
http://www.childrens-mercy.org/stats