On 11/9/2012 12:39 PM, Ahmed Abou-Setta, M.D. wrote:
> The problem is often in the literature we often see that some areas
> are dominated by certain groups of researchers, and in certain
> cases, the majority of the evidence come from these isolated pockets
> of researchers and by sheer numbers the evidence becomes in favor of
> whatever they state.
So let's make the example more concrete. Harvard does a lot of research,
so they might qualify as an "isolated pocket." You want to downgrade
research just because it comes from Harvard? The risk of biasing your
results because you are cherry picking far outweighs the risk that
Harvard has a systematic and predictable bias in research.
If one group of researchers do a lot of research in an area, we should
be grateful for their contributions and not suspicious. Having too much
data from a single source is not ideal, of course, but far too often we
suffer not from that but rather from having too little data from any source.
A systematic deviation between Harvard and everyone else could mean a
lot of things. It could represent a geographic effect, it could
represent the better quality of work done at Harvard, it could represent
a methodological approach to research that is unique to Harvard, or it
could be bias. I, for one, would hate to make an accusation of bias
without first investigating all the other options thoroughly.
Let the meta-analysts report various subgroups and comment on them as a
possible source of heterogeneity. Clearly having a subgroup of Harvard
research that differs from everyone else will limit generalizability of
the findings, with or without an accusation that Harvard is biased.
Furthermore, "isolated pockets" is so ill-defined as to lead to
potential spurious findings. Do you look at a "Harvard" effect or an
"Ivy League" effect or an "East Coast" effect or a "private school"
effect? Even if you hypothesize a "Harvard" effect, how do you define
it? Is it research conducted by researchers currently working at
Harvard, or should you include researchers who recently graduated from
Harvard as well? If so, how recent? How about all the people who tried
and failed to get tenure at Harvard and have moved elsewhere? Do you
look at the affiliation of the first author, or examine affiliation from
any of the authors? It sure sounds like the worst type of subgroup
analysis to me. Financial ties, though not completely unambiguous, are
still far better defined, by comparison.
So I'd encourage you to drop your effort to show that isolated pockets
of researchers are biasing our meta-analyses.
Steve Simon, [log in to unmask], Standard Disclaimer.
Sign up for the Monthly Mean, the newsletter that
dares to call itself average at www.pmean.com/news
|