Jon Brassey writes:
"I've just been reading about the controversy around a potentially
'dodgy' trial and it got me thinking about the limitations of critical
appraisal. This trial was deemed to have been fair and influenced
numerous guidelines around the globe."
I'm coming at this a bit late, but here are my thoughts for what it's
worth.
I don't know what a "dodgy" trial is, but I'm guessing that it makes you
uncomfortable in a way that you can't describe accurately. Or it
violates some norm (maybe lack of a plausible mechanism?) that is not
usually discussed in critical appraisal.
If it is the former, I think you need to check your internal biases. Are
you assuming that the trial is not as good as what is reported because
you are being asked to abandon a cherished belief? If you start
speculating on things that are unobservable, you leave yourself open for
all sorts of cognitive biases.
If you believe that fraud is rampant, then you have to discount all the
studies equally, not just the ones that make you uncomfortable.
If you believe that some studies are better than what is reported and
some studies are worse, then you can't say this study makes me
uncomfortable, so it must be because of some unobservable flaw unique to
this trial that is not apparent by just reading the paper.
In Statistics, if you have a quantity that is unknown and it is
uncorrelated with any of the observed values, you have to replace it
with its expected value. The expected value may be negative if you
believe that fraud and other factors cause the quality of the trial to
be lower on average than the quality of the report. Or it may be
positive if you believe that researchers often forget to mention things
that would have improved the credibility of the research report. Or it
might be zero if you think that things cancel out. But you can't replace
an unobserved value with +5 for some studies and -5 for other studies
because your gut tells you to do this.
Now if there is some norm that is being violated, but it is something
not usually discussed in critical appraisal, that's an easy fix. Just
change the critical appraisal standards. The science base medicine
movement is doing exactly this by arguing that the research community is
not sufficiently critical of certain treatments like homeopathy which
have no plausible mechanism.
The bottom line is that if critical appraisal is limited because it does
not factor in things that are not observable, then you can't do anything
about it because there really isn't any credible alternative.
That being said, I'm all for things like clinical trial registries that
decrease the number of unobservable factors.
--
Steve Simon, [log in to unmask], Standard disclaimer
I'm blogging now! http://blog.pmean.com
|