Dan - I agree that most fmri studies are probably underpowered, but I
don't beleive that loosening our Type I error rate is the way to fix
this problem. Rather, people need to run enough subjects that they
have power to find a sufficient effect size at a reasonable Type I
error rate. I think that this issue is particularly important for
studies that are exploratory, where any region that lights up will
have a story told about it. In this case it is critical to control
false positives. For studies where we are testing hypotheses about
particular regions, then I would agree that loosening the
constrictions on Type I error control could be more reasonable, but I
also worry that this gives people license to "p-value surf" until
they find a region of activation that they can tell a story about.
cheers
russ
On Feb 7, 2006, at 8:12 AM, Daniel Y Kimberg wrote:
> Alle Meije Wink wrote:
>> As you can read in Will's e-mail, it is perfectly legal to report
>> results based on a p=0.08 threshold, as long as you accurately
>> describe what you're doing. Your next big problem then is getting
>> your paper past the reviewers...
>
> This thread gives me an excuse to raise an issue I've been meaning to
> ask about. FDR and FWER are different standards, and provide
> different assurances. I don't really know much about the historical
> basis for the gold standard of FWER=0.05, but it's certainly in part
> cultural (how the standard is imposed varies across sub-fields even
> within a discipline). I wonder if there's been work (in fMRI or
> elsewhere) on which standard is "better" in the sense of maximizing
> the rate of growth of knowledge (or something like that). I can't
> imagine there's a way to do this uncontroversially, but it seems like
> a problem that could be modeled in fMRI, given a raft of assumptions
> and a decent amount of data for estimating parameters.
>
> One of the reasons I've wondered about this is that basic cognitive
> fMRI seems like an area that would benefit from relaxed standards in
> reporting. Most studies are under-powered, so it could be imagined
> that Type II error is more rampant than Type I (although dubious
> methods are also rampant). At the same time, if the major constraint
> is the rate at which journals can publish articles, then relaxed
> standards aren't going to help all that much -- the total number of
> articles will stay the same, but it will just allow some better
> decisions around the boundaries.
>
> To be a hair more on-topic, as Alle Meije noted above, there's no
> reason you can't use whatever standard appeals to you in reporting.
> Significance thresholds are just one piece of the puzzle in evaluating
> how scientifically informative a data set is. Reviewers are entitled
> to make decisions about what they consider meaningful, and journals
> are entitled to set policies as well. I'm personally willing to
> entertain whatever assurances a study provides, and then decide if I
> feel an article as a whole has sufficient impact in the sense of its
> effect on the state of scientific knowledge. But I've never had an
> article to review that used FDR, and I don't know what the relevant
> journal policies are, so I'll stop now.
>
> dan
---
Russell A. Poldrack, Ph.d.
Associate Professor
UCLA Department of Psychology
Franz Hall, Box 951563
Los Angeles, CA 90095-1563
phone: 310-794-1224
fax: 310-206-5895
email: [log in to unmask]
web: www.poldracklab.org
|