Hi all,

I agree with Dan *and* Russ; Type II errors are important too, but the
best solution is to collect sufficient number of subjects.  But the
one thing to remember about FDR is that it's not totally exploratory:

If the null hypothesis is true everywhere then the definition of FDR
and FWE coincide, and a FDR procedure controls FWE.  More informally,
if there's absolutely no effect, a 0.05 FDR procedure controls the
chance of one or more false positives at 0.05.  More formally, a
level-alpha FDR procedure has weak control of FWE.

So this is reassuring; if you've got total crap data, FDR shouldn't
find anything (with the usual specificity).

-Tom


     -- Thomas Nichols --------------------   Department of Biostatistics
        http://www.sph.umich.edu/~nichols     University of Michigan
        [log in to unmask]                     1420 Washington Heights
     --------------------------------------   Ann Arbor, MI 48109-2029

On Tue, 7 Feb 2006, Russ Poldrack wrote:

> Dan - I agree that most fmri studies are probably underpowered, but I don't 
> beleive that loosening our Type I error rate is the way to fix this problem. 
> Rather, people need to run enough subjects that they have power to find a 
> sufficient effect size at a reasonable Type I error rate. I think that this 
> issue is particularly important for studies that are exploratory, where any 
> region that lights up will have a story told about it.  In this case it is 
> critical to control false positives.  For studies where we are testing 
> hypotheses about particular regions, then I would agree that loosening the 
> constrictions on Type I error control could be more reasonable, but I also 
> worry that this gives people license to "p-value surf" until they find a 
> region of activation that they can tell a story about.
>
> cheers
> russ
>
> On Feb 7, 2006, at 8:12 AM, Daniel Y Kimberg wrote:
>
>> Alle Meije Wink wrote:
>>> As you can read in Will's e-mail, it is perfectly legal to report
>>> results based on a p=0.08 threshold, as long as you accurately
>>> describe what you're doing. Your next big problem then is getting
>>> your paper past the reviewers...
>> 
>> This thread gives me an excuse to raise an issue I've been meaning to
>> ask about.  FDR and FWER are different standards, and provide
>> different assurances.  I don't really know much about the historical
>> basis for the gold standard of FWER=0.05, but it's certainly in part
>> cultural (how the standard is imposed varies across sub-fields even
>> within a discipline).  I wonder if there's been work (in fMRI or
>> elsewhere) on which standard is "better" in the sense of maximizing
>> the rate of growth of knowledge (or something like that).  I can't
>> imagine there's a way to do this uncontroversially, but it seems like
>> a problem that could be modeled in fMRI, given a raft of assumptions
>> and a decent amount of data for estimating parameters.
>> 
>> One of the reasons I've wondered about this is that basic cognitive
>> fMRI seems like an area that would benefit from relaxed standards in
>> reporting.  Most studies are under-powered, so it could be imagined
>> that Type II error is more rampant than Type I (although dubious
>> methods are also rampant).  At the same time, if the major constraint
>> is the rate at which journals can publish articles, then relaxed
>> standards aren't going to help all that much -- the total number of
>> articles will stay the same, but it will just allow some better
>> decisions around the boundaries.
>> 
>> To be a hair more on-topic, as Alle Meije noted above, there's no
>> reason you can't use whatever standard appeals to you in reporting.
>> Significance thresholds are just one piece of the puzzle in evaluating
>> how scientifically informative a data set is.  Reviewers are entitled
>> to make decisions about what they consider meaningful, and journals
>> are entitled to set policies as well.  I'm personally willing to
>> entertain whatever assurances a study provides, and then decide if I
>> feel an article as a whole has sufficient impact in the sense of its
>> effect on the state of scientific knowledge.  But I've never had an
>> article to review that used FDR, and I don't know what the relevant
>> journal policies are, so I'll stop now.
>> 
>> dan
>
> ---
> Russell A. Poldrack, Ph.d.
> Associate Professor
> UCLA Department of Psychology
> Franz Hall, Box 951563
> Los Angeles, CA 90095-1563
>
> phone: 310-794-1224
> fax: 310-206-5895
> email: [log in to unmask]
> web: www.poldracklab.org
>
>