Dear Gabor,
>> Also, you can't limit your search region to only areas with a
>> significant F-test because that would be considered double
>> dipping.
>
> Why not? E.g. take a neuropsychological test battery with 20
> individual tests and three groups A, B, C. I would run post-hoc
> tests A vs. B, B vs. C, A vs. C for those neuropsychological tests
> only that showed a significant group effect?
I concur with you that this would be a logical procedure that follows
the ANOVA tradition.
>
>
>> For these reasons, people seem not to correct for the number of
>> multiple comparisons.
>
> But then it should be more conservative to run an F-test and limit
> any further post-hoc tests (uncorrected or corrected for multiple
> comparisons), to areas that already showed an effect? BTW, it also
> seems to be standard to conduct two one-sided t-tests A > B and B >
> A with e.g. p = .001 instead of using a corrected p = .0005.
It would indeed be more conservative to run an F-test first, but the
consequences of not doing this are not the same as when not correcting
for multiple testing across the multitude of voxels. The reason is
that if you do not correct for the multiple comparisons of the ANOVA,
your type I error is still bound at the alpha level (i.e., say 5%
false positives), but if you don't correct for the voxels, you'll
virtually always reject the null (~100% false positives when the null
is true). So these two sins should not be treated equally.
Importantly, you also need to strike a balance with type II errors.
Overall, not correcting for the ANOVA multiple comparisons seems
reasonable.
With respect of your previous questions about the neglect of F
clusters: remember that in an F test any effect could contribute to a
rejection of the null, meaning that an F cluster could be internally
dishomogenous with respect of the effect that is detecting --
theoretically at least. This makes F clusters somewhat unattractive:
because these clusters are not logically unambiguous, they might be
difficult to interpret. It's not much what the price of the added
conservativeness is buying you.
Incidentally, the ANOVA practice of carrying out an F-test first and
then going ahead with the planned comparisons at uncorrected levels
does not correct for multiple comparisons in a strong sense, as you
seem to imply. For this, you need the Scheffe's post-hoc technique.
The ANOVA tradition too strikes a balance between stringency of
correction and type II errors, obtained by introducing the idea that
these uncorrected contrasts should be 'planned'.
Best wishes,
Roberto Viviani
Dept. of Psychiatry III
University of Ulm, Germany
|