Alexis,
> 1) You mentioned that "permutation methods are exact". Does this
> statement mean that p-values can never be underestimated using
> permutations, or does it have another meaning? Could you suggest
> references where this issue is discussed?
Sorry for not qualifying that; when the condition of exchangeability
under the null is satisfied, permutation methods are exact.
P-values can be underestimated, to the extent that the permutation
P-values are discrete and valid; so if the "true" P-value is 0.00001,
and you only have 20 permutations, the permutation P-value can be no
smaller than 0.05.
Also, if a Monte Carlo permutation test is used (in SnPM we call
it an 'approximate' test), then the P-values are an estimate of
what you'd get by running all permutations. But if you run enough
permutations (say 10,000) you'll be very close. See
http://www.jiscmail.ac.uk/cgi-bin/wa.exe?A2=ind0501&L=spm&P=R22179&I=-1
In any event, the P-values are unbiased, since we're using a
randomly selected sample of all possible permutations.
As for references... I can't point to a specific paper or book on
permutation's exactness. I'd start with Good's "Permutation Tests"
(or his new 2005 book "Permutation, Parametric, and Bootstrap Tests of
Hypotheses"); if you want to see permutation methods justified from the
sigma-field and on up, check out Pesarin's "Multivariate Permutation
Tests".
> 2) The permutation method you suggested for random-effect analysis
> relies on a sign exchangeability assumption. As pointed out in one
> of your articles, this boils down to assuming that the effect is
> symetrically distributed in the population under investigation. When
> symmetry doesn't hold, we can no longer claim that the permutation
> method is "exact" -- am I correct?
Right; then all bets are off.
> 3) We, in my lab, have both sign permutation and bootstrap
> implementations of RFX analysis. In our experience, the Bootstrap is
> always (slightly or a lot) more conservative than sign permutations. But
> that's an empirical observation...
Interesting. But does this observation apply to univariate (uncorrected)
P-values, or FWE-corrected (max-based) P-values?
-Tom
-- Thomas Nichols -------------------- Department of Biostatistics
http://www.sph.umich.edu/~nichols University of Michigan
[log in to unmask] 1420 Washington Heights
-------------------------------------- Ann Arbor, MI 48109-2029
|