SnPM users,
Different SnPM users have emailed us about the following situation
1. Run SnPM Set Up, get results.
2. Run SnPM Set Up again, same data, get slightly different results.
What happened? This is a result of using an 'approximate test'. Ideally,
you should run all possible permutations. But if there are millions and
millions, or if you don't want to wait for, say, 10,000 permutations,
then you can opt for an approximate test, where a random sub-sample of
all possible permutations are used. Each time you run Set Up is run,
a different random sub-sample is used, and so your results may be
slightly different.
So how different will they be? And how many permutations is enough
to ensure the results won't differ too much?
If the total number of possible permutations is much larger than the
number of actually used, you can get standard errors on P-values from
an approximate permutation test. The formula is
SEp = sqrt(p*(1-p)/N)
where p is the P-value and N is the number of permutations.
For example, if I'm wondering about the variability of a P-value
around 0.05 when I use an approximate test with 2,000 permutations
(out of, say, 1 million possible permutations), then the standard error
is sqrt(0.05*(1-0.05)/2000)=0.0049; an approximate 95% CI would be
0.05+/-1.96*0.0049 = 0.05+/-0.0096. In otherwords the margin of error
is about 0.01 or 20% of 0.05, which may not be very satisfying.
If instead we did N=10,000 permutations, then the standard error for
a 0.05 P-value is 0.0022, and the margin of error 0.0042, or less than
10% of 0.05. If the total number of possible permutations isn't huge
relative to N (e.g. say N=10,000 out of 15,000 possible) then the above
results are pessimestic, and the actual standard errors and margin of
errors will be smaller.
To be clear: If you run all permutations, then there is no uncertainty,
the permutation P-value is the nonparametric P-value for that dataset,
that's that. If you just run a random sub-sample of permutations, you
can use the formula above to get a rough idea how much the results
will vary over repeated approximate test on the same data.
Hope this is helpful.
-Jun Ding & Tom Nichols
|