Raj,

> Here's what I'm doing in the plain FDR approach: Take the set of
> behavioural scores and ROI-activations, correlate everything with
> everything else using the standard parametric p-vals from the
> standard correlation formula, sort the resulting parametric p-vals
> in ascending order, and find where they intersect the line y = q*i/V
> (in Genovese, Lazar and Nichols notation, with c(V)=1). Then that
> intersection point is the threshold p-val that gives FDR
> multiple-comparison-corrected significance at, e.g. q=0.05. Nice
> and simple, and seems to work fine.

Minor note: there might be multiple intersection points; use the
largest one.

> In the permutation version of FDR, everything is exactly the same
> except that non-parametric p-vals are computed for each individual
> correlation, by using permutation. E.g. for a given behav measure
> and a given ROI activation, shuffle the subjects, calculate the rho
> correlation value using the standard equation but discard the
> accompanying parametric p-val, and add that rho to the pile of
> permuted rho-vals. Then find the proportion of permuted
> abs(rho-vals) that exceed the abs(rho) that comes from the
> non-permuted, unshuffled subjects,

Only thing I'm not clear on is if the total set of numbers you compare
abs(rho) to is over all correlation pairs, or only the permuted
correlations from the current pair of interest. If it is over all
pairs, then you are assuming that the null distribution is the same
for all pairs; if you do it pair-by-pair, you are making weaker
assumptions, allowing them to be different.

> and use that as the non-parametric p-val from that particular
> correlation pairing. Then take all those non-parametric p-vals,
> after having done the permutation-shuffle for each
> everything-with-everything correlation, sort them in order, and
> intersect them with y = q*i/V, just as above. The only difference
> is that instead of each p-val being the output of a single
> parametric [rho,p]=corr() calculation, it's the output of a few
> thousand permutation shuffles (five thousand seems to be about as
> low as I can cut it for n=14 subjects, and the program still takes
> well over 24h to run).
>
> Two quick questions:
> 1. Do the procedures, as described above, sound valid?

Yup, except as noted... that you're better off to compute the P-value
separately for each correlation pair (instead of using a monster
permutation distribution using all correlation pairs).

> 2. Given that the permutation derived non-parametric p-vals seem to
> end up giving very similar FDR-thresholds to when the regular
> parametric p-vals are used (except that they use up an extra day or
> two of CPU-time), it seems that my data must be satisfying some set
> of statistical conditions.

Yes. The data seem to be bivariate Normal.

> Some variances must be approximately homogeneous with some other
> variances, I think. But I'm not sure which. Is it the variance,
> across subjects, of all my measures?

Variances can be different... that's not a problem.

The homogenity I was referring two as in the permutation distribution
of the different correlations. If the data are Normal and the null is
true (\rho=0) then they all will have exactly the same theoretical
distribution. But if different variables are non-Normal and
heterogeneous, say with some variables having right skew, others left
skew, some outliers, then the correlation distributions will be
heterogeneous and you shouldn't be pooling it across different
correlation pairs.

> The behavioural scores are large integers around 100, whereas the
> ROI-activations are small numbers between, say, -0.5 and 0.5. That's
> some pretty inhomogeneous variance. But nonetheless the
> permutation-based p-vals seem to be yielding FDR-thresholds similar
> to the parametric p-vals. I am puzzled about why that should be.

Again, since the permutation and Normal-theory P-values agree, it
seems like your data are approximately normal. I'm just recommending
aginst pooling permutation distributions over different correlation
pairs.

-Tom


     -- Thomas Nichols -------------------- Department of Biostatistics
        http://www.sph.umich.edu/~nichols University of Michigan
        [log in to unmask] 1420 Washington Heights
     -------------------------------------- Ann Arbor, MI 48109-2029