Rajeev,
> Many thanks indeed for your feedback, it is greatly appreciated.
> I implemented the change you suggested of collating the abs(max_corr)
> values instead of the all the corrs from each permutation.
> The p-values that result are a great deal more conservative,
> which makes sense given your point they are the FWE-corrected values.
>
> Another e.mailer suggested to me that by collating all the corrs,
> without picking out just the max from each permutation, I am
> effectively calculating the false-positive rate.
I wouldn't put it that way exactly. By pooling all of the
correlations together and creating a permutation distribution from
them, you are making an assumption of homogeneity. Frankly I would
recommend against that. Better would be to create permutation
distribtuions for each correlation, and from those create P-values for
each correlation. (Note you don't actually have to keep around the
whole permutation distribution for each correlation, you can compute
P-values on the fly). This will produce a nonparametric uncorrected
P-values.
*Once* you have uncorrected P-values for each correlation, you could
then apply the generic Benjamini-Hochberg FDR just using the observed
P-values (i.e. you don't use the permutation distribution any more).
>> Also, since correlations have variable variance, you should use a
>> Fisher's Z transformation, as that will stabalize the variance.
>> (To see why that is important, see
>> http://www.sph.umich.edu/~nichols/Docs/NicholsHayasaka.pdf Fig 2)
>
> Many thanks, I will read about this in your paper. When I plot the
> histogram of the non-transformed corrs, from the code as it
> currently stands, the end result after even just 100 permutations
> looks like a nicely Gaussian shaped curve, although I haven't done
> any quantitative checks to see if it's as truly Gaussian as it
> looks. Does this Gaussian-looking end-result distribution indicate
> that the non-transformed corrs in my data have more or less the
> correct variance structure, even without Z-transforming, or would it
> be important to do the Z-transform in any case ?
(We're now back to concerns relating to FWE).
The issue isn't how the permutation distribution of any one
correlation looks. The issue is how are the *scale* of the
distributions may differ, or how the *skew* may differ after
Fisher's transformation. If they differ alot between different
correlation pairs, then the max distribution can have sensitivity that
is quite non-uniform.
Hope this all helps!
-Tom
-- Thomas Nichols -------------------- Department of Biostatistics
http://www.sph.umich.edu/~nichols University of Michigan
[log in to unmask] 1420 Washington Heights
-------------------------------------- Ann Arbor, MI 48109-2029
|