Dear All,
I have conducted a relatively 'exploratory' PPI analysis and I am concerned that I should be adjusting for multiple comparisons based on the independent models (seeds) tested. I would be very grateful for your advice on if and how I should be adjusting my p values for the number of seeds tested.
All seeds were identified using standard 2nd level GLM analysis of the same data. This resulted in 22 seeds, not all of which I had specific hypotheses for in terms of PPI coupling. Each seed was modelled seperately, as per https://www.jiscmail.ac.uk/cgi-bin/webadmin?A2=ind1106&L=SPM&P=R4950&1=SPM&9=A&J=on&X=138E0C08B1CF195E85&Y=C.D.GOULD%40SUSSEX.AC.UK&d=No+Match%3BMatch%3BMatches&z=4.
Given that I tested 22 different models, I expect a number my 2nd level results will be false positives. For example in one contrast, of the 22 seeds entered only one seed gave a significant FWE corrected cluster at 2nd level. What method should I use to confirm whether this result is a false positive?
I had considered Bonferroni correction of the peak/cluster p(FWE) values with the alpha adjusted for the 22 seeds, but this is rather conservative and leaves me very little to work with. I have also tried FDR correction, but I am uncertain of how to conduct the FDR so it is adjusting p values uniformly over my different contrasts. (I have 3 contrasts, all tested with the 22 different seeds. I have been using fdr.m available here: http://brainder.org/2011/09/05/fdr-corrected-fdr-adjusted-p-values).
Where I have only one significant PPI effect in the contrast (e.g. p(FWE-cluster) = .009), I can use this in FDR with all 21 other seeds having p=1, giving a total correction for the 22 hypotheses (seeds) tested and a FDR adjusted p = 0.198 for the single significant result. However, in some contrasts one seed has produced six significant targets. If I enter all six of these p(FWE-cluster) values into FDR with 21 other seeds at p=1, the FDR correction will account for 27 hypotheses tested and adjust the values accordingly. Would it be acceptable to adjust one contrast for 27 hypotheses and compare this to a separate contrast adjusted for 22 hypotheses? I had also considered using the set-level p value (so I have a single p values per seed), but this would give me a significance level related to how many target clusters the seed produces, rather than how significant the overall effect of the PPI is from that seed.
Alternatively, given that each cluster is significant after FWE correction in the 2nd level model, is it necessary to correct for the number for seeds tested?
I haven't been able to find any literature where others have corrected PPI results for the number of seeds tested. Most papers I found test no more than two seeds and there is no mention of correction for multiple comparisons. Admittedly I did go for a rather 'scatter gun' approach with my 22 seeds(!), but I was considering issues of orthogonality and the suggestion that results could be maximised by testing seeds on contrasts other than the GLM they were identified in (loosely here: http://www2.fmrib.ox.ac.uk/Members/joreilly/ppi-issues).
In your opinions and experience, is this multiple comparisons issue likely to be something reviewers (examiners!) are concerned with? In which case, which of the correction methods described above would be most appropriate, or is there some alternative which I should consider?
With Kind Regards,
Cass
|