This is a very interesting issue and I'm currently responding to similar concerns from a reviewer. In my case we've run TBSS analyses on 4 DTI measures (FA, MD, L1, L23). These measures are strongly related to each other but do provide complementary information. Do you think that we have to correct for multiple comparisons? P<0.0125 is very stringent especially after correcting using TFCE too. I'd be interested to know the communities feeling about this point. Thanks Kx > On 13 Nov 2013, at 20:53, Thomas Nichols <[log in to unmask]> wrote: > > Dear Natalie, > >> We are addressing some reviewer concerns. My block design is ABACABAC. At the subject first-level, I modeled each of my three conditions (3 separate EVs), and specified two contrasts: B>A, and C>A with standard cluster thresholding z-score of 2.3 and significance level of p=.05. For contrast B>A (the EV for C was set at 0). Likewise, for contrast C>A (the EV for B =0). >> >> At the higher-level, I used Flame 1+2, cluster Z>2.3, p<.05. >> >> The reviewer noted that two separate analyses were performed for B and C and said this doubles chances of finding a result by chance. I have been asked to either correct for repeat testing or incorporate both conditions into the model. >> >> How can I correct for repeat testing? I understand I can’t use p=0.025 for example as changing the p-value in this way would correspond to a voxel-wise uncorrected value and does not relate to the final cluster-level corrected p-value. > > A Bonferroni correction can indeed be applied to (familywise) error corrected P-values; you simply need to change the "p<.05" above to "p<.025". > > I'm not exactly sure what the reviewer is getting at. They *might* be wondering if a common effect in B>A and C>A is due to a *decrease* in A; if you have any rest scans in your design you can try to look at the pure effect of A<0 to check this. Or, they might be after the usual multiplicity issue, that any time you look at multiple tests you increase your risk of false positives. > > I should say this second concern is a fairly knotty issue. What if you had only reported on B>A and then passed your data on to a friend who published C>A? Should you have corrected for the inference across the two papers? There's no hard and fast rule, but one line of reasoning goes like this: If you needed to look at all of a set of contrasts to answer a scientific question, then you should be correcting for multiplicity. If each contrast answers a distinct question that you interpret in isolation (and could, conceivably, be written up on it's own as a scientific work), *and* there aren't *too* many of them in total, then you can get away with out correction. > > There's no hard rule on this, and I won't try to define "too many", but here's an example of the former: Say you fit an FIR HRF, where you have, say, 10 EV's that model the HRF response at each of 10 lags. I don't think any reasonable person would say each of the 10 COPEs are answering a distinct question, and you'd have to deal with the multiplicity over the 10 tests. > > Hope this helps! > > -Tom > > > > > -- > __________________________________________________________ > Thomas Nichols, PhD > Principal Research Fellow, Head of Neuroimaging Statistics > Department of Statistics & Warwick Manufacturing Group > University of Warwick, Coventry CV4 7AL, United Kingdom > > Web: http://go.warwick.ac.uk/tenichols > Email: [log in to unmask] > Phone, Stats: +44 24761 51086, WMG: +44 24761 50752 > Fax: +44 24 7652 4532 >