Dear Helmut, Tibor, Colin, and Christian,

Thank you so much for your helpful comments!



Apologies for not including all details regarding our initial analysis approach in the original post.

In fact, we did have some a priori specified ROIs, which were based on peak coordinates from previous studies. We used small volume correction (thresholded at voxel-wise p < .05 FWE) to investigate any group differences within these ROIs. A whole-brain analysis was also performed, using both corrected and uncorrected (p < .001) thresholds. We chose these thresholds as we thought they are appropriate and conventional.

As mentioned, no significant effects emerged.



When using a whole-brain threshold of p < .005 (cluster extent 50 voxels) like the reviewer suggested, we do find some significant group differences - one cluster is located within one of our a priori ROIs. 



We found it a bit difficult to know how to treat these results, as they are only significant at a threshold we obviously decided to be too liberal prior to the analyses. In hindsight, I guess it could be argued that we should have relaxed the threshold for the ROIs (i.e. using a p < .005 threshold), but this is not what we originally did.. (nor is it what the reviewer suggests, i.e. he wants a whole-brain look at the .005 threshold).



At this point, it is difficult to present these results in an unbiased way, and that's why we leaned towards interpreting these as "trend-level" differences (which is in fact true, given our original threshold). 



By the way I accept the fact that extracting volumes from the findings are circular and will avoid this (perhaps only for illustrative purposes).



I will take your suggestions into account, thanks again for the comments!



Best,

Lasse



Lasse Bang

Researcher / PhD Candidate

Regional Department for Eating Disorders (RASP) 

Oslo University Hospital, Ullevål HF

Oslo, Norway

E-mail: [log in to unmask]

Phone: +47 23 02 73 71 /+47 41 42 97 04

 

IKKE SENSITIVT INNHOLD



-----Opprinnelig melding-----

Fra: Christian Gaser [mailto:[log in to unmask]] 

Sendt: 13. januar 2016 09:06

Til: [log in to unmask]; Lasse Bang

Kopi: Christian Gaser

Emne: Re: Reviewer's comment: "Liberal" thresholds in VBM



Dear Lasse,



On Tue, 12 Jan 2016 10:19:24 +0000, Lasse Bang <[log in to unmask]> wrote:



>Dear experts,

>

>I was hoping someone could comment on the following:

>We have performed a VBM study (using the VBM8 toolbox, voxel dimensions are 1x1x1), where we used a two-sample t-test (whole-brain) to compare GM volumes of patients (n = 22) vs. controls (n = 22). We initially used a voxel-wise FWE threshold of p < .05, and then a more liberal threshold of p < .001, uncorrected for multiple comparisons. We were unable to detect any significant group differences using these thresholds.

>

>A reviewer is suggesting we use an even more liberal threshold (p < .005, with a minimum cluster extent of 50 voxels). Using this threshold, some group differences emerged in the supplementary motor area (439 voxels), superior frontal gyrus (78 voxels), middle frontal gyrus (83 voxels), and supramarginal gyrus (125 voxels). At least of two of these areas are of particular interest, as previous research have indicated that patients may have reduced GM volume in these regions.

It is somewhat weird that a reviewer suggest such an arbitrary threshold that is quite uncommon in a statistical sense (I would rather have some concerns reading that in a manuscript as reviewer). Although this is not the most elegant way to do this posthoc I would suggest to try the following:



1. Try a more liberal height threshold and report p-values corrected for multiple comparisons using cluster statistic.

2. Closely linked to #1 you can also try to use threshold-free cluster enhancement (TFCE, Smith. S. et al. Neuroimage 2009) which combines height and cluster threshold. For that purpose you can use randomise (FSL) or the TFCE toolbox (http://www.neuro.uni-jena.de/tfce/)

3. Follow the suggestion of the reviewer, but use a more common threshold of p<0.01 or p<0.05 and report your results as explorative. In case that you had a clear anatomical hypothesis (e.g. focusing on some regions that are previously reported) before doing the analysis it is also appropriate to report results using uncorrected p-values. Because you had a hypothesis you can now report your results as significant and not explorative.

4. If you have a clear anatomical hypothesis about expected regions you can also apply small volume correction (SVC). The smaller search volume (e.g. you don't expect to find something outside your hypothesized regions) will result in a more liberal threshold corrected for multiple comparisons.

5. FDR can be also used to correct for multiple comparisons. The default in SPM is the topological FDR, but you can also try the (old) voxel-wise FDR by setting the following in spm_defaults.m

defaults.stats.topoFDR      = 0;



>

>Initially, I was a bit reluctant to use this threshold, as I have rarely seen it in the VBM literature.

>So far, I have referred to these results as �trend-level� group differences, and treat them as such.

>

>What are your opinions on using such a threshold: Is it acceptable to present these results as trend-level differences?

>I was thinking that one way to elucidate these group differences in greater detail would be to extract the mean values from each cluster

>using the get_totals script; so that I could present an effect size measure of these results (and perhaps correlate regional volumes with other interesting variables). Does this approach seem like a reasonable effort to accommodate the reviewer's comment?

As Helmut already mentioned this posthoc ROI analysis will be biased. It is some kind of a circular analysis (double dipping) where you use the same dataset for selection and selective (posthoc) analysis (Kriegeskorte et al. Nature Neurocience 2009).



Best,



Christian

________________________________________



Christian Gaser, Ph.D.

Associate Professor of Computational Neuroscience/Neuroimaging

Biomagnetic Center

Structural Brain Mapping Group

Department of Neurology

Jena University Hospital

Erlanger Allee 101, D-07747 Jena, Germany

Tel: ++49-3641-9325778 Fax:   ++49-3641-9325772

e-mail: [log in to unmask]

http://www.neuro.uni-jena.de



>

>Any comments appreciated!

>

>Best,

>-Lasse

>Lasse Bang,

>Ph.D. candidate

>Regional Department for Eating Disorders

>Oslo University Hospital

>

>

>Lasse Bang

>Researcher / PhD Candidate

>Regional Department for Eating Disorders (RASP)

>Oslo University Hospital, Ullev�l HF

>Oslo, Norway

>E-mail: [log in to unmask]<[log in to unmask]" target="_blank">http:[log in to unmask]>

>Phone: +47 23 02 73 71 /+47 41 42 97 04

>

>IKKE SENSITIVT INNHOLD

>

>