Good idea Timmy. As I have nothing better to do:
The x-axis shows the "true" p-value. The y-axis shows in what fraction
of permutation-based experiments we claim to achieve p<0.05, given the
actual p-value on the x-axis. The different curves for the different
number of permutations show that the more permutations you do, the
more accurate is your reporting.
Tom interprets this to mean that you should really do at least 5000
permutations. I claim that generally, 500 is good enough - for
example, this means that if you do 500 permutations, and the true p-
value is actually 0.06, there is about a 20% chance that you will
claim that it is <0.05. I would say that for many applications this is
acceptable.....
Wrt the FDR question, yes, if you only have one voxel activated then
FDR will be 'conservative'.
Cheers, Steve.
On 13 Mar 2008, at 15:07, Tim Behrens wrote:
> Yes - I'm curious as well.
> It would nice to see the effects of #perms on a nice easily
> interpretable graph.
> T
>
> On 13 Mar 2008, at 15:00, Marenco, Stefano (NIH/NIMH) [E] wrote:
>
>> I am curious about the number of permutations necessary for TFCE.
>> 5000
>> takes a very long time (>3 days for us), yet I thought that
>> reducing the
>> number of permutations basically results in widely varying p values
>> if
>> the same analysis is run a second time. I remember snPM advertising
>> 2000
>> permutations as a reasonable number. What is the current empirical
>> understanding of this?
>>
>> Another question I have is the following. We have a single
>> significant
>> voxel FWE corrected in one analysis, but when we try to do a fdr
>> corrected analysis, nothing comes out significant. Is this possible?
>>
>> Stefano Marenco, NIMH
>>
>> -----Original Message-----
>> From: Steve Smith [mailto:[log in to unmask]]
>> Sent: Thursday, March 13, 2008 3:29 AM
>> To: [log in to unmask]
>> Subject: Re: [FSL] SVC for multiple comparison
>>
>> Hi - I'm afraid you're not allowed to reduce your set of considered
>> voxels (in order to reduce the effects of multiple comparisons) using
>> the same data before and after - that's cheating and will make Tom
>> cry.
>>
>> You _are_ for example, allowed to find a (corrected) significant ROI
>> in the FA, and then only test the MD in that ROI.
>>
>> But you're NOT allowed to find an (uncorrected) 'significant' ROI in
>> FA, and then only test within there for multiple comparisons - this
>> goes against the whole reason for needing to do multiple comparisons
>> on the original full set of voxels.
>>
>> If you're needing to boost significance I would just recommend
>> testing
>> the -tfce option (probably using H=1 and E=1 and just 500
>> permutations
>> to start with)
>>
>> Cheers.
>>
>>
>>
>> On 11 Mar 2008, at 20:59, Versace, Amelia wrote:
>>
>>> Dear FSL experts,
>>>
>>> I am trying to do small volume correction for multiple comparison in
>>> DTI data, because I got significant results just in tbss_*_voxtstat*
>>> image (uncorrected p value).
>>>
>>> I was wondering if the following steps are correct:
>>>
>>> 1. run tbss -i all_FA.nii.gz -o tbss_* -m mean_FA_skeletonized _mask
>>> -d design.mat -t design.con -c 3 -n 10000 -v 5
>>>
>>> 2. define 1-voxtstat* image (fslmaths tbss_*_voxtstat* -mul -1 -add
>>> 1 tbss_*_1-voxtstat*)
>>>
>>> 3. define a WM-mask (accordingly with mean_FA_skeleton_mask)
>>> surrounding a significant roi (group of voxels with p>0.999).
>>>
>>> 2. run fdr -i tbss_*_1-voxtstat* -m WM_mask -q 0.05
>>>
>>> 4.if the output is >0, can I consider that roi as small volume
>>> corrected for multiple comparison??
>>>
>>>
>>> About point 3, is there any size limitation of WM-mask in order to
>>> use FDR properly??
>>>
>>>
>>> If this is not the proper way, can anybody suggest a better one?
>>> Many thanks for your help!
>>> Amelia
>>>
>>
>>
>> ------------------------------------------------------------------------
>> ---
>> Stephen M. Smith, Professor of Biomedical Engineering
>> Associate Director, Oxford University FMRIB Centre
>>
>> FMRIB, JR Hospital, Headington, Oxford OX3 9DU, UK
>> +44 (0) 1865 222726 (fax 222717)
>> [log in to unmask] http://www.fmrib.ox.ac.uk/~steve
>> ------------------------------------------------------------------------
>> ---
>>
>
---------------------------------------------------------------------------
Stephen M. Smith, Professor of Biomedical Engineering
Associate Director, Oxford University FMRIB Centre
FMRIB, JR Hospital, Headington, Oxford OX3 9DU, UK
+44 (0) 1865 222726 (fax 222717)
[log in to unmask] http://www.fmrib.ox.ac.uk/~steve
---------------------------------------------------------------------------
|