First, let me apologize for a rather harsh tone of my language in my email below; it was meant to be an internal memo in our group but somehow it got posted on the SPM mailing list unedited.

 

It turned out, the strange phenomenon described below is caused by a mismatch between the number of in-mask voxels between spm_uc_FDR.m and spm_P_FDR.m. You can correct this problem by editing a line in spm_uc_FDR.m function. Alternatively you can edit two lines in wfu_spm_getSPM5.m function to achieve the same result.  The edited versions of both functions are attached. Either of these changes should correct the problem.

 

Finally, being a statistician, I don’t think it’s a good idea to relax the threshold until you see a blob. I know some people do this, but changing the threshold after seeing the results doesn’t sound like a good science.

 

-Satoru

 

---

From: SPM (Statistical Parametric Mapping) [mailto:[log in to unmask]] On Behalf Of Simon Jones
Sent: Monday, February 23, 2009 11:24 AM

To: [log in to unmask]
Subject: Re: [SPM] wfu pickatlas problem?

 

Dear Satoru and Tom

 

I had a similar issue as Wei-chun Wang.

 

I have analyzed some data and using FDR 05 threshold within an ROI in the PickAtlas and receive no suprathreshold clusters.  But subsequently using the standard SPM SVC clusters are detected. 

 

Page 1. Standard SPM analysis using uncorrected threshold p 001.  Some coordinates appear to have an FDR significance < 05.

Page 2. Standard SPM using FDR corrected threshold identifies the same coordinates.

Page 3. Apply SVC to page 2 on an ROI results makes the FDR corrected values slightly more significant.

Page 4. Same data using PickAtlas with the ROI in 3 and uncorrected threshold p 001. Similar results to 1.

Page 5. Same data using PickAtlas with the ROI and FDR corrected threshold p 05 and no significant voxels are found.

 

I think SPM SVC applies threshold to data already passed through to the results so the PickAtlas differs?  If I pick a higher FDR threshold than 0.05 such as 0.1 then the significant voxels are reported by the PickAtlas with values like 0.034 etc.  I attach a graph of the statistics similar to Satoru’s.  The blue SPM FDR 05 crosses once, the red PickAtlas 0.05 do not cross (so no clusters found), the green PickAtlas 0.1 threshold actually crosses the red wiggly line 3 times ( the point at 114 is off the scale shown) so this time it finds clusters.  What conclusions can be made? You can see in page 4 that some of the main blob is cut by the ROI.  I am not trying to tweak thresholds.

 

Thank you for your help

 

Simon

 

 

From: SPM (Statistical Parametric Mapping) [mailto:[log in to unmask]] On Behalf Of Thomas Nichols
Sent: 23 February 2009 13:23
To: [log in to unmask]
Subject: Re: [SPM] wfu pickatlas problem?

 

Dear Joe et al,

 

In fact there's no problem with the plot of P-values starting above the line.  For a level-alpha FDR procedure, all you need is one or more P-values that fall below the slope-alpha line, and the largest such P-value defines the FDR threshold.  In statistics lingo, this Benjamini & Hochberg FDR method is a "step-up" test, which essentially works from worst P-value and steps 'up' until it finds a significant P-value and rejects that null and all others with smaller P-values; this is in contrast to a step-down test which starts from the most significant P-value, and and stops at the first non-significant P-value.  (I'm Satoru remembers this, as we co-wrote a paper explaining the difference [1] :)

 

Anyway, I can't help explain Wei-chun's problem, but wanted to clarify this issue.

 

-Tom

 

[1] Thomas Nichols and Satoru Hayasaka

Controlling the familywise error rate in functional neuroimaging: a comparative review

Statistical Methods in Medical Research 2003; 12: 419-446

 

On Tue, Feb 10, 2009 at 9:52 PM, Joseph Maldjian <[log in to unmask]> wrote:

Wei-Chung,

Below is the response from Satoru Hayasaka in our group on your
dilemma...



So, Ben gave me a list of t-scores for this ROI, and I was able to
figure out why this person got strange results. As I said yesterday, the
FDR threshold is determined by plotting uncorrected p-values from the
smallest to the largest. Then you draw a straight line corresponding the
FDR value of your choice (0.05, 0.072, etc), and wherever the p-value
curve crosses this straight line is the FDR threshold. See the attached
figure for an illustration of this.

Now, in order for the FDR threshold to exist, the p-value curve should
start below the straight line, and crosses the straight line as it
increases. Unfortunately, this person's p-value curve doesn't start
below the straight line, so theoretically the FDR threshold is
undefined, and nothing survives the FDR correction. But this person
played with the FDR threshold value to cause a double-crossing (see
yellow line). This, in no way, makes the FDR-corrected p-values valid;
this FDR-threshold "fooled" (hopefully unintentionally) spm_P_FDR.m and
made it produce invalid p-values. I guess whoever wrote spm_P_FDR.m
wasn't prepared for a scenario like this.

So, this is the answer to this conundrum. Nothing really survived; but
this person tweaked the threshold and produced the results which are
invalid and misleading. So, I guess you have to call a spade a spade.
Nothing survived at FDR<0.05, and that's the end of the story. It has
nothing to do with PickAtlas.

-Satoru



------------------------------------------------------------------------
Joseph A. Maldjian, MD
Professor and Chief of Neuroradiology
Director Advanced Neuroscience Imaging Research Core (ANSIR)
Wake Forest University School of Medicine
Office:336 716-2815
fax:    336 716-2870
email: [log in to unmask]
website: www.fmri.wfubmc.edu/maldjian.htm
------------------------------------------------------------------------
-

-----Original Message-----
From: SPM (Statistical Parametric Mapping) [mailto:[log in to unmask]]
On Behalf Of Wei-chun Wang
Sent: Monday, February 09, 2009 7:38 PM
To: [log in to unmask]
Subject: [SPM] wfu pickatlas problem?

I'm trying to use pickatlas with SPM and I'm running into a problem. I
have
a ROI that appears to be just under the threshold for significant after
small volume correction (.049 FDR). However, when I use pickatlas and
.05
FDR as my threshold, my activation does not reach threshold. HOWEVER, if
I
use .072 FDR my activations show up, and in the table the corrected p
value
is properly listed as .05
To sum, my problem is that the threshold has to be relaxed by ~.022 in
order
for the actual threshold (.05) to be seen by SPM.

I've included 3 screen shots that illustrate my problem:
"uncorrected" shows what happens if i set an uncorrected p-value and
then
use SPM's SVC function. you can see that the FDR corrected is .05
"pickatlas_05" shows what happens if i use pickatlas and set the p-value
to
.05 FDR. no activations reach the threshold
and finally, "pickatlas_072" shows what happens if i use pickatlas and
set
the p-value to .072 FDR. at this point it appears most of the
activations
that reach the .05 threshold appear.

This isn't a rounding error. The activations do not show up if I try
.051 or
.06, only at .072. Also matlab shows that the FDR corrected p-value is
~.049. I was wondering if this is a bug in the software and if there is
a
workaround? Or is it a problem in my data? I was hoping to use this
program
in order to produce accurate activation maps of just the small volume
corrected ROI. So alternatively, if anyone can recommend a separate
method
for doing so I would be quite appreciative.
Wei-chun Wang

--
____________________________________________
Thomas Nichols, PhD
Director, Modelling & Genetics
GlaxoSmithKline Clinical Imaging Centre

Senior Research Fellow
Oxford University FMRIB Centre