IAIN T JOHNSTONE wrote:
> > Just to be a little more concrete, if I see a dozen low-powered
> > studies with similar task contrasts, the thresholded maps make it easy
> > for me to make generalizations such as "motor planning almost always
> > activates the SMA, but never visual cortex."
>
> I'm afraid I disagree strongly with this line of reasoning. Thresholded
> maps don't allow any conclusions to be drawn about areas that were not
> significantly activated, other than they failed to reach significance.
> To conclude that an area is never activated because it doesn't show up
> on thresholded maps is incorrect, and I would say is exactly the problem
> with thresholded maps - they tempt people to affirm the NULL without
> proper justification. Without at least a power analysis, based upon
> reasonable interval predictions (x % signal change, although deciding on
> x for fMRI studies is quite tricky), one is really in no position to
> conclude anything at all about brain regions that show no significant
> activation.

I didn't mean to imply that we could affirm the null hypothesis by
looking at maps.  But there is a big difference between taking a null
finding to affirm the null and considering a null finding informative,
even in the absence of a power analysis.

Let me try to argue this a different way.  If you observe that across
50 studies of the difference between two task conditions, analyzed in
similar ways, some particular area A always exceeds some threshold and
some particular other area B never does, then you can generalize about
differences between activation in these locations based on this
knowledge.  Of course you wouldn't be justified in claiming that there
is no true difference between conditions in area B.  But you would be
justified in predicting that in the next similar dataset, area A would
again be more reliably active than area B.  The degree of
justification for this prediction would depend on how many studies you
observed and how consistent the pattern was.  You will of course miss
many true differences between regions, and it could be argued that
thresholding to control the FWE is a particularly bad choice if this
is how you want to proceed.  But thresholding does restrict the number
of hypotheses you're willing to consider in a useful way (bearing in
mind that the studies in question were not designed to measure
differences between regions).  Of course, best would be if the authors
of some of the studies shared your interests and could have carried
out the relevant comparisons directly.

Note that this doesn't involve any commitment to the null hypothesis
being true in B.  Even so, knowing that B does not exceed threshold in
any of the studies is still informative.  If instead of saying that B
failed to exceed threshold I told you that we didn't collect data in
B, you wouldn't be justified in drawing the same conclusions.  So in
that sense, I consider the data informative.  We still need to
consider reasons why A may have been "active" (i.e., supra-threshold)
more often, including the possibility that our sensitivity in B is
much lower for trivial reasons.

As an aside, this kind of eyeball meta-analysis, which we all do to
some extent, doesn't even depend on having a formal definition for the
regions, perfect co-registration, etc.  It's obviously possible to
argue about whether it's best done with thresholded or unthresholded
maps.  I lean a bit towards thresholded, because I'm reluctant to
encourage presenting data that I know is basically all noise,
especially when large smoothing kernels make it look patternful.

dan