----- Forwarded message from [log in to unmask] -----
    Date: Wed, 29 Aug 2007 20:20:52 -0400
    From: [log in to unmask]
Reply-To: [log in to unmask]
 Subject: Re: [SPM] Q: Objective criteria for when small-volume
correction is valid? [Re: Q: Are small midbrain nuclei unfairly
penalised by multiple-comparison correction procedures?]
      To: Rajeev Raizada <[log in to unmask]>


Dear Rajeev,

>
> Many thanks for the comment.
> If I understand you correctly, what you're saying essentially is
> that
> a small-volume correction is valid only if the small-volume that
> you pick
> really truly did arise from a prior hypothesis that you thought
> up
> beforehand.

The question of mathematical validity of the p-value for a given
method is whether the probability of falsely rejecting the null
using that method is equal (or, in practice, close enough) to the
nominal or desired probability. One issue that will affect p-value
validity is random sampling. This is the relevant assumption to
consider when thinking about how placing ROIs for SVCs will affect
results. If the ROI one chooses is contingent on the data, then the
random sampling assumption has been violated. You now have a
distribution that is conditioned on the criteria used to choose the
ROI, but are instead using the marginal distribution to determine
nominal p-values.


>
> This highlights an aspect of small-volume correction that puzzles
> me.
> Naturally, we all as researchers have all sorts of expectations
> and
> hunches beforehand about what our dataset might show.
> In that sense, the "dredging PubMed" line was a bit of poetic
> license.


> Indeed, it's usually only the ROIs that make sense in light of
> our
> prior hunches that we choose to investigate further.
> That's true for this possible midbrain nucleus, as well.
>
> Does this mean that the statistical validity of a small-volume
> correction
> hinges upon the externally unverifiable factor of whether
> a researcher's prior hunch about possible activations
> was really truly felt beforehand. Or whether they wrote it down,
> and then signed and dated the paper?

It seems that there are two separate issues here. One
ethical/judicial and the other mathematical. The latter, which I
discussed above, concerned how the distribution one uses to
generate nominal p-values for a test (like SVC), might not be valid
if the data are conditioned on some criteria but the distribution is
not.
In terms of the former, whether one has to "write it down"
beforehand, this seems more to do with whether you want to convince
somebody, either yourself or someone else, after you've looked at
the data that you haven't actually conditioned your ROIs on the
data. In practice, I think it would be a good idea (tangentially,
related ideas have been suggested for mandatory registration of
studies with scientific organizations before they are done to force
people not to hide their negative findings), but mathematically it
does not matter per se one way or the other. Also, the temporal
order, as you discussed, does not per se matter. Though one could
see the logic in respecting it as one could not "cheat" if one
forces oneself to select tests before the data are examined.

Best,
Eric

>
> It doesn't seem correct that something as concrete as the
> validity
> of a corrected p-value should hinge on the temporal ordering
> of thoughts that happened to pass through a researcher's mind.
>
> Put another way: nobody ever runs a small-volume correction
> on regions of the T-map that look as if they are probably empty.
> But if the validity of the procedure hinges upon pre-stating
> hypotheses and then acting on them regardless of how the data
> look,
> then that is what people should be doing, no?
>
> Is there not some objective set of criteria for determining
> when it is, and when it is not, valid to run a small-volume
> correction?
>
> Many thanks,
>
> Raj
>



----- End forwarded message -----