Dear Mike,
Model comparison is not a solved problem unfortunately.
If you only aim to correct, I would suggest to use the Bonferroni
correction. We actually did a couple of tests recently (unpublished) and
it seemed the best approach. As it is quite conservative, you need to
run more permutations for each model.
Regarding the weight maps and ranks from different models: MKL is a
sparse approach, which means that it will not select correlated
information (e.g. if regions 1 and 2 have the same information, it will
only pick one). So you need to be very careful in interpreting and
comparing those maps. I would also suggest to be careful for false
positives when model performance is low (Haufe et al., 2014, Kia et al.,
2016), and please make no inference at all if the model is not
significant after permutation testing.
HTH,
Best,
Jessica
On 19/04/2018 09:05, Mike Myers wrote:
> Dear all
>
> I've a special case using the integrated MKL model in Pronto: For the
> MKL modelling, I've used beta images (from SPM GLM computations) of
> fMRI data (20 subjects) cointaining task- and fear-related brain
> activity.
> The target variables are questionnaire scores from different
> questionnaires and subscales assessing fear that have been filled out
> by the subjects (so I used the MKL Regression Approach using SVR).
> For each questionnaire (and subscales) I've trained a separate MKL
> model using correlation and MSE as performance measures (based on 1000
> permutations using nested cross-validation (hyperparameter
> optimization C: 0.1 1 10 100 1000) based on a leave one subject out
> scheme.
>
> Finally, for each MKL model, I get a correlation coefficient and a p
> value. I've trained 17 models. The respective regions weights show
> differential contributing brain regions and ranks to the predictions
> models of the questionnaires across the MKL models. I would like to
> use this info to interpret brain-questionnaire relationships
> (different fear questionnaires might be associated with different
> neural activity).
>
> My question now relates to the multiple comparisons problem. If I
> understand it correctly, I don't have to correct for multiple
> comparisons within the models (why exactly ?)
> But as I compare the 17 MKL models, I've to correct between the
> models. As Bonferroni correction would be to conservative (the
> dependent variables are the quesionnaires and they correlate), I've
> been thinking of FDR (Benjamin-Hochberg procedure). Would this be
> appropriate ? Do you see other solutons ?
>
> Many thanks for your help!
> mike
>
>
|