Dear Helmut,
Will might be answering this at the same time as I do, but I have been too
lazy to walk down the stairs to find out... let's see whether we have
e-mail crossings again ;-)
Q1: In principle, there is no need to worry about large Bayes factors. It
is perfectly reasonable that some models should be much better in
explaining the empirical data than others. I am more puzzled about two of
your models (mode 1 and 2) showing perfectly identical model fit. This
does seem unlikely to me, unless the models have identical structure.
Q2: If I understand you correctly you are asking for measures of "absolute"
model fit. You could, in principle, compute measures like "percent
variance accounted for", but such measures are purely descriptive. How to
derive a measure of absolute fit that has inferential meaning is something
that has been bugging me as well, and I do not have a good answer, I'm
afraid. One thing you could do within the Bayesian Model Comparison
framework is to compare your DCM against its GLM-equivalent, i.e. a model
in which all inputs affect all regions, but no connections exist between
regions (see Fig. 2 in Stephan 2004, J. Anat. 205:443-470).
Best wishes,
Klaas
>Dear Wil and list,
>
>having received quite encouraging input during the SPM short course the
>past week, we have embarqued on doing some DCM pilot analyses. Not
>everything seems completely hopeless...but we need some help in
>interpreting the results.
>
>1) background: when looking at the "output", i.e. data and model
>predictions, we can see that the model is _relatively_ poor. 'Poor' is
>based on the visual impression, 'relatively' is based on the observation
>that the fit of the conventional GLM is similarly poor but still gives
>highly (0.05 corr) significant results.
>Also, we do not think that the model is too bad to work with as comparing
>different models (that follow more or less two patterns based on which
>they are set up) can distinguish between a group of 'bad' and 'better'
>models (matching the 'pattern' followed when setting them up).
>
>2) Now: as can be seen in the output of the model comparison as pasted
>below, we get very strange BF values when comparing either of the
>'probable' models with any one 'bad one' (here, we pasted the output of
>models 1,2, and 3 [3 being the 'bad' model].
>
>Q1) What does it mean when we get such large BF values? Does is just tell
>us that the model involved is really inferior, or does it tell us that we
>are doing anything wrong conceptionally in the first place?
>
>Q2) In terms of judging the goodness of fit and thus the validity of the
>DCM approach given that fit - do any objective measures/approaches exist
>to justify going ahead with a model anyway, like e.g. relating the DCM
>goodness of fit to the GLM goodness of fit?
>
>Any help appreciated,
>
>Thanks,
>
>Helmut
>
>P.S. This was done using DCM as implemented in SPM2 (not SPM5).
>
>
>---------------------------------------------------------------
>Model 1:
>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_front_thalammat
> versus
>Model 2:
>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_parietal_thalam.mat
>
>All costs are in units of binary bits
>
>Region VOI_MDN_thalamusL: relative cost = 0.00, BF= 1.00
>Region VOI_prec: relative cost = 0.00, BF= 1.00
>AIC Penalty = 0.00, BF = 1.00
>BIC Penalty = 0.00, BF = 1.00
>AIC Overall = 0.00, BF = 1.00
>BIC Overall = 0.00, BF = 1.00
>
>No consistent evidence in favour of either model
>
>---------------------------------------------------------------
>---------------------------------------------------------------
>Model 1:
>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_front_thalammat
> versus
>Model 3:
>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_prec_thalam.mat
>
>All costs are in units of binary bits
>
>Region VOI_MDN_thalamusL: relative cost = 0.50, BF= 0.71
>Region VOI_prec: relative cost = -257.88, BF=
>426557236973423400000000000000000000000000000000000000000000000000000000000000.00
>AIC Penalty = 0.00, BF = 1.00
>BIC Penalty = 0.00, BF = 1.00
>AIC Overall = -257.38, BF =
>301488196972060720000000000000000000000000000000000000000000000000000000000000.00
>BIC Overall = -257.38, BF =
>301488196972060720000000000000000000000000000000000000000000000000000000000000.00
>
>Consistent evidence in favour of model 1
>Bayes factor >=
>301488196972060720000000000000000000000000000000000000000000000000000000000000.00
>
>---------------------------------------------------------------
>---------------------------------------------------------------
>Model 2:
>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_parietal_thalam.mat
> versus
>Model 1:
>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_front_thalammat
>
>All costs are in units of binary bits
>
>Region VOI_MDN_thalamusL: relative cost = 0.00, BF= 1.00
>Region VOI_prec: relative cost = 0.00, BF= 1.00
>AIC Penalty = 0.00, BF = 1.00
>BIC Penalty = 0.00, BF = 1.00
>AIC Overall = 0.00, BF = 1.00
>BIC Overall = 0.00, BF = 1.00
>
>No consistent evidence in favour of either model
>
>---------------------------------------------------------------
>---------------------------------------------------------------
>Model 2:
>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_parietal_thalam.mat
> versus
>Model 3:
>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_prec_thalam.mat
>
>All costs are in units of binary bits
>
>Region VOI_MDN_thalamusL: relative cost = 0.50, BF= 0.71
>Region VOI_prec: relative cost = -257.88, BF=
>426557236973423400000000000000000000000000000000000000000000000000000000000000.00
>AIC Penalty = 0.00, BF = 1.00
>BIC Penalty = 0.00, BF = 1.00
>AIC Overall = -257.38, BF =
>301488196972060720000000000000000000000000000000000000000000000000000000000000.00
>BIC Overall = -257.38, BF =
>301488196972060720000000000000000000000000000000000000000000000000000000000000.00
>
>Consistent evidence in favour of model 2
>Bayes factor >=
>301488196972060720000000000000000000000000000000000000000000000000000000000000.00
>
>---------------------------------------------------------------
>---------------------------------------------------------------
>Model 3:
>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_prec_thalam.mat
> versus
>Model 1:
>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_front_thalammat
>
>All costs are in units of binary bits
>
>Region VOI_MDN_thalamusL: relative cost = -0.50, BF= 1.41
>Region VOI_prec: relative cost = 257.88, BF= 0.00
>AIC Penalty = 0.00, BF = 1.00
>BIC Penalty = 0.00, BF = 1.00
>AIC Overall = 257.38, BF = 0.00
>BIC Overall = 257.38, BF = 0.00
>
>Consistent evidence in favour of model 1
>Bayes factor >=
>301488196972060720000000000000000000000000000000000000000000000000000000000000.00
>
>---------------------------------------------------------------
>---------------------------------------------------------------
>Model 3:
>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_prec_thalam.mat
> versus
>Model 2:
>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_parietal_thalam.mat
>
>All costs are in units of binary bits
>
>Region VOI_MDN_thalamusL: relative cost = -0.50, BF= 1.41
>Region VOI_prec: relative cost = 257.88, BF= 0.00
>AIC Penalty = 0.00, BF = 1.00
>BIC Penalty = 0.00, BF = 1.00
>AIC Overall = 257.38, BF = 0.00
>BIC Overall = 257.38, BF = 0.00
>
>Consistent evidence in favour of model 2
>Bayes factor >=
>301488196972060720000000000000000000000000000000000000000000000000000000000000.00
>
>---------------------------------------------------------------
|
