Dear Klaas,
> Will might be answering this at the same time as I do, but I have been too
> lazy to walk down the stairs to find out... let's see whether we have
> e-mail crossings again ;-)
Any one answer will be fine...
>
> Q1: In principle, there is no need to worry about large Bayes factors. It
> is perfectly reasonable that some models should be much better in
> explaining the empirical data than others. I am more puzzled about two of
> your models (mode 1 and 2) showing perfectly identical model fit. This
> does seem unlikely to me, unless the models have identical structure.
This is so because I selected two different regions (one in either model)
which are anatomically distant but functionally connected. This should
explain the result. Which at the same time brings up my next question: is
there any way in distinguishing which one of different regions for which
functional connectivity has been established (SVD) is the driving region? Or
are they all equivalent? So far, I have not included the within one model.
Would doing this be the key?
> Q2: If I understand you correctly you are asking for measures of
> "absolute" model fit. You could, in principle, compute measures like
> "percent variance accounted for", but such measures are purely
> descriptive. How to derive a measure of absolute fit that has inferential
> meaning is something that has been bugging me as well, and I do not have a
> good answer, I'm afraid. One thing you could do within the Bayesian Model
> Comparison framework is to compare your DCM against its GLM-equivalent,
> i.e. a model in which all inputs affect all regions, but no connections
> exist between regions (see Fig. 2 in Stephan 2004, J. Anat. 205:443-470).
Thank you for this hint which sounds promising. I will have a look. The
basic question is: if I was ever to report this in a paper, I could - as
others have done - just not talk about the fit ("don't wake a sleeping
dog" - gibt's das?), but if I were - how could I make the point that in our
subjects this is similar to the - equally bad :-( - fit of the GLM for those
regions.
Thanks again,
Helmut
>
> Best wishes,
> Klaas
>
>
>>Dear Wil and list,
>>
>>having received quite encouraging input during the SPM short course the
>>past week, we have embarqued on doing some DCM pilot analyses. Not
>>everything seems completely hopeless...but we need some help in
>>interpreting the results.
>>
>>1) background: when looking at the "output", i.e. data and model
>>predictions, we can see that the model is _relatively_ poor. 'Poor' is
>>based on the visual impression, 'relatively' is based on the observation
>>that the fit of the conventional GLM is similarly poor but still gives
>>highly (0.05 corr) significant results.
>>Also, we do not think that the model is too bad to work with as comparing
>>different models (that follow more or less two patterns based on which
>>they are set up) can distinguish between a group of 'bad' and 'better'
>>models (matching the 'pattern' followed when setting them up).
>>
>>2) Now: as can be seen in the output of the model comparison as pasted
>>below, we get very strange BF values when comparing either of the
>>'probable' models with any one 'bad one' (here, we pasted the output of
>>models 1,2, and 3 [3 being the 'bad' model].
>>
>>Q1) What does it mean when we get such large BF values? Does is just tell
>>us that the model involved is really inferior, or does it tell us that we
>>are doing anything wrong conceptionally in the first place?
>>
>>Q2) In terms of judging the goodness of fit and thus the validity of the
>>DCM approach given that fit - do any objective measures/approaches exist
>>to justify going ahead with a model anyway, like e.g. relating the DCM
>>goodness of fit to the GLM goodness of fit?
>>
>>Any help appreciated,
>>
>>Thanks,
>>
>>Helmut
>>
>>P.S. This was done using DCM as implemented in SPM2 (not SPM5).
>>
>>
>>---------------------------------------------------------------
>>Model 1:
>>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_front_thalammat
>> versus
>>Model 2:
>>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_parietal_thalam.mat
>>
>>All costs are in units of binary bits
>>
>>Region VOI_MDN_thalamusL: relative cost = 0.00, BF= 1.00
>>Region VOI_prec: relative cost = 0.00, BF= 1.00
>>AIC Penalty = 0.00, BF = 1.00
>>BIC Penalty = 0.00, BF = 1.00
>>AIC Overall = 0.00, BF = 1.00
>>BIC Overall = 0.00, BF = 1.00
>>
>>No consistent evidence in favour of either model
>>
>>---------------------------------------------------------------
>>---------------------------------------------------------------
>>Model 1:
>>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_front_thalammat
>> versus
>>Model 3:
>>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_prec_thalam.mat
>>
>>All costs are in units of binary bits
>>
>>Region VOI_MDN_thalamusL: relative cost = 0.50, BF= 0.71
>>Region VOI_prec: relative cost = -257.88, BF=
>>426557236973423400000000000000000000000000000000000000000000000000000000000000.00
>>AIC Penalty = 0.00, BF = 1.00
>>BIC Penalty = 0.00, BF = 1.00
>>AIC Overall = -257.38, BF =
>>301488196972060720000000000000000000000000000000000000000000000000000000000000.00
>>BIC Overall = -257.38, BF =
>>301488196972060720000000000000000000000000000000000000000000000000000000000000.00
>>
>>Consistent evidence in favour of model 1
>>Bayes factor >=
>>301488196972060720000000000000000000000000000000000000000000000000000000000000.00
>>
>>---------------------------------------------------------------
>>---------------------------------------------------------------
>>Model 2:
>>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_parietal_thalam.mat
>> versus
>>Model 1:
>>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_front_thalammat
>>
>>All costs are in units of binary bits
>>
>>Region VOI_MDN_thalamusL: relative cost = 0.00, BF= 1.00
>>Region VOI_prec: relative cost = 0.00, BF= 1.00
>>AIC Penalty = 0.00, BF = 1.00
>>BIC Penalty = 0.00, BF = 1.00
>>AIC Overall = 0.00, BF = 1.00
>>BIC Overall = 0.00, BF = 1.00
>>
>>No consistent evidence in favour of either model
>>
>>---------------------------------------------------------------
>>---------------------------------------------------------------
>>Model 2:
>>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_parietal_thalam.mat
>> versus
>>Model 3:
>>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_prec_thalam.mat
>>
>>All costs are in units of binary bits
>>
>>Region VOI_MDN_thalamusL: relative cost = 0.50, BF= 0.71
>>Region VOI_prec: relative cost = -257.88, BF=
>>426557236973423400000000000000000000000000000000000000000000000000000000000000.00
>>AIC Penalty = 0.00, BF = 1.00
>>BIC Penalty = 0.00, BF = 1.00
>>AIC Overall = -257.38, BF =
>>301488196972060720000000000000000000000000000000000000000000000000000000000000.00
>>BIC Overall = -257.38, BF =
>>301488196972060720000000000000000000000000000000000000000000000000000000000000.00
>>
>>Consistent evidence in favour of model 2
>>Bayes factor >=
>>301488196972060720000000000000000000000000000000000000000000000000000000000000.00
>>
>>---------------------------------------------------------------
>>---------------------------------------------------------------
>>Model 3:
>>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_prec_thalam.mat
>> versus
>>Model 1:
>>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_front_thalammat
>>
>>All costs are in units of binary bits
>>
>>Region VOI_MDN_thalamusL: relative cost = -0.50, BF= 1.41
>>Region VOI_prec: relative cost = 257.88, BF= 0.00
>>AIC Penalty = 0.00, BF = 1.00
>>BIC Penalty = 0.00, BF = 1.00
>>AIC Overall = 257.38, BF = 0.00
>>BIC Overall = 257.38, BF = 0.00
>>
>>Consistent evidence in favour of model 1
>>Bayes factor >=
>>301488196972060720000000000000000000000000000000000000000000000000000000000000.00
>>
>>---------------------------------------------------------------
>>---------------------------------------------------------------
>>Model 3:
>>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_prec_thalam.mat
>> versus
>>Model 2:
>>R:\p0414EEG\spike\analyses\spm2\appended_sess12_block_24\DCM_parietal_thalam.mat
>>
>>All costs are in units of binary bits
>>
>>Region VOI_MDN_thalamusL: relative cost = -0.50, BF= 1.41
>>Region VOI_prec: relative cost = 257.88, BF= 0.00
>>AIC Penalty = 0.00, BF = 1.00
>>BIC Penalty = 0.00, BF = 1.00
>>AIC Overall = 257.38, BF = 0.00
>>BIC Overall = 257.38, BF = 0.00
>>
>>Consistent evidence in favour of model 2
>>Bayes factor >=
>>301488196972060720000000000000000000000000000000000000000000000000000000000000.00
>>
>>---------------------------------------------------------------
>
>
|