Dear colleague,
As a short follow up to Karl’s response:
Fixed effects and random effects procedures exist both for inference about model structure and inference about parameter estimates, respectively. It is important to keep these two things separate in order to avoid confusion.
As Karl pointed out, the first step in any DCM study is usually to perform a BMS procedure to establish which of several alternative model structures best accounts for the experimental observations. This can be a fixed effects procedure (i.e. group Bayes factor) if one assumes that the same model generated the data for all subjects (i.e. treating model structure as a fixed effect in the population). Alternatively, if one assumes that model structure is itself probabilistically distributed in the population (e.g. when studying patients in which pathophysiological processes can differ or when examining cognitive processes for which different subjects may adopt different strategies) one should use a random effects BMS procedure using the hierarchical Bayesian model we described recently. Once you have established the most likely model, you may or may not wish to proceed to making inferences about the parameter estimates of that optimal model. Note that,
as Karl already pointed out, in many instances the primary inference is about model structure, which means that no further inference step following BMS is required and all you need to do is to report some statistic (e.g. mean) of the parameter estimates for the group.
If you do wish to make inferences about particular parameters in your optimal model, again both fixed effects and random effects approaches exist; and the rationale for choosing between them is the same as for BMS above. Fixed effects parameter inference can be done in several ways, e.g. using the “DCM average” function is SPM (i.e. a Bayesian average). Random effects inference on parameters proceeds simply by taking the subject-specific maximum a posteriori estimates (MAP) and entering them into a classical frequentist test, e.g. a t-test or ANOVA.
I hope you find this short summary useful. As Karl said, we are currently preparing a tutorial paper in which we make some recommendations for good practice and which explains these issues in some more depth. This will hopefully be on its way soon.
Best wishes,
Klaas
________________________________
Von: Karl Friston <[log in to unmask]>
An: [log in to unmask]
CC: [log in to unmask]; Klaas Enno Stephan <[log in to unmask]>
Gesendet: Donnerstag, den 18. Juni 2009, 14:35:01 Uhr
Betreff: Re: Some difficulties in using DCM
Dear Jiangang,
I got somewhat confused of the
definition and selection of the fixed-effects and the random-effects
analysis in the DCM analysis, especially after reading some recent papers
about DCM analysis. In such papers, I could not make sure which analysis,
fixed- or random- effect was used in a study. Three examples are
illustrated in the following:
Example 1:
In Summerfield et al. (2006, Science 314:1311-1314), only one model was
used (thus no Bayesian model selection), and the intrinsic and bilinear
parameters were averaged across subjects, whose statistical significance
was assessed using a t-tests with a threshold p < 0.05. My question
is: Was the Bayesian random effect approach used in this
paper?
No, this paper used classical inference based on the summary statistic
approach to make an inference about the parameters
of a single model (i.e., summarize the effect of interest with a
subject-specific statistic; here coupling parameter estimates from
single-subject analyses and then use classical tests).
Example 2:
In Acs and Greenlee (2008, NeuroImage 41:380-388), fifteen models with
the same intrinsic connections but different bilinear connections were
used, and the average intrinsic and bilinear parameters were obtained
using DCM-average routine provided by SPM5. In this study, individual
model comparison was performed by Bayesian model selection, and the group
model comparison was determined based on Group Bayes Factor and Positive
Evidence Ration. My question is: Was the Bayesian fixed effect
approach used in this paper?
No; this used the conventional (fixed-effects) Bayesian model selection
(BMS). This is fine if, a priori, you think that
each subject's data were generated under the same (unknown) model. Random
effects analyses allow for random
model effects.
Example 3:
In Fairhall and Ishai (2007, Cereb. Cortex 17:2400-2406), a number of
models were used, and the intrinsic and bilinear parameters were averaged
across subjects, whose statistical significance was assessed using a
t-tests with a threshold p < 0.05. In this study, individual model
comparison was performed by Bayesian model selection, and the group model
comparison was determined based on Group Bayes Factor. My question is: Which approach, fixed effect or random effect approach was used in
this paper?
Again, because they used the group Bayes factor, this is a conventional
BMS procedure; pooling data from different subjects.
(i.e. fixed effects). Note that the tests of the model parameters
represent an inference on the parameters (conditional on a
specific model), not an inference on models.
It seems that the fixed effect
approach was usually used in combination with Bayesian model selection
and DCM-average method provided by SPM5, while random effect approach was
used in combination with t test for investigation of the statistical
significance of intrinsic and bilinear parameters (sometimes, only one
model was assumed). Is it true?.
I will not try to answer this question became Klaas Stephan et al are
preparing a Ten Simple Rules paper for DCM that
will cover good practice for inference at the group level. I will
say:
i) If there is only one model, then between-subject inferences are based
on the parameters of that model. This can
proceed using classical (e.g. t-tests or ANOVA) or Bayesian (e.g. DCM
averaging) techniques.
ii) If there are a number of models, one has to first make an inference
about models (and, if necessary, proceed to
inference on the parameters of the model selected, as above). Inference
on models can again use classical (e.g., ANOVA
of the log-evidences for each model per subject) or Bayesian (fixed or
random effects) procedures. One would only use
random effects BMS if there was some reason to suppose that different
subjects might use different models (e.g
responders and non-responders in a drug study). Fixed effects BMS reduces
to comparing the group Bayes factors,
whereas random effects analyses are newer and add an extra (group) level
to the generative model. If you use a fixed
effects BMS, it is not unusual to supplement this with a classical
analysis to ensure there are no outliers (in terms of the
Free-energy approximation to the log-evidences).
With respect
to our work, we are now using DCM analysis to investigate the effective
connectivity between six cortical regions that were distributed in
primary and higher visual cortex, and frontal cortex. The method of DCM
analysis in our study is described as following:
First, a model with reciprocal and hierarchical intrinsic
connection was assumed.
Second, on the basis of such model, several alternative
models were generated by varying the number and location of bilinear
effect. As a result, all the models assumed in our study have the same
intrinsic connection but different assumption of bilinear
effect.
Third, Bayesian selection was used for the selection of the
best explanatory model.
Finally, the parameters of the best model were averaged and
assessed across subjects. Here, my question is: how to assess
statistical significance of these parameters across subjects, using
DCM-average or t test?
I believe that many new DCM users have the similar confusion to me.
There is no need to establish that parameters are significant because you
have already made the inference that the model with these parameters
was more likely than any competing model - this is the inference.
However, as with all inferences it is good practice to report
quantitatively what you
are making an inference about, This would entail reporting the
average coupling strengths (or % change for bilinear effects) in a graph
or table.
Having said this, you may want to make the inference that the coupling
strengths (or changes) were in the same direction over subjects.
This is an inference on parameters (given the best model selected). In
this instance, one could use either classical or Bayesian inference.
Although
it would be more consistent to use Bayesian inference, most readers will
find a simple t-test (or ANOVA) easier to understand and I personally,
would be happy to use classical inference.
I hope this helps - Karl
