Bayesian experts:
I'd like to be able to calculate a number that is a pure measure of
relative model fit. Not AIC or DIC, not something that balances fit with
complexity, but rather something that is a pure measure of fit. I could use
r-squared, but that's pretty inappropriate for non-normal data. If I was
fitting via maximum likelihood methods, I could use the maximized
log-likelihood (or alternatively, the minimum deviance). However, I'm being
Bayesian. I could use D-bar, but that contains a small penalty for
complexity (see http://www.mrc-bsu.cam.ac.uk/bugs/winbugs/dicpage.shtml#q17
). I could use D-hat, and I feel that I should be able to do so, but that's
not working for me.
An example: I have a model with lots of different pieces. Some are fixed
effects, some are random effects, one has to do with temporal
autocorrelation. I wrote model code with a number of step functions, so
that I could enable/disable pieces at will by setting the appropriate
flags. I then ran this model using eight different combinations of model
pieces, and looked at the results. Those results are in the table below.
Each of the first six columns notes whether a given element was included
(one) or not (zero). The latter four columns list model-fitness-related
metrics.
TAC year by.year spcov by.tran by.comm dhat dbar pD DIC
0 0 0 1 1 1 1220.5 1535.3 314.8 1850.1
1 0 0 1 1 1 1228.7 1535.8 307.1 1842.9
0 1 0 1 1 1 1223.4 1535.5 312.1 1847.6
1 1 0 1 1 1 1229.1 1535.9 306.8 1842.7
0 0 1 1 1 1 1226.2 1525.1 298.9 1824.0
1 0 1 1 1 1 1242.7 1525.2 282.5 1807.7
0 1 1 1 1 1 1231.7 1526.7 295.0 1821.7
1 1 1 1 1 1 1244.4 1527.2 282.8 1810.0
As D-hat is analogous to minimum deviance, and minimum deviance decreases
monotonically with complexity, I expected that D-hat would also decrease
monotonically with complexity. Note that I'm using pretty broad priors,
which should mean that this model is pretty close to an ML model,
strengthening the similarity between D-hat and minimum deviance. However,
in this example, D-hat increases monotonically with complexity. FYI, both
JAGS and OpenBUGS give the same values for D-bar, D-hat, pD, and DIC, for
all the above models.
Any ideas on what's going on here? Is there a better fitness metric than
D-bar? Thanks,
Tim Handley
Research Assistant
Mediterranean Network, NPS
805-658-5759 (CHIS)
805-370-2396 (SAMO)
-------------------------------------------------------------------
This list is for discussion of modelling issues and the BUGS software.
For help with crashes and error messages, first mail [log in to unmask]
To mail the BUGS list, mail to [log in to unmask]
Before mailing, please check the archive at www.jiscmail.ac.uk/lists/bugs.html
Please do not mail attachments to the list.
To leave the BUGS list, send LEAVE BUGS to [log in to unmask]
If this fails, mail [log in to unmask], NOT the whole list
|