Forwarded post on behalf of Leigh Callinan ([log in to unmask])
Please reply to Leigh, and not to me.
----------------------------------
Some weeks ago I posted this question to the list.
I am trying to understand why in WINBUGS, a model with an extra
explanatory variable, can have a deviance greater than the model
without it.
I suspect it's because deviance accuracy assumes a very good
convergence; and an added explanatory variable might reduce overall
convergence.
Any advice on this would be much appreciated.
I got very useful advice from several people. Even more people
couldn’t help but asked me to post a summary of these responses.
The common answer was that the posterion distribution of the deviance
was skewed and comparisons of mean deviance between models was
consequently unreliable.
Here’s the summary of answers.
Mick McCarthy: Another possibility is that the minimum deviance of the
model with the extra variable should be less, but the average of the
posterior distribution of the deviance might be greater if the
posterior distribution is skewed.
Mick Deviney: How are you comparing deviance? Mean vs. mean? median
vs. median? Minimum vs. minimum? Or some other?
My reply: I used Dbar = the posterior mean of the deviance, from The
DIC Tool dialog box for the deviance comparisons. Perhaps the
distribution of the deviance was skewed such that comparisons of
means are unreliable.
Frank Deviney: I think that's highly likely, and that the convergence
issue you mentioned originally is a likely culprit. I've thought about
this issue a little bit, but I haven't come to any conclusion about
what to do about it. I think sometime you can get a good Rhat when
things haven't really converged.
Victor Ssempijja: It is possible that the model with more variables
will have parts of the likelihood that will be almost flat and this
definitely causes variance problems, refer to
Hill,B 1977. Exact and approximate Bayesian solutions for inference
about variance components and multivariate inadmissibility
Nicky Welton: I've always imagined it to be due to changes in the
shape of the posterior distribution for the deviance. The reported
value is the *mean* from this typically skewed distribution, and so
very sensitive to changes in shape. Certainly if the burn-in period
isn't long enough, then there will likely be a longer tail of the
sampled distribution causing the mean to increase. So I'd definitely
suggest increasing your burn-in period to see if the issue goes away.
Joachim Büschken wrote that he’d previously received advice on this
problem, viz:
- The covariate added might be related to the random effect in the model.
- A respondent pointed at possible problems with E(LogL) and referred
me to the Aitken paper in JRSSB (1991). However, the issues discussed
there do not explain why E(-2Log) increases with an additional
explanatory variable. It seems to me that this discussion is more
concerned with the use of E(LogL) in general.
- One respondent suggested that adding an additional variable
increases the parameter space and flattens the priors. The effect on
the posterior and, hence, the likelihood could be that observations in
the tails get more weight (?).
Gilbert Marzolin suggested I might have a look in Gelman & Hill (2007)
at page 480: "Adding a predictor can increase the residual variance".
Mike Sweeting referred me to this item in the BUGS Project Manual -
Frequently Asked Questions about DIC
If Dbar measures 'lack of fit', why can it increase when I add a covariate?
Suppose Yi is assumed to be N(0,1) under model 1, and consider a
covariate xi with mean 0 and which is uncorrelated with Y. Then it is
straightforward to show that fitting a more complex model 2: Yi ~ N(b
xi,1) leads to Dbar increasing by 1. The crucial idea is that Dbar
should perhaps not really be considered a measure of fit (in spite of
the title of Spiegelhalter et al (2002)!). Fit might better be
measured by Dhat. As emphasised by van der Linde (2005) (also
available from here ), Dbar is more a measure of model 'adequacy', and
already incorporates a degree of penalty for complexity
David Mangen: It sounds like this variable really contributed nothing
to the model. I suspect (although I do not know) that this is much
like the adjusted R-Square for a model, which will typically decrease
if irrelevant explanatory variables are added to a model.
Leigh Callinan, PhD, AStat
Biometrician, Department of Primary Industries
PO Box 3100
Bendigo DC Victoria 3554, Australia
Phone: 03 54 304342
Fax: 03 54 304304
Email: [log in to unmask]
-------------------------------------------------------------------
This list is for discussion of modelling issues and the BUGS software.
For help with crashes and error messages, first mail [log in to unmask]
To mail the BUGS list, mail to [log in to unmask]
Before mailing, please check the archive at www.jiscmail.ac.uk/lists/bugs.html
Please do not mail attachments to the list.
To leave the BUGS list, send LEAVE BUGS to [log in to unmask]
If this fails, mail [log in to unmask], NOT the whole list
|
