Replies to “WinBUGS limit? ":
I would like to thank all who contributed with replies and suggestions. The actual suggestions are attached.
I would like to thank Ben Stewart-Koster and Wayne Thogmartin for suggesting the following papers that use CAR/other models with sample sizes in excess of 1000:
Building Statistical Models To Analyze Species Distributions by Andrew M. Latimer, Shanshan Wu, Alan E. Gelfand, And John A. Silander, Jr.
Thogmartin, W. E., J. R. Sauer, and M. G. Knutson. 2004. A hierarchical spatial model of avian abundance with application to Cerulean Warblers. Ecological Applications 14:1766–1779.
Thogmartin, W. E., M. G. Knutson, and J. R. Sauer. 2006. Predicting regional abundance of rare grassland birds with a hierarchical spatial count model. Condor 108:25-46.
Thogmartin, W. E., and M. G. Knutson. 2007. Scaling local species-habitat relations to the larger landscape with a hierarchical spatial count model. Landscape Ecology 22:61–75.
Thogmartin, W. E., J. R. Sauer, and M. G. Knutson. 2007. Modeling and mapping abundance of American woodcock across their breeding range in the United States. Journal of Wildlife Management 71:376–382.
Also, Congdon's three books, as suggested by Barrie Stokes, provide a favorable view regarding WinBUGS.
Jamie: thanks for the “Diggle's geostatistical approach predicting ten's of thousands of
locations” example and Besag’s example on autoregressive correlations.
Thank you once again for your help and sympathy.
Below is a repost of my query and the actual responses…
My post was, as follows:
Hi all,
I received this comment on an NIH (R-21) grant regarding the use of WinBUGS:
"... The application suggests a Bayesian hierarchical model, using WinBUGS to handle spatial-temporal structures in the multi-level analysis. This is a freeware with many limitations, especially when the sample size is large, i.e., >1000. ..."
Has anyone else encountered similar criticism regarding WinBUGS? Also, I run space-time models that exceed 1000 space time units regularly w/o encountering any problems. E.g. I used 304 spatial units along with 5 time periods = 1520 space-time units successfully. Are there any (peer-reviewed) citations on this matter? If NIH rejects WinBUGS then what are the other options? How should one reply to such comments?
Thank you and Regards,
Rudy Banerjee, PhD
Associate Research Scientist
Prevention Research Center
Berkeley, CA
The replies are, as follows (in order as received):
1 Hi Rudy,
Good question. I would be interested in any response you received.
Thanks,
David
________________________
David Pawel
U.S. Environmental Protection Agency
Radiation Protection Division (6608J)
1200 Pennsylvania Av., NW
Washington DC 20460-0001
202-343-9202
[log in to unmask]
2 Hi Rudy:
I am also working with spatio-temporal data in WinBUGS and I have run a
model on 9720 units (18 periods x 540 geographical units) without any
problem. Therefore the limitation that you mention may be due to the model
used instead to the amount of data.
Hope this helps.
Miguel.
"Miguel Angel Martinez Beneito" [log in to unmask]
3 Hi Rudy,
I have attached a paper that discusses this somewhat, from the perspective of ecology. They cover spatial models mainly, but I am pretty sure they suggest datasets over 10000 may cause a problem, not 1000. I tried to run a three level hierarchical model in winBUGS once, with a large number of observations (up near 10000) and I could get it to compile but I had trouble getting it to run. This was a model with no correlation structure included, and normally distributed response. So as far as I can tell it was the most simple you could run. The GeoBUGS program allows the incorporation of spatial information, but I am sure you will have received other info about that. I don't know a whole lot about it, having only played with it briefly, it is something I intend on getting to in a few months time. I am going to be using MatlaS where possible.
I hope this helps.
Regards
Ben.
PS. I exchanged emails with David Spiegelhalter about this and I am pretty sure he told me that BUGS can handle model 3 in the attached paper when they state that it cannot, just something to be aware of.
Ben Stewart-Koster
PhD Candidate
eWater CRC
Australian Rivers Institute
Griffith University
Kessels Rd, Nathan, 4111
Queensland, Australia.
Ph. +61 (0)7 3735 7337
Fax. +61 (0)7 3735 7615
"Those who dam streams, can build fountains,
But those of us who just let them run free,
We can move mountains."
- Michael Franti
Attachment title: BUILDING STATISTICAL MODELS TO ANALYZE SPECIES DISTRIBUTIONS
ANDREW M. LATIMER,1,4 SHANSHAN WU,2 ALAN E. GELFAND,3 AND JOHN A. SILANDER, JR.1
4 Rudy,
I look forward to seeing your summary of responses to this question; it's a
good one!
Nick
Nicholas Horton <[log in to unmask]>
5 Rudy:
Unfortunately, it sounds as though you have a reviewer who knows only enough to be dangerous. This sort of size limitation IS important when dealing with several of the spatial correlation structures, primarily because the computations scale N^3. However, that's why in spatial epidemiology, wildlife ecology, and elsewhere we such prevalent use of the spatial.car and spatial.l1 models. If I were you, I'd simply respond with a list of citations that include conditional autoregressive examples where sample sizes exceed 1000. Here are a few of mine:
Thogmartin, W. E., J. R. Sauer, and M. G. Knutson. 2004. A hierarchical spatial model of avian abundance with application to Cerulean Warblers. Ecological Applications 14:1766–1779.
Thogmartin, W. E., M. G. Knutson, and J. R. Sauer. 2006. Predicting regional abundance of rare grassland birds with a hierarchical spatial count model. Condor 108:25-46.
Thogmartin, W. E., and M. G. Knutson. 2007. Scaling local species-habitat relations to the larger landscape with a hierarchical spatial count model. Landscape Ecology 22:61–75.
Thogmartin, W. E., J. R. Sauer, and M. G. Knutson. 2007. Modeling and mapping abundance of American woodcock across their breeding range in the United States. Journal of Wildlife Management 71:376–382.
Wayne Thogmartin
Wayne E. Thogmartin, PhD
USGS Upper Midwest Environmental Sciences Center
2630 Fanta Reed Road
La Crosse, WI 54603 USA
608-781-6309 (off)
608-783-6066 (fax)
[log in to unmask]
http://www.umesc.er.usgs.gov/terrestrial/migratory_birds/bird_conservation.html
The contents of this message are mine personally and do not reflect any portion of the US government.
6 Hi Rudy
My full sympathies to you, receiving an amazing comment like that.
First Point: A colleague I just talked to says that he has a very
successful WinBUGS program which reads in 2800 (two thousand eight
hundred) real number instrumental measurements, and estimates 2800
corresponding posteriors, and another 2800 posterior predictive, so 5600
parameters in all. No sweatsky. And on a standard issue PC. I hope you
get many other similar testimonies.
Second Point: Just how would the NIH reviewer handle your modelling
situation? Notwithstanding that the Bayesian approach is the one true
statistics (don't start me!), what other modelling approach could he
suggest that could handle your spatial-temporal hierarchical model -
SAS? Don't think so ... .
IMHO, referring to WinBUGS as "freeware with many limitations" is
unkind and misinformed - to my mind it is if anything more marvellous
*because* it is free. I have Congdon's three books; the number of
different statistical models he demonstrates is amazing, and in every
case the results either accord with or go beyond those from the
available conventional model. I think WinBUGS is less limited than
almost any other package. Writing your own custom MCMC algorithms in
your favourite programming language is a very unattractive alternative
if you're a busy subject matter investigatorrrie
Barrie Stokes
Discipline of Clinical Pharmacology
School of Medicine and Public Health
Faculty of Health
University of Newcastle
NSW Australia
ph (+61) 02 4921 1832
fax (+61) 02 4960 2088
Clinical Pharmacology
Room 5.43 Level 5
NewMed II Building,
Newcastle Mater Misericordiae Hospital
Waratah NSW 2298
Australia
7 I don't know your example, but I have run multivariate categorical data sets with 66 variables on samples of 1100.
Also was analyzing simualted MV binary data with 5,000 obs.
No idea where this crit. came from.
Lindon J Eaves/FS/VCU <[log in to unmask]>
L.
8 I have no reference but have run 4000 cases in a regression and 1500
cases in a 3-level hierarchical model with no problems.
"Lucke, Joseph F" <[log in to unmask]>
9 Rudy
I have no references for you - but I regularly run models where the number
of observations exceed 1000 (sometimes 10 times this amount). I am
presuming by space-time units you are talking about: 1xspace-time unit = 1
observed datum which has 2 dimensions.
WinBUGS does, apparently, have an upper limit on RAM useage - which we get
around by thinning and refreshing the chains when required. I'm not sure if
the open BUGS version has this limitation - but this could be an alternative
for your NIH grant.
Sounds like the reviewer is being a little harsh.
Steve
--
Professor Stephen Duffull
Chair of Clinical Pharmacy
School of Pharmacy
University of Otago
PO Box 913 Dunedin
New Zealand
E: [log in to unmask]
P: +64 3 479 5044
F: +64 3 479 7034
Design software: www.winpopt.com
10 Hi,
It sounds like the reviewer is saying two things. 1) Fit the Bayesian
spatio-temporal model using Winbugs and 2) Winbugs might give you
problems with large samples. These seem conflicting, but that's nothing
new for NIH reviewers. I have never had a problem with 2, though
convergence is sometimes very slow with many random effects. I have
never seen anything published on this issue.
My response would be to "agree" with the reviewer and fit the
spatio-temporal model in WinBUGS. Just mention in the limitations
section of the proposal that some models are slow to converge with large
N.
Good luck,
Garnett
Garnett P. McMillan, PhD
Biostatistician / Research Scientist
Behavioral Health Research Center of the Southwest
612 Encino Pl NE
Albuquerque, New Mexico 87102
(505) 244-3099
(503) 320-2029
www.bhrcs.org
11 Rudy,
thanks for this interesting post. I am working with a hierarchical model with 3000 (spatial) x 83 (temporal) space time units. While the model runs slowly on a desktop (20 seconds / iteration) it has given me no indication of erroneous results. I did present a previous, smaller iteration of the model (3000 spatial x 21 temporal) at the Society for Conservation Biology meetings in July, but the abstract does not mention winbugs (see below for citation). However, I haven't (yet) published anything, so I'm not sure this communication will be of use to you. If anyone replies in private with a published example, could you forward the citation to me too (or send the reply back to the list)?
It is possible, but laborious, to run similar models in R. But you have to code your own gibbs and/or metropolis hastings samplers, set up the data structure, etc. (all the things winbugs does for you). Depending on your R knowledge, it can take a long time to get it running. There are a few packages that may be helpful (see UMacs). I have been considering moving to R for my project, but mostly because of speed.
Wilson, A., A. Latimer and J. Silander (2007). The Fire-Weather Relationship in the South African Fynbos: Implications under Climate Change (#5608). Society for Conservation Biology 2007 Meeting, Port Elizabeth, South Africa, Society for Conservation Biology.
Abstract available at: http://www.nmmu.ac.za/scb/posterabs.pdf
Adam
Adam Wilson
http://hydrodictyon.eeb.uconn.edu/people/wilson/
Department of Ecology and Evolutionary Biology
Bitiotemporal models similar to what you are describing, with between ~900 to ~2000 spatial units at 15 time points. With up to 30,000 space-time units, things get intractable, although presumably with enough computer memory and enough time, the models could be fit in WinBUGS. I haven't seen any citations about this in the literature.
For what you're doing, with the ~1500 space-time units, I can't see that there should be any problems in WinBUGS. I think the reviewer was overstating things by referring to this as a "limitation" of WinBUGS. It may be "freeware" (so is R for that matter), but it *is* state-of-the-art (short of writing your own MCMC sampler). I would just cite your own experience running models with over 1000 space-time-units in your response to the reviewers.
Good luck.
-jarvis chen
Jarvis T. Chen, ScD
Research Scientist
Harvard School of Public Health
Department of Society, Human Development, and Health
401 Park Drive, Rm 403N
Boston, MA 02215
Tel. 617-384-8707
Fax. 617-384-8713
email:[log in to unmask]
13 Hi Rudy,
This is an interesting issue and we'd be grateful if you'd send a digest
of responses that you receive to the list.
Many thanks
Hayley Jones
List manager
14 I think the most effective way to avoid such unwarranted criticism is,
if possible, to include in Section C (Preliminary Studies) a comparable
analysis you have successfully completed. Sorry I don't have a reference
of the kind you are looking for.
Purushottam W. (Prakash) Laud, PhD
Professor and
Director of Graduate Studies
Division of Biostatistics
Medical College of Wisconsin
Tel. 414-456-8781
Fax. 414-456-6513
________________________________________
15 Hi Rudy,
I feel your pain. I think some of the problem stems from a cryptic
comment in the first version of the GeoBUGS manual that warned about the
huge dimensions of spatial prediction matrices.
I, too, model some very big datasets with hierarchical approaches. I
have been getting funded from NIH under an R03. I have applied
Diggle's geostatistical approach predicting ten's of thousands of
locations. The matrix dimensions are huge but things seem to work out.
Using autoregressive correlations (e.g. Besag) even bigger data sets are
being modeled (in my hands). I could suggest a couple of publications
that used Diggle's methods and large prediction matrices. I'm not sure
how much support that will be.
I would be interested in seeing any technical support you get from the
group.
Jamie
J.A. Thompson, DVM, DVSc, DACT, DACVPM (Epidemiology)
Professor of Toxicology and Clinical Science
College of Veterinary Medicine, Texas A&M University
Aniruddha "Rudy" Banerjee, B. Arch, MSU&RP, PhD.
Associate Research Scientist
Prevention Research Center
1995 University Ave, Ste 450
Berkeley, CA 94704
510.883.5740
510.644.0594 (Fax)
[log in to unmask]
web site: http://www.prev.org/aboutprc_staffandfellows_banerjee.html
Aniruddha "Rudy" Banerjee, B. Arch, MSU&RP, PhD.
Associate Research Scientist
Prevention Research Center
1995 University Ave, Ste 450
Berkeley, CA 94704
510.883.5740
510.644.0594 (Fax)
[log in to unmask]
web site: http://www.prev.org/aboutprc_staffandfellows_banerjee.html
-------------------------------------------------------------------
This list is for discussion of modelling issues and the BUGS software.
For help with crashes and error messages, first mail [log in to unmask]
To mail the BUGS list, mail to [log in to unmask]
Before mailing, please check the archive at www.jiscmail.ac.uk/lists/bugs.html
Please do not mail attachments to the list.
To leave the BUGS list, send LEAVE BUGS to [log in to unmask]
If this fails, mail [log in to unmask], NOT the whole list
|
