[moved to stan-dev; removed cc list]
If I have N parameters and M posterior draws,
the amount of memory required to store the draws in
R should be roughly (N * M * 8) bytes + overhead.
Does anyone know what the overhead is in RStan as a
function of N and M? And if the total is much more
than (N * M * 8), which it seems to be if I understand
user reports correctly, where is the extra memory is
being used?
My understanding is that it's possible to generate the
draws, store to a CSV file, then read them back into R
using less memory than would be required to store them
in R in the first place. Is that right?
- Bob
> On Nov 27, 2015, at 12:55 PM, Jiqiang Guo <
guo...@gmail.com> wrote:
>
> If one really don't want to save the draws into R's memory, one can set argument pars=character(0), and specify argument sample_file to save draws into external files.
>
> Jiqiang
>
> On Thu, Nov 26, 2015 at 8:58 PM, Andrew Gelman <
gel...@stat.columbia.edu> wrote:
> cc-ing Ben, Jonah, and Jiqiang, since the issue is R’s memory hogging. Also cc-ing Hadley in case he has any thoughts on this>
> A
>
> > On Nov 26, 2015, at 11:13 AM, Bob Carpenter <
ca...@alias-i.com> wrote:
> >
> > These queries can go to our users list.
> >
> > Yes, CmdStan is very economical with memory compared
> > to other interfaces. It produces CSV file outputs.
> >
> > We really need to fix the memory hogging issues in
> > R, but I don't know the first thing about it.
> >
> > - Bob
> >
> >
> >> On Nov 26, 2015, at 4:15 AM, Grant, Robert L <
Robert...@sgul.kingston.ac.uk> wrote:
> >>
> >> I now have the scaling-up run on Stata 14.1 and rstan (in Rgui). JAGS is next but here's a question for you all. Stata has the same (or similar) memory issue as before, bombing out at i=20, p=5000. I think I'll scale down to p=100 to get at least three points to suggest a trend (currently aiming for 500, 1000, 5000, 10000). And rstan uses more than my puny PC will allow at that size too, when it tries the hrasch. Unfortunately that's the only computer that has both Stata and the freedom to compile for Stan. Such are the challenges enjoyed by isolated statisticians. So I was wondering about trying StataStan as it's just a CmdStan wrapper. I suspect CmdStan uses less memory by writing results out to text files as it goes, but would be interested to hear your thoughts. I can re-run that and JAGS straight away to get this paper going.
> >>
> >> And happy thanksgiving to y'all!
> >>
> >> Robert Grant
> >> Senior Lecturer in health & social care statistics
> >> Kingston University & St George's, University of London
> >>
www.robertgrantstats.co.uk
>
>
>