memory usage in R

[Bob Carpenter]

[moved to stan-dev]

On 11/12/12 4:44 PM, Andrew Gelman wrote:
> We really shouldn't be running out of memory in R for models that can be fit in Jags!

We gathered that from the original poster!

Do you have anything more constructive about HOW to
conserve memory in R?

One issue where we could lose a memory battle with JAGS is
with iterative matrix operations. Those consume a lot of
memory for us during auto-dif but wouldn't in JAGS. This
is dynamic/peak memory used to generate samples on top of
whatever's being used to store the data.

One possibility is to write all the data to file, then
read it back in.

- Bob

[Andrew Gelman]

I don't have any brilliant ideas, but, sure, writing to file is a possibility. Another possibility is thinning. I recall that in Bugs/Jags, I was running into memory problems when I had models with thousands of latent variables. I don't know what is causing Stan to crash R here.
A

> --
>

[Jiqiang Guo]

[to stan-dev]

1. reading csv files of samples in RStan

Currently, no functionality in RStan to do that.  I guess I should work on it so maybe in the release after 1.0.4, we can have it. 

2. keeping `samples so far'

I do not know now.  I have never encounter an error in R of `out of memory`.  I do not know how R or OS deals with this case.  And in the case if it is R calling a C++ function that is running out of memory, I am not sure we can do something in R then.  In short, it is difficult.  Unless this happens a lot, I would not put the investigation on top priority. 

--

Jiqiang 

---------- Forwarded message ----------
From: Bob Carpenter <ca...@alias-i.com>
Date: Mon, Nov 12, 2012 at 5:20 PM
Subject: Re: [stan-users] Re: defining a covariance matrix from a correlation matrix
To: stan-...@googlegroups.com

On 11/12/12 5:14 PM, Jiqiang Guo wrote:

The gradient is not stored in the fitted object.

The only issue with the gradient is the space needed
for the expression tree gets stored in memory in order
to compute gradients.  It's transient memory that's
allocated and used per sample and then freed.

We do store some other information such as the tree_depth, stepsize,
lp__.  I do not think it will be an issue in general.  My guess about that there is no sample in the fitted object for
your case if because of the error; in the whole sampling process, whenever error is encountered, an empty object is
returned instead.

I think this makes sense to keep consistency.

If there's a way to keep the "samples so far" in
a consistent state that might help, but I don't think it
should be a high priority.

Another option is that set pars to be as smallest as possible (currently, it does not allow to be empty) and specify
sample_file argument for stan(), so that the samples are written to text files as using Stan from the command line.

Is there an easy way to read the results back into
a stanfit object?  I looked at the rstan doc index
and didn't see anything.  If we could do that, then we
could use sflist2stanfit to put multiple samples together.

Does anyone recall how JAGS reads data?  I could always
download r2jags and look at the source.

- Bob

--

[Bob Carpenter]

On 11/12/12 5:37 PM, Jiqiang Guo wrote:
> [to stan-dev]
>
> 1. reading csv files of samples in RStan
>
> Currently, no functionality in RStan to do that. I guess I should work on it so maybe in the release after 1.0.4, we
> can have it.

This would be really useful for scalability in R or even
just for running from Stan command line and then
reading into R for later analysis.

> 2. keeping `samples so far'
>
> I do not know now. I have never encounter an error in R of `out of memory`.

Lucky you.

If you use 32-bit R and try to run JAGS models,
you'll run into it fast enough. I rewrote some models in
Stan that took up way less memory than their JAGS counterparts
in R.

Any model that calls iterative solvers on matrices may
take up a lot of memory during sampling the way things are
implemented now. The issue is that we build up the entire
expression tree over the iterations before propagating the
chain rule backwards for calculating gradients. We plan to
make this more efficient by extracting the double values,
calling the iterative algorithsm on those, then partially
evaluating the gradients using all the rules in The Matrix
Cookbook and Magnus and Neudecker's book.

> I do not know how R or OS deals with this
> case. And in the case if it is R calling a C++ function that is running out of memory, I am not sure we can do
> something in R then.

It's going to be handled by process. When R calls the
C function, it's all in the same process, and so sharing
all the memory from the OSes point of view. Thus if you run
R in 32-bit mode, the OS is going to put a low ceiling on
how much memory can be used. A C function called from R
might not be well behaved with respect to R -- it can call its
own malloc() and play havoc with R's memory model.

I don't know how well Rcpp deals with these issues and how
much is up to the user.

You can try something like object.size() in R. memory.profile()
also looks useful.

There may also be extra objects hanging around in the R environment
taking up space that have nothing to do with Stan. R uses
garbage collection, but doesn't appear to do it very well. So
one probably shouldn't use long-running R sessions even if you get
rid of local variables and call gc() to try to clean things up.

> In short, it is difficult. Unless this happens a lot, I would not put the investigation on top
> priority.

We should probably test the scalability to make sure we're
not doing something sub-optimally. If it's a deep issue with
R, I agree that it's not a priority for us to try to fix it!

- Bob

[Ben Goodrich]

On Monday, November 12, 2012 4:51:07 PM UTC-5, Bob Carpenter wrote:

[moved to stan-dev]

On 11/12/12 4:44 PM, Andrew Gelman wrote:
> We really shouldn't be running out of memory in R for models that can be fit in Jags!

We gathered that from the original poster!

Do you have anything more constructive about HOW to
conserve memory in R?

There is the issue of us storing both triangles of a covariance / correlation matrix.

Ben

[Bob Carpenter]

We could change that. The three places it touches
are (1) data, (2) inits, and (3) output. Right now,
(1) and (2) use the Stan dump format, which is a full
matrix. For (3), there's less dependency and less we'd
break by only outputting the lower triangular part.

We'd then have to be clever in extract() to turn it
back into a matrix for analysis in R.

- Bob