memory usage in RStan?

[Bob Carpenter]

What happens at the end of an RStan run to cause a spike in memory?
Or is this just R allocating space? I was running some experiments
and tracking the memory just through the system tool and it was under
200MB until it ended and then it shot up to 850MB. Is this all just
in creating the final object?

For what it's worth, R2jags took 1.7GB for the same model and same number
of iterations (with all other save parameters the same).

Oddly, R2jags, when given the same starting point, looked like it
converged (R-hats all less than 1.05, effective sample sizes all
> 100), but the answers weren't anywhere near Stan's and had overall
lower log likelihoods. Stan's answers matched what Sophia got in
Stata, but JAGS's didn't. Note that this was with R2jags' estimate of
R-hat and n_eff, not Stan's. I still haven't figured out how to
transfer output from R2jags into RStan. JAGS was about three times
as fast per iteration, though :-)

- Bob

[Ben Goodrich]

http://cran.r-project.org/doc/manuals/R-exts.html#Profiling-R-code-for-memory-use

[Andrew Gelman]

Interesting example. Worth saving for our Stan paper perhaps.
A

> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Jiqiang Guo]

On Thu, Aug 29, 2013 at 4:18 PM, Bob Carpenter <ca...@alias-i.com> wrote:

What happens at the end of an RStan run to cause a spike in memory?
Or is this just R allocating space?  I was running some experiments
and tracking the memory just through the system tool and it was under
200MB until it ended and then it shot up to 850MB.  Is this all just
in creating the final object?

I don't think the memory usage would have a spike when combining multiple

chains into a stanfit object if the samples are just used by reference in R.

What I thought is the memory for a chain is allocated before sampling, 

so the total memory usage should increase as chains 

are sampled sequentially.  I run an example and use Instruments in Mac

to monitor the memory usage over time.  I do not see the increasing of memory 

usage as more chains are sampled and I do not see a spike at the end always.

Attached is one one instance of monitoring the memory.  I am assuming that

"%System Load" are reflecting the real memory usage as others do not change much. 

The sampling happened after 00:20, before which the compiling indeeds uses a lot of memory

for the eight schools example. 

--

Jiqiang 

 

[Bob Carpenter]

Thanks for looking into this. I didn't get the attachment.

I have no idea what R and Mac are up to. I'm trying to
view memory through the Mac Activity Monitor. After I
start up the R GUI, but before I run anything, the memory
bounces around between 25MB and 81MB.

Then after calling stan() (which loads the library), memory
goes up to

100MB

and the C++ compiler uses just under 1GB in
a separate process. Then after iterations start going, memory
goes to

160MB

The model has roughly 3K parameters and I'm
running 4 chains of 1000 iterations each, so that's a total of
about 12M values, or about 100MB if you store them as double
objects. So the memory hit here is consistent with R using
60MB and Stan using 100MB --- that's storing them efficiently!

But now, the second chain has started, and memory usage goes up to

200MB

So it looks like the memory may be allocated before a chain runs
or after one finishes? When the third chain starts, that goes up to

215MB

Then when the fourth chain starts, memory usage goes up to:

230MB

And then when the fourth chain finishes, memory usage by the R process
as reported by the Activity Monitor goes up to:

450MB

And that doesn't go back down sitting around for 10 minutes.
Then, running extract() and assigning it takes memory usage up to:

1GB

- Bob

[Ben Goodrich]

On Friday, August 30, 2013 1:13:47 PM UTC-4, Bob Carpenter wrote:

And then when the fourth chain finishes, memory usage by the R process
as reported by the Activity Monitor goes up to:

   450MB

And that doesn't go back down sitting around for 10 minutes.

You should try a manual garbage collection at that point

gc(TRUE)

If that causes the memory to go down, then it is basically okay. If R needs to garbage collect in order to do some subsequent calculation, then it probably will.

Ben

[Bob Carpenter]

That took the total memory usage of R down to 627MB.

But then reading these things through system utilities isn't
ideal. R reports this:

> gc(TRUE)
Garbage collection 153 = 62+11+80 (level 2) ...
27.1 Mbytes of cons cells used (16%)
306.9 Mbytes of vectors used (49%)
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 506386 27.1 3170326 169.4 4953636 264.6
Vcells 40225372 306.9 82274108 627.8 76757609 585.7


(That's after the extract().) If I set the extracted
value to 0 and then gc again, I get back to 595MB in
the Activity Monitor and R reports this:

> gc(TRUE)
Garbage collection 155 = 62+11+82 (level 2) ...
27.1 Mbytes of cons cells used (25%)
272.0 Mbytes of vectors used (43%)
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 506345 27.1 2029008 108.4 4953636 264.6
Vcells 35643354 272.0 82274108 627.8 76757609 585.7

Basically the same numbers. So either extract() isn't using
any extra memory (very cool) or R's doing something unexpected
for GC (not out of the realm of possibility).

- Bob

[Ben Goodrich]

On Friday, August 30, 2013 1:25:59 PM UTC-4, Bob Carpenter wrote:

Basically the same numbers.  So either extract() isn't using
any extra memory (very cool) or R's doing something unexpected
for GC (not out of the realm of possibility).

I think a pure extract wouldn't really use a noticeable amount of RAM since the "new" array would just be a pointer to the "old" array that is already allocated by the stanfit object. But if you (for some weird reason) changed any of the cells in the "new" array, it would probably copy the whole array.

Ben

[Jiqiang Guo]

The screenshot is attached again.  Your observation of memory 

increasing with more chains get sampled is consistent to what I 

thought it is supposed to be.  All memory for a chain is allocated 

before sampling.  (But I did not see it in the attached figure.)

I do not know if extract() would increase the memory usage.  But there 

are some manipulations such as discarding warmup samples and combining 

multiple chains in some cases.  So I would think it is ok if it increases 

memory usage. 

--

Jiqiang 

- Bob

--
You received this message because you are subscribed to the Google Groups "stan development mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+unsubscribe@googlegroups.com.