estimating memory requirements

[Jack Tanner]

I'm about to run a model on a multi-core Linux box with 16G RAM. I'd
like to run multiple chains in parallel, as in Reference Manual sec 4.2.
But I want to avoid running out of RAM / swapping to disk.

Is it reasonable to estimate the memory requirements for one chain by
running a chain for just some small number of iterations? In other
words, does a Stan model's memory use grow over time, or does it
allocate all the memory it'll need by the end of so-many early
iterations? And if the latter is true, how many iterations do I need for
this estimate?

[Marcus Brubaker]

Your plan for estimating memory usage should work just fine for Stan.  The amount of memory used shouldn't grow over iterations, although it's a good idea to let it run for a few iterations before checking memory usage to ensure that the full NUTS tree depth has been hit.  That said, for most complex models the amount of memory used by the sampler is dwarfed by the amount used by the autodiff so even one iteration should get you pretty close.

Cheers,

Marcus

On Wed, Jun 5, 2013 at 10:37 AM, Jack Tanner <ih...@hotmail.com> wrote:

I'm about to run a model on a multi-core Linux box with 16G RAM. I'd like to run multiple chains in parallel, as in Reference Manual sec 4.2. But I want to avoid running out of RAM / swapping to disk.

Is it reasonable to estimate the memory requirements for one chain by running a chain for just some small number of iterations? In other words, does a Stan model's memory use grow over time, or does it allocate all the memory it'll need by the end of so-many early iterations? And if the latter is true, how many iterations do I need for this estimate?

--
You received this message because you are subscribed to the Google Groups "stan users mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Ben Goodrich]

Also look into the sample_file argument to stan() if you are calling from R, which will write draws to a file rather than holding everything in RAM. Just be sure to index the filename by chain_id to avoid having multiple processes write to the same file!

Ben