monitor memory requirements?

[Ben Bolker]

Hi, folks,

I have a memory-intensive TMB-based calculation. I want to run
experiments scaling the size of the problem to estimate how much memory
I will need in order to do my actual problem of interest.

I thought that the peakRAM package (which runs R's gc() and records
the results to get an estimate of total and peak RAM used during the
execution of a specified expression) would be useful, but as far as I
can tell it doesn't capture memory used by TMB.

In particular, I am running a series of experiments where peakRAM
reports at most a few 100 Mb used, but my R session is crashing when my
machine (with 60G of RAM) runs out of memory.

So I'm looking for a different way to do this, ideally within R, but I
guess I could also try to run an external script to watch the memory
used and match it up with recorded timestamps from my R session ...

Does anyone have suggestions?

Ben Bolker

[Rob]

Hi Ben,

This is something we've also run into and I've tried a couple of approaches. For an external script I've had success using something like [memusg script](https://gist.github.com/netj/526585) I've also ran several profiles with valgrind massif and got very similar results to the memusg script linked. That is the most reliable way we have found.

I did also write a small package for doing profiling from R which works for TMB https://github.com/mrc-ide/memprof

If your R session is crashing you can write out the profile to a file and read it in a new session.

From first session

memprof::with_monitor(my_func(), monitor_file = "myprofile")

In a new session

profile <- memprof::monitor_read("myprofile")
plot(profile)

This will monitor the current R process and any spawned processes meaning the TMB fit should be captured too. `profile` is a data frame of memory usage over time, you can read the max `utils::format.object_size(max(profile$rss), "auto")`

Note that I don't think the profiles from `memprof` are as reliable as memusg/massif in all cases, I've seen weird results when trying to profile anything which knits an R Markdown document but with TMB it has had comparable results.

[jointprof](https://github.com/r-prof/jointprof) also looks really promising for some more detailed output but I haven't tried this in anger. I'd be interested if you have any success with this.

Rob

[Kelli Johnson - NOAA Federal]

I asked around and Matthew Supernaw suggested the following:

use Valgrind and lldb or gdb.
valgrind will detect any memory leaks and lldb will catch the source of the crash.

Valgrind:
valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes --verbose  --log-file=valgrind-out.txt R -f myscript.R

LLDB:
R -d lldb
run -f myscript.R

I note that Matthew works on a linux machine, though a Windows equivalent of Valgrind is available called Deleaker (see this blog post https://www.deleaker.com/blog/2020/01/04/valgrind-for-windows/).

Best,

Kelli

[Ben Bolker]

Thanks.

As far as I know, this is *not* a memory leak problem -- just a
problem that takes more memory than I have on my machine.

cheers
Ben

On 3/6/25 11:33, 'Kelli Johnson - NOAA Federal' via TMB Users wrote:
> I asked around and Matthew Supernaw suggested the following:
> use Valgrind and lldb or gdb.
> valgrind will detect any memory leaks and lldb will catch the source of
> the crash.
>
> Valgrind:
> valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes --
> verbose  --log-file=valgrind-out.txt R -f myscript.R
>
> LLDB:
> R -d lldb
> run -f myscript.R
>
> I note that Matthew works on a linux machine, though a Windows
> equivalent of Valgrind is available called Deleaker (see this blog post
> https://www.deleaker.com/blog/2020/01/04/valgrind-for-windows/).
>
> Best,
> Kelli
> On Thursday, March 6, 2025 at 1:19:59 AM UTC-8 roberta...@gmail.com wrote:
>
> Hi Ben,
>
> This is something we've also run into and I've tried a couple of
> approaches. For an external script I've had success using something

> like [memusg script](https://gist.github.com/netj/526585 <https://
> gist.github.com/netj/526585>) I've also ran several profiles with

> valgrind massif and got very similar results to the memusg script
> linked. That is the most reliable way we have found.
>
> I did also write a small package for doing profiling from R which

> works for TMB https://github.com/mrc-ide/memprof <https://
> github.com/mrc-ide/memprof>

>
> If your R session is crashing you can write out the profile to a
> file and read it in a new session.
>
> From first session
> memprof::with_monitor(my_func(), monitor_file = "myprofile")
>
> In a new session
> profile <- memprof::monitor_read("myprofile")
> plot(profile)
>
> This will monitor the current R process and any spawned processes
> meaning the TMB fit should be captured too. `profile` is a data
> frame of memory usage over time, you can read the max
> `utils::format.object_size(max(profile$rss), "auto")`
>
> Note that I don't think the profiles from `memprof` are as reliable
> as memusg/massif in all cases, I've seen weird results when trying
> to profile anything which knits an R Markdown document but with TMB
> it has had comparable results.
>

> [jointprof](https://github.com/r-prof/jointprof <https://github.com/
> r-prof/jointprof>) also looks really promising for some more

> --
> To post to this group, send email to us...@tmb-project.org. Before
> posting, please check the wiki and issuetracker at https://github.com/
> kaskr/adcomp/ <https://github.com/kaskr/adcomp/>. Please try to create a
> simple repeatable example to go with your question (e.g issues 154, 134,
> 51). Use the issuetracker to report bugs.
> ---
> You received this message because you are subscribed to the Google
> Groups "TMB Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to tmb-users+...@googlegroups.com <mailto:tmb-
> users+un...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/tmb-
> users/5c9dc82f-1b10-4627-8d88-8b99fa363916n%40googlegroups.com <https://
> groups.google.com/d/msgid/tmb-
> users/5c9dc82f-1b10-4627-8d88-8b99fa363916n%40googlegroups.com?
> utm_medium=email&utm_source=footer>.

--
Dr. Benjamin Bolker
Professor, Mathematics & Statistics and Biology, McMaster University
Director, School of Computational Science and Engineering
* E-mail is sent at my convenience; I don't expect replies outside of
working hours.

[Ben Bolker]

  For the record, I couldn't get memusg to work properly for my application - I was trying to loop over multiple processes and spawn batch jobs (by calling "./memusg Rscript MY_SCRIPT PARAMS" either via system() in R, or from a loop in a bash shell), and things were wonky [presumably because of something about how subprocesses are counted that I don't understand]. However, the memprof package worked perfectly; I looped over problem sizes, saving the cumulative output to a checkpoint file as I went along, so that by the time the script stopped because I maxed out memory at 60G and the R session crashed, I had a nice table of the memory requirements for a range of smaller problems. I ran a log-log regression to estimate the scaling, and think that I have the answer to my question (I'll need about 600G and 3 hours to run my actual problem -- the local HPC cluster has a few nodes this big, hopefully I won't have to wait too long to get access to them).

  cheers

   Ben Bolker