Memory Usage Spike

[Paddy O'Toole]

Hi,

Firstly, thanks for the amazing software, which is heavily used by the HIV Inference Group at Imperial College London.

We have developed a Bayesian spatio-temporal model for circumcision in TMB. An example of it's use can be found here. I won't go into too much detail on this model here, as I feel this repo provides a reasonable reproducible example.

One issue that we have found is that during the "optimising tape ..." stage of running our optimiser, we experience a substantial memory usage spike (< 500GB for large models) (again, this is better elaborated on in our example repo). This spike appears to be associated with the TMB code from this line down, which involves calculations using several high-dimensional sparse matrices representing our random effects. The (temporarily) large memory usage of this model means it is very demanding on our
high-performance cluster.

We’re trying to understand:
- Why does the memory spike like this?
- Can we predict the size of the memory spike from size in input data, number of parameters etc.
- Can we save out the optimised tape and then use this for multiple model fits? This could allow us to do the costly optimisation on a beefy node but then run the rest of the simulation on smaller nodes.

Any help would be greatly appreciated.

Many Thanks,
Paddy O'Toole,
Research Assistant,
Imperial College London

[Kasper Kristensen]

You may be able to get rid of the memory spike simply by switching AD library - see ?TMB::compile.

I suggest you try:

- Add framework='TMBad' to compilation.

- Remove flags '-O0 -g' (slows down the model)

Then recompile the package.

[Paddy O'Toole]

This appears to have fixed the issue, thanks so much! Is there anywhere I might have come across this myself in the documentation? I didn't realise that using the TMBad framework performs sparse hessian compression.

[Paddy O'Toole]

Apart from ??TMB::compile, which doesn't specify why you might use CPPad over TMBad, or vice versa.

[Kasper Kristensen]

Yes, TMBad framework can perform sparse hessian compression. However, this *not* needed/recommended except for very special model structures (has to be enabled using 'TMB::config()'). In fact the sparse hessian compression might make things worse!

Memory reductions and speedups can be expected by simply switching the framework - nothing else.

You are right that the documentation doesn't yet contain the appropriate information.