Memory allocation work arounds/tips

[ada...@uw.edu]

Hi Everyone,

 Does anyone have any guidance/tips for working around memory allocation failures during optimization:

opt <- nlminb(obj$par, obj$fn, obj$gr)

Error in sparseHessianFun(env, skipFixedEffects = skipFixedEffects) : 

  Memory allocation fail in function 'MakeADHessObject2'

 I'm developing a multispecies age-structured model with normally distributed random effects (recruitment deviates) for fisheries applications and it works fine when "fixing" the variance of the random effects and running a penalized likelihood approach. However, when I try to estimate the variance and treat the random deviates as random effects I get memory allocation failures for a few of the 16 sensitivity runs I am doing (I clear the environment and memory using "gc" and "FreeADFun between runs). The odd part to me is that one sensitivity run will work while another sensitivity run with the same parameters (fixed and random), but parameterized slightly different will not work. 

I've tried a few things that don't quite get me where I want to be for all the sensitivity runs: 1) sensitivity runs with shorter time series work (e.g. reduced number of estimable parameters), 2) reducing the size of the model helps (the multispecies formulation is solved via iteration using a for loop around the dynamics and reducing the number of iterations helps, but the multispecies model is not technically converged), 3) moving from a 32 to 64 GB memory computer did not help, 4) changing "tape.parallel" didn't help. I have a few 7 dim arrays of derived quantities and perhaps reducing those may help or reducing some 4D fixed effects parameter arrays? Perhaps using a different inner optimizer?

If anyone has come across this issue and has a few pointers or tips that would be much appreciated!

Cheers,

Grant

[Grant Adams]

Following up on this, I found a bit of success for a specific application of the model if it helps anyone. 1. I removed any parameters/code that was not used by the specific application of the model I am currently working on, 2. I removed ADREPORT for everything and REPORT for everything except what was absolutely necessary, and 3. switched 7D arrays to 5D arrays with updated indexing. I'm not sure what exactly did the trick. After doing most of 1, and all of 2 and 3 it still wasnt working, then removing 1 last parameter_vector that was entirely mapped out did the trick. I ideally would not like to gut the model as I did, but it is working so I dont need to throw my computer out the window.

--
To post to this group, send email to us...@tmb-project.org. Before posting, please check the wiki and issuetracker at https://github.com/kaskr/adcomp/. Please try to create a simple repeatable example to go with your question (e.g issues 154, 134, 51). Use the issuetracker to report bugs.
---
You received this message because you are subscribed to a topic in the Google Groups "TMB Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tmb-users/7xRQWBPeywU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tmb-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tmb-users/db0e5b47-48f5-42a6-997e-e09cdc949850n%40googlegroups.com.

--

Grant Adams

School of Aquatic Fishery Sciences

University of Washington

https://grantdadams.wordpress.com/

[Cole Monnahan - NOAA Federal]

Hi Grant,

It looks like this is failing during optimization, not during the Hessian/SE calcs? You could try profiling the memory usage. I got it to work with VAST based on some guidance from Kasper I think. You could adapt this procedure for your model and run it in to profile memory. I don't know if this would help, and didn't really understand the output too well myself. But is something to try. 

https://github.com/James-Thorson-NOAA/VAST/wiki/Speed-test#profiling

-Cole

[Cole Monnahan - NOAA Federal]

I had similar problem and Kasper suggested the following as a way to help diagnose the issue, copied from an email:

It's often useful to look at which operators are on the AD tapes, and their frequencies (requires 'TMBAD_FRAMEWORK').

> TMB:::op_table(fit1$env$ADFun)

> TMB:::op_table(fit1$env$ADGrad)

> TMB:::op_table(environment(fit1$env$spHess)$ADHess)