oneStepPredict parallel=TRUE

[Mollie Brooks]

Hi,

I’m trying to get OSA residuals from a large glmmTMB model and it’s taking a long time. I checked that the model has converged. I tried to speed it up using the parallel argument, but the documentation that I found is limited and I think I may be using it incorrectly. I requested 20 nodes from DTU’s cluster, but it only used 2380 MB maximum out of 122880.00 MB that I requested.

My script is basically just this
library(TMB)
library(glmmTMB)
library(parallel)

load("cormorants_topmodel.Rdata")

res_m0dd= TMB::oneStepPredict(m0dd$obj,observation.name="yobs",data.term.indicator="keep",discrete=TRUE,range=c(0,100),method="oneStepGeneric",trace=FALSE,parallel=TRUE)

It runs for the whole allocated 24 hours and doesn’t give any errors. Do I need to do anything more to get the parallelization to work?

Thanks in advance.

Cheers,
Mollie

[Kasper Kristensen]

Before submitting a giant job, you should probably do some interactive tests to get an idea of the total runtime. Change your current oneStepPredict call to only calculate the, say, 10 first residuals:

subset=1:10

trace=TRUE

parallel=FALSE

Knowing the total number of observations you can then guess approximately the time of the full run.

You should also double check the 'range' argument based on the range of your observations. The 'range' should cover the majority of the probability mass.

Finally, I would try the parallel run on a small subset of observations, before submitting the full job.

[Mollie Brooks]

Just to follow up on this. I confirmed that the code wasn’t using all 20 cores that I requested from the cluster until I added this to my R code:

options(mc.cores=20)

cheers,

Mollie

--
To post to this group, send email to us...@tmb-project.org. Before posting, please check the wiki and issuetracker at https://github.com/kaskr/adcomp/. Please try to create a simple repeatable example to go with your question (e.g issues 154, 134, 51). Use the issuetracker to report bugs.
---
You received this message because you are subscribed to the Google Groups "TMB Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tmb-users+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/tmb-users/57942b50-a0f1-4137-aa8a-673441b34240n%40googlegroups.com.