stuck dll

[Andras Farkas]

Hello nimble-users,

would like to ask for input on the following:

I have a model that is inside a function that run in a "loop-ed" fashion where all files created by nimble goes into a dirName randomly created prior to setting up model. At the end of this function my preferred plan is to delete the folder and all related files then start the whole process again in the next loop. As a side effect parts of the files/objects created appear to "stuck" and prevents the model compilation in the subsequent loop. Have tried clearCompiled() but no success. here is the process as inside the function:

looping_function <- function(...) {

  #some calculations here....
 
  model_code <- nimbleCode({})
 
  constants <- list(#constants here")
  obs_data <- list(#constants here")
  inits <- list(#constants here")

  model_nimb <- nimbleModel(model_code,data = obs_data,inits = inits,constants = constants,buildDerivs = TRUE)
  set.seed(runif(1,10,10000))
  nimblefolder<-paste0("NM","1",password(n = 5, numbers = TRUE, case = TRUE,special = F))
  dir.create(nimblefolder)

  Cmodel <- compileNimble(model_nimb,dirName=file.path(getwd(), nimblefolder))
  conf <- configureMCMC(Cmodel)
  mcmc <- buildMCMC(conf)
  Cmcmc <- compileNimble(mcmc, project = Cmodel,dirName=file.path(getwd(), nimblefolder))
  final <- runMCMC(Cmcmc, niter = 5000, nburnin =500,  samplesAsCodaMCMC = TRUE)
  #some calculations here....
 
  clearCompiled(Cmcmc)
  clearCompiled(Cmodel)
  sapply(list.files(file.path(getwd(), nimblefolder), "\\.dll$", full.names = TRUE), function(dll) {
  try(dyn.unload(dll), silent = TRUE)})
  unlink(file.path(getwd(), nimblefolder), recursive = T)
  gc()
  invisible(return(NULL))
}

and the loop over the data frame intervals rows:

for (ii in 1:nrow(intervals)) {
  x <- intervals[ii, ]
  looping_function  (...)
}

the 1st loop runs fine, but then getting this error message as the second loop around proceeds, the break happens at (as far as I can tell):

Cmodel <- compileNimble(model_nimb,dirName=file.path(getwd(), nimblefolder))

error message:

Error in FUN(X[[i]], ...) :
  no such symbol RNimble_Ptr_ManualFinalizer in package C:/.../app/shiny/NM2FYWXh/dynamicRegistrations_09_05_05_32_27.dll
Calls: <Anonymous> ... $ -> $.DLLInfo -> getNativeSymbolInfo -> lapply -> FUN

NM2FYWXh in this case is the name of the folder that was created in the 1st loop that gets deleted appropriately at the end of loop yet the system "holds on" to dynamicRegistrations_09_05_05_32_27.dll from that 1st loop.

my question is that are there some best practices to do the clean up so all parts and pieces of the model gets removed to prevent the break? Interrupting R session and start a new one is not an option for me. 

All the following nimble "components" are subject to change from loop to loop: model_code, constant , obs_data , inits, conf and of course nimblefolder.

appreciate all your thoughts,

thanks,

Andras

[Andras Farkas]

minor correction to code above:

  constants <- list(#constants here")

  obs_data <- list(# obs_data   here")
  inits <- list(# inits here")

[Perry de Valpine]

Hi Andras,

Please try the following two changes:

Only use one of the clearCompiled calls. Each one should attempt to clear everything in the project, so using both should be unnecessary. 

Try adding this before clearCompiled:

Cmodel <- model_nmb <- conf <- mcmc <- Cmcmc <- NULL

gc()

This attempts to force garbage collection of everything involved in the compilation prior to clearing the DLL. I suspect that the error message is arising from R's garbage collector attempting to call a finalizer (for destroying the C++ side of objects) AFTER clearCompiled, so that the finalizer no longer exists. When you reassign to the same names, it may trigger R's garbage collector (which normally operates without user management needed), and then hits the problem that the needed DLL has already been unloaded. If you can successfully prod R to garbage collect everything before removing the DLL, that might solve the problem.

This is among the kinds of reasons that clearCompiled() is not highly stable or safe.

It's only a guess at what is going on. Of course, if the model structure does not change (and only the data change), you could reuse the compiled objects instead of rebuilding and recompiling each time, so I'm assuming that is not the case.

HTH

Perry

--
You received this message because you are subscribed to the Google Groups "nimble-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/nimble-users/73d602d3-a491-4a30-84ab-288f39ccc826n%40googlegroups.com.

[moty...@gmail.com]

Hi Perry

Thank you for the response... unfortunately looks like system still holds on to the dlls even with your proposal... I solved my issue with the system2() workaround... working on windows am sure you already suspected... great package(s) though, love the flexibility and overall speed ...

Thanks

Andras

Yahoo Mail: Search, Organize, Conquer