Seemingly strange RAM/configure time behavior

[baug...@vt.edu]

Hello Nimble Crew,

I'm reporting what seems like strange behavior to me. I wrote an MCMC sampler for an inhomogeneous density spatial capture-recapture model that uses the equivalent of a zeros trick to use a continuous instead of a discrete state space to avoid a categorical sampler.

It appeared to be working great--very fast. But then, I increased the resolution of the discrete habitat covariates and compilation time and RAM use blew up. These should increase, but my intuition tells me they increase more than they should. For example, there is no problem using 4096 cells. But moving to just 4489 causes roughly a 10x increase in compilation time and RAM use. Perhaps this is expected, but it seems excessive.

I isolated the problem to configuring D.beta0 and D.beta1. If I tell Nimble to not configure these and then add random walk samplers for them afterwards, I'm back to very fast compilation and low RAM use.

Do you all find this consistent with expected behavior? If interested, the code is below and a reproducible example is here (if you tell Nimble to configure all nodes):

I don't really need a solution since I found a decent one, but this seemed odd enough to report.

Thanks,

Ben

https://github.com/benaug/SCR-Dcov

NimModel <- nimbleCode({
  #--------------------------------------------------------------
  # priors
  #--------------------------------------------------------------
  #Dcov priors
  D.beta0 ~ dnorm(0,sd=10)
  D.beta1 ~ dnorm(0,sd=10)
  #detection priors
  lam0 ~ dunif(0,10)
  sigma ~ dunif(0,100)
  #--------------------------------------------------------------
  #Density model
  for(c in 1:n.cells) {
    lambda.cell[c] <- exp(D.beta0 + D.beta1*D.cov[c])*cellArea #expected N in cell c
    pi.cell[c] <- lambda.cell[c] / lambda.N #expected proportion of total N in cell c
  }
  lambda.N <- sum(lambda.cell[1:n.cells]) #expected N in state space
  N ~ dpois(lambda.N) #realized N in state space
 
  for(i in 1:M) {
    #dunif() here implies uniform distribution within a grid cell
    #also tells nimble s's are in continuous space, not discrete
    s[i,1] ~  dunif(xlim[1],xlim[2])
    s[i,2] ~  dunif(ylim[1],ylim[2])
    #get cell s_i lives in using look-up table
    s.cell[i] <- cells[trunc(s[i,1]/res)+1,trunc(s[i,2]/res)+1]
    dummy.data[i] ~ dCell(pi.cell[s.cell[i]]) #categorical likelihood for this cell, equivalent to zero's trick
    #Observation model, skipping z_i=0 calculations
    lam[i,1:J] <- GetDetectionRate(s = s[i,1:2], X = X[1:J,1:2], J=J,sigma=sigma, lam0=lam0, z=z[i])
    y[i,1:J] ~ dPoissonVector(lam=lam[i,1:J]*K1D[1:J],z=z[i]) #vectorized obs mod
  }
})
#custom Metropolis-Hastings update for N/z

[Chris Paciorek]

Hi Ben,

It looks like the memory explosion is coming from conjugacy checking.
Your work-around avoids that problematic conjugacy checking for
`D.beta0` and `D.beta`. Alternatively you could set `checkConjugacy =
FALSE` in your `configureMCMC` call. I don't think the resolution is
the key thing here - I see very high memory use (more than 100 GB)
even with the 4096 cells. It may just be that you have a machine with
a lot of memory and it only starts to run out of memory when you
increase the resolution by a bit.

We'll put on our to-do list to see if there is some pathology here in
conjugacy checking that we can fix.

-chris

-chris

> --
> You received this message because you are subscribed to the Google Groups "nimble-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/92fd9491-d09d-460b-a51b-9bc2de156628n%40googlegroups.com.