Occupancy model convergence issues

84 views
Skip to first unread message

Kelley Erickson

unread,
Apr 20, 2020, 10:52:12 AM4/20/20
to nimble-users
Hi, I'm trying to get an occupancy model for a large dataset to converge and am running into memory usage issues. I can't run the chains for enough iterations to achieve convergence without getting an error along the lines of "Can't allocate vector of size X GB".  I was wondering if anyone had any suggestions, whether it's increasing sampling efficiency or reducing the amount of memory used.

Thank you in advance!

Kelley

I have included the model code below, along with some details about the size of the data.

code <- nimbleCode({

    #Estimate psi for each site
    for (i in 1:nsites) {
        logit(psi[i]) <- alpha0 + beta*X1[i] + gamma*X2[i] + delta*X3[i] 
        z[i] ~ dbern(psi[i])
    }

    #Estimate p for each observation
    for (j in 1:nsites.visits) {
        logit(p[j]) <- obs[IDobs[j]] + zeta*Y1[j] + eta*Y2[j] + mos[IDmos[j]] + yr[IDyr[j]]
        p.temp[j]  <- z[Site[j]] * p[j]
        h[j] ~ dbern(p.temp[j])
    }

    #Priors
    alpha0 ~ dunif(-5, 5) #intercept for occupancy
    beta ~ dunif(-5, 5) # occupancy coefficient associated with population density
    gamma ~ dunif(-5, 5) #occupancy coefficient associated with temperature
    delta ~ dunif(-5, 5) # occupancy coefficient associated with area
    zeta ~ dunif(-5, 5) # detection coefficient associated with whether collector has previously collected the species
    eta ~ dunif(-5, 5) # detection coefficient associated with whether collector is a citizen scientist

    #Random effect of collector
    for(j in 1:nobs) {
      obs[j] ~ dnorm(0, tau=tauobs)
    }

    #Random effect of month
    for(k in 1:nmos){
      mos[k] ~ dnorm(0, tau=taumos)
    }

    #Random effect of year
    for(l in 1:nyr){
      yr[l] ~ dnorm(0, tau=tauyr)
    }

    #Hyperpriors
    tauobs ~ dgamma(0.01, 0.01)
    taumos ~ dgamma(0.01, 0.01)
    tauyr  ~ dgamma(0.01, 0.01)
   
    #Derived parameter
    N <- sum(z[])
})

monitors <- c("N","psi",  "p", "alpha0","beta", "gamma", "delta", "zeta",
                   "eta", "obs", "mos", "yr", "tauobs",
                   "taumos", "tauyr") # parameters to monitor

#N: the number of sites occupied (1 value)
#psi: Occupancy (67 values)
#p: Detection probability (24240 values, one for each site visit)
#alpha0: Occupancy intercept (1 value)
#beta: occupancy coefficient (1 value)
#gamma: occupancy coefficient (1 value)
#delta: occupancy coefficient (1 value)
#zeta: detection coefficient (1 value)
#eta: detection coefficient (1 value)
#obs: random effect of collector (762 values)
#mos: random effect of month (12 values)
#yr: random effect of year (124 values)
#tauobs: hyperprior associated with random effect of collector
#taumos: hyperprior associated with random effect of month
#tauyr: hyperprior associated with random effect of year

initsFx <- function() {

    list(
   
      z = rep(1, nsites),
        beta = rnorm(1),
        gamma = rnorm(1),
        delta = rnorm(1),
        zeta = rnorm(1),
        eta = rnorm(1),
        alpha0 = rnorm(1),
        obs = rnorm(nobs),
        mos = rnorm(nmos),
        yr = rnorm(nyr),
        tauobs = rlnorm(1),
        taumos = rlnorm(1),
        tauyr = rlnorm(1)
    )
}

nimbleOptions(clearNimbleFunctionsAfterCompiling=T)
mcmc1 <- nimbleMCMC(
    code = code, constants = constants, data = data,
    monitors = monitors, inits = initsFx,
    nchains = 1, niter = 102686, nburnin = 1000, thin = 1,
    summary = T, WAIC = F, samplesAsCodaMCMC = TRUE
)


Perry de Valpine

unread,
Apr 20, 2020, 11:17:07 AM4/20/20
to Kelley Erickson, nimble-users
Hi Kelley,

Thanks for the question.

It looks like you are trying to monitor 25215 parameters for 102686 iterations.  That's about 2.6 billion numbers.  Each double-precision number is 8 bytes, and a gigabyte is 2^30 bytes.  Putting it together, it looks like you are requesting 19.3 gigabytes in MCMC output.  Sometimes nimble gets into other kinds of memory use issues, but in this case I think you'll want to reduce what you are requesting from the MCMC.  Options would include thinning, not monitoring p, or using the monitor2/thin2 system to monitor p at much higher thinning than the other parameters.  To use monitors2/thin2, you'll have to switch to building the model and MCMC yourself.  It would look like:

m <- nimbleModel(code = code, constants = constants, data = data, inits = initsFx)
MCMC <- buildMCMC(m, monitors = monitors, monitors2 = monitors2) ## take "p" out of monitors and put it in monitors2
comp <- compileNimble(m, MCMC)
samples <- runMCMC(comp$MCMC, niter = 100000, nburnin = 1000, thin = 1, thin2 = 100, summary = TRUE, WAIC = FALSE, samplesAsCodaMCMC = TRUE)

-Perry


--
You received this message because you are subscribed to the Google Groups "nimble-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/96e648fa-e84a-47b9-b3c9-cf4e3cda091a%40googlegroups.com.

Andrew Crosby

unread,
Apr 20, 2020, 4:07:32 PM4/20/20
to Kelley Erickson, nimble-users
HI Kelley, 
I've run into this problem a lot, since I work with extremely large datasets as well. It gets especially bad when you are fitting multi-species models. 

My number 1 solution is to save only the stochastic nodes that bear on the value you need to know (i.e., the coefficients, the z values, and the random effect values). If you know these values, you can calculate psi and p for each iteration after the fact. 


--
You received this message because you are subscribed to the Google Groups "nimble-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/96e648fa-e84a-47b9-b3c9-cf4e3cda091a%40googlegroups.com.


--
Andrew D. Crosby
Postdoctoral Fellow
Boreal Avian Modelling Project
University of Alberta
Edmonton, AB
Reply all
Reply to author
Forward
0 new messages