Parallel chains with Nimble error...any ideas why?

[Michael Roast]

Hi all,

I'm trying to run MCMC chains in parallel for a hidden markov model for a simulated capture recapture data set, but I am getting an error message than I cannot solve.

Error in checkForRemoteErrors(val) : 

  4 nodes produced errors; first error: unused argument (X[[i]])

I think the X argument is necessary to specify the required n chains?

This is the first time I'm trying to run chains in parallel so I may be missing something obvious, but the worked example from the website works perfectly but not my HMM, and my function seems to work fine on a single chain. 

Thanks in advance for any wisdom!

Michael

Error in checkForRemoteErrors(val) : 

  4 nodes produced errors; first error: unused argument (X[[i]])

#### The model that causes this error #### 

library(nimble)

library(parallel)

detectCores()

>8

# simple 'capture history' dataset simulated.

ymat <- matrix(sample(c(0,1),120,replace = TRUE),ncol = 6, nrow = 20)

this_cluster <- makeCluster(4) 

# function containing all model code

paraChain_code <- function(y){

library(nimble)

  

first <- apply(y, 1, function(x) min(which(x!= 0),na.rm = TRUE))

  

hmm.model<- nimbleCode({

  # Priors

  phi ~ dunif(0,1)    # prior for survival φ

  p ~ dunif(0,1)      # prior for detection p

  

  # Vector of initial state probabilities δ

  delta[1] <- 1     # Alive

  delta[2] <- 0     # Dead

  

  # Transition matrix Γ

  gamma[1,1] <- phi     # transition Pr(alive t --> alive t+1)

  gamma[1,2] <- 1-phi    # transition Pr(alive t --> dead t+1)

  gamma[2,1] <- 0     # transition Pr(dead t --> alive t+1)

  gamma[2,2] <- 1   # transition Pr(dead t --> dead t+1)

  

  # Observation matrix Ω

  omega[1,1] <- 1 - p # observation Pr(alive t --> not detected t)

  omega[1,2] <- p     # observation Pr(alive t --> detected t)

  omega[2,1] <- 1 # observation Pr(dead t --> not detected t)

  omega[2,2] <- 0     # observation Pr(dead t --> detected t)

  

  # Likelihood

  for(i in 1:N){    # Loops over each individual (row)

    z[i,first[i]] ~ dcat(delta[1:2])  # Draws an initial state   

    for(j in (first[i]+1):T){    # Loops through each time point (column)

      z[i,j] ~ dcat(gamma[z[i,j-1], 1:2]) # Draws current state based on previous state

      y[i,j] ~ dcat(omega[z[i,j], 1:2])  # Draws an observation based on current state.

    }

  }

})

# Defines constants in the model

my.constants <- list(N = nrow(y),   # N individual

                     T = ncol(y),   # N of time points

                     first = first) # occasions of first capture

# Defines data in the model

# To remove 0s. 1 is non-detected, 2 is detected

my.data <- list(y = y+1)

# Defines initial values for states and probabilities

# Sets initial latent state to either Resident  or Transient

zinits <- y + 1

zinits[zinits == 2] <- 1

initial.values <-  list(phi = runif(1,0,1),

                        p = runif(1,0,1),

                        z = zinits)

# Specfies the parameters of interest to record in output

parameters.to.save <- c("phi","p")

#MCMC model details

n.iter <- 1000

n.burnin <- 200

myModel <- nimbleModel(code = hmm.model,        

                               constants = my.constants,      

                               data = my.data,

                               inits = initial.values)

CmyModel <- compileNimble(myModel)

myMCMC <- buildMCMC(CmyModel, monitors = parameters.to.save)

CmyMCMC <- compileNimble(myMCMC)

results <- runMCMC(CmyMCMC, niter = n.iter, nburnin = n.burnin)

return(results)

}

paraChain_code(y = ymat) 

# Function works correctly to create a single chain

#Then to run in parallel fails each time.

chain_output <- parLapply(cl = this_cluster, X = 1:4,

                          fun = paraChain_code,

                          y = ymat)

>Error in checkForRemoteErrors(val) : 

  4 nodes produced errors; first error: unused argument (X[[i]])

stopCluster(this_cluster)

##### Directly Copied from Website works fine #####

library(parallel)

this_cluster <- makeCluster(4)

set.seed(10120)

myData <- rgamma(1000, shape = 0.4, rate = 0.8)

# Create a function with all the needed code

run_MCMC_allcode <- function(seed, data) {

  library(nimble)

  

  myCode <- nimbleCode({

    a ~ dunif(0, 100)

    b ~ dnorm(0, 100)

    

    for (i in 1:length_y) {

      y[i] ~ dgamma(shape = a, rate = b)

    }

  })

  

  myModel <- nimbleModel(code = myCode,

                         data = list(y = data),

                         constants = list(length_y = 1000),

                         inits = list(a = 0.5, b = 0.5))

  

  CmyModel <- compileNimble(myModel)

  

  myMCMC <- buildMCMC(CmyModel)

  

  CmyMCMC <- compileNimble(myMCMC)

  

  results <- runMCMC(CmyMCMC, niter = 10000, setSeed = seed)

  

  return(results)

}

chain_output <- parLapply(cl = this_cluster, X = 1:4, 

                          fun = run_MCMC_allcode, 

                          data = myData)

stopCluster(this_cluster)

https://r-nimble.org/nimbleExamples/parallelizing_NIMBLE.html

[Chris Paciorek]

hi Michael,

Your `paraChain_code` function needs to take two arguments. The first is just an index, the second is your data.

Note that in our example code, we use that index to set a separate seed for each chain. And aside from that, in general any parallel lapply type statement

needs to either take a list  or a vector over which the function will be run in parallel. If a vector, that will often be a vector of indexes as in our example code.

-chris

--
You received this message because you are subscribed to the Google Groups "nimble-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/100b12a3-9c33-442d-b25f-1a84dce1966bn%40googlegroups.com.