runCrossValidate bug in NIMBLE versions 0.10.0 - 0.13.1

[Chris Paciorek]

We recently uncovered a bug that produces incorrect results when using runCrossValidate() in versions of nimble starting with 0.10.0 (released in October 2020). The bug affects the default MSE loss function and any user-defined loss functions, but not the 'predictive' loss function. Unfortunately the bug caused incorrect MCMC sampling when data were held out, so any cross-validation results obtained with runCrossValidate should be recalculated.

The bug fix will be in our upcoming release of version 1.0.0 and is also included in the development version of NIMBLE on our 'devel' branch, which can be installed from GitHub.

-Chris, for the NIMBLE Development Team

[fabiank...@gmail.com]

Hey NIMBLE team, 

I recently used runCrossValidate() on a model that uses the Dirichlet process (DP).  In previous NIMBLE versions, CV worked without any error but having updated NIMBLE to the development version to recheck analysis, I have an error message. Below is the NIMBLE DP example given in the manual and this gives the same error I obtained with my model.

code <- nimbleCode({
  z[1:N] ~ dCRP(alpha, size = N)
  alpha ~ dgamma(1, 1)
  for(i in 1:M) {
    thetatilde[i] ~ dnorm(0, var = 100)
    s2tilde[i] ~ dinvgamma(1, 1)
  }
  for(i in 1:N)
    y[i] ~ dnorm(thetatilde[z[i]], var = s2tilde[z[i]])
})
set.seed(1)
constants <- list(N = 100, M = 50)
data <- list(y = c(rnorm(50, -5, sqrt(3)), rnorm(50, 5, sqrt(4))))
inits <- list(thetatilde = rnorm(constants$M, 0, 10),
              s2tilde = rinvgamma(constants$M, 1, 1),
              z = sample(1:10, size = constants$N, replace = TRUE),
              alpha = 1)
model <- nimbleModel(code, constants, data, inits)
model$calculate()

foldFunction <- function(i){
  foldNodes_i <- paste0('y[i]')  # will return 'y[1]' for i = 1 e.g.
  return(foldNodes_i)
}

dyesMCMCconfiguration <- configureMCMC(model)

crossValOutput <- runCrossValidate(MCMCconfiguration = dyesMCMCconfiguration,
                                   k = 2,
                                   foldFunction = foldFunction,
                                   lossFunction = "MSE",
                                   MCMCcontrol = list(niter = 5000,
                                                      nburnin = 500), silent = T)

The error I obtained is Error in buildMCMC(modelMCMCConf) :

sampler_CRP: At least one variable has to be clustered for each cluster membership ID.

I think this has to do with my initial values for z?  Many thanks for the help. 

Kind regards,

Fabian

[Chris Paciorek]

Hi Fabian,

The error you're getting is not directly related to runCrossValidate. Rather it relates to a change in how we handle predictive nodes in MCMC sampling as of version 0.13.0. A

As of 0.13.0, nodes that have no data nodes as child nodes are only sampled at the end of each iteration of an MCMC and do not affect sampling of the other unknowns in the model. In general this improves MCMC mixing.

The effect of this in a BNP model such as yours is that if there are elements of 'y' that are not known values (i.e., not data) (such as the held-out elements in cross-validation), then the corresponding elements of the cluster membership vector ('z' in your case) do not have any dependents determined when MCMC samplers are assigned. We don't allow this in our CRP sampler (this is to protect against unanticipated model structures that might invalidate our CRP sampler), hence the message you are getting. It would also occur in a non-cross-validation context, e.g., if you simply set some of the 'y' elements to be missing and therefore not flagged as data nodes.

The solution is to revert to NIMBLE's previous handling of posterior predictive nodes, in which the nodes with missing values (those that if they had values would be data nodes) are treated as part of the full set of unknown parameters and sampling of other parameters is conditioned on the current value(s) of those nodes. You can do this by setting the following two options at the beginning of your code.

nimbleOptions(MCMCusePredictiveDependenciesInCalculations = TRUE)
nimbleOptions(MCMCorderPosteriorPredictiveSamplersLast = FALSE)

Let us know if you have questions or that doesn't fix things for you.

-chris

--
You received this message because you are subscribed to the Google Groups "nimble-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/0ee05bf0-247c-430a-8e17-c1ecfe2134e9n%40googlegroups.com.

[Fabian Ketwaroo]

Hey Chris, 

Thanks for the fast response and clear explanation. It makes sense and I understand why I got this error. Adding those NIMBLE options did the trick for me.

Cheers,

Fabian