31 views

Skip to first unread message

Apr 24, 2023, 11:52:46 AM4/24/23

to nimble-users

We recently uncovered a bug that produces incorrect results when
using runCrossValidate() in versions of nimble starting with 0.10.0
(released in October 2020). The bug affects the default MSE
loss function and any user-defined loss functions, but not the
'predictive' loss function. Unfortunately the bug caused incorrect
MCMC sampling when data were held out, so any cross-validation results
obtained with runCrossValidate should be recalculated.

The
bug fix will be in our upcoming release of version 1.0.0 and is also
included in the development version of NIMBLE on our 'devel' branch, which can be installed from GitHub.

-Chris, for the NIMBLE Development Team

May 16, 2023, 12:08:30 PM5/16/23

to nimble-users

Hey NIMBLE team,

I recently used runCrossValidate() on a model that uses the Dirichlet process (DP). In previous NIMBLE versions, CV worked without any error but having updated NIMBLE to the development version to recheck analysis, I have an error message. Below is the NIMBLE DP example given in the manual and this gives the same error I obtained with my model.

code <- nimbleCode({

z[1:N] ~ dCRP(alpha, size = N)

alpha ~ dgamma(1, 1)

for(i in 1:M) {

thetatilde[i] ~ dnorm(0, var = 100)

s2tilde[i] ~ dinvgamma(1, 1)

}

for(i in 1:N)

y[i] ~ dnorm(thetatilde[z[i]], var = s2tilde[z[i]])

})

set.seed(1)

constants <- list(N = 100, M = 50)

data <- list(y = c(rnorm(50, -5, sqrt(3)), rnorm(50, 5, sqrt(4))))

inits <- list(thetatilde = rnorm(constants$M, 0, 10),

s2tilde = rinvgamma(constants$M, 1, 1),

z = sample(1:10, size = constants$N, replace = TRUE),

alpha = 1)

model <- nimbleModel(code, constants, data, inits)

model$calculate()

foldFunction <- function(i){

foldNodes_i <- paste0('y[i]') # will return 'y[1]' for i = 1 e.g.

return(foldNodes_i)

}

dyesMCMCconfiguration <- configureMCMC(model)

crossValOutput <- runCrossValidate(MCMCconfiguration = dyesMCMCconfiguration,

k = 2,

foldFunction = foldFunction,

lossFunction = "MSE",

MCMCcontrol = list(niter = 5000,

nburnin = 500), silent = T)

The error I obtained is Error in buildMCMC(modelMCMCConf) :

sampler_CRP: At least one variable has to be clustered for each cluster membership ID.z[1:N] ~ dCRP(alpha, size = N)

alpha ~ dgamma(1, 1)

for(i in 1:M) {

thetatilde[i] ~ dnorm(0, var = 100)

s2tilde[i] ~ dinvgamma(1, 1)

}

for(i in 1:N)

y[i] ~ dnorm(thetatilde[z[i]], var = s2tilde[z[i]])

})

set.seed(1)

constants <- list(N = 100, M = 50)

data <- list(y = c(rnorm(50, -5, sqrt(3)), rnorm(50, 5, sqrt(4))))

inits <- list(thetatilde = rnorm(constants$M, 0, 10),

s2tilde = rinvgamma(constants$M, 1, 1),

z = sample(1:10, size = constants$N, replace = TRUE),

alpha = 1)

model <- nimbleModel(code, constants, data, inits)

model$calculate()

foldFunction <- function(i){

foldNodes_i <- paste0('y[i]') # will return 'y[1]' for i = 1 e.g.

return(foldNodes_i)

}

dyesMCMCconfiguration <- configureMCMC(model)

crossValOutput <- runCrossValidate(MCMCconfiguration = dyesMCMCconfiguration,

k = 2,

foldFunction = foldFunction,

lossFunction = "MSE",

MCMCcontrol = list(niter = 5000,

nburnin = 500), silent = T)

The error I obtained is Error in buildMCMC(modelMCMCConf) :

I think this has to do with my initial values for z? Many thanks for the help.

Kind regards,

Fabian

May 16, 2023, 8:31:07 PM5/16/23

to fabiank...@gmail.com, nimble-users

Hi Fabian,

The error you're getting is not directly related to runCrossValidate. Rather it relates to a change in how we handle predictive nodes in MCMC sampling as of version 0.13.0. A

As of 0.13.0, nodes that have no data nodes as child nodes are only sampled at the end of each iteration of an MCMC and do not affect sampling of the other unknowns in the model. In general this improves MCMC mixing.

The effect of this in a BNP model such as yours is that if there are elements of 'y' that are not known values (i.e., not data) (such as the held-out elements in cross-validation), then the corresponding elements of the cluster membership vector ('z' in your case) do not have any dependents determined when MCMC samplers are assigned. We don't allow this in our CRP sampler (this is to protect against unanticipated model structures that might invalidate our CRP sampler), hence the message you are getting. It would also occur in a non-cross-validation context, e.g., if you simply set some of the 'y' elements to be missing and therefore not flagged as data nodes.

The solution is to revert to NIMBLE's previous handling of posterior predictive nodes, in which the nodes with missing values (those that if they had values would be data nodes) are treated as part of the full set of unknown parameters and sampling of other parameters is conditioned on the current value(s) of those nodes. You can do this by setting the following two options at the beginning of your code.

nimbleOptions(MCMCusePredictiveDependenciesInCalculations = TRUE)

nimbleOptions(MCMCorderPosteriorPredictiveSamplersLast = FALSE)

nimbleOptions(MCMCorderPosteriorPredictiveSamplersLast = FALSE)

Let us know if you have questions or that doesn't fix things for you.

-chris

--

You received this message because you are subscribed to the Google Groups "nimble-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/0ee05bf0-247c-430a-8e17-c1ecfe2134e9n%40googlegroups.com.

May 17, 2023, 5:31:54 AM5/17/23

to paci...@stat.berkeley.edu, nimble-users

Hey Chris,

Thanks for the fast response and clear explanation. It makes sense and I understand why I got this error. Adding those NIMBLE options did the trick for me.

Cheers,

Fabian

Reply all

Reply to author

Forward

0 new messages

Search

Clear search

Close search

Google apps

Main menu