Error "object 'dep_dcat_identity_value_sizeMax' not found" in v1.4.0

[PierGianLuca]

Dear wonderful Nimble devs,

I updated to v1.4.0 from CRAN, and tested whether my proto-package "inferno" works with the new version. But I got this error:

> object 'dep_dcat_identity_value_sizeMax' not found

which seems to occur directly after this Note from Nimble:

> [Note] Reordering posterior_predictive samplers to execute last

So I suspect that's related to my calling `confnimble$setSamplerExecutionOrder`, here:

https://github.com/pglpm/inferno/blob/f661cdde167579de1e586afada16244c8fce43db/R/learn.R#L2679

I checked the Nimble NEWS, but can't find any relevant items, especially in "DEVELOPER LEVEL CHANGES".

While preparing a minimal working example, I thought I could start mentioning this, in case you already have a hint of what could be the cause.

Thank you for version 1.4.0! :)

Kind regards,
Luca

[Daniel Turek]

Luca, thank you very much for this error report.  There were some updates made to the MCMC system of conjugacy processing in version 1.4.0, which are probably what's causing the problem.  I would not have suspected his has anything to do with the setSamplerExecutionOrder method, but clearly something is happening that we didn't foresee.  If you're please able to put together a reproducible example of the problem (something that worked fine for earlier versions of nimble, and now causes an error when using version 1.4.0), then we'll take a look and see what's happening.

Thanks,

Daniel

[PierGianLuca]

Dear Daniel,

Enclosed is a minimal script that works with Nimble 1.3.0 (with warnings about log-probabilities, but they're expected) but doesn't work with 1.4.0.

I suspect the problem may come from the indexing with `Nprob` or `Ni` or `Nf`.

Please note that the implementation you see in this minimal example may not make much sense, but it does make sense in the full package; see comments in line 37.

Thank you for looking into this!
Luca

[Daniel Turek]

Luca, thank you so much for sending the simplified reproducible example.  That was extremely helpful for figuring out what was happening.

This error does not relate to your code, but in fact to an update that was made to the conjugacy system added in nimble release v1.4.0.  In that release, the conjugacy system was generalized to handle a new conjugate relationship for the "tau" parameter of dcar_normal distributions.  However, with that update, some assumptions that were being made in the conjugacy processing were no longer valid, which caused the error you encountered.  You can read a little bit more detail about the situation here, but for anyone else reading this I'll add that this error should be uncommon, only relates to multivariate conjugate nodes, and will trigger an Error (a hard stoppage) if encountered - as Luca experienced.

Luca, assuming I got the fix right, are you able to try installing nimble from the GitHub branch containing this fix?  The branch is called "conj_sizes_fix".  You can install nimble from that branch using the code below:

remove.packages('nimble')
library(remotes)
remotes::install_github('nimble-dev/nimble', ref = 'conj_sizes_fix', subdir = 'packages/nimble')
library(nimble)

Please let me know if this seems to work for you, and we can go from there.

Cheers!

Daniel

[PierGianLuca]

Hi Daniel,

Thank you so much! From a couple of tests, I get no errors and numerical results exactly like those with v1.3.0!

I read the comments on GitHub. If you think this change is not important for Nimble, can you suggest how I can change the code to make it work again? For me that would also work.

Incidentally, thanks to this bug I discovered another bug not related to Nimble :)

Cheers,
Luca

On 251219 15:27, Daniel Turek wrote:
> Luca, thank you so much for sending the simplified reproducible example.  That was extremely helpful for figuring out what was happening.
>

> This error does not relate to your code, but in fact to an update that was made to the conjugacy system added in nimble release v1.4.0.  In that release, the conjugacy system was generalized to handle a new conjugate relationship for the "tau" parameter of dcar_normal distributions.  However, with that update, some assumptions that were being made in the conjugacy processing were no longer valid, which caused the error you encountered.  You can read a little bit more detail about the situation here <https://github.com/nimble-dev/nimble/pull/1605>, but for anyone else reading this I'll add that this error should be uncommon, only relates to multivariate conjugate nodes, and will trigger an Error (a hard stoppage) if encountered - as Luca experienced.

[Daniel Turek]

I'm so glad it helped you find another bug.  Similarly, your code uncovered this oversight in the conjugacy updates, so it was beneficial all around.

I think this fix (currently only on branch "conj_sizes_fix") is important enough for us to incorporate it into the released version of nimble.  I'll touch base with the nimble team to see when we can make that happen.  In the meantime, if you're able to use "useConjugacy = FALSE" as an argument to configureMCMC() in your package code, that would avoid the problem - but it would also prevent any conjugate sampling.  You also could do a little more careful work, to assign conjugate samplers to any other parts of the model, but avoid conjugate samplers for the problem-causing nodes, which are the "Nprob" nodes following the Dirichlet distribution.  I'm sorry, I don't have a much better solution for you in the meantime.  Let me know if you decide to use either of those two workarounds, and need any help doing so.

Cheers,

Daniel

[PierGianLuca]

Dear Daniel,

I'm glad to hear the fix will be incorporated in the future! It seems no-one is really using the inferno package (😢), so I'm in no hurry :)

Nimble's conjugacy is what makes the particular nonparametric representation that I'm using so fast, despite being nonparametric. I tried from time to time to set up the conjugate samplers by hand, but Nimble does a much better job automatically. This tells a lot about the amazing work you all have put in the code...

Thank you again and happy upcoming festivities!
Luca