Using Compiled Nimble Functions in R packages

[philipcindamix]

Hi,

I'm wondering if anyone has any familiarity with using compiled NimbleFunctions in R packages (i.e. not necessarily for model estimation)?

For example, it is very helpful that CompileNimble effectively translates the R code to C++ so I would like to use this compiled code in a standalone R package. I know that I can get the underlying C++ code saved to a directory using the dirname, however, I'm not sure how to implement it an R package. 

I have used Rcpp code before in R packages, however, I imagine the steps would be slightly different?

Many Thanks,

Philip 

 

[Scott Brown]

I am just about to start working through this problem as well.  I do not have any advice currently, but am happy to share anything that I learn.  Currently reading through https://r-pkgs.org/structure.html

I DO know that there is currently (or at least was quite recently) a bug where `compileNimble` complains if you have a "." in your package name, so beware of that.  :)  Possibly if the compilation happens *before* packageification as you suggest then this isn't a problem?

Cheers,

-Scott

[Scott Brown]

One particular issue that I am currently struggling with is this contention: """Finally, it’s important to note that library() should NEVER be used inside a package. Packages and scripts rely on different mechanisms for declaring their dependencies and this is one of the biggest adjustments you need to make in your mental model and habits.""" [1]

I have found that calling everything with `nimble::` is difficult and often fails due to nimble functions that call other nimble functions, and my code complains that "cannot find 'so-and-soNimbleMethod'" because I haven't called `library(nimble)` in my package....

[1] https://r-pkgs.org/structure.html

[Chris Paciorek]

With regard to Scott's question about `library(nimble)`, you'll need to put `nimble` into your package `Depends` (in `DESCRIPTION) (as discussed in this GitHub issue). CRAN doesn't recommend this (they'd prefer the use of `Imports`), but they do allow it. If you do this then you don't need `nimble::`. 

--
You received this message because you are subscribed to the Google Groups "nimble-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/4b05366d-b72f-4689-9626-0c62ba8024adn%40googlegroups.com.

[Chris Paciorek]

Philip,

If you are simply trying to use nimble's functionality (in particular nimbleFunctions) in your package, then yes this is possible and there are various examples on CRAN (e.g., `BayesNSGP`, `nimbleEcology`, etc.)

But it sounds like you may want to use nimble to generate C++ and then have that as part of your package? If so then you can take that C++ and make it part of your package, either with or without using Rcpp (in the latter case, the Writing R Extensions manual has lots of (potentially hard to read) information). Using Rcpp would probably make the package creation easier --  you'd then have the internal computations use the C++ you generated via nimble, while the interfaces between R and C++ would come from Rcpp.  If you can say more about what you are trying to do, we may be able to give more guidance. 

-chris

[philipcindamix]

Thanks Chris and Scott,

@Chris: Yes it is the second option. I want the Cpp code to be compiled for the source (or binary) package at install time. For the purpose of illustration it could be any function using solveLeastSquares as an example from the manual.

Once I run the compileNimble I get a number of files .o, .h, .cpp etc. I'm presuming I place them in the src folder in the package. I doubt that it is as simple as that as the Makevars.win file links to my installation of Nimble which would not be portable for a package. If there was some sort of minimal package structure that somebody had to demo that would be great. In my opinion it would be another great application for Nimble. As you helpfully suggested I can look at the Writing R extension manual and figure it out eventually. 

Kind Regards,

Philip 

[Perry de Valpine]

Hi Philip,

Unfortunately it's not something we have a good workflow for right now. It is something we're working on as part of longer-term developments.

One issue, as you point out, is that compiling nimble-generated C++ currently includes nimble .h files and links to a nimble-generated DLL or .so. In fact, the first time you call compileNimble in an R session, it first builds a DLL or .so to link against within that session. Another issue is that the R interface objects (functions and class definitions that allow you to seamlessly use the compiled results) are generated from functions within the nimble package.

I hope we can provide better tools in a refactored version of our compiler tools in the future, but sorry to say we don't have that now. I'm very glad to see your interest in the idea.

Perry

To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/94d40cb1-7247-4060-a979-e2d5a04e7e7an%40googlegroups.com.

[Scott Brown]

So, one issue that I was having with creating packages for my nimble functions is that they wanted to compile at *load* time.  This is not good if I have lots of nimble functions, and users only need one.

To help with this, I created this:

```
memoiseNimbleCompilation <- function(nimbleF) {
  # not sure why we need this, but nimble complains otherwise
  nimbleF <- nimbleF
  compiledFunction <- NULL
  maybeCompiledFunction <- function(...) {
    if (is.null(compiledFunction)) {
      compiledFunction <<- compileNimble(nimbleF)
    }
    return(compiledFunction(...))
  }
  return(maybeCompiledFunction)
}

```

You can use it like this:

`theCompiledFunction <- memoiseNimbleCompilation(theUncompiledFunction)`

This makes it so that compilation is "lazy", and gets deferred until the moment a user attempts to call the function.  Of course, this only works for nimble functions, but I suppose a similar thing could be arranged for models, or mcmc builds or what have you.

[Perry de Valpine]

Thanks Scott.

I suspect the 

nimbleF <- nimbleF

can be equivalently replaced with

force(nimbleF)

the documentation of which explains why it can be useful (related to R's lexical scoping, not related to nimble).

Perry

To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/60cfc8bc-0d30-4bea-b003-c03dbcac4711n%40googlegroups.com.

[Chris Paciorek]

Hi Scott, in terms of your question about "they wanted to compile at *load* time", could you describe in more detail what you are doing / trying to do in terms of use of nimbleFunctions and compilation in your package? That would help us give more guidance here. 

In various packages that depend on nimble, the packages define nimbleFunctions but don't actually call `compileNimble` in their source code. In other cases, one might have R functions that call `compileNimble` but in that case, that should happen when a user of your package executes the relevant R function, not at load time.

-chris

To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/CABeTmqihVfm6HdNa3QiG7xLc4bexa%2B9hfAxiKpAa1T493ioAvg%40mail.gmail.com.

[Chris Paciorek]

Ah, that makes sense -- I'm not seeing the exact situation in which a user would call the R function multiple times, but I can imagine that could be the case.

Sounds like you've got this under control.

-Chris

On Wed, Nov 1, 2023 at 10:57 AM Scott Brown <gitpushor...@gmail.com> wrote:

"one might have R functions that call `compileNimble` but in that case, that should happen when a user of your package executes the relevant R function" 

Exactly, but if we are repeatedly calling this function, we don't want to repeatedly compile the nimbleFunction.

So, if we want to "hide" the compilation step from the user, the two naive solutions are to 1) compile once on library load or 2) compile each time we call the R "wrapper" for our nimbleFunction+compile.  These are both unfortunate scenarios, so having a lazy-compile-but-only-once sort of thing is what this memoize function tries to solve.