Best Way to Distribute Functions that Use nimbleExternalCall in R Packages

[joec...@googlemail.com]

Dear list,

I am currently writing an R package that, as part of implementation, extends some of NIMBLE's functionality with a set of custom distributions.  A couple of the functions added require linkage to C++ code.

My question is a conceptual one.

How is the best way to package these functions in a portable way during development?  I've currently investigated two options but both have some significant drawbacks:

*Option 1*:

I put the "myCustomFunc.cpp" and "myCustomFunc.h" files in the /inst/nimble_custom folder of the package so that they survive the installation process.  I then add some code to the .onLoad function that firstly compiles the source files to create a .o file in a temporary session directory (as the installation path will most likely not have write permissions for most users) and then makes a new global object that is the result of the nimbleExternalCall function using the header file location and newly create .o file as inputs.

The downside to this approach is that the source files need to be compiled every time the package namespace is loaded which could result in a significant load time if I start to write lots of external functions.  It also has the downside that the external function wrapper is not available when the package is being built and so you get loads of warnings from each call of the nimbleFunction that uses your external function saying something like "myCustomFunc appears to be an R function and so must be defined as a nimble function in order to compile this function".  I have suppress these warnings to pass a CRAN check.

*Option 2*

I put the "myCustomFunc.cpp" and "myCustomFunc.h" files the src subfolder of my package.  These functions then get compiled during the normal building of the package and can be exposed through the dll/so object.  This feels like the best way to go as I'm not fighting R's normal packaging routines but I just can't get it to work.  The nimbleExternalCall can only take .o files as inputs (as far as I can see) and not use .dll/.so files.  The .o intermediate products are all deleted upon installation.

I've tried implementing some kludges like adding a target onto the makevars files that copies the myCustomFunc.o file to /inst/nimble_custom but that opens up a whole host of other problems: not least of which it causes my package to fail the CRAN checker because there is compiled code in a place it shouldn't be.

I was wondering if anyone here had had experience of extended nimble functionality with packages that use the nimbleExternalCall interface.  How did you do it and keep your package passing the checker?

Many thanks in advance for people's time:

All the best,

Joe

[Perry de Valpine]

Hi Joe,

Thanks for the clear explanation. You've hit upon some genuine challenges. I can share some thoughts but not as satisfying an answer as you might hope for.

nimble itself has the same challenge. As far as I know, there is no way to distribute compiled objects in a package that can be linked against reliably and for each operating system. And I believe some of the reasons for that have to do with safety and security (on some operating systems, you can't routinely move a compiled library and then link against it).

nimble's (also non-ideal) solution combines a custom build process and a one-time compilation that happens in each session. The custom build process copies some of the files from inst/CppCode to src, where they can be compiled into nimble.so  (or .dll on windows). That allows a user to load the package and call for example some of nimble's own distributions, just like for any package's compiled components from src. Then on the first call to compileNimble in a session, nimble builds a library to link against for that and any further calls to compileNimble in the session. That library uses some of the same inst/CppCode files and more. That one-time compilation in each session is the reason the first call to compileNimble always takes longer. Creating a custom build is not fully necessary for this approach; you could instead have two copies of some C++ code, one in src and one in inst.

A hard part is that as a package user of nimble, you don't have control to trigger a step like the one-time compilation when compileNimble is called (unless your package provides a wrapper for calling it), and you note it would not be great to do it on package loading. There may be some clever way to use a lazy object (via delayedAssign) to accomplish this, if you can arrange for an object that is touched only when a user needs one of your compiled pieces.

Another potential solution would be to look into R_GetCCallable. I believe this is what Rcpp arranges to use from the Rcpp::interfaces attribute in C++, which allows one package to use another package's compiled functions. This does not use linking but rather R's mechanism for sharing function pointers across packages.

I'm hoping some of the big changes coming in "nimble 2.0" will make this easier, but that won't help you right away.

HTH

Perry

--
You received this message because you are subscribed to the Google Groups "nimble-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/nimble-users/747a798d-9a58-418a-8016-e877eb11f50bn%40googlegroups.com.