how to include external c++ code in stan model for r package based on rstan?

[Glenn Goldsmith]

Hi there, 

I have a (r)stan model that uses an external .hpp file to define a function with custom autodiff, which I'm calling from an R wrapper function. The model compiles fine using the approach described here: https://cran.r-project.org/web/packages/rstan/vignettes/external.html. However, the constant recompilation needed to avoid crashing R is making testing extremely slow, so I'm trying to turn it into a R package with a precompiled stan model. 

I've successfully run rstantools::rstan_package_skeleton, made the cleanup and cleanup.win files executable, added the .hpp file to the inst directory (not sure this is the right place for it?) and edited the NAMESPACE file to include:

useDynLib(reffreg, .registration = TRUE)

import(Rcpp)

import("methods")

export(reffreg)

However, when I try to build the package using devtools::build, I (unsurprisingly) get an error saying that the custom autodiff function is declared but not defined. I'm assuming I somehow need to let the stan compiler know where find the .hpp file (in tools/make_cpp.R?), but am not sure how to do this. 

Any advice very gratefully received,

Glenn.

[Ben Goodrich]

On Thursday, May 4, 2017 at 10:23:33 PM UTC-4, Glenn Goldsmith wrote:

I've successfully run rstantools::rstan_package_skeleton, made the cleanup and cleanup.win files executable, added the .hpp file to the inst directory (not sure this is the right place for it?)

I think inst/include/myheader.hpp is fine.
 

However, when I try to build the package using devtools::build, I (unsurprisingly) get an error saying that the custom autodiff function is declared but not defined. I'm assuming I somehow need to let the stan compiler know where find the .hpp file (in tools/make_cpp.R?), but am not sure how to do this. 

Edit line 23 of tools/make_cpp.R where it calls stanc_builder() and add the argument allow_undefined = TRUE. Then back up two lines and add a "#include myheader.hpp".

Let us know if you run into more problems.

Ben

[Ben Goodrich]

On Friday, May 5, 2017 at 2:04:19 AM UTC-4, Ben Goodrich wrote:

On Thursday, May 4, 2017 at 10:23:33 PM UTC-4, Glenn Goldsmith wrote:

I've successfully run rstantools::rstan_package_skeleton, made the cleanup and cleanup.win files executable, added the .hpp file to the inst directory (not sure this is the right place for it?)

I think inst/include/myheader.hpp is fine.

Upon further review, I think .hpp files should go under src/ .
You can get a new make_cpp.R file that implements all this from

https://raw.githubusercontent.com/stan-dev/rstanarm/master/tools/make_cpp.R

but you also need new versions of the cleanup files

https://raw.githubusercontent.com/stan-dev/rstanarm/master/cleanup
https://raw.githubusercontent.com/stan-dev/rstanarm/master/cleanup.win

Ben

[Glenn Goldsmith]

Thanks Ben, that's super helpful. 

Unfortunately, however, I'm still struggling to get this to work. Further details below; once again, any suggestions very gratefully received.

I think the issue stems from the fact that the cpp code I'm using is an edited version of stan-generated cpp code for a user-defined function. As a result, its reliant on both the headers required by the rest of the stan-generated code, as well as quite a number of other user-defined functions in my .stan file. 

This isn't an issue when using the includes argument in rstan::stan_model() (as described here) because the relevant code is inserted inside the model namespace after the other user defined functions, so I had thought the easiest way to address this would be to edit the make_cpp.R file to replicate this behaviour. 

My edited code that attempts to do this (with edits vs. your original version in red) is:

 

stan_files <- dir("exec", pattern = "stan$", full.names = TRUE)
# include_files <- dir("src", pattern = "hpp$")
include_files <- dir("src", pattern = "hpp$", full.names = TRUE)
include_code <- paste(unlist(sapply(include_files, readLines)),"\n",sep = "",collapse = "")
cat(readLines(file.path("inst", "chunks", "license.stan")),
    "#ifndef MODELS_HPP", "#define MODELS_HPP",
    "#define STAN__SERVICES__COMMAND_HPP", "#include <rstan/rstaninc.hpp>",
    # if (length(include_files)) paste0('#include "', include_files, '"'),
    sapply(stan_files, FUN = function(f) {
      cppcode <- rstan::stanc_builder(f, allow_undefined = TRUE,
                                      isystem = file.path("inst", "chunks"))$cppcode
      cppcode <- gsub("typedef.*stan_model.*;", "", cppcode, perl = TRUE)
      if (length(include_files)) cppcode <- sub("(class[[:space:]]+[A-Za-z_][A-Za-z0-9_]*[[:space:]]*: public prob_grad \\{)",
                                                paste(include_code," \\1"),
                                                cppcode)
      return(cppcode)
    }), "#endif", file = file.path("src", "include", "models.hpp"), 
    sep = "\n", append = FALSE)

I've tested this by running each of the individual lines in the cleanup file from the terminal, and I think the revised make_cpp.R is generating a models.hpp file that contains the cpp code I want.

However, when I run the line in cleanup that calls roxygenize, I get a 'Function declared but not defined' error, with the 'ERROR at line xxx' pointing to the end of the functions block in the original .stan file (rather than anything in the cpp files).

I also get the same error calling devtools::build() even if I comment out the call to roxygenize in the cleanup files.

Glenn. 

--
You received this message because you are subscribed to a topic in the Google Groups "Stan users mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/stan-users/pspLrs0oi9c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to stan-users+unsubscribe@googlegroups.com.
To post to this group, send email to stan-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ben Goodrich]

On Sunday, May 7, 2017 at 9:20:12 AM UTC-4, Glenn Goldsmith wrote:

I've tested this by running each of the individual lines in the cleanup file from the terminal, and I think the revised make_cpp.R is generating a models.hpp file that contains the cpp code I want.

If you do

R CMD INSTALL --preclean

you can look at the src/include/models.hpp file.
 

However, when I run the line in cleanup that calls roxygenize, I get a 'Function declared but not defined' error, with the 'ERROR at line xxx' pointing to the end of the functions block in the original .stan file (rather than anything in the cpp files).

I also get the same error calling devtools::build() even if I comment out the call to roxygenize in the cleanup files.

In your R/stanmodels.R, there is another call to rstan::stanc_builder() that needs to add an allow_undefined = TRUE.

Ben

[Glenn Goldsmith]

Ah, yes, that was it - all working now. 

Thanks Ben, I would never have found that by myself!

As an aside, does anyone happen to know why devtools::install() and devtools::install_local() with INSTALL_opts="--preclean" would fail with a "shared object ‘mymodel.so’ not found" error while R CMD install --preclean works fine from the terminal?)

Cheers,

Glenn.