trouble deploying pre-compiled mod

[William Wolf]

Hi all. First question here. I've been at the following problem for hours. I've been sure to read all relevant threads in this group before posing this question.

I compile a stan model with the following code:

model <- stan(file = "model.stan", data = data, save_dso = FALSE, chains = 4, cores = 4, iter = 2000, warmup = 1000)

When I ls the current directory, I see model.rds whereas I didn't see it before. All is good thus far.

Subsequently, I refit this model with new parameters via:

fit <- stan(file = "model.stan",  data = new.data, chains = 1, cores = 1, iter = 4000, warmup = 2000)

This code is running in a Shiny app. Locally, all works as expected.

When I deploy this app to shinyapps.io, the model tries to recompile (resulting in a g++ error - presumably from lack of sufficient memory). Irrespective, I don't want this model to recompile in the first place: I'd like it to use the model.rds object I'd already built. Yes, I do include this object in the files uploaded to the shinyapps.io server.

I feel like I've tried all the things - including loading model.rds explicitly with loadRDS and passing it into the stan(fit = model ... call. What am I missing?

For completeness, below is the error I received on the shinyapps.io end. Again, I don't wish for this model to go about recompiling in the first place.

Compilation ERROR, function(s)/method(s) not created! In file included from /usr/local/lib/R/site-library/BH/include/boost/config.hpp:39:0,

                 from /usr/local/lib/R/site-library/BH/include/boost/math/tools/config.hpp:13,

                 from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/rev/core/var.hpp:7,

                 from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/rev/core/gevv_vvv_vari.hpp:5,

                 from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/rev/core.hpp:12,

                 from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/rev/mat.hpp:4,

                 from /usr/local/lib/R/site-library/StanHeaders/include/stan/math.hpp:4,

                 from /usr/local/lib/R/site-library/StanHeaders/include/src/stan/model/model_header.hpp:4,

                 from file137729fa09.cpp:8:

/usr/local/lib/R/site-library/BH/include/boost/config/compiler/gcc.hpp:186:0: warning: "BOOST_NO_CXX11_RVALUE_REFERENCES" redefined [enabled by default]

 #  define BOOST_NO_CXX11_RVALUE_REFERENCES

 ^

<command-line>:0:0: note: this is the location of the previous definition

g++: internal compiler error: Killed (program cc1plus)

Please submit a full bug report,

with preprocessed source if appropriate.

See <file:///usr/share/doc/gcc-4.8/README.Bugs> for instructions.

make: *** [file137729fa09.o] Error 4

[Ben Goodrich]

On Thursday, February 2, 2017 at 5:07:08 PM UTC-5, William Wolf wrote:

When I deploy this app to shinyapps.io, the model tries to recompile (resulting in a g++ error - presumably from lack of sufficient memory). Irrespective, I don't want this model to recompile in the first place: I'd like it to use the model.rds object I'd already built. Yes, I do include this object in the files uploaded to the shinyapps.io server.

I have never tried this before. It might be the case that if you upload the .stan file first and the .rds file second, it will get past the timestamp check that triggers a recompile. Otherwise, maybe it is possible to use clang++ instead of g++ on shinyapps.io. The best answer is that if you want predictable behavior with precompiled models, you should just make a R package that contains them. See the rstan_package.skeleton function in the rstantools package or one of several packages like rstanarm that does it this way.

Ben

[William Wolf]

Ben,

Thanks for the speedy response. In order, and for others:

1. There appears to be no way to upload/scp/etc. a file to a shinyapps.io server: it must all be bundled into the initial deploy.

2. Putting the .rds file on S3 then downloading this file after the application starts was successful in and of itself, but the `stan` called ignored it and tries to recompile.

3. I've been at the "build an R package" for several hours now. I create the package as such: 

library(rstantools)

rstan_package_skeleton(name = "my_package", stan_files = "model.stan", path = "~/Documents/r-packages/")

The results of `R CMD check my_package` are as follows:

** testing if installed package can be loaded

Error in .doLoadActions(where, attach) :

  error in load action .__A__.1 for package RescueTimeEstimation: is(module, "character"): object 'm' not found

I tried following the relevant thread in this group, which lead me to adding `useDynLib(RescueTimeEstimation, .registration = TRUE)` to the NAMESPACE file. From there, everything works. However (and crucially?), roxygen refuses to overwrite this file, so my NAMESPACE is left as is, instead of having all of the nice import statements I'm assuming I ultimately need.

4. Finally, while I have you: how do I actually use the model within the package? For example, how should the `stan(file = ...)` call look? I'm guessing it's not `stan(file = ../exec/model.stan)`. I assume this is an easy one; I'm not so experienced with C++ myself.

[Ben Goodrich]

On Friday, February 3, 2017 at 9:10:12 AM UTC-5, William Wolf wrote:

I tried following the relevant thread in this group, which lead me to adding `useDynLib(RescueTimeEstimation, .registration = TRUE)` to the NAMESPACE file. From there, everything works. However (and crucially?), roxygen refuses to overwrite this file, so my NAMESPACE is left as is, instead of having all of the nice import statements I'm assuming I ultimately need.

If you have a my_package.R file, such as

https://github.com/stan-dev/rstanarm/blob/master/R/rstanarm-package.R

that contains

#' @useDynLib my_package, .registration = TRUE

and you run roxygen2::roxygenize(clean = TRUE), it should overwrite the NAMESPACE file to include this line and any other @import statements.
 

4. Finally, while I have you: how do I actually use the model within the package? For example, how should the `stan(file = ...)` call look? I'm guessing it's not `stan(file = ../exec/model.stan)`. I assume this is an easy one; I'm not so experienced with C++ myself.

There will be an unexported named list called stanmodels. You can pass stanmodels$modelname to sampling(), optimizing(), and / or vb().

Ben
 

[William Wolf]

No luck with the first part. Even when copying the rstanarm-package.R file verbatim, I get the following error:

> roxygen2::roxygenize(clean=TRUE)

First time using roxygen2. Upgrading automatically...

Warning: The existing 'NAMESPACE' file was not generated by roxygen2, and will not be overwritten.

When I delete and recreate (as blank) the NAMESPACE file itself, I get the same error.

[Ben Goodrich]

On Friday, February 3, 2017 at 11:15:26 AM UTC-5, William Wolf wrote:

No luck with the first part. Even when copying the rstanarm-package.R file verbatim, I get the following error:

> roxygen2::roxygenize(clean=TRUE)

First time using roxygen2. Upgrading automatically...

Warning: The existing 'NAMESPACE' file was not generated by roxygen2, and will not be overwritten.

When I delete and recreate (as blank) the NAMESPACE file itself, I get the same error.

Try just deleting the NAMESPACE file and not creating a blank new one.

Ben

[William Wolf]

That worked. (And strange - I could have sworn it didn't work before).

I now have the package on GitHub and can successfully install it back to my Mac (via devtools::install_github). However, when the package tries to build on shinyapps, I get the following error:

Error in dyn.load(file, DLLpath = DLLpath, ...) : 

  unable to load shared object '/usr/local/lib/R/site-library/RescueTimeEstimator/libs/RescueTimeEstimator.so':

  /usr/local/lib/R/site-library/RescueTimeEstimator/libs/RescueTimeEstimator.so: invalid ELF header

At this point, I realize this is no longer within Stan's territory. Any ideas anyway? I understand this is due to a "misalignment" inherent in trying to use a shared object on Linux (shinyapps runs Linux) that was compiled on my Mac.

[Ben Goodrich]

On Friday, February 3, 2017 at 2:10:56 PM UTC-5, William Wolf wrote:

That worked. (And strange - I could have sworn it didn't work before).

I now have the package on GitHub and can successfully install it back to my Mac (via devtools::install_github). However, when the package tries to build on shinyapps, I get the following error:

Error in dyn.load(file, DLLpath = DLLpath, ...) : 

  unable to load shared object '/usr/local/lib/R/site-library/RescueTimeEstimator/libs/RescueTimeEstimator.so':

  /usr/local/lib/R/site-library/RescueTimeEstimator/libs/RescueTimeEstimator.so: invalid ELF header

At this point, I realize this is no longer within Stan's territory. Any ideas anyway? I understand this is due to a "misalignment" inherent in trying to use a shared object on Linux (shinyapps runs Linux) that was compiled on my Mac.

I think you need to compile it on the same Linux setup.

Ben

[William Wolf]

Alrighty. Will report back. Thanks for all the help.

[William Wolf]

Ben,

I successfully compiled the model within a Docker container (which I'm hoping is "same" enough), but I get the same errors from shinyapps. My code pushed to master, shinyapps cache cleared, etc. Any further ideas? My next move is to Dockerize the application then deploy it myself.

[Ben Goodrich]

Do you have a link to a repo? I could maybe take a look at it.

[William Wolf]

Ben,

I saw (by email) that you had replied. The link is here! Help would be supremely appreciated.

[Ben Goodrich]

I made a PR on GitHub. Let me know how it goes.

Ben

[William Wolf]

Thank you so, so much Ben. I'm kind've floored that you made this effort. So much appreciated.

Unfortunately I'm still getting the same error (I did nothing more than merge the PR and re-deploy to shinyapps). I have no easy way to ensure that the `--preclean` flag is on on their end, but I'll follow up with them on this issue tomorrow.

Just to be certain, I should pre-compile this package before pushing to GitHub - i.e. run `R CMD INSTALL -l RescueTimeEstimator/packrat/lib/x86_64-pc-linux-gnu/3.3.2/ --no-multiarch --with-keep.source RescueTimeEstimator` in terminal, thus creating `src/RescueTimeEstimator.so` and `src/Modules.o` - then include both `src/RescueTimeEstimator.so` and `src/Modules.o` in the commit, correct? Furthermore, I should ensure that I do this compilation on an architecture identical to that of the shinyapps server, i.e. a Docker container running Ubuntu 14.04, yeah?

Thanks once more.

[Ben Goodrich]

On Wednesday, February 8, 2017 at 5:50:09 PM UTC-5, William Wolf wrote:

Unfortunately I'm still getting the same error (I did nothing more than merge the PR and re-deploy to shinyapps). I have no easy way to ensure that the `--preclean` flag is on on their end, but I'll follow up with them on this issue tomorrow.

Yeah, I think you need to talk a RStudio person into letting you pass

args = "--preclean"

to devtools::install_github. Let us know if you need us to try to talk to them because we definitely want people to be using Stan on shinyapps.io.
 

Just to be certain, I should pre-compile this package before pushing to GitHub - i.e. run `R CMD INSTALL -l RescueTimeEstimator/packrat/lib/x86_64-pc-linux-gnu/3.3.2/ --no-multiarch --with-keep.source RescueTimeEstimator` in terminal, thus creating `src/RescueTimeEstimator.so` and `src/Modules.o` - then include both `src/RescueTimeEstimator.so` and `src/Modules.o` in the commit, correct? Furthermore, I should ensure that I do this compilation on an architecture identical to that of the shinyapps server, i.e. a Docker container running Ubuntu 14.04, yeah?

Maybe. Or maybe the opposite. Have your src directory on GitHub look like

src$ ls
include  Makevars  Makevars.win  Modules.cpp

without any .o or .so files. Then I should think the regular devtools::install_github would build / work from shinyapps.io. It does for me with rstanarm locally at least.

Ben

[William Wolf]

With both `src/RescueTimeEstimator.so` and `src/Modules.o` omitted from the project, I am able to successfully install and run the package on a fresh Docker container running Ubuntu 14.04 (dmccloskey/r-base image). Ubuntu 14.04 is the same operating system as that of the shinyapps.io servers.

Additionally, this works for both: 

• `install_github("cavaunpeu/RescueTimeEstimator", args = "--preclean", build_vignettes = FALSE)`
  
• `install_github("cavaunpeu/RescueTimeEstimator")
  

Unfortunately, when I deploy to shinyapps, I get the same errors as before.

• With the cache off, I get a message saying `Error: Unhandled Exception: Child Task 340213605 failed: Error building image: Error building RescueTimeEstimator (1.0)` and nothing more.
• Subsequently, with the cache then turned on, I get the same ELF header error as before.

I've attached both logs.

Crucially, with both `src/RescueTimeEstimator.so` and `src/Modules.o` now omitted from the project, the shinyapps server is tasked with two things:

1. Compiling the RStan code
2. Building the package

In my initial attempt a week ago - where I simply included the RStan code in my application project, and didn't build a separate R package as I'm doing now - the server was tasked with #1 just the same, and failed (presumably) because of a simple memory error (shinyapps unpaid plans are limited to 1GB of RAM). In fact, this is probably the simple reason the GitHub install is failing as well.

This is likely more of a shinyapps issue from this point forward. I'll link this thread on their Google Group. The relevant thread I've created over there can be found here.

[William Wolf]

Problem solved: the commit from which `devtools::install_github` pulls (when installing the R package I've built, named RescueTimeEstimator) pertains to the most recent commit that is installed on the machine from which you deploy to Shinyapps. In my case, that was the initial commit (so, while I made plenty of incremental changes and pushed them diligently to GitHub/master, none of them were being sent to the shinyapps server).

Finally, for information, the `RescueTimeEstimator.so` and `Modules.o` files should not be included on GitHub. (These are the files that are generated when the Stan code is compiled into C++ code, but before the encompassing R package itself is built).

Thanks so much for your time, feedback and pull request. Happy Monday!

[William Wolf]

I just got an email asking for more detail re: deploying a Shiny app that contains an RStan model to shinyapps.io, so I figured I'd respond here such that all can see.

In short, there are two steps:

1. Create a secondary package containing your RStan model, and push the code to GitHub
2. Install this secondary package via `devtools::install_github` into your project (i.e. your Shiny application)

Once you complete Step 2, you proceed as normal with the deployment to shinyapps.io. Here is the more detailed breakdown of how to complete Step 1:

1.  `library(rstantools);  rstan_package_skeleton(name = "MySecondaryPackageName", stan_files = "/path/to/model.stan", path = "path/to/secondary/package")` (in R console)
   
2. ` cd path/to/secondary/package` (on the command line)
3. `chmod u+x cleanup; chmod u+x cleanup.win` (on the command line)
4. manually write `useDynLib(MySecondaryPackageName, .registration = TRUE)` in the `NAMESPACE` file, as well as the methods you'd like to export, i.e. be able to use externally as  `MySecondaryPackageName::myMethod`, and the packages (dependencies) you need to import.
5. `install.packages("devtools"); library(devtools); load_all()` (in R console)
6. press `Check` in the RStudio console
7. ensure that `Module.o` and `MySecondaryPackageName.so` are not found in `src/`; these files will be built when we install this package from our GitHub via `devtools::install_github("me/MySecondaryPackageName")`. if they're already in `src/`, this step will fail.

I forget what Step 5 is supposed to do. I believe the desired outcome of Step 6 is simply "no horrible errors." I apologize for the ambiguity here: this is how I have it in my notes. I'll clarify these points the next time I undertake this process.

Here are two examples of packages containing RStan models which were then included in Shiny apps:

• OrderedCategoricalGLM
  
• RescueTimeEstimator
  

Finally, nota bene:

I'm yet unsuccessful at freshly installing RStan, <all other packages I'm using>, and MySecondaryPackageName from a packrat spec. Therein, the build of MySecondaryPackageName always fails at "no member named 'max_step_checker' in namespace 'boost::numeric::odeint'." At this point, I believe this is a problem with the way in which packrat installs RStan itself.

Alternatively, when installing MySecondaryPackageName using the RStan I have installed globally on my machine, it works. This is all to say: before thinking about installing MySecondaryPackageName, ensure that you have RStan itself installed as intended (as described in their docs), and that packrat is possibly incapable of doing this within its normal usage.

Hope this helps.