RStan installation

[Jiqiang Guo]

Currently, if we get rstan installed, we can do what stan can do in R.

That is, we would end up with a csv file for samples etc.  

I will go ahead trying to hook up the mcmc::chains object and return 

analysis and samples from c++ to R. 

I would like to bring up the installation of rstan by talking about what I did to

make it work.  That might bring up better ways to do it. 

rstan have two places that need stan:

First, stanc.  As we need to compile stanc once, I think we should just

compile it during installation (in the case of installation from source).  

(The other way is just to compile it after the package is loaded, which adds 

time to compile model -- only once but every session.)

So I have source code for stanc in R as follows:

rstan/src/stanc.hpp

rstan/src/stanc.cpp 

and Makevars files in rstan/src, which tells R how to compile them during 

installation. 

Currently in Makevars, I have 

PKG_LIBS = `$(R_HOME)/bin/Rscript -e "Rcpp:::LdFlags()"` -L$(STAN_HOME)/bin -lstanc

PKG_CPPFLAGS = -I$(STAN_HOME)/lib -I$(STAN_HOME)/src -I./include/

Note that the libstanc is 

CXX_ARGS="-O3 -Wall -fpic -pipe -g" # this is just copied from what are used in R CMD SHLIB

g++ -I src -I lib $CXX_ARGS -c src/stan/gm/grammars/var_decls_grammar_inst.cpp  -o bin/stan/gm/grammars/var_decls_grammar_inst.o

g++ -I src -I lib $CXX_ARGS -c src/stan/gm/grammars/whitespace_grammar_inst.cpp -o bin/stan/gm/grammars/whitespace_grammar_inst.o

g++ -I src -I lib $CXX_ARGS -c src/stan/gm/grammars/program_grammar_inst.cpp    -o bin/stan/gm/grammars/program_grammar_inst.o

g++ -I src -I lib $CXX_ARGS -c src/stan/gm/grammars/statement_grammar_inst.cpp  -o bin/stan/gm/grammars/statement_grammar_inst.o

g++ -I src -I lib $CXX_ARGS -c src/stan/gm/grammars/expression_grammar_inst.cpp -o bin/stan/gm/grammars/expression_grammar_inst.o

g++ -I src -I lib $CXX_ARGS -c src/stan/gm/ast_def.cpp  -o bin/stan/gm/ast_def.o

g++ -shared -o ./bin/libstanc.so src/stan/agrad/agrad.o src/stan/math/matrix.o src/stan/agrad/matrix.o \

               bin/stan/gm/grammars/statement_grammar_inst.o  bin/stan/gm/grammars/expression_grammar_inst.o bin/stan/gm/ast_def.o \

               bin/stan/gm/grammars/var_decls_grammar_inst.o \

               bin/stan/gm/grammars/whitespace_grammar_inst.o \

               bin/stan/gm/grammars/program_grammar_inst.o

So I am suggesting we have libstanc.a (or libstanc.so).  In order to make it loadable, I did

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$STAH_HOME/bin.  

Second, we need Stan for compiling models.  I need stan's head files and binary libraries (libstan). 

This step is easier.  For the time being, I am just using the STAN_HOME and assuming that

libstan is in LD_LIBRARY_PATH. 

Simply put, I need to set environment variables STAN_HOME and LD_LIBRARY_PATH

to allow the installation of rstan from source assuming that stan is built with libstan 

and libstanc in STAN_HOME/bin. 

By the way, rstan is in branch rstan.  We could try to use it now. 

--

Jiqiang 

[Bob Carpenter]

On 5/24/12 3:23 PM, Jiqiang Guo wrote:
> Currently, if we get rstan installed, we can do what stan can do in R.

That's great. Can you read the data from R
or are you relying on the main that's already there?

I put in the flag in stanc to suppress generating a main
function.

> That is, we would end up with a csv file for samples etc.
> I will go ahead trying to hook up the mcmc::chains object and return
> analysis and samples from c++ to R.

Do you mean reading the mcmc::chains object from the CSV
file? That'd be great. It's a real challenge with the
arbitrarily dimensioned arrays and the need to swap row-order
and column order.

I've been tapping Matt for auto-correlation, but I'm about
to finish the changes to the model to make writing to the
chains object easier.

> I would like to bring up the installation of rstan by talking about what I did to
> make it work. That might bring up better ways to do it.

I'd talk to Ben about this -- I know next to
nothing about R. I've inserted some opinions/guesses
below.

> rstan have two places that need stan:
>
> First, stanc. As we need to compile stanc once, I think we should just
> compile it during installation (in the case of installation from source).

Agreed.

I'm not sure how portable executables are across platforms,
but we could also distribute a pre-made version.

The other pre-req for all of this is having
the whole C++ tool chain in place. What you need is
documented in an appendix in the manual.

> (The other way is just to compile it after the package is loaded, which adds
> time to compile model -- only once but every session.)
>
> So I have source code for stanc in R as follows:
>
> rstan/src/stanc.hpp
> rstan/src/stanc.cpp

Did you put it in a namespace? If not, I'd suggest
stan::stanc if that's not too much trouble in interacting
with R.

> and Makevars files in rstan/src, which tells R how to compile them during
> installation.

Great.

> Currently in Makevars, I have

This you'll need to run by Daniel!

> PKG_LIBS = `$(R_HOME)/bin/Rscript -e "Rcpp:::LdFlags()"` -L$(STAN_HOME)/bin -lstanc
> PKG_CPPFLAGS = -I$(STAN_HOME)/lib -I$(STAN_HOME)/src -I./include/
>
> Note that the libstanc is
>
> CXX_ARGS="-O3 -Wall -fpic -pipe -g" # this is just copied from what are used in R CMD SHLIB
>
> g++ -I src -I lib $CXX_ARGS -c src/stan/gm/grammars/var_decls_grammar_inst.cpp -o
> bin/stan/gm/grammars/var_decls_grammar_inst.o
> g++ -I src -I lib $CXX_ARGS -c src/stan/gm/grammars/whitespace_grammar_inst.cpp -o
> bin/stan/gm/grammars/whitespace_grammar_inst.o
> g++ -I src -I lib $CXX_ARGS -c src/stan/gm/grammars/program_grammar_inst.cpp -o
> bin/stan/gm/grammars/program_grammar_inst.o
> g++ -I src -I lib $CXX_ARGS -c src/stan/gm/grammars/statement_grammar_inst.cpp -o
> bin/stan/gm/grammars/statement_grammar_inst.o
> g++ -I src -I lib $CXX_ARGS -c src/stan/gm/grammars/expression_grammar_inst.cpp -o
> bin/stan/gm/grammars/expression_grammar_inst.o
> g++ -I src -I lib $CXX_ARGS -c src/stan/gm/ast_def.cpp -o bin/stan/gm/ast_def.o
>
> g++ -shared -o ./bin/libstanc.so src/stan/agrad/agrad.o src/stan/math/matrix.o src/stan/agrad/matrix.o \
> bin/stan/gm/grammars/statement_grammar_inst.o bin/stan/gm/grammars/expression_grammar_inst.o
> bin/stan/gm/ast_def.o \
> bin/stan/gm/grammars/var_decls_grammar_inst.o \
> bin/stan/gm/grammars/whitespace_grammar_inst.o \
> bin/stan/gm/grammars/program_grammar_inst.o
>
> So I am suggesting we have libstanc.a (or libstanc.so).

Won't the usage in R determine whether you need a .a or .so/.dll?

> In order to make it loadable, I did
>
> LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$STAH_HOME/bin.

> Second, we need Stan for compiling models. I need stan's head files and binary libraries (libstan).

You'll also need the library headers in stan/lib.
Should we be calling those something else?

> This step is easier. For the time being, I am just using the STAN_HOME and assuming that
> libstan is in LD_LIBRARY_PATH.

The other issue is whether this environment var works cross platform?

> Simply put, I need to set environment variables STAN_HOME and LD_LIBRARY_PATH
> to allow the installation of rstan from source assuming that stan is built with libstan
> and libstanc in STAN_HOME/bin.

It would be better if there's a way we could code
that without an environment variable. In my experience,
they really confuse users (including me). Could that
path just be specified in an R command somewhere?

> By the way, rstan is in branch rstan. We could try to use it now.

I don't know how others feel, but I'm fine with you just
merging back into the main branch. There's no competing
version of rstan to contend with. The only issue is whether
we want to include whatever form it's in in the first release.

- Bob

[Daniel Lee]

On Thu, May 24, 2012 at 6:45 PM, Bob Carpenter <ca...@alias-i.com> wrote:
> On 5/24/12 3:23 PM, Jiqiang Guo wrote:
>> That is, we would end up with a csv file for samples etc.
>> I will go ahead trying to hook up the mcmc::chains object and return
>> analysis and samples from c++ to R.
>
>
> Do you mean reading the mcmc::chains object from the CSV
> file?  That'd be great.  It's a real challenge with the
> arbitrarily dimensioned arrays and the need to swap row-order
> and column order.

I wrote that portion today. There are a few public functions sitting
in stan::mcmc (in stan/mcmc/chains.hpp) that read from a file and
populate a chains object. For a sample, look at the test.

>> Currently in Makevars, I have
>
>
> This you'll need to run by Daniel!

Jiqiang, we can talk about this soon. I'm working on something now,
but we can talk about it either tomorrow or Tuesday.



Daniel

[Ben Goodrich]

On Thursday, May 24, 2012 6:45:19 PM UTC-4, Bob Carpenter wrote:

> rstan have two places that need stan:
>
> First, stanc.  As we need to compile stanc once, I think we should just
> compile it during installation (in the case of installation from source).

Agreed.

I'm not sure how portable executables are across platforms,
but we could also distribute a pre-made version.

CRAN distributes binaries built from source on their servers for Windows and Mac. So, agreed that stanc should be a (but probably the only) executable that gets built.

> So I have source code for stanc in R as follows:
>
> rstan/src/stanc.hpp
> rstan/src/stanc.cpp

Did you put it in a namespace?  If not, I'd suggest
stan::stanc if that's not too much trouble in interacting
with R.

C++ namespaces shouldn't be a problem, but Bob's advice also applies to R NAMESPACES, which should hide all functions that the useR doesn't need to call directly.

> Currently in Makevars, I have

This you'll need to run by Daniel!

> PKG_LIBS = `$(R_HOME)/bin/Rscript -e "Rcpp:::LdFlags()"` -L$(STAN_HOME)/bin -lstanc
> PKG_CPPFLAGS = -I$(STAN_HOME)/lib -I$(STAN_HOME)/src -I./include/
>
> Note that the libstanc is
>
> CXX_ARGS="-O3 -Wall -fpic -pipe -g" # this is just copied from what are used in R CMD SHLIB

Although the CRAN servers mostly have g++, they disallow g++-specific options like -Wall -march, etc.

> So I am suggesting we have libstanc.a (or libstanc.so).

Won't the usage in R determine whether you need a .a or .so/.dll?

Yes, and the file suffix gets appended automatically depending on the platform.

> This step is easier.  For the time being, I am just using the STAN_HOME and assuming that
> libstan is in LD_LIBRARY_PATH.

The other issue is whether this environment var works cross platform?

R is self-aware enough to know where loaded packages are installed. So, you can have something like this

rstanDir <- dirname(system.file(package = "rstan"))

and then drill down from there in a cross-platform way with

rstanHeaders <- file.path(rstanDir, "rstan", "include")
rstanLibstan <- file.path(rstanDir, "rstan", "libs")

> Simply put, I need to set environment variables STAN_HOME and LD_LIBRARY_PATH
> to allow the installation of rstan from source assuming that stan is built with libstan
> and libstanc in STAN_HOME/bin.

It would be better if there's a way we could code
that without an environment variable.  In my experience,
they really confuse users (including me).  Could that
path just be specified in an R command somewhere?

The above should enable rstan to work without environmental variables once it is installed. Getting the paths right during installation might take a bit of work, but Rcpp is supposed to pretty much automate that.
 

> By the way, rstan is in branch rstan.  We could try to use it now.

I don't know how others feel, but I'm fine with you just
merging back into the main branch.  There's no competing
version of rstan to contend with.  The only issue is whether
we want to include whatever form it's in in the first release.

We don't want to have too many long-term branches on Google Code because it gets confusing and we have to remember to keep them in sync. Right now, we just have master, mixed_type, and rstan but I think rstan could be merged into master and mixed_type as soon as it is minimally working.

Ben
 

[Jiqiang Guo]

Are you assuming we have all the stan source included in the rstan package? 

In that case, it would be easy.  But I am afraid that stan's source is too large so that 

it must be separated from rstan in order to put it on CRAN. 

If separated, we have got a way to figure out where stan is. 

Jiqiang 

[Jiqiang Guo]

I meant to say that

we have to figure out a way to find where stan is. 

[Ben Goodrich]

On Friday, May 25, 2012 12:34:11 AM UTC-4, Jiqiang Guo wrote:

Are you assuming we have all the stan source included in the rstan package? 

In that case, it would be easy.  But I am afraid that stan's source is too large so that 

it must be separated from rstan in order to put it on CRAN.

The big problem is boost. Eigen is small and we can strip out gtest, src/test/, src/models, etc.

We could say that boost is a system requirement. But how much of the boost/ directory does stan actually use?

Ben

[Ben Goodrich]

It appears as if the CRAN servers have boost installed because, e.g., this package

http://cran.r-project.org/web/packages/mspath/index.html

lists boost >= 1.31 as a System Requirement and their binaries build.

So, I'd say cut boost from the R package unless we are only using boost/math and boost/accumulators and a small number of other things.

Ben

[Bob Carpenter]

I don't care how we do it as long as we don't
have to support multiple versions of Boost!
We're pretty much at the limits of our manpower
testing what we already have.

With our own release, we can control the version.
What could we expect from R for Boost versions?

It's not just the CRAN servers that will need
Boost, but any client installs.

We're using lots of Boost. The problem is that
boost::math isn't standalone -- it includes lots
of other pieces of Boost. The other big thing
we're using is the Boost parsers, which bring in
every bit of Boost having to do with higher-order
functors, maps and lambdas. We're also using other
little bits and pieces like the accumulators (as
you mentioned) and template metaprogramming (for
propto calcs).

I tried a while ago to see if I could trim down which
pieces of Boost we could include, but it's pretty
spaghetti-like. Maybe someone else would have better
luck teasing out just the pieces we need. I couldn't
make much headway removing top-level packages.

- Bob

[Ben Goodrich]

On Friday, May 25, 2012 3:11:43 AM UTC-4, Bob Carpenter wrote:

With our own release, we can control the version.
What could we expect from R for Boost versions?

I don't know for a fact, but I would guess latest stable Boost. Maybe -.01 from latest stable.

It's not just the CRAN servers that will need
Boost, but any client installs.

Really? I could understand anyone who builds from source would need Boost. But does the typical Mac / Windows user who downloads the rstan binary (built on CRAN servers) need Boost to compile a model? I was operating under the assumption that they would have stanc and perhaps libstan + precompiled headers at their disposal. In which case, I would think they could parse a .stan file and compile the C++ file without linking directly to Boost (or Eigen), but maybe that is not right.

We're using lots of Boost.  

OK. Given that they have to install a C++ compiler anyway, adding Boost to the list of System Dependencies is not too much of an additional burden.

Ben

[Bob Carpenter]

On 5/25/12 1:36 PM, Ben Goodrich wrote:
> On Friday, May 25, 2012 3:11:43 AM UTC-4, Bob Carpenter wrote:
>
> With our own release, we can control the version.
> What could we expect from R for Boost versions?
>
>
> I don't know for a fact, but I would guess latest stable Boost. Maybe
> -.01 from latest stable.

That we could live with. I thought you were
mentioning Boost 1.31, which is 18 versions out of
date!

> It's not just the CRAN servers that will need
> Boost, but any client installs.
>
>
> Really?

Alas, yes. At the very least we need the headers
in order to compile. But the parts of Boost we're
using is header only.

> I could understand anyone who builds from source would need
> Boost. But does the typical Mac / Windows user who downloads the rstan
> binary (built on CRAN servers) need Boost to compile a model? I was
> operating under the assumption that they would have stanc and perhaps
> libstan + precompiled headers at their disposal.

We could use precompiled headers here, but they're almost
as big as Boost itself! I'm not sure how to precompile them.
Daniel was playing around with it to see if we could get
a speed boost.

> In which case, I would
> think they could parse a .stan file and compile the C++ file without
> linking directly to Boost (or Eigen), but maybe that is not right.
>
> We're using lots of Boost.
>
>
> OK. Given that they have to install a C++ compiler anyway, adding Boost
> to the list of System Dependencies is not too much of an additional burden.

It's only 40MB or so :-)

The problem isn't so much that any of these things is
by itself that much of a burden, it's that they interact
and the combinatorics quickly gets unmanageable (e.g., which
version of Boost and Eigen compile on which version of clang++
in 32-bit environments and that kind of thing).

- Bob