Proble installing rstah under Sage's R

[Emmanuel Charpentier]

Dear list,

I use both R as distributed y Debian and Sage, which includes its own version of R. Using sage 6.3beta6, which has R 3.1.0, I have been able to install a ton of packages.

but Rstan does not install :

> install_rstan()
Removing package from ‘/usr/local/sage-6.3beta6/local/lib/R/library’
(as ‘lib’ is unspecified)
--- Please select a CRAN mirror for use in this session ---
essai de l'URL 'http://cran.univ-paris1.fr/src/contrib/inline_0.3.13.tar.gz'
Content type 'application/x-gzip' length 14224 bytes (13 Kb)
URL ouverte
==================================================
downloaded 13 Kb

essai de l'URL 'http://cran.univ-paris1.fr/src/contrib/BH_1.54.0-2.tar.gz'
Content type 'application/x-gzip' length 6422875 bytes (6.1 Mb)
URL ouverte
==================================================
downloaded 6.1 Mb

essai de l'URL 'http://cran.univ-paris1.fr/src/contrib/RcppEigen_0.3.2.1.2.tar.gz'
Content type 'application/x-gzip' length 1254206 bytes (1.2 Mb)
URL ouverte
==================================================
downloaded 1.2 Mb

* installing *source* package ‘inline’ ...
** package ‘inline’ correctement décompressé et sommes MD5 vérifiées
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (inline)
* installing *source* package ‘BH’ ...
** package ‘BH’ correctement décompressé et sommes MD5 vérifiées
** inst
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (BH)
* installing *source* package ‘RcppEigen’ ...
** package ‘RcppEigen’ correctement décompressé et sommes MD5 vérifiées
** libs
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG   -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include"  -I../inst/include -fpic   -c RcppEigen.cpp -o RcppEigen.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG   -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include"  -I../inst/include -fpic   -c fastLm.cpp -o fastLm.o
g++ -shared -L/usr/local/sage-6.3beta6/local/lib/ -o RcppEigen.so RcppEigen.o fastLm.o -L/usr/local/sage-6.3beta6/local/lib -llapack -lcblas -L/usr/local/sage-6.3beta6/local/lib -lf77blas -latlas -lgfortran -lm -lquadmath -L/usr/local/sage-6.3beta6/local/lib/R//lib -lR
installing to /usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/libs
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (RcppEigen)

The downloaded source packages are in
    ‘/tmp/Rtmp0xOKsZ/downloaded_packages’
essai de l'URL 'http://cran.univ-paris1.fr/src/contrib/Rcpp_0.11.2.tar.gz'
Content type 'application/x-gzip' length 2004313 bytes (1.9 Mb)
URL ouverte
==================================================
downloaded 1.9 Mb

* installing *source* package ‘Rcpp’ ...
** package ‘Rcpp’ correctement décompressé et sommes MD5 vérifiées
** libs
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I../inst/include/     -fpic   -c Date.cpp -o Date.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I../inst/include/     -fpic   -c Module.cpp -o Module.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I../inst/include/     -fpic   -c Rcpp_init.cpp -o Rcpp_init.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I../inst/include/     -fpic   -c api.cpp -o api.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I../inst/include/     -fpic   -c attributes.cpp -o attributes.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I../inst/include/     -fpic   -c barrier.cpp -o barrier.o
g++ -shared -L/usr/local/sage-6.3beta6/local/lib/ -o Rcpp.so Date.o Module.o Rcpp_init.o api.o attributes.o barrier.o -L/usr/local/sage-6.3beta6/local/lib/R//lib -lR
installing to /usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/libs
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (Rcpp)

The downloaded source packages are in
    ‘/tmp/Rtmp0xOKsZ/downloaded_packages’

Attaching package: ‘Rcpp’

L'objet suivant est masqué from ‘package:inline’:

    registerPlugin

essai de l'URL 'http://rstan.org/repo/src/contrib/rstan_2.4.0.tar.gz'
Content type 'application/x-gzip' length 1399541 bytes (1.3 Mb)
URL ouverte
==================================================
downloaded 1.3 Mb

* installing *source* package ‘rstan’ ...
** package ‘rstan’ correctement décompressé et sommes MD5 vérifiées
** libs
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c stanc.cpp -o stanc.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c chains.cpp -o chains.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c misc.cpp -o misc.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c init.cpp -o init.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c gm__grammars__var_deccls_grammar_inst.cpp -o gm__grammars__var_deccls_grammar_inst.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c gm__ast_def.cpp -o gm__ast_def.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c gm__grammars__statement_grammar_inst.cpp -o gm__grammars__statement_grammar_inst.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c gm__grammars__program_grammar_inst.cpp -o gm__grammars__program_grammar_inst.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c gm__grammars__statement_2_grammar_inst.cpp -o gm__grammars__statement_2_grammar_inst.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c gm__grammars__expression_grammar_inst.cpp -o gm__grammars__expression_grammar_inst.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c gm__grammars__functions_grammar_inst.cpp -o gm__grammars__functions_grammar_inst.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c gm__grammars__expression07_grammar_inst.cpp -o gm__grammars__expression07_grammar_inst.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c gm__grammars__term_grammar_inst.cpp -o gm__grammars__term_grammar_inst.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c gm__grammars__whitespace_grammar_inst.cpp -o gm__grammars__whitespace_grammar_inst.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c gm__grammars__bare_type_grammar_inst.cpp -o gm__grammars__bare_type_grammar_inst.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c agrad__rev__var_stack.cpp -o agrad__rev__var_stack.o
ar qc libstan.a agrad__rev__var_stack.o
g++ -shared -L/usr/local/sage-6.3beta6/local/lib/ -o rstan.so stanc.o chains.o misc.o init.o gm__grammars__var_deccls_grammar_inst.o gm__ast_def.o gm__grammars__statement_grammar_inst.o gm__grammars__program_grammar_inst.o gm__grammars__statement_2_grammar_inst.o gm__grammars__expression_grammar_inst.o gm__grammars__functions_grammar_inst.o gm__grammars__expression07_grammar_inst.o gm__grammars__term_grammar_inst.o gm__grammars__whitespace_grammar_inst.o gm__grammars__bare_type_grammar_inst.o -L/usr/local/sage-6.3beta6/local/lib/R//lib -lR
installing via 'install.libs.R' to /usr/local/sage-6.3beta6/local/lib/R/library/rstan
installing rstan libs to /usr/local/sage-6.3beta6/local/lib/R/library/rstan/libs
installing libstan.a to /usr/local/sage-6.3beta6/local/lib/R/library/rstan/libstan
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
Error in dyn.load(file, DLLpath = DLLpath, ...) :
  impossible de charger l'objet partagé '/usr/local/sage-6.3beta6/local/lib/R/library/rstan/libs/rstan.so':
  /usr/local/sage-6.3beta6/local/lib/R/library/rstan/libs/rstan.so: undefined symbol: _ZN4stan5agrad18var_nochain_stack_E
Erreur : le chargement a échoué
Exécution arrêtée
ERROR: loading failed
* removing ‘/usr/local/sage-6.3beta6/local/lib/R/library/rstan’

The downloaded source packages are in
    ‘/tmp/Rtmp0xOKsZ/downloaded_packages’
Erreur dans library(rstan) : there is no package called ‘rstan’
De plus : Message d'avis :
In install.packages("rstan", type = "source") :
  installation of package ‘rstan’ had non-zero exit status

Any hint ? I note that rstan seems to need its very own versin of Rcpp and RcppEigen. Could this be the cause of the problems ?

HTH,

--
Emmanuel Charpentier

[Bob Carpenter]

What's your g++ version?

Can you get the latest version of Rcpp to install
and run a simple example? (Is the simple example still
in our install instructions?)

I don't know what you mean by "very own version of Rcpp".
Is RStan somehow unusual in how it depends on other R
packages?

- Bob

> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Ben Goodrich]

On Wednesday, July 23, 2014 4:12:03 PM UTC-4, Emmanuel Charpentier wrote:

** testing if installed package can be loaded
Error in dyn.load(file, DLLpath = DLLpath, ...) :
  impossible de charger l'objet partagé '/usr/local/sage-6.3beta6/local/lib/R/library/rstan/libs/rstan.so':
  /usr/local/sage-6.3beta6/local/lib/R/library/rstan/libs/rstan.so: undefined symbol: _ZN4stan5agrad18var_nochain_stack_E
Erreur : le chargement a échoué
Exécution arrêtée
ERROR: loading failed
* removing ‘/usr/local/sage-6.3beta6/local/lib/R/library/rstan’

The downloaded source packages are in
    ‘/tmp/Rtmp0xOKsZ/downloaded_packages’
Erreur dans library(rstan) : there is no package called ‘rstan’
De plus : Message d'avis :
In install.packages("rstan", type = "source") :
  installation of package ‘rstan’ had non-zero exit status

Any hint ? I note that rstan seems to need its very own versin of Rcpp and RcppEigen. Could this be the cause of the problems ?

They are unmodified Rcpp and RcppEigen from CRAN. My guess is that there is something going on with Sage that causes it to not link correctly. I'll try to figure it out. Can you install rstan outside of R via Sage? And you are using the g++ that comes with Sage or the system one (and if so what version)?

Ben

[Ben Goodrich]

On Wednesday, July 23, 2014 4:24:36 PM UTC-4, Ben Goodrich wrote:

They are unmodified Rcpp and RcppEigen from CRAN. My guess is that there is something going on with Sage that causes it to not link correctly. I'll try to figure it out. Can you install rstan outside of R via Sage? And you are using the g++ that comes with Sage or the system one (and if so what version)?

I meant "Can you install rstan outside of Sage via R?"

Ben

[Emmanuel Charpentier]

Le mercredi 23 juillet 2014 22:18:22 UTC+2, Bob Carpenter a écrit :

What's your g++ version?

System :
charpent@asus16-ec:~$ g++ --version
g++ (Debian 4.9.1-1) 4.9.1

Under Sage's R :
system("g++ --version")
g++ (Debian 4.9.1-1) 4.9.1

Same difference...

Can you get the latest version of Rcpp to install
and run a simple example?  (Is the simple example still
in our install instructions?)

I ran "update.packages(ask=FALSE, checkBuilt=TRUE)" just before trying to install Sage 2.4.0.

The Rcpp update was uneventful. *After* my failed install_rstan() attempt, the same update attempt yelded nothing to update.

[ BTW : I wonder about the necessity for the install_rstan script to install an already installed.package, if the latest available package on CRAN is sufficent... ]
 

I don't know what you mean by "very own version of Rcpp".
Is RStan somehow unusual in how it depends on other R
packages?

That's what your install script seems to believe. Why not make it depend on sufficienciently recent version of CRAN packages instead ?
 

[Emmanuel Charpentier]

Answers below.

What do you mean ? Rstan installed uneventfully in system's R (i. e. as packaged by Debian for testing/jessie).
 

And you are using the g++ that comes with Sage or the system one (and if so what version)?

Sage uses system's R (currently, it does not try to force it's own compilers if the system ones are sufficiently recent).

HTH,

--
Emmanuel Charpentier

Ben

[Ben Goodrich]

On Wednesday, July 23, 2014 4:37:35 PM UTC-4, Emmanuel Charpentier wrote:

[ BTW : I wonder about the necessity for the install_rstan script to install an already installed.package, if the latest available package on CRAN is sufficent... ]

We had some problems attributable to the binary version of Rcpp being compiled on CRAN with a different compiler than that which is being used to build rstan. So, we need to install Rcpp from source with the same compiler, rather than merely having it be installed.

I'll build the sage beta tonight and see if I can figure out why it is acting differently.

Ben

[Emmanuel Charpentier]

Additional information : the same problem, exacly arises on another (much smaller) laptop with Ubuntu 14.04 which has g++ 4.8.2 coming from Ubuntu and a locally-compiled sage 6.3beta6.

HTH,

--
Emmanuel Charpentier

[Emmanuel Charpentier]

Le mercredi 23 juillet 2014 22:54:36 UTC+2, Ben Goodrich a écrit :

On Wednesday, July 23, 2014 4:37:35 PM UTC-4, Emmanuel Charpentier wrote:

[ BTW : I wonder about the necessity for the install_rstan script to install an already installed.package, if the latest available package on CRAN is sufficent... ]

We had some problems attributable to the binary version of Rcpp being compiled on CRAN with a different compiler than that which is being used to build rstan.

That's under Windows, right ? That's the only platform where there is an incentive to use precompiled R packages rather than source packages and local compilation.

In that case, you have other sources of problems to be worried with... ;-{]=
 

So, we need to install Rcpp from source with the same compiler, rather than merely having it be installed.

I'll build the sage beta tonight and see if I can figure out why it is acting differently.

Word to the wise : do NOT use 'SAGE_FAT_BINARY="YES"', which is a nice shortcut to avoid the Atlas optimization, but will give you a very problematic Sage's R (Rscript is nonfunctional : see trac#16694 on trac.sagemath.org).

[Ben Goodrich]

On Wednesday, July 23, 2014 4:56:25 PM UTC-4, Emmanuel Charpentier wrote:

Additional information : the same problem, exacly arises on another (much smaller) laptop with Ubuntu 14.04 which has g++ 4.8.2 coming from Ubuntu and a locally-compiled sage 6.3beta6.

I was able to replicate this installation problem under Sage. But since it works with Debian's R and with r-devel compiled from source, I'm inclined to think it is yet another Sage bug with R. But I will keep investigating.

The only thing I have noticed so far is that outside of Sage, I get this

g++ -I/opt/r-devel/include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256 -I/usr/local/include -I"/opt/r-devel/library/Rcpp/include" -I"/opt/r-devel/library/RcppEigen/include" -I"/opt/r-devel/library/BH/include"   -fpic  -g -O2  -c gm__ast_def.cpp -o gm__ast_def.o
In file included from gm__ast_def.cpp:1:0:
../inst/include/stansrc/stan/gm/ast_def.cpp: In constructor ‘stan::gm::function_signatures::function_signatures()’:
../inst/include/stansrc/stan/gm/ast_def.cpp:378:5: note: variable tracking size limit exceeded with -fvar-tracking-assignments, retrying without
     function_signatures::function_signatures() {
     ^

But under Sage, I do not get the "variable tracking size limit exceeded with -fvar-tracking-assignments, retrying without".

Also, does it build for you under Sage with clang++? It does for me. Just put

CC=clang
CXX=clang++

into a file at ~/.R/Makevars and then in R under Sage do

install.packages("rstan", type = 'source')

Ben

[Jiqiang Guo]

It looks like this error is the one built with cpp optimization level 0 with g++, see here
https://travis-ci.org/stan-dev/rstan/builds/30479408

Jiqiang

[Ben Goodrich]

On Wednesday, July 23, 2014 11:27:26 PM UTC-4, Jiqiang Guo wrote:

It looks like this error is the one built with cpp optimization level 0 with g++, see here
https://travis-ci.org/stan-dev/rstan/builds/30479408

Good catch! Indeed, Sage is for some reason building R without any optimization when it has -O2 by default. It is a bit disturbing that we can't build rstan without optimization (which would be useful for debugging at some point), but it is more disturbing that Sage would turn off optimization by default when building R.

Ben

[Emmanuel Charpentier]

Thank you all for these data. A quick look at the Sage build tree didn't reveal a possible cause. I'll try to rebuild sage's R with -O2 forced and report theresults, but that won't be before tonight (mid-afternoon for you...).

--
Emmanuel Charpentier

[Emmanuel Charpentier]

Three more data points : I upgraded Sage's R (uneventfully, see attached log) to 3.1.1, reinstalled/upgraded the 251 packages of my current working set.

1) R was compiled with "-O2" without any intervention from me.

2) some packages, such as nloptr, did not install when I tried  Sys.setenv(MAKEFLAGS = "-j4"). They did install uneventfully after restarting a new session without this call...

3) R gives the same problem as before : compiles, but gives a nonfunctional package (see log below).

HTH,

--
Emmanuel Charpentier

> install_rstan()
Removing package from ‘/usr/local/sage-6.3beta6/local/lib/R/library’
(as ‘lib’ is unspecified)

    ‘/tmp/RtmpWW4dN8/downloaded_packages’

    ‘/tmp/RtmpWW4dN8/downloaded_packages’

Attaching package: ‘Rcpp’

The following object is masked from ‘package:inline’:

    registerPlugin

essai de l'URL 'http://rstan.org/repo/src/contrib/rstan_2.4.0.tar.gz'
Content type 'application/x-gzip' length 1399541 bytes (1.3 Mb)
URL ouverte
==================================================
downloaded 1.3 Mb

* installing *source* package ‘rstan’ ...
** package ‘rstan’ correctement décompressé et sommes MD5 vérifiées
** libs
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c stanc.cpp -o stanc.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c chains.cpp -o chains.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c misc.cpp -o misc.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c init.cpp -o init.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c gm__grammars__var_deccls_grammar_inst.cpp -o gm__grammars__var_deccls_grammar_inst.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c gm__ast_def.cpp -o gm__ast_def.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c gm__grammars__statement_grammar_inst.cpp -o gm__grammars__statement_grammar_inst.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c gm__grammars__program_grammar_inst.cpp -o gm__grammars__program_grammar_inst.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c gm__grammars__statement_2_grammar_inst.cpp -o gm__grammars__statement_2_grammar_inst.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c gm__grammars__expression_grammar_inst.cpp -o gm__grammars__expression_grammar_inst.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c gm__grammars__functions_grammar_inst.cpp -o gm__grammars__functions_grammar_inst.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c gm__grammars__expression07_grammar_inst.cpp -o gm__grammars__expression07_grammar_inst.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c gm__grammars__term_grammar_inst.cpp -o gm__grammars__term_grammar_inst.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c gm__grammars__whitespace_grammar_inst.cpp -o gm__grammars__whitespace_grammar_inst.o
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c gm__grammars__bare_type_grammar_inst.cpp -o gm__grammars__bare_type_grammar_inst.o

g++ -shared -L/usr/local/sage-6.3beta6/local/lib/ -o rstan.so stanc.o chains.o misc.o init.o gm__grammars__var_deccls_grammar_inst.o gm__ast_def.o gm__grammars__statement_grammar_inst.o gm__grammars__program_grammar_inst.o gm__grammars__statement_2_grammar_inst.o gm__grammars__expression_grammar_inst.o gm__grammars__functions_grammar_inst.o gm__grammars__expression07_grammar_inst.o gm__grammars__term_grammar_inst.o gm__grammars__whitespace_grammar_inst.o gm__grammars__bare_type_grammar_inst.o -L/usr/local/sage-6.3beta6/local/lib/R//lib -lR
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c agrad__rev__var_stack.cpp -o agrad__rev__var_stack.o
ar qc libstan.a agrad__rev__var_stack.o
installing via 'install.libs.R' to /usr/local/sage-6.3beta6/local/lib/R/library/rstan
installing rstan libs to /usr/local/sage-6.3beta6/local/lib/R/library/rstan/libs
installing libstan.a to /usr/local/sage-6.3beta6/local/lib/R/library/rstan/libstan
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
Error in dyn.load(file, DLLpath = DLLpath, ...) :
  impossible de charger l'objet partagé '/usr/local/sage-6.3beta6/local/lib/R/library/rstan/libs/rstan.so':
  /usr/local/sage-6.3beta6/local/lib/R/library/rstan/libs/rstan.so: undefined symbol: _ZN4stan5agrad18var_nochain_stack_E
Erreur : le chargement a échoué
Exécution arrêtée
ERROR: loading failed
* removing ‘/usr/local/sage-6.3beta6/local/lib/R/library/rstan’

The downloaded source packages are in

    ‘/tmp/RtmpWW4dN8/downloaded_packages’

[Ben Goodrich]

On Thursday, July 24, 2014 3:12:54 PM UTC-4, Emmanuel Charpentier wrote:

essai de l'URL 'http://rstan.org/repo/src/contrib/rstan_2.4.0.tar.gz'

Content type 'application/x-gzip' length 1399541 bytes (1.3 Mb)
URL ouverte
==================================================
downloaded 1.3 Mb

* installing *source* package ‘rstan’ ...
** package ‘rstan’ correctement décompressé et sommes MD5 vérifiées
** libs
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c stanc.cpp -o stanc.o

This should have a -O2 somewhere as in

g++ -I/opt/r-devel/include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256 -I/usr/local/include -I"/opt/r-devel/library/Rcpp/include" -I"/opt/r-devel/library/RcppEigen/include" -I"/opt/r-devel/library/BH/include"   -fpic  -g -O2  -c stanc.cpp -o stanc.o

In the absence of that, it is implicitly -O0. Is there a Sage ticket for this issue?

Ben
 

[Emmanuel Charpentier]

Le jeudi 24 juillet 2014 21:37:31 UTC+2, Ben Goodrich a écrit :

On Thursday, July 24, 2014 3:12:54 PM UTC-4, Emmanuel Charpentier wrote:

essai de l'URL 'http://rstan.org/repo/src/contrib/rstan_2.4.0.tar.gz'

Content type 'application/x-gzip' length 1399541 bytes (1.3 Mb)
URL ouverte
==================================================
downloaded 1.3 Mb

* installing *source* package ‘rstan’ ...
** package ‘rstan’ correctement décompressé et sommes MD5 vérifiées
** libs
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c stanc.cpp -o stanc.o

This should have a -O2 somewhere as in

g++ -I/opt/r-devel/include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256 -I/usr/local/include -I"/opt/r-devel/library/Rcpp/include" -I"/opt/r-devel/library/RcppEigen/include" -I"/opt/r-devel/library/BH/include"   -fpic  -g -O2  -c stanc.cpp -o stanc.o

In the absence of that, it is implicitly -O0. Is there a Sage ticket for this issue?

Not yet. I would rather diagnose this before yelling "wolf !".

Do you see how to pass this flag to GCC through R ? E. g. via Sys.setenv() ? or via Makevars ? This could allow to pinpoint the problem.

[Ben Goodrich]

On Thursday, July 24, 2014 5:15:17 PM UTC-4, Emmanuel Charpentier wrote:

Le jeudi 24 juillet 2014 21:37:31 UTC+2, Ben Goodrich a écrit :

On Thursday, July 24, 2014 3:12:54 PM UTC-4, Emmanuel Charpentier wrote:

essai de l'URL 'http://rstan.org/repo/src/contrib/rstan_2.4.0.tar.gz'

Content type 'application/x-gzip' length 1399541 bytes (1.3 Mb)
URL ouverte
==================================================
downloaded 1.3 Mb

* installing *source* package ‘rstan’ ...
** package ‘rstan’ correctement décompressé et sommes MD5 vérifiées
** libs
g++ -I/usr/local/sage-6.3beta6/local/lib/R//include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256  -I"/usr/local/sage-6.3beta6/local/lib/R/library/Rcpp/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/RcppEigen/include" -I"/usr/local/sage-6.3beta6/local/lib/R/library/BH/include"   -fpic   -c stanc.cpp -o stanc.o

This should have a -O2 somewhere as in

g++ -I/opt/r-devel/include -DNDEBUG -I"../inst/include/stansrc"   -I"../inst/include"  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -ftemplate-depth-256 -I/usr/local/include -I"/opt/r-devel/library/Rcpp/include" -I"/opt/r-devel/library/RcppEigen/include" -I"/opt/r-devel/library/BH/include"   -fpic  -g -O2  -c stanc.cpp -o stanc.o

In the absence of that, it is implicitly -O0. Is there a Sage ticket for this issue?

Not yet. I would rather diagnose this before yelling "wolf !".

Do you see how to pass this flag to GCC through R ? E. g. via Sys.setenv() ? or via Makevars ? This could allow to pinpoint the problem.

You can override it by putting

CXXFLAGS = -O2 # or -03

into ~/.R/Makevars and then execute

  options(repos = c(getOption("repos"), 
          rstan = "http://rstan.org/repo/"))

  install.packages("rstan", type = 'source')

but, assuming that is successful, the wolf-like question would remain why R under Sage is configured differently than the R defaults?

Ben

[Emmanuel Charpentier]

Turns out tha the olf might have been your dog. I found this in .R/Makevars :

# created by rstan at Sun Jul  6 16:57:16 2014
CXXFLAGS =  $(LTO) -O3  #set_by_rstan
R_XTRA_CPPFLAGS =  -I$(R_INCLUDE_DIR)   #set_by_rstan

After correction to :

## Original state, as found on 2014-07-25
## # created by rstan at Sun Jul  6 16:57:16 2014
## CXXFLAGS =  $(LTO) -O3  #set_by_rstan
## R_XTRA_CPPFLAGS =  -I$(R_INCLUDE_DIR)   #set_by_rstan
CXXFLAGS =  $(LTO) -O2  #set_by_rstan
R_XTRA_CPPFLAGS =  -I$(R_INCLUDE_DIR)   #set_by_rstan

stan install like a charm under sage's R :

> install.packages("rstan", repos="http://rstan.org/repo")

[ ... ]

installing rstan libs to /usr/local/sage-6.3beta6/local/lib/R/library/rstan/libs
installing libstan.a to /usr/local/sage-6.3beta6/local/lib/R/library/rstan/libstan
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded

* DONE (rstan)

And the resulting library compiles and fits the 8 schools example.

It seems that my problem (and problems with installing other libraries after enabling parallel compilation) might have been caused by this (undocumented) leftover of the installation of Rstan 3.2 -O3 is a bit *too* optimizing for complex tasks...

BTW : could you explicitely state this in the rstan install document, make rstan depend on CRAN's Rcpp, inline and BH, use a simpler install command (see above) AND GET RID OF THE DAMN INSTALL SCRIPT ?

Just my two cents... What do you think ?

HTH,

--

Emmanuel Charpentier

Le mercredi 23 juillet 2014 22:12:03 UTC+2, Emmanuel Charpentier a écrit :

[ Bandwidth savings ... ]

[Bob Carpenter]

Thanks for bearing with this and for the feedback.
More below.

On Jul 25, 2014, at 2:34 AM, Emmanuel Charpentier <emanuel.c...@gmail.com> wrote:

> Turns out tha the olf might have been your dog.

olf?

I take it that's RStan 2.3. There's a 2.4 release linked from the
install instructions right now.

Can you compile models using -O3? That's the important part.

>
> BTW : could you explicitely state this in the rstan install document, make rstan depend on CRAN's Rcpp, inline and BH,

RStan 2.3 didn't depend on BH, but I believe 2.4 (or the release
after) will depend on BH and RcppEigen (although I hate the dependencies,
it seems the only way to get it into CRAN, which should hopefully simplify
some of the installs).

> use a simpler install command (see above) AND GET RID OF THE DAMN INSTALL SCRIPT ?
>
> Just my two cents... What do you think ?

For one, Andrew would kill us if we got rid of the install script.
That's because it's cleaned up most of the install procedures for
"standard" installations, which are very fiddly and error prone.

We figured people running non-standard installs would be able to
deal with a little more customization on their own.

Having said that, I have no idea what Sage is (other than what
I see on their home page: http://www.sagemath.org). Maybe we just need to
build Stan into Sage so nobody has to install it manually. Isn't that
the point of these mega-integrations?

Having said that, we can add special instructions for special versions
of R.

- Bob

[Ben Goodrich]

On Friday, July 25, 2014 2:34:46 AM UTC-4, Emmanuel Charpentier wrote:

It seems that my problem (and problems with installing other libraries after enabling parallel compilation) might have been caused by this (undocumented) leftover of the installation of Rstan 3.2 -O3 is a bit *too* optimizing for complex tasks...

-O3 may be too extreme for some other R packages, but I don't think that is the problem with the rstan installation. If R from Sage were picking up the -O3 flag, then there would have been a -O3 in each call to g++ while it was compiling rstan. There was no -O at all, which is equivalent to -O0, which does not work for rstan with g++ for some unknown reason, like Mav mentioned when he linked to

https://travis-ci.org/stan-dev/rstan/builds/30479408

The install_rstan() function explicitly ignores ~/.R/Makevars because people (mostly Windows people) mess that file up by including -g in the CXXFLAGS, in which case mingw is unable to compile it. By ignoring ~/.R/Makevars, install.packages() reverts to the CXXFLAGS that were used to compile R, which is somehow empty when run from the Sage beta (after I mv ~/.R/Makevars out of the way)

> system(paste0(Sys.getenv('R_HOME'), "/bin/R", " CMD config CXXFLAGS"))

BTW : could you explicitely state this in the rstan install document

yes
 

make rstan depend on CRAN's Rcpp, inline and BH,

it does (for rstan 2.4.0)

> source("http://mc-stan.org/rstan/install.R", max.deparse.length = 1500, echo = TRUE)

> install_rstan <- function() {
+   on.exit(Sys.unsetenv("R_MAKEVARS_USER"))
+   on.exit(Sys.unsetenv("R_MAKEVARS_SITE"), add = TRUE)
+
+   try(remove.packages("rstan"), silent = TRUE)
+   Sys.setenv(R_MAKEVARS_USER = "foobar")
+   Sys.setenv(R_MAKEVARS_SITE = "foobar")
+   install.packages(c("inline", "BH", "RcppEigen"))
+   install.packages("Rcpp", type = "source")
+   library(inline)
+   library(Rcpp)
+   src <- '
+     std::vector<std::string> s;
+     s.push_back("hello");
+     s.push_back("world");
+     return Rcpp::wrap(s);
+   '
+   hellofun <- cxxfunction(body = src, includes = '', plugin = 'Rcpp', verbose = FALSE)
+   test <- try(hellofun())
+   if(inherits(test, "try-error")) stop("hello world failed; ask for help on Rcpp list")
+
+   options(repos = c(getOption("repos"),
+           rstan = "http://rstan.org/repo/"))
+   install.packages("rstan", type = 'source')
+   library(rstan)
+   set_cppo("fast")
+   if (any(grepl("^darwin", R.version$os, ignore.case = TRUE))) {
+     cat('\nCC=clang', 'CXX=clang++ -arch x86_64 -ftemplate-depth-256',
+         file = "~/.R/Makevars", sep = "\n", append = TRUE)
+   }
+   return(invisible(NULL))
+ }

 

use a simpler install command (see above) AND GET RID OF THE DAMN INSTALL SCRIPT ?

Just my two cents... What do you think ?

I think I have been advocating getting rstan onto CRAN for more than two years

https://groups.google.com/d/msg/stan-dev/wIpu_otqK10/H5OZnHOX5PQJ

and am the only Stan developer that is strongly in favor of getting rstan onto CRAN, so you are shouting at the wrong person. BH 1.54.0-x was the first version of the BH package that would build rstan and rstan 1.0.1 was rejected from CRAN on September 15, 2012 largely for bundling the necessary parts of Boost. Since then, we have dealt with one installation problem after another on Windows and Mac OSX (and occasionally Linux) that has culminated in the script you are keen to get rid of, which has reduced installation problems of rstan 2.3.0 and rstan 2.4.0 to a trickle. For rstan 2.4.0, we switched to depend on the R packages BH and RcppEigen instead of bundling Boost and Eigen to see if there were any problems, so that we could try again to get rstan onto CRAN for the next rstan version. That seems to be going fine and rstan 2.4.0 builds on the CRAN server (because the CRAN server uses the default CXXFLAGS), but there are a few more things we would have to do to make rstan fully CRAN-compliant.

Ben

[Emmanuel Charpentier]

Dear Bob,

Answers below.

Le vendredi 25 juillet 2014 09:08:08 UTC+2, Bob Carpenter a écrit :

Thanks for bearing with this and for the feedback.
More below.

On Jul 25, 2014, at 2:34 AM, Emmanuel Charpentier <emanuel.c...@gmail.com> wrote:

> Turns out tha the olf might have been your dog.

olf?

Typo for "wolf" (my typing worth zilch fefore the second cup of coffee (oand not much afterwards...). See posts above to get the point.

No. The point is that RStan 2.3 (?) left these droppings after installation.

AN INSTALL SCRIPT SHOULD NOT MAKE PERMANENT AND UNDOCUMENTED  CHANGES TO THE USER'S SYSTEM, dammit !

Can you compile models using -O3?  That's the important part.

As told above, rstan as compiled (with -O2 in .R/Makevars) compiles and fits 8-schools.

>
> BTW : could you explicitely state this in the rstan install document, make rstan depend on CRAN's Rcpp, inline and BH,

RStan 2.3 didn't depend on BH, but I believe 2.4 (or the release
after) will depend on BH and RcppEigen (although I hate the dependencies,
it seems the only way to get it into CRAN, which should hopefully simplify
some of the installs).

In this case, I second CRAN's attitude : this standard greatly simplifies the life of people needing to use R for other purposes than Stan's frontend.

A practicing biostatiscian  has to solves problems (legitimate or not) not yet in Stan's reach. Examples : propensity score matching for observational cohorts comparisons (I hear that J. Hill is on this case, but I found nothing practically useful a few weeks ago), producing p-values to appease an (illiterate) reviewer's  fear (illegitiate but frequent).

To solve these problems, an armada of tools exists, whose use is the bread-and-butter of biostatistics. R is the "greatest common denominator" of these tools. CRAN's standard ensure ome form of consistency between these tools, thus allowing their use in the same installation : one does not have one installation for descriptive statistics, one installation for nonlinear (frequentist) regression, one for Bayesian modeling, etc...

> use a simpler install command (see above) AND GET RID OF THE DAMN INSTALL SCRIPT ?
>
> Just my two cents... What do you think ?

For one, Andrew would kill us if we got rid of the install script.

You might shoot back ;-)
 

That's because it's cleaned up most of the install procedures for
"standard" installations, which are very fiddly and error prone.  

He's right : a standardized procedure is important. But anything that needs to be done to install Stan properly should be done *from the package* and should use only the tools CRAN standardized on (modulo possible dependencies on other CRAN packages for specific tasks).

If a needed tool is missing from CRAN collection,, the only logical thing to do is ... to package it for the CRAN collection 8--)

We figured people running non-standard installs would be able to
deal with a little more customization on their own.

Having said that, I have no idea what Sage is (other than what
I see on their home page: http://www.sagemath.org).

You've read their blurb. Itn fact is a great heap of mathematically-oriented software (with an accent on algebra and algebra-related subjects), but with very useful links to various software such as symbolic computation (Maxima, but also sympy and others, Mathematica if available), numerical computation (octave or Matlab if available), statistics (R ;-), graphics, etc ... and a way to use them from a common "place". It is an extremely enticing set of tools when you happen to have the use of at least two of these tools (in my case, R and Maxima+/-sumpy/scipy +/- octave) : Sage allows some interoperability between these tools (for some values of "some":-) ad a way to interact easily with \LaTeX (much more important than it seems ! I consider siome form of "literate programming" a basic method).
 

 Maybe we just need to
build Stan into Sage so nobody has to install it manually.

R has part of Sage for several years. Since most of the Sage "active team" is mor pure math otriented, R versions tend to be lag, sometimes quite a bit. So it depends on lesser minds with time for more mundane tasks to get updated versions of R on a regular basis. That's what yours truly is doing since about 1 year...

 Isn't that
the point of these mega-integrations?

It's a bit more (and a bit less) than that.
 

Having said that, we can add special instructions for special versions
of R.

I'd rather have a strong standard package that does not depend on special care and feeding to get installed.

HTH,

--
Emmanuel Charpentier

- Bob

[Emmanuel Charpentier]

Dear Ben,

Just saw your answer. I might have an answer, but I'll ave to check a couple things first, and that won't be before tonight.

Just a couple of comments :

Le vendredi 25 juillet 2014 16:12:00 UTC+2, Ben Goodrich a écrit :

[ ... ]

> system(paste0(Sys.getenv('R_HOME'), "/bin/R", " CMD config CXXFLAGS"))

*This* is the right way the install script (if any) should do intallation, not permanenlly dropping an undocumented configuration file !
 

BTW : could you explicitely state this in the rstan install document

yes
 

make rstan depend on CRAN's Rcpp, inline and BH,

it does (for rstan 2.4.0)

[ ... ]
 

+   install.packages(c("inline", "BH", "RcppEigen"))
+   install.packages("Rcpp", type = "source")

No : here, you unconditionally install whatever Rcpp and BH version happens to be available on CRAN at install time. You should install use the "Depends" field of the DESCRIPTION file, which would install the packages *if necessary* (even with version constraints if necessary).

use a simpler install command (see above) AND GET RID OF THE DAMN INSTALL SCRIPT ?

Just my two cents... What do you think ?

I think I have been advocating getting rstan onto CRAN for more than two years

Didn't know that. You have my support (for reasons explained in another post).

https://groups.google.com/d/msg/stan-dev/wIpu_otqK10/H5OZnHOX5PQJ

and am the only Stan developer that is strongly in favor of getting rstan onto CRAN, so you are shouting at the wrong person.

I'm not shouting at anybody (if so understood, please accept my apologies) !). I'm shouting at undocumented permanent modifications of my everyday system. THis cannot be accepted, since there exists a way (that you demonstrated) to achieve the same result without touching any other file.

And it should be done from the damn package, not from something else you have to download from God-if-any-knows-where with no authentication...

BH 1.54.0-x was the first version of the BH package that would build rstan and rstan 1.0.1 was rejected from CRAN on September 15, 2012 largely for bundling the necessary parts of Boost. Since then, we have dealt with one installation problem after another on Windows and Mac OSX (and occasionally Linux) that has culminated in the script you are keen to get rid of, which has reduced installation problems of rstan 2.3.0 and rstan 2.4.0 to a trickle. For rstan 2.4.0, we switched to depend on the R packages BH and RcppEigen instead of bundling Boost and Eigen to see if there were any problems, so that we could try again to get rstan onto CRAN for the next rstan version. That seems to be going fine and rstan 2.4.0 builds on the CRAN server (because the CRAN server uses the default CXXFLAGS), but there are a few more things we would have to do to make rstan fully CRAN-compliant.

I think that this (CRAN compliance) is a worthy goal, susceptible to boost (pun not intended) the availability of Stan to mere biostatiscicians mortals.

HTH,

--
Emmanuel Charpentier

[Ben Goodrich]

On Friday, July 25, 2014 11:20:02 AM UTC-4, Emmanuel Charpentier wrote:

Dear Ben,

Just saw your answer. I might have an answer, but I'll ave to check a couple things first, and that won't be before tonight.

Just a couple of comments :

Le vendredi 25 juillet 2014 16:12:00 UTC+2, Ben Goodrich a écrit :

[ ... ]

> system(paste0(Sys.getenv('R_HOME'), "/bin/R", " CMD config CXXFLAGS"))

*This* is the right way the install script (if any) should do intallation, not permanenlly dropping an undocumented configuration file !

We don't permanently drop anything. We ignore ~/.R/Makevars by instead searching for foobar inside the install_rstan() function and then read from it thereafter

  ...
  on.exit(Sys.unsetenv("R_MAKEVARS_USER"))

  on.exit(Sys.unsetenv("R_MAKEVARS_SITE"), add = TRUE)

  try(remove.packages("rstan"), silent = TRUE)
  Sys.setenv(R_MAKEVARS_USER = "foobar")
  Sys.setenv(R_MAKEVARS_SITE = "foobar")
  ...

The rstan package should install with R's CXXFLAGS. It is a very minor rstan bug that it currently does not install if R's CXXFLAGS are empty, in which case g++ compiles at -O0. It is a less minor bug in Sage that it builds R with empty CXXFLAGS. You can work around both bugs by filling out ~/.R/Makevars however you like (excluding CXXFLAGS=-O0 if CXX=g++) and installing rstan without the install_rstan() function.

I think you are objecting to the set_cppo() function, which is called by install_rstan() after rstan is installed and modifies ~/.R/Makevars so that it is correctly formatted and by default uses -O3 because that makes Stan programs run slightly faster than -O2 and much faster than any other optimization level. Setting -O3 probably breaks some other R packages, which I would be sympathetic to if that happened on your system, but it changes a  non-executable file, is documented and also prints a message to the console when it is called.
 

[ ... ]
 

+   install.packages(c("inline", "BH", "RcppEigen"))
+   install.packages("Rcpp", type = "source")

No : here, you unconditionally install whatever Rcpp and BH version happens to be available on CRAN at install time. You should install use the "Depends" field of the DESCRIPTION file, which would install the packages *if necessary* (even with version constraints if necessary).

No:

http://cran.r-project.org/doc/manuals/R-exts.html#Package-Dependencies

"Field ‘Depends’ should nowadays be used rarely, only for packages which are intended to be put on the search path to make their facilities available to the end user (and not to the package itself)"

and later on in that same section

A package that wishes to make use of header files in other packages needs to declare them as a comma-separated list in the field ‘LinkingTo’ in the DESCRIPTION file. For example

LinkingTo: link1, link2

As from R 3.0.2 the ‘LinkingTo’ field can have a version requirement which is checked at installation. (In earlier versions of R it would cause the specification to be ignored.)

Specifying a package in ‘LinkingTo’ suffices if these are C++ headers containing source code or static linking is done at installation: the packages do not need to be (and usually should not be) listed in the ‘Depends’ or ‘Imports’ fields. This includes CRAN packages BH and almost all users of RcppArmadillo and RcppEigen.

which is exactly what the DESCRIPTION file in rstan does

https://raw.githubusercontent.com/stan-dev/rstan/develop/rstan/rstan/DESCRIPTION

Depends: R (>= 3.0.2), Rcpp (>= 0.9.10), utils, inline, methods
LinkingTo: Rcpp, RcppEigen, BH (>= 1.54)
Suggests: RUnit, RcppEigen, BH (>= 1.54), parallel

with the added Suggests line to work around the undocumented fact that the rstan vignette would not build without it.

Ben

[Bob Carpenter]

On Jul 25, 2014, at 10:55 AM, Emmanuel Charpentier <emanuel.c...@gmail.com> wrote:

> AN INSTALL SCRIPT SHOULD NOT MAKE PERMANENT AND UNDOCUMENTED CHANGES TO THE USER'S SYSTEM, dammit !

Agreed. See Ben's reply. No need to shout, though.

If you want to shout, please direct it at the other R package
writers who modify these things and caused the problems for us
in the first place. And at the people who configure R itself
for Windows (the installer doesn't match their recommendations
on paths, for example).

...

> In this case, I second CRAN's attitude : this standard greatly simplifies the life of people needing to use R for other purposes than Stan's frontend.

First, see Ben's response again.

We haven't been able to finagle our package onto CRAN (yet) due
to their rather severe limits on size, compile times, and
test times.

The BH package is not a complete version of the Boost lib, again
due to CRAN's restrictions. It's also not up to date, being based
on Boost 1.54 rather than the now 9-month old Boost 1.55.

I hate hate hate that depending on BH now hobbles Stan's ability
to keep up with Boost improvements.

Plus, every time we want to use something in Boost that's not in
BH, we have to ask the maintainers of BH to add it.

Bioconductor (bio's equivalent of CRAN) is just as bad as CRAN
in terms of restrictions on what they'll package.

Basically, R is a huge pain. We haven't had any of these
problems with Python (at least as far as I'm aware).

- Bob

[Emmanuel Charpentier]

Dear Ben :
A couple more datapoints :
1) Sages'R indeed needs -O2 (or -O3, but I'd settle for -O2 since -O3 leads to invalid objects for other libraries, such as nloptr, which is the workhorse of a lot of other packages...).
2) I did not (yet) found a way to pass it from R to an installation od rstan othar than via "./RMakevars". GCC mysteries...
3) System's R does not seem to *need* -O2|3, but gives a couple of warnings about 'exceeded limits. Retrying without limits".
4) GCC mysteries : any compilation of rstan seem to need about 2 Gb (!!!) of memory. Parallelizing it seriously accelerates the process, but needs as much as 11,2 Gb (!!!!) of RAM.
5) Consequences of 4) : any model compilation (even a small one) on a 2 Gb machine leads to swapping.

(Yes, according to my employer, I'm supposed to work on *that*.
With Windows XP. And Word.
I managed to dtch the software, but I'm bound hardware-wise).

Speculation :

There is indeed a problem with Sage's R choice of CXXFLAGS. But given the very small alterations made to original's R tarball, I wonder where is the problem.

I'll file a ticket (against Sage's R) when I understand the problem fully. If you have ideas...

Comments below.

HTH,

--
Emmanuel Charpentier

Le vendredi 25 juillet 2014 17:54:49 UTC+2, Ben Goodrich a écrit :

On Friday, July 25, 2014 11:20:02 AM UTC-4, Emmanuel Charpentier wrote:

Dear Ben,

Just saw your answer. I might have an answer, but I'll ave to check a couple things first, and that won't be before tonight.

Just a couple of comments :

Le vendredi 25 juillet 2014 16:12:00 UTC+2, Ben Goodrich a écrit :

[ ... ]

> system(paste0(Sys.getenv('R_HOME'), "/bin/R", " CMD config CXXFLAGS"))

*This* is the right way the install script (if any) should do intallation, not permanenlly dropping an undocumented configuration file !

We don't permanently drop anything. We ignore ~/.R/Makevars by instead searching for foobar inside the install_rstan() function and then read from it thereafter

 

Rstan 2.3 still left me with .//Makevars... (notwithstanding the set_cppo hypothesis).

Shouldn't setting this option be *internal* to rstan's wrapper functions ? Again, the idea is to avoid to modify the state of thge user's computer. To your garden-variety biostatistician, GCC is something like an endemic disease (fortunately non-mortal (usually...)) ; the CXXFLAGS some sort of prognostic factor. The Stan USER should NOT have to worry with such technical details -but these should be documented for the special cases).
 

  ...
  on.exit(Sys.unsetenv("R_MAKEVARS_USER"))
  on.exit(Sys.unsetenv("R_MAKEVARS_SITE"), add = TRUE)

  try(remove.packages("rstan"), silent = TRUE)
  Sys.setenv(R_MAKEVARS_USER = "foobar")
  Sys.setenv(R_MAKEVARS_SITE = "foobar")
  ...

The rstan package should install with R's CXXFLAGS. It is a very minor rstan bug that it currently does not install if R's CXXFLAGS are empty, in which case g++ compiles at -O0. It is a less minor bug in Sage that it builds R with empty CXXFLAGS. You can work around both bugs by filling out ~/.R/Makevars however you like (excluding CXXFLAGS=-O0 if CXX=g++) and installing rstan without the install_rstan() function.

Indeed !

I think you are objecting to the set_cppo() function, which is called by install_rstan() after rstan is installed and modifies ~/.R/Makevars

That might be the case. I didn't think of that... But this should be documented as a potentially serious side effect. And, if possible at all, limited in scope by the R wrapper functions.
 

so that it is correctly formatted and by default uses -O3 because that makes Stan programs run slightly faster than -O2 and much faster than any other optimization level. Setting -O3 probably breaks some other R packages, which I would be sympathetic to if that happened on your system,

It did. More than once ...

but it changes a  non-executable file, is documented and also prints a message to the console when it is called.

Not during installation !

 

[ ... ]
 

+   install.packages(c("inline", "BH", "RcppEigen"))
+   install.packages("Rcpp", type = "source")

No : here, you unconditionally install whatever Rcpp and BH version happens to be available on CRAN at install time. You should install use the "Depends" field of the DESCRIPTION file, which would install the packages *if necessary* (even with version constraints if necessary).

No:

http://cran.r-project.org/doc/manuals/R-exts.html#Package-Dependencies

"Field ‘Depends’ should nowadays be used rarely, only for packages which are intended to be put on the search path to make their facilities available to the end user (and not to the package itself)"

and later on in that same section

A package that wishes to make use of header files in other packages needs to declare them as a comma-separated list in the field ‘LinkingTo’ in the DESCRIPTION file. For example

LinkingTo: link1, link2

As from R 3.0.2 the ‘LinkingTo’ field can have a version requirement which is checked at installation. (In earlier versions of R it would cause the specification to be ignored.)

Specifying a package in ‘LinkingTo’ suffices if these are C++ headers containing source code or static linking is done at installation: the packages do not need to be (and usually should not be) listed in the ‘Depends’ or ‘Imports’ fields. This includes CRAN packages BH and almost all users of RcppArmadillo and RcppEigen.

which is exactly what the DESCRIPTION file in rstan does

https://raw.githubusercontent.com/stan-dev/rstan/develop/rstan/rstan/DESCRIPTION

Depends: R (>= 3.0.2), Rcpp (>= 0.9.10), utils, inline, methods
LinkingTo: Rcpp, RcppEigen, BH (>= 1.54)
Suggests: RUnit, RcppEigen, BH (>= 1.54), parallel

with the added Suggests line to work around the undocumented fact that the rstan vignette would not build without it.

Okay. So why unconditionally install BH and RcppEigen ?

Emmanuel

Ben

[Jiqiang Guo]

On Fri, Jul 25, 2014 at 2:53 PM, Emmanuel Charpentier <emanuel.c...@gmail.com> wrote:

1) Sages'R indeed needs -O2 (or -O3, but I'd settle for -O2 since -O3 leads to invalid objects for other libraries, such as nloptr, which is the workhorse of a lot of other packages...).
2) I did not (yet) found a way to pass it from R to an installation od rstan othar than via "./RMakevars"

By default, file `Rscript -e "cat(R.home('etc'))"`/Makeconf should have CXXFLAGS set to O2. 

Jiqiang 

[Emmanuel Charpentier]

Dear Bob,

I mostly agree. Some comments below.

THT,

--
Emmanuel Charpentier

Le vendredi 25 juillet 2014 20:43:59 UTC+2, Bob Carpenter a écrit :

On Jul 25, 2014, at 10:55 AM, Emmanuel Charpentier <emanuel.c...@gmail.com> wrote:

> AN INSTALL SCRIPT SHOULD NOT MAKE PERMANENT AND UNDOCUMENTED  CHANGES TO THE USER'S SYSTEM, dammit !

Agreed.  See Ben's reply.  No need to shout, though.

If you want to shout, please direct it at the other R package
writers who modify these things and caused the problems for us
in the first place.  And at the people who configure R itself
for Windows (the installer doesn't match their recommendations
on paths, for example).  

I see the problem... It seems to originate in Digital (VMS), by way of Redmond...

...

> In this case, I second CRAN's attitude : this standard greatly simplifies the life of people needing to use R for other purposes than Stan's frontend.

First, see Ben's response again.

We haven't been able to finagle our package onto CRAN (yet) due
to their rather severe limits on size, compile times, and
test times.  

Aha. As well I support their imposed limits on what tools to use, etc... memory and time limits are less receivable. Is'nt an exemption negociable (maybe at the price of a "18-wheeler" warning sticker slapped on the CRAN page of the  package, advertising its weight) ?

The BH package is not a complete version of the Boost lib, again
due to CRAN's restrictions.  It's also not up to date, being based
on Boost 1.54 rather than the now 9-month old Boost 1.55.  

This is more troublesome (again : possible negotiation ?).

I hate hate hate that depending on BH now hobbles Stan's ability
to keep up with Boost improvements.  

Plus, every time we want to use something in Boost that's not in
BH, we have to ask the maintainers of BH to add it.  

Bioconductor (bio's equivalent of CRAN) is just as bad as CRAN
in terms of restrictions on what they'll package.

Basically, R is a huge pain.

From developer's point of view.

But the user's point of view is different : as already said, R is the "swiss army chainsaw" of applied biostatistics : it offers to the practitioner a huge lot of tools he can apply to ,practical problems. To him/her, the consistency of R libraries is paramount.

The developer has, by definition, another worldview : he/her aims to the best solution of his/her problem, and consistency with other problems is not a concern.

Again CRAN is a huge success from biostatistician's point of view because of the availability of a large, consistent palette of tools. It is also of benefits to theoretician and developers, sinc it gives the users : the success of WinBUGS, which was an atrocity from a Unix software architecture point of view, gave MCMC a lot of  previously untought-of applications. Negotiation between those point of viewsd *could* alleviate the pain (on both sides) and increase the scope of these success.

  We haven't had any of these
problems with Python (at least as far as I'm aware).

Not the same audience : Python users are probably more "engeenering" or "finance" types. In my other life (medical), I've had students that seemed to believe that an electron was something with the shape, color and size of a pea... That didn't forbid them to be good physicians/dentists. The Python crowd is quite different, if I judge by what I read in newsgroups/mailing lists.

- Bob

[Bob Carpenter]

> ...

> We haven't been able to finagle our package onto CRAN (yet) due
> to their rather severe limits on size, compile times, and
> test times.
>
> Aha. As well I support their imposed limits on what tools to use, etc... memory and time limits are less receivable. Is'nt an exemption negociable (maybe at the price of a "18-wheeler" warning sticker slapped on the CRAN page of the package, advertising its weight) ?

If you want to get into a shouting match, suggest that
to the CRAN maintainers :-) Their point is that they run
all the automated tests and only have limited resources for
the 1000s of packages they "maintain."

> The BH package is not a complete version of the Boost lib, again
> due to CRAN's restrictions. It's also not up to date, being based
> on Boost 1.54 rather than the now 9-month old Boost 1.55.
>
> This is more troublesome (again : possible negotiation ?).

See above. Good luck.

> ...

> Basically, R is a huge pain.
>
> From developer's point of view.
>
> But the user's point of view is different : as already said, R is the "swiss army chainsaw" of applied biostatistics : it offers to the practitioner a huge lot of tools he can apply to ,practical problems. To him/her, the consistency of R libraries is paramount.

Agreed. Which is why we're trying to deal with the pain.

>
> The developer has, by definition, another worldview : he/her aims to the best solution of his/her problem, and consistency with other problems is not a concern.
>
> Again CRAN is a huge success from biostatistician's point of view because of the availability of a large, consistent palette of tools. It is also of benefits to theoretician and developers, sinc it gives the users : the success of WinBUGS, which was an atrocity from a Unix software architecture point of view, gave MCMC a lot of previously untought-of applications. Negotiation between those point of viewsd *could* alleviate the pain (on both sides) and increase the scope of these success.

I take it you never tried to install R2WinBUGS or any of the other
R interfaces to BUGS. What a pain! And not fully encapsulated in
CRAN --- just the interface was on CRAN, then you had to coordinate
the install of R2WinBUGS in R with the install of WinBUGS on Windows,
fiddling with paths and so on. R2jags has gotten better over the years,
to the point where it's relatively straightforward to install.

The real problem for us is that we need the C++ tool chain available
at run time so that users can compile models.

Overall, in standard R distributions, I've never had a problem installing
RStan. It's when people want to run R through Cygwin or SAGE or some
other combination of tools that want to run their own way.

> We haven't had any of these
> problems with Python (at least as far as I'm aware).
>
> Not the same audience : Python users are probably more "engeenering" or "finance" types.

...

> The Python crowd is quite different, if I judge by what I read in newsgroups/mailing lists.

Yup. It's also computer scientists, physicists, and
bioinformaticians.

But that's not the perspective I'm talking about. I'm talking
about the developers. They're statisticians for R, whereas
they're computer scientists for Python.

- Bob

[Emmanuel Charpentier]

Aha ! Now, it makes sense, indeed... See below for the diff.

Thank you, Jidiang, for the diagnosis. Any suggestions for patching Sage's R ?

charpent@asus16-ec:~$ diff -u /usr/local/sage-6.3beta6/local/lib/R//etc/Makeconf /usr/lib/R/etc/Makeconf
--- /usr/local/sage-6.3beta6/local/lib/R//etc/Makeconf    2014-07-24 22:18:31.082753022 +0200
+++ /usr/lib/R/etc/Makeconf    2014-04-10 13:58:38.000000000 +0200
@@ -4,26 +4,26 @@
 #
 # R was configured using the following call
 # (not including env. vars and site configuration)
-# configure  '--prefix=/usr/local/sage-6.3beta6/local' '--libdir=/usr/local/sage-6.3beta6/local/lib' '--with-x=yes' '--enable-R-shlib' '--with-readline=yes' '--with-blas=-L/usr/local/sage-6.3beta6/local/lib -lf77blas -latlas' '--with-lapack=-L/usr/local/sage-6.3beta6/local/lib -llapack -lcblas' 'MAKE=make -j8' 'CC=gcc' 'CFLAGS=-g -O2 ' 'LDFLAGS=-L/usr/local/sage-6.3beta6/local/lib/ ' 'CPPFLAGS=' 'F77=gfortran' 'CXX=g++' 'CXXFLAGS=' 'FC=gfortran' 'FCFLAGS=-g -O2 '
+# configure  '--prefix=/usr' '--with-cairo' '--with-jpeglib' '--with-readline' '--with-tcltk' '--with-system-bzlib' '--with-system-pcre' '--with-system-zlib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--datadir=/usr/share/R/share' '--includedir=/usr/share/R/include' '--with-blas' '--with-lapack' '--enable-R-profiling' '--enable-R-shlib' '--enable-memory-profiling' '--without-recommended-packages' '--build' 'x86_64-linux-gnu' 'build_alias=x86_64-linux-gnu' 'R_PRINTCMD=/usr/bin/lpr' 'R_PAPERSIZE=letter' 'R_BROWSER=xdg-open' 'LIBnn=lib' 'CC=gcc -std=gnu99' 'CFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g' 'LDFLAGS=-Wl,-z,relro' 'CPPFLAGS=' 'F77=gfortran' 'FFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4' 'CXX=g++' 'CXXFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g' 'FC=gfortran' 'FCFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4'
 
 ## This fails if it contains spaces, or if it is quoted
 include $(R_SHARE_DIR)/make/vars.mk
 
 AR = ar
 ## Used by packages 'maps' and 'mapdata'
-AWK = gawk
-BLAS_LIBS = -L/usr/local/sage-6.3beta6/local/lib -lf77blas -latlas
+AWK = /usr/bin/awk
+BLAS_LIBS = -lblas
 C_VISIBILITY = -fvisibility=hidden
 CC = gcc -std=gnu99
-CFLAGS = -g -O2  $(LTO)
+CFLAGS = -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g $(LTO)
 CPICFLAGS = -fpic
 CPPFLAGS =
 CXX = g++
 CXXCPP = $(CXX) -E
-CXXFLAGS =  $(LTO)
+CXXFLAGS = -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g $(LTO)
 CXXPICFLAGS = -fpic
 CXX1X = g++
-CXX1XFLAGS =
+CXX1XFLAGS = -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g
 CXX1XPICFLAGS = -fpic
 CXX1XSTD =  -std=c++11
 DYLIB_EXT = .so
@@ -36,37 +36,37 @@
 ECHO_T =
 ## NB, set FC before F77 as on Solaris make, setting FC sets F77
 FC = gfortran
-FCFLAGS = -g -O2  $(LTO)
+FCFLAGS = -g -O2 -fstack-protector --param=ssp-buffer-size=4 $(LTO)
 ## additional libs needed when linking with $(FC), e.g. on Solaris
 FCLIBS =
 F77 = gfortran
 F77_VISIBILITY = -fvisibility=hidden
-FFLAGS = -g -O2 $(LTO)
+FFLAGS = -g -O2 -fstack-protector --param=ssp-buffer-size=4 $(LTO)
 FLIBS =  -lgfortran -lm -lquadmath
 FCPICFLAGS = -fpic
 FPICFLAGS = -fpic
 FOUNDATION_CPPFLAGS =
 FOUNDATION_LIBS =
-JAR =
-JAVA =
-JAVAC =
-JAVAH =
+JAR = /usr/bin/jar
+JAVA = /usr/bin/java
+JAVAC = /usr/bin/javac
+JAVAH = /usr/bin/javah
 ## JAVA_HOME might be used in the next three. 
 ## They are for packages 'JavaGD' and 'rJava'
-JAVA_HOME =
-JAVA_CPPFLAGS =
-JAVA_LIBS =
-JAVA_LD_LIBRARY_PATH =
-LAPACK_LIBS = -L/usr/local/sage-6.3beta6/local/lib -llapack -lcblas
+JAVA_HOME = /usr/lib/jvm/java-7-openjdk-amd64/jre
+JAVA_CPPFLAGS = -I$(JAVA_HOME)/../include
+JAVA_LIBS = -L$(JAVA_HOME)/lib/amd64/server -ljvm
+JAVA_LD_LIBRARY_PATH = $(JAVA_HOME)/lib/amd64/server
+LAPACK_LIBS = -llapack
 ## we only need this is if it is external, as otherwise link to R
 LIBINTL=
 LIBM = -lm
 LIBR = -L$(R_HOME)/lib$(R_ARCH) -lR
-LIBS =  -llzma -lrt -ldl -lm
+LIBS =  -lpcre -llzma -lbz2 -lz -lrt -ldl -lm
 ## needed by R CMD config
 LIBnn = lib
 LIBTOOL = $(SHELL) "$(R_HOME)/bin/libtool"
-LDFLAGS = -L/usr/local/sage-6.3beta6/local/lib/
+LDFLAGS = -Wl,-z,relro
 LTO =
 ## needed to build applications linking to static libR
 MAIN_LD = $(CC)
@@ -79,7 +79,7 @@
 OBJCXX =
 R_ARCH =
 RANLIB = ranlib
-SAFE_FFLAGS = -g -O2 -ffloat-store
+SAFE_FFLAGS = -g -O2 -fstack-protector --param=ssp-buffer-size=4 -ffloat-store
 SED = /bin/sed
 SHELL = /bin/bash
 SHLIB_CFLAGS =
@@ -102,8 +102,8 @@
 SHLIB_OPENMP_FFLAGS = -fopenmp
 STRIP_LIBS = strip --strip-unneeded
 STRIP_STATIC_LIBS = strip --strip-debug
-TCLTK_CPPFLAGS = -I/usr/include/tcl8.6 -I/usr/include/tcl8.6
-TCLTK_LIBS = -L/usr/lib/x86_64-linux-gnu -ltcl8.6 -L/usr/lib/x86_64-linux-gnu -ltk8.6 -lX11 -lXss -lXext
+TCLTK_CPPFLAGS = -I/usr/include/tcl8.5 -I/usr/include/tcl8.5
+TCLTK_LIBS = -L/usr/lib/x86_64-linux-gnu -ltcl8.5 -L/usr/lib/x86_64-linux-gnu -ltk8.5 -lX11 -lXss -lXext
 
 ## for linking to libR.a
 STATIC_LIBR = # -Wl,--whole-archive "$(R_HOME)/lib$(R_ARCH)/libR.a" -Wl,--no-whole-archive $(BLAS_LIBS) $(FLIBS)  $(LIBINTL) -lreadline  $(LIBS)

[Jiqiang Guo]

On Fri, Jul 25, 2014 at 3:44 PM, Emmanuel Charpentier <emanuel.c...@gmail.com> wrote:

Aha ! Now, it makes sense, indeed... See below for the diff.

Thank you, Jidiang, for the diagnosis. Any suggestions for patching Sage's R ?

The head of this Makeconf file says:

$  head -n 2 `Rscript -e "cat(R.home('etc'))"`/Makeconf

# etc/Makeconf.  Generated from Makeconf.in by configure.

#

I think configure building R in sage would require passing O2 to CXXFLAGS and probably others as what are typically done for Linux.  From the diff you gave, CXXFLAGS=''.

Jiqiang 

[Ben Goodrich]

On Friday, July 25, 2014 2:53:59 PM UTC-4, Emmanuel Charpentier wrote:

Dear Ben :
A couple more datapoints :
1) Sages'R indeed needs -O2 (or -O3, but I'd settle for -O2 since -O3 leads to invalid objects for other libraries, such as nloptr, which is the workhorse of a lot of other packages...).

OK
 

2) I did not (yet) found a way to pass it from R to an installation od rstan othar than via "./RMakevars". GCC mysteries...

I suspect there is something suboptimal in the Sage build. A plain ./configure on the r-devel tree results in

R is now configured for x86_64-unknown-linux-gnu

  Source directory:          .
  Installation directory:    /usr/local

  C compiler:                gcc -std=gnu99  -g -O2
  Fortran 77 compiler:       gfortran  -g -O2

  C++ compiler:              g++  -g -O2
  C++ 11 compiler:           g++  -std=c++11 -g -O2
  Fortran 90/95 compiler:    gfortran -g -O2

So, someone would have to do something other than a plain ./configure to remove the -O2.
 

3) System's R does not seem to *need* -O2|3, but gives a couple of warnings about 'exceeded limits. Retrying without limits".

I get those also with g++, and I think they are harmless because it succeeds on the retry. They do not appear with clang++.
 

4) GCC mysteries : any compilation of rstan seem to need about 2 Gb (!!!) of memory. Parallelizing it seriously accelerates the process, but needs as much as 11,2 Gb (!!!!) of RAM.

Not a mystery or at least this is something we have known to be the case for 3 years. With clang++, it is much better in terms of RAM usage. With g++-4.9, you can add the flags -flto and -ffat-lto-objects (see https://gcc.gnu.org/gcc-4.9/changes.html) to CXXFLAGS in ~/.R/Makevars and it is a lot better but not quite as good as clang++.
 

5) Consequences of 4) : any model compilation (even a small one) on a 2 Gb machine leads to swapping.

I would install clang++ from the Debian repositories and set CC=clang and CXX=clang++ in ~/.R/Makevars

[...]

Rstan 2.3 still left me with .//Makevars... (notwithstanding the set_cppo hypothesis).

Yes, so there is some chance that it will break other packages, and I would understand if CRAN makes us get rid of set_cppo() if / when we retry to upload it to CRAN. But set_cppo() in rstan 2.3.0 should have produced a valid ~/.R/Makevars for installing rstan. It is just we can't ensure that ~/.R/Makevars is valid on every user's machine so install_rstan() ignores it in favor of whatever settings R was built with (which are amenable to installing rstan for any binary install of R or a sane source install of R but apparently not Sage's).
 

Shouldn't setting this option be *internal* to rstan's wrapper functions ? Again, the idea is to avoid to modify the state of thge user's computer. To your garden-variety biostatistician, GCC is something like an endemic disease (fortunately non-mortal (usually...)) ; the CXXFLAGS some sort of prognostic factor. The Stan USER should NOT have to worry with such technical details -but these should be documented for the special cases).

I don't think there is a way to pass a particular set of flags when internally calling stan_model() (to compile the program) that wouldn't get superseded by what is in ~/.R/Makevars because the compilation is actually done with a separate R process. For most users that do not want to worry about such technical details, calling set_cppo() once solves that problem, but it is somewhat evil to the extent it breaks other R packages.
 

 [...]

I think you are objecting to the set_cppo() function, which is called by install_rstan() after rstan is installed and modifies ~/.R/Makevars

That might be the case. I didn't think of that... But this should be documented as a potentially serious side effect. And, if possible at all, limited in scope by the R wrapper functions.
 

so that it is correctly formatted and by default uses -O3 because that makes Stan programs run slightly faster than -O2 and much faster than any other optimization level. Setting -O3 probably breaks some other R packages, which I would be sympathetic to if that happened on your system,

It did. More than once ...

Sorry. We were aware of the potential for this problem, but were not aware of particular packages that do not work under -O3. If you could tell us what those were, we could add a warning about that to the documentation. If you call set_cppo("presentation2"), it should set it back to -O2.
 

but it changes a  non-executable file, is documented and also prints a message to the console when it is called.

Not during installation !

It isn't read during installation, but it is modified by install_rstan() after the installation. There should be a message like "mode fast without NDEBUG for compiling C++ code is set".
 

 

[ ... ]
 

+   install.packages(c("inline", "BH", "RcppEigen"))
+   install.packages("Rcpp", type = "source")

No : here, you unconditionally install whatever Rcpp and BH version happens to be available on CRAN at install time. You should install use the "Depends" field of the DESCRIPTION file, which would install the packages *if necessary* (even with version constraints if necessary).

No:

http://cran.r-project.org/doc/manuals/R-exts.html#Package-Dependencies

"Field ‘Depends’ should nowadays be used rarely, only for packages which are intended to be put on the search path to make their facilities available to the end user (and not to the package itself)"

and later on in that same section

A package that wishes to make use of header files in other packages needs to declare them as a comma-separated list in the field ‘LinkingTo’ in the DESCRIPTION file. For example

LinkingTo: link1, link2

As from R 3.0.2 the ‘LinkingTo’ field can have a version requirement which is checked at installation. (In earlier versions of R it would cause the specification to be ignored.)

Specifying a package in ‘LinkingTo’ suffices if these are C++ headers containing source code or static linking is done at installation: the packages do not need to be (and usually should not be) listed in the ‘Depends’ or ‘Imports’ fields. This includes CRAN packages BH and almost all users of RcppArmadillo and RcppEigen.

which is exactly what the DESCRIPTION file in rstan does

https://raw.githubusercontent.com/stan-dev/rstan/develop/rstan/rstan/DESCRIPTION

Depends: R (>= 3.0.2), Rcpp (>= 0.9.10), utils, inline, methods
LinkingTo: Rcpp, RcppEigen, BH (>= 1.54)
Suggests: RUnit, RcppEigen, BH (>= 1.54), parallel

with the added Suggests line to work around the undocumented fact that the rstan vignette would not build without it.

Okay. So why unconditionally install BH and RcppEigen ?

Because BH and RcppEigen are not only needed when the package is built / installed but also when stan_model() is called to compile the user's program. So, you could install rstan but not use it unless BH and RcppEigen are on the disk. CRAN's DESCRIPTION file does not have an idiom for this case. (Re)Installing them unconditionally isn't harmful but wastes a little bandwidth if they are already installed. I suppose I could write some logic to check whether they are installed and what version they have, but I haven't gotten around to it.

Ben