boost headers and rstan

[Ben Goodrich]

Thinking again about how we could get RStan onto CRAN, if we made boost a system requirement to build stanc (only) and created a dependency on the BH package

http://cran.r-project.org/web/packages/BH/index.html

in order to build both libstan and the user models, then I think we could avoid the purported reason (excessive size) that RStan was rejected the first time. There is an issue that the BH package is based on boost 1.51, but presumably we could ask Jay to update it and presumably we want Stan to stick with boost 1.51 for a while due to the ongoing modularization / git-ification of boost.

However, the BH package currently does not include enough of boost to build libstan or the user models. AFAICT, these headers (plus whatever they pull in) are needed by some part of (the master branch of) Stan but are not included in the BH package

goodrich@CYBERPOWERPC:/opt/stan$ cat /tmp/missing_boost_headers.txt
boost/accumulators/accumulators.hpp
boost/algorithm/string.hpp
boost/exception/all.hpp
boost/fusion/include/adapt_struct.hpp
boost/math/constants/constants.hpp
boost/math/distributions.hpp
boost/math/policies/error_handling.hpp
boost/math/special_functions/acosh.hpp
boost/math/special_functions/asinh.hpp
boost/math/special_functions/atanh.hpp
boost/math/special_functions/beta.hpp
boost/math/special_functions/cbrt.hpp
boost/math/special_functions/digamma.hpp
boost/math/special_functions/erf.hpp
boost/math/special_functions/gamma.hpp
boost/math/special_functions/hypot.hpp
boost/math/special_functions/owens_t.hpp
boost/math/special_functions/round.hpp
boost/math/special_functions/trunc.hpp
boost/multi_array.hpp
boost/random/additive_combine.hpp
boost/random/bernoulli_distribution.hpp
boost/random/binomial_distribution.hpp
boost/random/cauchy_distribution.hpp
boost/random/chi_squared_distribution.hpp
boost/random/exponential_distribution.hpp
boost/random/gamma_distribution.hpp
boost/random/lognormal_distribution.hpp
boost/random/negative_binomial_distribution.hpp
boost/random/normal_distribution.hpp
boost/random/poisson_distribution.hpp
boost/random/student_t_distribution.hpp
boost/random/uniform_01.hpp
boost/random/uniform_real_distribution.hpp
boost/random/weibull_distribution.hpp
boost/spirit/include/qi.hpp
boost/variant/apply_visitor.hpp
boost/variant/recursive_variant.hpp

I obtained this list by moving my /usr/include/boost out of the way and calling

make clean-all && make O=0 CC=clang++ BOOST=/tmp/BH/inst/include/ -j7 -k test-all 2> /tmp/boost_errors.txt
grep -F "file not found" /tmp/boost_errors.txt | sed 's/^.*error: //g' | sed 's/ file not found//g' | sort | uniq | sed s/\'//g > /tmp/missing_boost_headers.txt

Of the above, I suspect the following are "stanc only"

boost/fusion/include/adapt_struct.hpp
boost/spirit/include/qi.hpp
boost/variant/apply_visitor.hpp
boost/variant/recursive_variant.hpp


Is that correct? If I exclude those and use bcp to copy everything that the remaining headers pull in, it adds 8M of files that were not already in the BH package, which could be a problem for them so if it is possible to cut down further that would be good.

Ben

[Jiqiang Guo]

In current makefile for rstan, bcp can be used to make a temporary stanlib file used instead of the lib under stan.  But the size of the folder that created by bcp are way larger than the size of package BH.  So I doubt the correctness of the above list of header files that are needed for both stanc and models (though if header files for stanc are excluded, there would be fewer files). 

And if I add files in this list into BH's package, there are still files missing from 'make test-all'

--

Jiqiang 

Ben

--
You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Ben Goodrich]

On Monday, April 8, 2013 5:03:33 PM UTC-4, Jiqiang Guo wrote:

In current makefile for rstan, bcp can be used to make a temporary stanlib file used instead of the lib under stan.  But the size of the folder that created by bcp are way larger than the size of package BH.  So I doubt the correctness of the above list of header files that are needed for both stanc and models (though if header files for stanc are excluded, there would be fewer files). 

And if I add files in this list into BH's package, there are still files missing from 'make test-all'

I suppose it could miss a few things if the same file required multiple missing boost headers and only the first error got reported. If you do make test-all with -k, what are the boost header files that are still missing?

Ben

[Jiqiang Guo]

The problem is that some boost header files are included by boost header files, which you cannot catch by this way. 

--

Jiqiang 

[Ben Goodrich]

On Monday, April 8, 2013 8:14:21 PM UTC-4, Jiqiang Guo wrote:

The problem is that some boost header files are included by boost header files, which you cannot catch by this way. 

Right, but why does bcp not bring them in when you run bcp on that list?

Ben

[Jiqiang Guo]

Sorry that I did not get the whole picture of the very first post. Now I see what you meant. 

By my original brute force, there would be fewer files than using bcp.  For example, rstan 1.1.1 (not using bcp) is about 3M while rstan 1.2.0 (bcp is used) is about 4M. 

--

Jiqiang 

[Ben Goodrich]

On Tuesday, April 9, 2013 11:26:00 AM UTC-4, Jiqiang Guo wrote:

Sorry that I did not get the whole picture of the very first post. Now I see what you meant. 

By my original brute force, there would be fewer files than using bcp.  For example, rstan 1.1.1 (not using bcp) is about 3M while rstan 1.2.0 (bcp is used) is about 4M. 

That would be even better. But one way or another, do you think the plan would be feasible?

Ben

[Jiqiang Guo]

Hate to say that I do not see it would satisfy cran.  First, we still some header files that would not be small.  Second, rstan.so for one platform is about 4.7M.  Last, it takes some CPU time to build rstan. 

--

Jiqiang 

[Ben Goodrich]

On Tuesday, April 9, 2013 2:37:07 PM UTC-4, Jiqiang Guo wrote:

Hate to say that I do not see it would satisfy cran.  First, we still some header files that would not be small.  Second, rstan.so for one platform is about 4.7M.  Last, it takes some CPU time to build rstan. 

I wouldn't be surprised if they found some excuse to reject it, but I don't think these are necessarily dealbreakers. I have several packages that are more than 4M

4.0M    RcppArmadillo
5.2M    RcppEigen
5.5M    PBSmapping
5.9M    Rgraphviz2
6.4M    accuracy
9.0M    highlight
11M     spatstat
12M     ElemStatLearn
20M     BH
21M     speedRlibs
33M     qtbase
56M     rstan

I haven't timed a serial R CMD check recently, but I think they permit a package to have 5 minutes with no questions asked.

Ben

[Jiqiang Guo]

R CMD build and check would take about 20 mins for us, http://d1m1s1b1.stat.columbia.edu:8080/. 

I do not really want to start a dialog with CRAN.  But if anybody wants to try, I would be glad to help. 

--

Jiqiang 

[Ben Goodrich]

On Wednesday, April 10, 2013 2:32:51 PM UTC-4, Jiqiang Guo wrote:

R CMD build and check would take about 20 mins for us, http://d1m1s1b1.stat.columbia.edu:8080/. 

I do not really want to start a dialog with CRAN.  But if anybody wants to try, I would be glad to help. 

I would be willing to try if we were able to get the time for check down to a reasonable number. Due to the \Sexp in the .Rd files, R CMD check has to install rstan twice, which probably accounts for about 1/3 of the time. We could replace those \Sexp with sed calls, but that still doesn't get us particularly close. Just compiling the parser with g++ takes me 3 minutes and 20 seconds, which doesn't leave a lot of time for anything else. That is a shame because clang++ can do it in 1 minute and 30 seconds, which would probably be manageable.

Ben