Intel C++ Compiler

[Ben Goodrich]

Some rare good news on the compiler front. I sullied my otherwise mostly pure laptop by installing the closed source but free-as-in-beer-for-noncommercial-use-iff-you-register-so-we-can-later-attempt-to-sell-you-stuff-you-do-not-want Intel C++ compiler from

http://software.intel.com/en-us/articles/non-commercial-software-download/

This compiler is popular in certain circles. And make test-all (eventually) passes all tests with 55MB of warnings output, most of which are repetitive and pertain to Eigen or boost rather than stan specifically. The only useful stuff I could grep in the warnings was

goodrich@CYBERPOWERPC:/tmp$ grep "^src" warnings.txt
src/test/prob/distributions/multivariate/continuous/lkj_corr_test.cpp(59): warning #265: floating-point operation result is out of range
src/test/prob/distributions/multivariate/continuous/lkj_cov_test.cpp(59): warning #265: floating-point operation result is out of range
src/stan/math/error_handling.hpp(460): warning #186: pointless comparison of unsigned integer with zero
src/stan/math/error_handling.hpp(460): warning #186: pointless comparison of unsigned integer with zero
src/stan/math/error_handling.hpp(460): warning #186: pointless comparison of unsigned integer with zero
src/test/math/matrix_test.cpp(194): warning #68: integer conversion resulted in a change of sign
src/test/math/matrix_test.cpp(194): warning #68: integer conversion resulted in a change of sign
src/test/math/matrix_test.cpp(197): warning #68: integer conversion resulted in a change of sign
src/test/math/matrix_test.cpp(198): warning #68: integer conversion resulted in a change of sign
src/test/math/matrix_test.cpp(200): warning #68: integer conversion resulted in a change of sign
src/test/math/matrix_test.cpp(200): warning #68: integer conversion resulted in a change of sign

Perhaps we should make it a habit of occasionally testing stan with Intel's C++ compiler and / or mention that it is as supported as recent versions of clang are?

Ben

[Bob Carpenter]

I hear the Intel compiler produces faster code
than g++ or clang++, so it'd be nice to see the speed.

I think we should claim we tested the Intel compiler
and it worked, not that we're planning on supporting it
going forward. At least until we get the compilation
more under control in the context of R and across
Win/Linux/Mac for g++ and clang++. I just don't want to
have to field questions about it.

Thanks for the truncated report.

The pointless comparison of unsigned int with
zero is from a templated class instantiated with
"unsigned int", but I suppose the comparison could
be statically removed the way we do with propto calcs
in the case of unsigned int.

I'll check out the matrix and LKJ density tests.

- Bob

[Ben Goodrich]

On Thursday, March 29, 2012 10:37:20 AM UTC-4, Bob Carpenter wrote:

I hear the Intel compiler produces faster code
than g++ or clang++, so it'd be nice to see the speed.

It's complicated (and I didn't test run-time speed). It definitely can be said that it compiles slowly; it took over an hour to run through test-all with one core. The benchmarks I have seen often don't make it 100% clear if they are talking about using the Intel Math Kernel Library (MKL) standalone or MKL with the Intel C++ compiler. So, I never have been able to get a clear picture of how much of the gain to attribute to the compiler itself. FWIW, the next version of Eigen will supposedly support using the MKL backend for some operations.
 

I think we should claim we tested the Intel compiler
and it worked, not that we're planning on supporting it
going forward.  At least until we get the compilation
more under control in the context of R and across
Win/Linux/Mac for g++ and clang++.  I just don't want to
have to field questions about it.

OK, although in general I would say that stan should support whatever compilers are supported by the intersection of boost and Eigen.

Ben

[Bob Carpenter]

[Bob Carpenter]

Sorry -- I keep accidentally sending out empty
e-mails.

It's because I have to hit reply-to in order to
see the formatting in the messages from Ben and
from Marcus. As you can see below, I only get
spaces in their emails, and Thunderbird conveneniently
deletes line-initial spaces when showing messages.

- Bob

On 3/29/12 12:19 PM, Ben Goodrich wrote:

> On Thursday, March 29, 2012 10:37:20 AM UTC-4, Bob Carpenter wrote:
>
> I hear the Intel compiler produces faster code
> than g++ or clang++, so it'd be nice to see the speed.
>
>
> It's complicated (and I didn't test run-time speed). It definitely can be said that it compiles slowly; it took over an
> hour to run through test-all with one core. The benchmarks I have seen often don't make it 100% clear if they are
> talking about using the Intel Math Kernel Library (MKL) standalone or MKL with the Intel C++ compiler. So, I never have
> been able to get a clear picture of how much of the gain to attribute to the compiler itself. FWIW, the next version of
> Eigen will supposedly support using the MKL backend for some operations.
>
> I think we should claim we tested the Intel compiler
> and it worked, not that we're planning on supporting it
> going forward. At least until we get the compilation
> more under control in the context of R and across
> Win/Linux/Mac for g++ and clang++. I just don't want to
> have to field questions about it.
>
> OK, although in general I would say that stan should support whatever compilers are supported by the intersection of
> boost and Eigen.

Maybe we mean different things by "support". Right now,
our support is at the level of "works on my machine".

Here's what the Eigen FAQ has to say about their compiler
support:

------------------------------
Compiler support

Eigen is standard C++98 and so should theoretically be compatible with any compliant compiler. Whenever we use some
non-standard feature, that is optional and can be disabled.
Eigen is being successfully used with the following compilers:

GCC, version 3.4 and newer. Very good performance with GCC 4.2 and newer.
MSVC (Visual Studio), 2005 and newer. Vectorization is enabled with 2008 and newer, which also gives good performance.
Intel C++ compiler. Very good performance.
LLVM/CLang++ (2.8 and newer).
MinGW, recent versions. Very good performance when using GCC 4.
QNX's QCC compiler.
--------------------------

If that's all you mean by support, then by all means. Let's list
everything where we've been able to get it to compile. Anyone want
to volunteer to try MinGW on Windows?

The same version of g++ behaves very differently on the different platforms,
and what workedin 4.8 doesn't seem to work in 4.9. It's been driving
us nuts. It even behaves differently in terms of being able to succeed
with compilation at different optimization levels on the same platform.

Right now, our situation with Windows is that MSVC can compile the
parser, but not the models, whereas a suitably binary-edited verison of
g++ can compile the models but not the parser, and then only at
optimization level 3. The other optimization levels cause the compiler
to output an empty object file.

- Bob

[Ben Goodrich]

On Thursday, March 29, 2012 5:15:00 PM UTC-4, Bob Carpenter wrote:

> OK, although in general I would say that stan should support whatever compilers are supported by the intersection of
> boost and Eigen.

I just meant "known to pass test-all, which is run somewhat regularly", in which case we haven't achieved this goal yet.
 

Anyone want to volunteer to try MinGW on Windows?

CRAN will partially test MinGW for us, which is unfortunately to say that we need to get MinGW 4.5.x to compile stanc.

 

The same version of g++ behaves very differently on the different platforms,
and what workedin 4.8 doesn't seem to work in 4.9.  It's been driving
us nuts.  It even behaves differently in terms of being able to succeed
with compilation at different optimization levels on the same platform.

I know about the 4.9 thing but I think we are good on 4.2 -- 4.8. Do we know of discrepancies based on the optimization level for !Windows?

Right now, our situation with Windows is that MSVC can compile the
parser, but not the models, whereas a suitably binary-edited verison of
g++ can compile the models but not the parser, and then only at
optimization level 3.  The other optimization levels cause the compiler
to output an empty object file.

I can't help much with Windows, but unless there are unusual circumstances, CRAN won't accept the R package if it doesn't compile on Windows with the gcc toolchain. Their Windows server might be set up to allow a larger stack limit, but I doubt they would tolerate having packages binary-edit an executable.

Ben

[Ben Goodrich]

On Thursday, March 29, 2012 7:43:21 PM UTC-4, Ben Goodrich wrote:

I can't help much with Windows

... but I have managed to set up a cross-compiling environment to target Windows using

http://mingw-w64.sourceforge.net/

which is also used by CRAN.

After a bit of fiddling, I can get this far

make AR=i686-w64-mingw32-ar CC=i686-w64-mingw32-g++ test-all
i686-w64-mingw32-g++ -I src -I lib -O0 -Wall -std=gnu++0x -g -static   -c -I lib/gtest/include -I lib/gtest -g -DGTEST_HAS_PTHREAD=0 -o test/gtest.o lib/gtest/src/gtest-all.cc
i686-w64-mingw32-ar rv test/libgtest.a test/gtest.o
i686-w64-mingw32-ar: creating test/libgtest.a
a - test/gtest.o
i686-w64-mingw32-g++ -I src -I lib -O0 -Wall -std=gnu++0x -g -static   -c -I lib/gtest/include -I lib/gtest -g -DGTEST_HAS_PTHREAD=0 -o test/io/csv_writer.o src/test/io/csv_writer_test.cpp
------------------------------------------------------------
i686-w64-mingw32-g++ -I src -I lib -O0 -Wall -std=gnu++0x -g -static    -I lib/gtest/include -I lib/gtest -g -DGTEST_HAS_PTHREAD=0 lib/gtest/src/gtest_main.cc test/libgtest.a -o test/io/csv_writer test/io/csv_writer.o
test/io/csv_writer --gtest_output="xml:test/io/csv_writer.xml"
wine: cannot find 'test/io/csv_writer'
make: *** [test/io/csv_writer] Error 2

The problem appears to be that wine (which is used by mingw to not emulate Windows) does not know how to handle relative paths. Thus, I can subsequently do

master!stan *> wine /opt/stan/test/io/csv_writer
Running main() from gtest_main.cc
[==========] Running 7 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 7 tests from io_csv_writer
[ RUN      ] io_csv_writer.csv_writer_int
[       OK ] io_csv_writer.csv_writer_int (0 ms)
[ RUN      ] io_csv_writer.csv_writer_double
[       OK ] io_csv_writer.csv_writer_double (0 ms)
[ RUN      ] io_csv_writer.csv_writer_vector
[       OK ] io_csv_writer.csv_writer_vector (0 ms)
[ RUN      ] io_csv_writer.csv_writer_row_vector
[       OK ] io_csv_writer.csv_writer_row_vector (0 ms)
[ RUN      ] io_csv_writer.csv_writer_matrix
[       OK ] io_csv_writer.csv_writer_matrix (0 ms)
[ RUN      ] io_csv_writer.csv_writer_matrix_col_major
[       OK ] io_csv_writer.csv_writer_matrix_col_major (0 ms)
[ RUN      ] io_csv_writer.csv_writer_strings
[       OK ] io_csv_writer.csv_writer_strings (0 ms)
[----------] 7 tests from io_csv_writer (1 ms total)

[----------] Global test environment tear-down
[==========] 7 tests from 1 test case ran. (2 ms total)
[  PASSED  ] 7 tests.

Daniel, is there some canonical way for the makefile to prefix $STAN_HOME to all paths so that they are absolute?

Thanks,
Ben

[Ben Goodrich]

On Friday, March 30, 2012 1:19:56 PM UTC-4, Ben Goodrich wrote:

is there some canonical way for the makefile to prefix $STAN_HOME to all paths so that they are absolute?

I figured it out. Plz verify that commit e6bc010c73f8a12ff2a49f6ea1f15b71838ee078 does not bork everything on your OS.

Now, test-all with mingw on -O0 hits a (I think known) problem with the parser making a file that is too big. But with -03 test-all runs for about half an hour, through all the main tests, and then fails with what looks like to be a cross-compiling specific issue that I hopefully just fixed and am trying again.

Ben

 

[Bob Carpenter]

Hopefully the size issue for the parser will go away when
I've finished the header/implementation refactor.

It's a ton of work. I've spent about 10 hours on it
already and there's probably another 4 or 5 left if all
goes well. So maybe early next week. This is something
to do on a big monitor, not at home.

Should've listened to Matt harder in the beginning when
he said there were advantages to splitting. I just followed
Eigen and Boost's practice, which apparently doesn't scale
very well.

The same kind of pain awaits in adding "const" declarations
everywhere they should go. I was very bad about not doing
that back in the days when I could hardly even get hello world
to compile in C++.

- Bob

[Ben Goodrich]

Well, the good news is that i686-w64-mingw32-g++ with -03 on my laptop can get through test-all and is now going through test-bugs and has made it through a couple of BUGS models now.

So, the question is why does this not work on Windows? Does mingw need the default system g++ (which is 4.6.x in my case) in any way? Maybe your Windows machine has a different version? I have not tried cross-compiling with -m32 yet, so maybe it is a problem with a 64 bit system trying to generate 32 bit binaries?

Daniel and I probably need to create proper make options when the compiler is i686-w64-mingw32-g++. The make/detect_os concludes that I am Linux but make/detect_cc concludes that I am using g++ rather than i686-w64-mingw32-g++. So, what we really want in this case is a mix of the existing Linux and Windows options.

Ben

[Bob Carpenter]

On 3/30/12 6:10 PM, Ben Goodrich wrote:
> Well, the good news is that i686-w64-mingw32-g++ with -03 on my laptop can get through test-all and is now going through
> test-bugs and has made it through a couple of BUGS models now.

That's fantastic news.

I know Daniel was playing around with which tests
ran, but if you could run all the tests that are
turned on AND build stanc and run the models, that's
real progress.

> So, the question is why does this not work on Windows? Does mingw need the default system g++ (which is 4.6.x in my
> case) in any way? Maybe your Windows machine has a different version? I have not tried cross-compiling with -m32 yet, so
> maybe it is a problem with a 64 bit system trying to generate 32 bit binaries?

Daniel was wondering similar things
today about which versions of the compilers are getting
used and how they all depend on each other.

I think he's running 4.6 on Windows. Which version
does R expect?

> Daniel and I probably need to create proper make options when the compiler is i686-w64-mingw32-g++. The make/detect_os
> concludes that I am Linux but make/detect_cc concludes that I am using g++ rather than i686-w64-mingw32-g++. So, what we
> really want in this case is a mix of the existing Linux and Windows options.

Am I ever glad I'm not the only person working on
this project!

- Bob

[Ben Goodrich]

On Saturday, March 31, 2012 1:10:39 AM UTC-4, Bob Carpenter wrote:

On 3/30/12 6:10 PM, Ben Goodrich wrote:
> Well, the good news is that i686-w64-mingw32-g++ with -03 on my laptop can get through test-all and is now going through
> test-bugs and has made it through a couple of BUGS models now.

That's fantastic news.

I can also report that i686-w64-mingw32-g++ with -03 also looks fine with -m32. The bad news is that I seem to have broken test-unit on Jenkins. See

https://groups.google.com/d/msg/stan-buildbot/JpBsOoJAvqc/ipdgN7yBofQJ

Toward the end, there is a

/jobs/Stan/workspaceEXE = .exetest/prob/transform --gtest_output="xml:test/prob/transform.xml";

/bin/sh: -c: line 1: syntax error near unexpected token `('
/bin/sh: -c: line 1: ` echo '------------------------------------------------------------'; /cygdrive/c/Program Files (x86)/Jenkins/jobs/Stan/workspaceEXE = .exetest/agrad/agrad -

The only thing I can think of is that my laptop uses /bin/bash and Jenkins seems to use /bin/sh which is usually an alias but might not be an alias for bash. So maybe I have a bashism somewhere? A likely candidate is the addprefix mojo in make/tests that I added to get cross-compiling to work. Can Jenkins just use bash like normal people? Otherwise, we might have to go another route in the makefile.
 

I know Daniel was playing around with which tests
ran, but if you could run all the tests that are
turned on AND build stanc and run the models, that's
real progress.

Albeit progress that raises the question of why Linux is more successful at building Windows binaries than Windows is. It wouldn't be the first time that happened, but it is rare.
 

> So, the question is why does this not work on Windows? Does mingw need the default system g++ (which is 4.6.x in my
> case) in any way? Maybe your Windows machine has a different version? I have not tried cross-compiling with -m32 yet, so
> maybe it is a problem with a 64 bit system trying to generate 32 bit binaries?

Daniel was wondering similar things
today about which versions of the compilers are getting
used and how they all depend on each other.

I think he's running 4.6 on Windows.  Which version
does R expect?

4.5.x for now; see

http://cran.r-project.org/web/checks/check_flavors.html#r-release-windows-ix86_x86_64

Although Rtools (which Windows users will need because it is the only toolchain officially supported by Rcpp )

http://cran.r-project.org/bin/windows/Rtools/

seems to say that the latest version uses a prerelease of 4.6.3.
 

> Daniel and I probably need to create proper make options when the compiler is i686-w64-mingw32-g++. The make/detect_os
> concludes that I am Linux but make/detect_cc concludes that I am using g++ rather than i686-w64-mingw32-g++. So, what we
> really want in this case is a mix of the existing Linux and Windows options.

Am I ever glad I'm not the only person working on
this project!

I started implementing this, and it seems fine (aside from the Jenkins thing).

Ben

[Bob Carpenter]

> I know Daniel was playing around with which tests
> ran, but if you could run all the tests that are
> turned on AND build stanc and run the models, that's
> real progress.
>
> Albeit progress that raises the question of why Linux is more successful at building Windows binaries than Windows is.
> It wouldn't be the first time that happened, but it is rare.

In the best case, this would signal a problem with
our Windows machine (the one running Jenkins), not
Windows in general.

I'll try to build under Windows using an old notebook
I have lying around and see if my mileage varies.

- Bob

[Ben Goodrich]

Try to use whatever versions of all the tools come with the latest Rtools because that is closest to what CRAN will have. Also, in make/tests, I am pretty sure you will need to uncomment adding -static-libgcc and -static-libstdc++ and maybe they should be CFLAGS rather than LDFLAGS. These two options may be implied by the -static option, but then again the -static option may be too static.

Also, I thought I got Jenkins working again last night ("Jenkins build is back to normal") but this morning it says it is broken again in a different place with no intervening commits. Is Jenkins quantum?

Ben

[Bob Carpenter]

Is it OK with everyone if we only support the latest
version of R? There's a new one out as of yesterday,
2.15, and they changed versions of GCC.

More below inline.

On 3/31/12 3:17 PM, Ben Goodrich wrote:
> On Saturday, March 31, 2012 1:47:45 PM UTC-4, Bob Carpenter wrote:
>
> I'll try to build under Windows using an old notebook
> I have lying around and see if my mileage varies.
>
> Try to use whatever versions of all the tools come with the latest Rtools because that is closest to what CRAN will
> have. Also, in make/tests, I am pretty sure you will need to uncomment adding -static-libgcc and -static-libstdc++ and
> maybe they should be CFLAGS rather than LDFLAGS. These two options may be implied by the -static option, but then again
> the -static option may be too static.

Thanks for the tip. This page

http://cran.r-project.org/bin/windows/Rtools/

says to use the latest R and latest Rtools.

The latest R appears to be 2.15 (I'm already
a release behind and I just reinstalled last week!).
The toolchain Rtools215.exe is listed as "not frozen".

I take it that means they're giving us a moving target?

And do you think 215 is the one to use?

And what in the world do they mean by "releases after 2.14.1 will
use a new toolchain based on pre-4.6.3 gcc"? Are they really
using a pre-relese version of an old GCC?

> Also, I thought I got Jenkins working again last night ("Jenkins build is back to normal") but this morning it says it
> is broken again in a different place with no intervening commits. Is Jenkins quantum?

No, but I'm pretty sure there can be false positives. It
might be a timeout issue? I'm pretty sure I've seen false
positives as in I commit some doc and it says the build's fixed
or vice versa (submit doc and it says build is broken).

- Bob

[Ben Goodrich]

On Saturday, March 31, 2012 3:37:16 PM UTC-4, Bob Carpenter wrote:

Is it OK with everyone if we only support the latest
version of R?  There's a new one out as of yesterday,
2.15, and they changed versions of GCC.

There are "hard" and "soft" version requirements for R, and I would be fine with a soft one. In other words, we can say in the README or whatever that stan will (eventually) work on the latest released version of R *for Windows*, presumably with the latest released version of Rtools. This is not too much of a burden because it is pretty easy to update R on a personal Windows machine.

Alternatively, the hard route would require a certain version of R for *all operating systems* by specifying R 2.15 in the Requires section of the DESCRIPTION file. I think this would be overkill because it would preclude Linux / Mac / Solaris / etc. users with older versions of R but new enough versions of g++ from using stan. And these people are more likely to be using R via a server where they may not have administrative rights to update R.
 

On 3/31/12 3:17 PM, Ben Goodrich wrote:
> On Saturday, March 31, 2012 1:47:45 PM UTC-4, Bob Carpenter wrote:
The toolchain Rtools215.exe is listed as "not frozen".

I take it that means they're giving us a moving target?

And do you think 215 is the one to use?

215 is definitely the one to use in conjunction with the latest R. The timestamps on the files inside multi.zip are mostly December with a few January. It may become frozen by the time rstan is released.
 

And what in the world do they mean by "releases after 2.14.1 will
use a new toolchain based on pre-4.6.3 gcc"?  Are they really
using a pre-relese version of an old GCC?

Yes, although I don't know why. I am guessing that they cannot afford downtime on the CRAN servers so if they find a version of a compiler that mostly works with all 3000 R packages they have to build and rebuild, then they stick with it until forced to upgrade.
 

> Also, I thought I got Jenkins working again last night ("Jenkins build is back to normal") but this morning it says it
> is broken again in a different place with no intervening commits. Is Jenkins quantum?

No, but I'm pretty sure there can be false positives.  It
might be a timeout issue?  I'm pretty sure I've seen false
positives as in I commit some doc and it says the build's fixed
or vice versa (submit doc and it says build is broken).

Okay, now it is saying that MSVC makes a file that is too big when trying to build the parser, but I don't think I did anything to affect the settings in the MSVC case.

Ben

[Bob Carpenter]

Thanks for all the feedback on R versions. The "soft"
version makes most sense to me. We'll hear about it from
users no matter what we say, and it's not like we're giving
them support contracts.

On 3/31/12 4:04 PM, Ben Goodrich wrote:

> Okay, now it is saying that MSVC makes a file that is too big when trying to build the parser, but I don't think I did
> anything to affect the settings in the MSVC case.

Hmm. That's the problem we were having before.

These C++ compilers sure are squirrely.

- Bob

[Jiqiang Guo]

I should have said that I could help test building on windows.  I have windows 7 64bits and windows xp 32 bits.  But I guess the issue to me is that there is not a shell on which `make` could run as on Unix-like system. 

--

Jiqiang 

[Ben Goodrich]

On Saturday, March 31, 2012 6:51:22 PM UTC-4, Jiqiang Guo wrote:

I should have said that I could help test building on windows.  I have windows 7 64bits and windows xp 32 bits.  But I guess the issue to me is that there is not a shell on which `make` could run as on Unix-like system. 

If you install the latest Rtools from

http://cran.r-project.org/bin/windows/Rtools/

it puts the command-line tools (plus a compiler and other stuff) listed here

http://cran.r-project.org/doc/manuals/R-admin.html#The-command-line-tools

on your PATH. So, that should be enough to get you started once you also download git for Windows from

http://git-scm.com/download

Ben

[Jiqiang Guo]

As instructed, I installed Rtools215 and installed git.  Under the git-bash, `make test-all' fails with log message at http://justpaste.it/uwb. `make test-bugs' is still testing models after stanc is built successfully. 

Gcc in Rtools215: 

$ gcc -v

Using built-in specs.

COLLECT_GCC=c:\Rtools\gcc-4.6.3\bin\gcc.exe

COLLECT_LTO_WRAPPER=c:/rtools/gcc-4.6.3/bin/../libexec/gcc/i686-w64-mingw32/4.6.3/lto-wrapper.exe

Target: i686-w64-mingw32

Configured with: /data/gannet/ripley/Sources/mingw-test3/src/gcc/configure --host=i686-w64-mingw32 --build=x86_64-linux-gnu --target=i686-w64-mingw32 --with-sysroot=/data/gannet/ripley/Sources/mingw-test3/mingw32mingw32/mingw32 --prefix=/data/gannet/ripley/Sources/mingw-test3/mingw32mingw32/mingw32 --with-gmp=/data/gannet/ripley/Sources/mingw-test3/mingw32mingw32/prereq_install --with-mpfr=/data/gannet/ripley/Sources/mingw-test3/mingw32mingw32/prereq_install --with-mpc=/data/gannet/ripley/Sources/mingw-test3/mingw32mingw32/prereq_install --disable-shared --enable-static --enable-targets=all --enable-languages=c,c++,fortran --enable-libgomp --enable-sjlj-exceptions --enable-fully-dynamic-string --disable-nls --disable-werror --enable-checking=release --disable-win32-registry --disable-rpath --disable-werror CFLAGS='-O2 -mtune=core2 -fomit-frame-pointer' LDFLAGS=

Thread model: win32

gcc version 4.6.3 20111208 (prerelease) (GCC) 

Jiqiang 

[Bob Carpenter]

This is very encouraging. It gets through all the
unit tests other than some of the model compilations.

I have no idea what a "STATUS_ACCESS_VIOLATION" is,
but it looks like it's related to permissions:

http://gaoithe.blogspot.com/2011/03/solved-cygwin-windows7.html

That would make sense as these tests need to read rather
than just run.

- Bob

On 3/31/12 10:57 PM, Jiqiang Guo wrote:
> As instructed, I installed Rtools215 and installed git. Under the git-bash, `make test-all' fails with log message at
> http://justpaste.it/uwb. `make test-bugs' is still testing models after stanc is built successfully.
>
> Gcc in Rtools215:
>
> $ gcc -v
> Using built-in specs.
> COLLECT_GCC=c:\Rtools\gcc-4.6.3\bin\gcc.exe
> COLLECT_LTO_WRAPPER=c:/rtools/gcc-4.6.3/bin/../libexec/gcc/i686-w64-mingw32/4.6.3/lto-wrapper.exe
> Target: i686-w64-mingw32
> Configured with: /data/gannet/ripley/Sources/mingw-test3/src/gcc/configure --host=i686-w64-mingw32
> --build=x86_64-linux-gnu --target=i686-w64-mingw32
> --with-sysroot=/data/gannet/ripley/Sources/mingw-test3/mingw32mingw32/mingw32
> --prefix=/data/gannet/ripley/Sources/mingw-test3/mingw32mingw32/mingw32
> --with-gmp=/data/gannet/ripley/Sources/mingw-test3/mingw32mingw32/prereq_install
> --with-mpfr=/data/gannet/ripley/Sources/mingw-test3/mingw32mingw32/prereq_install
> --with-mpc=/data/gannet/ripley/Sources/mingw-test3/mingw32mingw32/prereq_install --disable-shared --enable-static
> --enable-targets=all --enable-languages=c,c++,fortran --enable-libgomp --enable-sjlj-exceptions
> --enable-fully-dynamic-string --disable-nls --disable-werror --enable-checking=release --disable-win32-registry
> --disable-rpath --disable-werror CFLAGS='-O2 -mtune=core2 -fomit-frame-pointer' LDFLAGS=
> Thread model: win32
> gcc version 4.6.3 20111208 (prerelease) (GCC)
>
>
>
> Jiqiang
>

[Ben Goodrich]

On Saturday, March 31, 2012 10:57:15 PM UTC-4, Jiqiang Guo wrote:

As instructed, I installed Rtools215 and installed git.  Under the git-bash, `make test-all' fails with log message at http://justpaste.it/uwb. `make test-bugs' is still testing models after stanc is built successfully. 

That is good. If stanc compiles and can usually do the right thing, we can probably get CRAN to accept rstan.

I can't get anything useful from Jenkins. Under -O0, the usual error occurs where "File is too large". Under any other optimization level, Jenkins runs out of RAM with -j4.

Ben

[Daniel Lee]

Sorry I've been pretty silent on this issue.

I'll take a look at our Jenkins box and try to figure out what's going on.

Ben, do you want administrative access over the Jenkins process?

[Ben Goodrich]

Can we just slow its roll to -j2?

On Saturday, March 31, 2012 11:25:50 PM UTC-4, Daniel Lee wrote:

Sorry I've been pretty silent on this issue.

I'll take a look at our Jenkins box and try to figure out what's going on.

Ben, do you want administrative access over the Jenkins process?

[Daniel Lee]

Of course. Looks like the machine is down. I'll go do it on the box on Monday.

[Ben Goodrich]

On Saturday, March 31, 2012 11:02:43 PM UTC-4, Bob Carpenter wrote:

I have no idea what a "STATUS_ACCESS_VIOLATION" is,
but it looks like it's related to permissions:

http://gaoithe.blogspot.com/2011/03/solved-cygwin-windows7.html

Possibly permissions (maybe try as Administrator?), possibly anti-virus (disable internet and turn-off AV?), possibly conflicting with a previous installation of Cygwin stuff (uninstall old or try rebaseall), possibly trying to access uninitialized memory (I just pushed a change to make/win that unsurpressed some uninitialized warnings on Windows), or possibly something else. But most likely we will be able to fix it somehow.

Ben

[Ben Goodrich]

With -01 only, I am getting the test failure below with mingw or pure g++ but not clang.

Ben

master!stan> wine test/math/error_handling.exe
Running main() from gtest_main.cc
[==========] Running 21 tests from 1 test case.

[----------] Global test environment set-up.

[----------] 21 tests from MathErrorHandling
[ RUN      ] MathErrorHandling.CheckNotNanDefaultPolicy
[       OK ] MathErrorHandling.CheckNotNanDefaultPolicy (0 ms)
[ RUN      ] MathErrorHandling.CheckNotNanErrnoPolicy
src/test/math/error_handling_test.cpp:33: Failure
Value of: std::isnan (result)
  Actual: true
Expected: false
check_not_nan should not have returned nan: inf
src/test/math/error_handling_test.cpp:37: Failure
Value of: std::isnan (result)
  Actual: true
Expected: false
check_not_nan should have returned nan: -inf
[  FAILED  ] MathErrorHandling.CheckNotNanErrnoPolicy (1 ms)
[ RUN      ] MathErrorHandling.CheckNotNanDefaultPolicyDefaultResult
[       OK ] MathErrorHandling.CheckNotNanDefaultPolicyDefaultResult (0 ms)
[ RUN      ] MathErrorHandling.CheckFiniteDefaultPolicy
[       OK ] MathErrorHandling.CheckFiniteDefaultPolicy (0 ms)
[ RUN      ] MathErrorHandling.CheckFiniteErrnoPolicy
[       OK ] MathErrorHandling.CheckFiniteErrnoPolicy (1 ms)
[ RUN      ] MathErrorHandling.CheckFiniteVectorDefaultPolicy
[       OK ] MathErrorHandling.CheckFiniteVectorDefaultPolicy (0 ms)
[ RUN      ] MathErrorHandling.CheckFiniteVectorErrnoPolicy
[       OK ] MathErrorHandling.CheckFiniteVectorErrnoPolicy (0 ms)
[ RUN      ] MathErrorHandling.CheckBoundedDefaultPolicyX
[       OK ] MathErrorHandling.CheckBoundedDefaultPolicyX (0 ms)
[ RUN      ] MathErrorHandling.CheckBoundedDefaultPolicyLow
[       OK ] MathErrorHandling.CheckBoundedDefaultPolicyLow (0 ms)
[ RUN      ] MathErrorHandling.CheckBoundedDefaultPolicyHigh
[       OK ] MathErrorHandling.CheckBoundedDefaultPolicyHigh (0 ms)
[ RUN      ] MathErrorHandling.CheckBoundedErrnoPolicyX
[       OK ] MathErrorHandling.CheckBoundedErrnoPolicyX (0 ms)
[ RUN      ] MathErrorHandling.CheckBoundedErrnoPolicyLow
[       OK ] MathErrorHandling.CheckBoundedErrnoPolicyLow (0 ms)
[ RUN      ] MathErrorHandling.CheckBoundedErrnoPolicyHigh
[       OK ] MathErrorHandling.CheckBoundedErrnoPolicyHigh (0 ms)
[ RUN      ] MathErrorHandling.CheckGreaterDefaultPolicy
[       OK ] MathErrorHandling.CheckGreaterDefaultPolicy (0 ms)
[ RUN      ] MathErrorHandling.CheckGreaterErrnoPolicy
[       OK ] MathErrorHandling.CheckGreaterErrnoPolicy (0 ms)
[ RUN      ] MathErrorHandling.CheckGreaterOrEqualDefaultPolicy
[       OK ] MathErrorHandling.CheckGreaterOrEqualDefaultPolicy (0 ms)
[ RUN      ] MathErrorHandling.CheckGreaterOrEqualErrnoPolicy
[       OK ] MathErrorHandling.CheckGreaterOrEqualErrnoPolicy (0 ms)
[ RUN      ] MathErrorHandling.CheckLessDefaultPolicy
[       OK ] MathErrorHandling.CheckLessDefaultPolicy (1 ms)
[ RUN      ] MathErrorHandling.CheckLessErrnoPolicy
[       OK ] MathErrorHandling.CheckLessErrnoPolicy (0 ms)
[ RUN      ] MathErrorHandling.CheckLessOrEqualDefaultPolicy
[       OK ] MathErrorHandling.CheckLessOrEqualDefaultPolicy (0 ms)
[ RUN      ] MathErrorHandling.CheckLessOrEqualErrnoPolicy
[       OK ] MathErrorHandling.CheckLessOrEqualErrnoPolicy (0 ms)
[----------] 21 tests from MathErrorHandling (4 ms total)

[----------] Global test environment tear-down

[==========] 21 tests from 1 test case ran. (5 ms total)
[  PASSED  ] 20 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] MathErrorHandling.CheckNotNanErrnoPolicy

 1 FAILED TEST
master!stan>

[Jiqiang Guo]

Whatever the issue cause that (Exception: STATUS_
ACCESS_VIOLATION), it is from mingw32 (or more specifically, the mingw32 packaged by Git on windows).  I tried to build just using windows `cmd', all went well (uname is not found, but it is fine). 

This is a link to a snapshot of cmd and gitbash, giving you an idea of what are on windows:

https://lh4.googleusercontent.com/-KK_eGTwsORg/T3iHfiyh00I/AAAAAAAAADE/sb5eOxa91yQ/w566-h400-k/gitbash_n_cmd.png

Jiqiang 

[Ben Goodrich]

On Sunday, April 1, 2012 12:52:50 PM UTC-4, Jiqiang Guo wrote:

Whatever the issue cause that (Exception: STATUS_
ACCESS_VIOLATION), it is from mingw32 (or more specifically, the mingw32 packaged by Git on windows).  I tried to build just using windows `cmd', all went well (uname is not found, but it is fine). 

This is a link to a snapshot of cmd and gitbash, giving you an idea of what are on windows:

https://lh4.googleusercontent.com/-KK_eGTwsORg/T3iHfiyh00I/AAAAAAAAADE/sb5eOxa91yQ/w566-h400-k/gitbash_n_cmd.png

If it is just an issue with gitbash, that is not a problem that would prevent us from putting it on CRAN. But when you do it with gitbash, does sh.exe.stackdump do or say anything informative? Probably not, judging by the other reports I see when Googling

STATUS_ACCESS_VIOLATION git Windows

But maybe one of the first few links will help you?

Ben

[Daniel Lee]

That's not good. I'm having some memory issues under O=1 (unrelated).

[Daniel Lee]

I'm moving -static-libgcc and -static-libstdc++ to LDFLAGS. The makefile defaults treat those as linker options. Let me know if it breaks on your machine and I'll just change it back since there is no functional difference.

[Ben Goodrich]

That works fine on my machine. Although it seems that on Jenkins, -static-libstdc++ is a hard-coded default, so providing as a CFLAG results in an error.

[Daniel Lee]

Sorry, I'm confused. Where do you see this error?

[Ben Goodrich]

https://groups.google.com/d/msg/stan-buildbot/bnfSPfX87kE/nRlcVSCgO2YJ

I did that over the weekend when I put -static-libstdc++ under CFLAGS and then it worked downthread when I reverted. I also Googled something that explained it to me but I don't remember exactly what.

Ben

On Monday, April 2, 2012 4:14:41 PM UTC-4, Daniel Lee wrote:

Sorry, I'm confused. Where do you see this error?

[Daniel Lee]

Ah. I let most of that slide.

I think one of the issues is O=1. I am having that out of memory problem with the optimization level set to 1.

[Ben Goodrich]

Yes, I hit out of memory problems with -j4 and 0 > 0 on Jenkins. But I would imagine -j2 is sufficient to see the other errors :) .

Ben

[Daniel Lee]

The issue on the Jenkins configuration is O > 0. cc1plus.exe runs out of memory (that's what g++ calls).

[Ben Goodrich]

Even with serial compilation? Maybe we should stuff more RAM into Jenkins.

On Monday, April 2, 2012 5:09:24 PM UTC-4, Daniel Lee wrote:

The issue on the Jenkins configuration is O > 0. cc1plus.exe runs out of memory (that's what g++ calls).

[Daniel Lee]

It's not a Jenkins problem. It's a problem with the version of g++ I have.

[Daniel Lee]

Looks like it's a 32 bit compiler. Hopefully the rtools g++ is 64 bit. I'll try to install it this week.

[Ben Goodrich]

On Monday, April 2, 2012 5:40:56 PM UTC-4, Daniel Lee wrote:

Looks like it's a 32 bit compiler. Hopefully the rtools g++ is 64 bit. I'll try to install it this week.

Rtools comes with both and is based directly on

http://mingw-w64.sourceforge.net/

rather than the slightly less direct

http://tdm-gcc.tdragon.net/

See

http://cran.r-project.org/bin/windows/Rtools/

for more details.

Ben

[Bob Carpenter]

So anyone with a 32-bit Windows machine will
be out of luck trying to compile multivariate
models at -O3. We just watched it crash at the 2GB
mark, which is all 32-bit hardware supports in
terms of process size (the OS/motherboard reserves
the other 2GB of potential address space).

The key to using less space in the compiler is
cutting down on the size of our translation units.
That doesn't play nicely with all of our header-only
template libs (Boost, Eigen, and the way we currently
have Stan organized).

I'm pretty sure it's possible to refactor the parser.
There are examples in the Boost parser doc:

http://www.boost.org/doc/libs/1_49_0/libs/spirit/example/qi/compiler_tutorial/

I haven't been able to refactor the parser so that
it uses explicit instantiation rather than being
header only. There's a simple working example of how to
do explicit instantiation with doc in

src/demo/templ_inst_demo

So far, I can't get it to link. The refactoring of the
grammars is in the subdirectory

stan/gm/grammars

I just added a couple of files that aren't being
used:

src/stan/gm/ast_header_def_unused_attempt.hpp
src/stan/gm/ast_header_unused_attempt.hpp

in which I tried to refactor the abstract
syntax tree (the data structure created by the parser
to represent a Stan program).

I can compile .o files, but I can't link. Rather than duplicate
defs, I get missing defs this time. I think it may
have something to do with the remaining templates in
ast.hpp, which could either be explicitly instantiated
during compilation with another file like the .cpp files
in the stan/gm/grammars directory or
replaced with their instantiations directly in the code.

- Bob

[Ben Goodrich]

On Monday, April 2, 2012 6:09:37 PM UTC-4, Bob Carpenter wrote:

The key to using less space in the compiler is
cutting down on the size of our translation units.

Don't totally understand but am all in favor of using less RAM. A complimentary route might be to figure out why clang++ uses a fraction of the RAM that g++ or especially mingw use and maybe there is a way to hint the compiler in the right direction?

Ben
 

[Bob Carpenter]

Let me try to explain.

C and C++ compilation involves two stages:

1. Building one or more relocatable object files (.o)

> clang++ -c foo.cpp
> ls foo.o

> clang++ -c bar.cpp
> ls bar.o

2. Linking the relocatable object files into an executable:

> clang++ foo.o bar.o -o myprog
> ls myprog

The unit through which a relocatable object file is
built is called a "translation unit". In the first
example, that's the contents of "foo.cpp", with all of
the (recursive) includes folded in. In the second, it's
the same for "bar.cpp".

Now, to achieve this, we need to split definitions and
declarations.

A. Declaration (traditionally in a .h file)
int foo(int x);

B. Definition (traditionally in a .c file)
int foo(int x) { return x + 1; }

In order to generate relocatable object files, we only need
the declarations of functions/variables on which we depend.

Here's the key:

In order to link, every function and constant needs to be
DEFINED in EXACTLY ONE of the .o files.

Templates mess all of this up. They're halfway between
declarations and definitions. It's only when you specify the
type for a template that you get a definition (or declaration).

So template libs, like Eigen and Boost, are often what's
called "header only", meaning that they put the declarations
and definitions in a single file and everything gets built
as one big translation unit.

That's what Stan does now. For instance, the following
does both the compilation and linking:

> clang++ baz.cpp -o myprog

but it requires baz to import DEFINITIONS of everything
it needs (other than the std libs, which presumably get
linked in as necessary).

The header guards on the .hpp files make sure that each
item is only defined once.

So what we want to do is break the parser and perhaps
the models into pieces, one with a declaration, and one
with a definition.

Because templates don't have a definition until they're
instantiated, C++ lets you create a third type of file called
a template instantiation file. This is usually a .cpp
file that indicates which instantiations of the templates
you want. It then plays the role of the definition file,
getting compiled separately and then linked in later against
a .o file compiled by including only the declarations.

The simplest way to understand this in action is to look
at the example in:

stan/src/demo/templ_inst_demo

where I provide a very simple textbook example.

- Bob

[Ben Goodrich]

On Monday, April 2, 2012 6:09:37 PM UTC-4, Bob Carpenter wrote:

So anyone with a 32-bit Windows machine will
be out of luck trying to compile multivariate
models at -O3.  We just watched it crash at the 2GB
mark, which is all 32-bit hardware supports in
terms of process size (the OS/motherboard reserves
the other 2GB of potential address space).

You can unlock another GB on Jenkins by following the instructions at

http://msdn.microsoft.com/en-us/library/bb613473%28VS.85%29.aspx

although that does not really solve the fundamental problem with templates (which I understand a bit more now).

Ben

[Matt Hoffman]

Fascinating. This trick doesn't seem to be particularly well
documented on the internet—I guess there's a good reason to buy books
after all!

I'm getting tired of waiting for small programs to compile, so maybe
I'll start breaking things out according to this template over the
next few days.

Matt

[Bob Carpenter]

Good news. I finished the refactoring of the
parser with explicit instantiation. It took some
fighting because it was pretty template-heavy and
recursive code, but it works.

More inline below.

On 4/7/12 2:26 PM, Matt Hoffman wrote:
> Fascinating. This trick doesn't seem to be particularly well

> documented on the internet�I guess there's a good reason to buy books
> after all!

It was only in the uber-template book (by two guys who
were on the spec team) that's sitting on Daniel's desk
(Vandevoorde and Josuttis).

>
> I'm getting tired of waiting for small programs to compile, so maybe

I also think it's a big wall time disadvantage against BUGS
or JAGS, because they load small models quickly. But BUGS and JAGS
slow down very quickly as data sets get larger.

> I'll start breaking things out according to this template over the
> next few days.

We'll need to change the makefiles to accomodate.
I'm going to wait until I can get Daniel in person
to fix the build as I check it in so we don't kill
everyone's compiler.

And if I didn't mention it in the rant below, there's
a fully worked out example in a directory under src/demo.
It was just the way I got hello world going, so I
thought I'd check it in.

I'm pretty sure the way we're using Eigen through
a bunch of explicitly defined functions (in
stan/math/matrix.hpp and stan/agrad/matrix.hpp)
will let us do that with Eigen, too.

I could probably also do something with the code
generator so we don't even include the Eigen libraries
if we don't have any vector or matrix data types in
the Stan program. It should be pretty easy.

Although I don't think he ever brought it up on the
list, Daniel got precompiled headers working, but with
these header-only libs, that only reduced compile time
by 5 or 10%, so didn't seem worth it.

It might be a bigger percentage reduction if we use the
inclusion model, but probably still won't be a huge
amount of wall time.

- Bob

[Ben Goodrich]

On Saturday, April 7, 2012 8:17:10 PM UTC-4, Bob Carpenter wrote:

Although I don't think he ever brought it up on the
list, Daniel got precompiled headers working, but with
these header-only libs, that only reduced compile time
by 5 or 10%, so didn't seem worth it.

CRAN won't accept pre-compiled anything for security reasons, although maybe would could make it a disabled-by-default option for people who build rstan from source.

Ben

[Matt Hoffman]

I think the "pre" in "precompiled" here just means prior to link time.
Users and/or CRAN should (hopefully!) be able to precompile their own
headers, so we won't have to distribute binaries.

Matt

[Bob Carpenter]

It's a moot point because we decided not to use them.

The name "precompiled headers" is misleading. You'd think
it would mean what Matt said, namely precompiled object (.o) files.
But it doesn't. Instead, it's a way for the
compiler to cache the parsing it does of #includes.
That way, the next translation unit you get to, you can lookup
the parsed form of all the header includes up to the point where
the includes diverge.

We're going to use "explicit template instantiation". That's
just the template version of separating out a pure
declaration header file from the definition source. This
will let us compile a bunch of smaller .o files separately
and then link them together.

I'm hoping that's kosher on CRAN -- they don't require
header-only libs, do they?

- Bob

[Ben Goodrich]

On Sunday, April 8, 2012 6:27:39 PM UTC-4, Bob Carpenter wrote:

We're going to use "explicit template instantiation".  That's
just the template version of separating out a pure
declaration header file from the definition source.  This
will let us compile a bunch of smaller .o files separately
and then link them together.

I'm hoping that's kosher on CRAN -- they don't require
header-only libs, do they?

That should be fine. CRAN just requires that uploaded packages be source only. The CRAN servers compile binaries for Windows (both 32bit and 64bit) and Macs (32bit and 64bit for Intel and 32bit PPC). In our case, probably the only executable that needs to be compiled is stanc.

Ben

[Daniel Lee]

On Sun, Apr 8, 2012 at 6:27 PM, Bob Carpenter <ca...@alias-i.com> wrote:

It's a moot point because we decided not to use them.

They are still in there. We had to comment it out of clang++ for mac and g++ for windows because the compilers would choke. I wonder if the Rtools 2.15 g++ can handle it now.

[Daniel Lee]

On Sat, Apr 7, 2012 at 8:17 PM, Bob Carpenter <ca...@alias-i.com> wrote:

Good news.  I finished the refactoring of the
parser with explicit instantiation.  It took some
fighting because it was pretty template-heavy and
recursive code, but it works.

Awesome!
 

We'll need to change the makefiles to accomodate.
I'm going to wait until I can get Daniel in person
to fix the build as I check it in so we don't kill
everyone's compiler.

We'll take care of the makefile this week.
 

Although I don't think he ever brought it up on the
list, Daniel got precompiled headers working, but with
these header-only libs, that only reduced compile time
by 5 or 10%, so didn't seem worth it.

Yeah, we didn't see too much improvement in compile time. Although, we didn't go through all the permutations of OS, compiler, and optimization level.