make test-bugs

[Daniel Lee]

Hi all,

The test-bugs target has changed a bit. The examples now run within
google tests. This allows us to record execution time to be displayed
through Jenkins. See:
http://d1m1s1b1.stat.columbia.edu:8080/job/Stan%20Model%20Tests/

There is also a test-models target that will run tests for the other
*.stan files in src/models/

The downside is that the unit tests are generated by hand. test-models
will indicate whether any Stan models are in src/models but don't have
associated tests. Once we have the chains object up and running, we
can get the automatic testing we've been planning. The performance
plugin for Jenkins tracks percentage of failed tests, so perhaps we
don't need to worry too much about "failures" of individual tests.






Daniel

[Ben Goodrich]

I lost the ability to do test-models in parallel via

make CC=clang++ O=3 -j8 test-models

If I do that I get

make[1]: warning: jobserver unavailable: using -j1.  Add `+' to parent make rule.

Googling points me to the end of

https://www.gnu.org/software/make/manual/html_node/Error-Messages.html

Ben

[Daniel Lee]

Do you have the same behavior for CC=g++? I'm not seeing the problem
with g++ for both windows and mac.



Daniel

[Ben Goodrich]

It occurs at least with g++-4.7 and clang 3.1 but not right away. make (version 3.81) test-models goes in parallel for a while but then reverts to serial once it hits an error. Here is the first occurrence:

cp: cannot stat `src/models/transforms/correlation_matrix.Rdata': No such file or directory
make: [models/transforms/correlation_matrix] Error 1 (ignored)

make[1]: warning: jobserver unavailable: using -j1.  Add `+' to parent make rule.

Ben

On Friday, May 25, 2012 1:00:41 PM UTC-4, Daniel Lee wrote:

Do you have the same behavior for CC=g++? I'm not seeing the problem
with g++ for both windows and mac.



Daniel

[Daniel Lee]

It's interesting that it would do that after hitting the first ignored error.  I'll take a closer look. 

There may be a way to rewrite the rule differently that would be more make friendly.

[Daniel Lee]

Ben, I just updated the Makefiles. The copy is now conditional on the file existing. Please let me know if you still see those errors. 

On May 25, 2012 2:40 PM, "Daniel Lee" <djsy...@gmail.com> wrote:

It's interesting that it would do that after hitting the first ignored error.  I'll take a closer look. 

There may be a way to rewrite the rule differently that would be more make friendly.

On May 25, 2012 2:26 PM, "Ben Goodrich" <goodri...@gmail.com> wrote:

[Ben Goodrich]

Now it just errors out with

--- Copying data ---
cp src/models/basic_estimators/normal_loc.Rdata models/basic_estimators/normal_loc.Rdata
cp: cannot stat `src/models/basic_estimators/normal_loc.Rdata': No such file or directory
make: *** [models/basic_estimators/normal_loc.stan] Error 1

Also, make/tests (re)defines CFLAGS_GTEST so whatever make/os_* does to CFLAGS_GTEST (like adding -lpthread in my case) has no effect.

Ben

On Tuesday, May 29, 2012 5:38:07 PM UTC-4, Daniel Lee wrote:

Ben, I just updated the Makefiles. The copy is now conditional on the file existing. Please let me know if you still see those errors. 

On May 25, 2012 2:40 PM, "Daniel Lee" wrote:

It's interesting that it would do that after hitting the first ignored error.  I'll take a closer look. 

There may be a way to rewrite the rule differently that would be more make friendly.

[Daniel Lee]

Sorry about that. Both issues should be fixed now.

By the way, do you have a linux box that's up and running most of the
time? Maybe it's time to think about working on parametrized builds
across different platforms.

[Daniel Lee]

On Tue, May 29, 2012 at 8:44 PM, Daniel Lee <bea...@alum.mit.edu> wrote:
> Sorry about that. Both issues should be fixed now.
>
> By the way, do you have a linux box that's up and running most of the
> time? Maybe it's time to think about working on parametrized builds
> across different platforms.
>

To give a little more context, I'm thinking about having a few Jenkins slaves:
https://wiki.jenkins-ci.org/display/JENKINS/Distributed+builds#Distributedbuilds-Havemasterlaunchslaveagentviassh

[Wei Wang]

I have a Ubuntu box sitting in my office mostly up.

[Bob Carpenter]

Is it possible to use virtual servers on a single
machine as slaves? I don't see why not, as they
should look just like everything else. Then we
might be able to get by with a single box.

- Bob

[Daniel Lee]

I'll give it a shot tomorrow. If we can't get it up and running in a
day, I think slaving will be easier.

[Ben Goodrich]

On Tuesday, May 29, 2012 8:44:14 PM UTC-4, Daniel Lee wrote:

Sorry about that. Both issues should be fixed now.

Unfortunately, neither were. In makefile, -include make/os_detect needs to be called *after* CC_TYPE is defined; otherwise it has no effect.

After fixing that, the models compile and link in parallel, but fallback to serial when running.

Running main() from gtest_main.cc
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from Models_BasicDistributions_NormalMixture
[ RUN      ] Models_BasicDistributions_NormalMixture.RunModel

make[1]: warning: jobserver unavailable: using -j1.  Add `+' to parent make rule.

STAN SAMPLING COMMAND
data = models/basic_estimators/normal_mixture.Rdata
init = random initialization
samples = models/basic_estimators/normal_mixture1.csv
append_samples = 0
seed = 1338343001 (randomly generated)
chain_id=1 (default)
iter = 2000
warmup = 1000
thin = 1
unit_mass_matrix = 0
leapfrog_steps = -1
max_treedepth = 10
epsilon = -1
epsilon_pm = 0
delta = 0.5
gamma = 0.05
 

By the way, do you have a linux box that's up and running most of the
time? Maybe it's time to think about working on parametrized builds
across different platforms.

I do, but wouldn't Hotfoot be better for that?

Ben

[Daniel Lee]

Thanks for letting me know about the CC_TYPE problem. I've pushed a
fix for that.

I'm glad it's building and linking in parallel. I don't think make
will try running in parallel. All the tests are run through the
test-models target in a for loop. test-models is a phony target, which
means that it runs the for loop every time it is called.

We had the tests run at the same time as it compiled, which would
allow for parallel execution. Unfortunately, that caused make to run
each test twice.

Any ideas on how to run the models in parallel?

[Ben Goodrich]

On Tuesday, May 29, 2012 10:18:03 PM UTC-4, Daniel Lee wrote:

Any ideas on how to run the models in parallel?

If you apply this patch:

diff --git a/make/tests b/make/tests
index d42df92..5a63cca 100644
--- a/make/tests
+++ b/make/tests
@@ -22,7 +22,7 @@ test-models: $(addsuffix $(EXE),$(MODEL_TESTS))
        @set -e;\
        $(foreach test,$(MODEL_TESTS), \
          echo '------------------------------------------------------------';\
-         $(addsuffix $(EXE),$(test)) --gtest_output="xml:$(test).xml";)
+         $(addsuffix $(EXE),$(test)) --gtest_output="xml:$(test).xml" & )
 ifneq ($(MISSING_MODEL_TESTS),)
        @echo
        @echo '-------------------- Missing $(words $(MISSING_MODEL_TESTS)) tests --------------------'

then the loop backgrounds the execution of the model and goes onto the next test. This is not ideal because

-- It does not respect the -j argument, so it ties up the machine
-- For me, it launches 16 threads across 8 virtual cores at about 50% CPU each, so not super-efficient
-- It makes a mess out of the output that gets piped to the shell

But, it is better than doing it serial particularly since these models take more than 25 minutes of CPU time

-- vol1/oxford
-- vol1/lsat
-- vol2/t_df/simt4
-- vol1/inhalers

Ben
 
 

[Daniel Lee]

Thanks for finding a solution. Unfortunately, I don't think running
the models in parallel is what we want.

If we did want to run them in parallel, I think the correct thing to
do is for test-models to depend on a bunch of phony targets. There
should be one phony target per test. Each phony target would depend on
the test it is responsible for running. The phony target's action
should be to run the test.

Unfortunately, that would still cause the output to get mangled, but
it would do what you want. Want to give it a try? Maybe give it a
different target -- like test-models-parallel?



Daniel

[Ben Goodrich]

On Wednesday, May 30, 2012 3:47:14 PM UTC-4, Daniel Lee wrote:

Thanks for finding a solution. Unfortunately, I don't think running
the models in parallel is what we want.

Well, I still don't think running 2000 iterations of these models automatically is telling us that much at the moment. But if someone wanted to do that, it would take most of a day to do them all serially. Maybe that is okay for Jenkins, but I doubt for a human.
 

If we did want to run them in parallel, I think the correct thing to
do is for test-models to depend on a bunch of phony targets. There
should be one phony target per test. Each phony target would depend on
the test it is responsible for running. The phony target's action
should be to run the test.

Unfortunately, that would still cause the output to get mangled, but
it would do what you want. Want to give it a try? Maybe give it a
different target -- like test-models-parallel?

I would have to read up a lot on makefiles. I've never done a complicated one.

Ben

[Daniel Lee]

On Wed, May 30, 2012 at 3:54 PM, Ben Goodrich <goodri...@gmail.com> wrote:
> Well, I still don't think running 2000 iterations of these models
> automatically is telling us that much at the moment. But if someone wanted
> to do that, it would take most of a day to do them all serially. Maybe that
> is okay for Jenkins, but I doubt for a human.

The tests were never designed with normal users in mind. I don't think
it's clear whether we will be shipping the tests with the standard
download.

> I would have to read up a lot on makefiles. I've never done a complicated
> one.

I don't think it's worth it. If you absolutely need to run things in
parallel, let me know.

[Daniel Lee]

The tests can now run in parallel.

We're able to do that now by having a phony target that looks like runtest/%. That target depends on whatever is in %, then runs the test. I should have thought about that before.



Daniel