Testing has changed on Stan develop

[Daniel Lee]

See https://github.com/stan-dev/stan/wiki/Testing-Stan-using-Gnu-Make-and-Python

I forgot to notify everyone when I merged the pull request earlier today.

[Bob Carpenter]

I just verified that it works.

Although the new version's much more verbose for running single tests,
the upside is that now I can get autocomplete from the shell.

- Bob

On Sep 28, 2014, at 6:45 PM, Daniel Lee <bea...@alum.mit.edu> wrote:

> See https://github.com/stan-dev/stan/wiki/Testing-Stan-using-Gnu-Make-and-Python
>
> I forgot to notify everyone when I merged the pull request earlier today.
>
>

> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Ben Goodrich]

On Sunday, September 28, 2014 6:45:31 PM UTC-4, Daniel Lee wrote:

See https://github.com/stan-dev/stan/wiki/Testing-Stan-using-Gnu-Make-and-Python

I forgot to notify everyone when I merged the pull request earlier today.

This looks good. Although as long as we are requiring python for developers, we might as well use python to shuffle the distribution tests when running them on a single machine.

What needs to be done to make this work on Yeti? Basically, what I was doing before was to build on the dependencies on the submit node and then pass each individual test to a Torque job array. Ideally, it would also be best if I could send each batch of 100 distribution tests to one worker of a job array rather than sending all the distribution tests for one distribution to a worker of a job array.

Ben
 

[Daniel Lee]

On Sun, Sep 28, 2014 at 7:11 PM, Ben Goodrich <goodri...@gmail.com> wrote:

On Sunday, September 28, 2014 6:45:31 PM UTC-4, Daniel Lee wrote:

See https://github.com/stan-dev/stan/wiki/Testing-Stan-using-Gnu-Make-and-Python

I forgot to notify everyone when I merged the pull request earlier today.

This looks good. Although as long as we are requiring python for developers, we might as well use python to shuffle the distribution tests when running them on a single machine.

Sure -- for building or for executing?

 

What needs to be done to make this work on Yeti? Basically, what I was doing before was to build on the dependencies on the submit node and then pass each individual test to a Torque job array. Ideally, it would also be best if I could send each batch of 100 distribution tests to one worker of a job array rather than sending all the distribution tests for one distribution to a worker of a job array.

What do you mean "build on the dependencies"? Can you spec out what you want to do in more concrete detail? I say we build a separate script for yeti, but I'm not exactly sure what you want.

[Daniel Lee]

Ben, btw, I was able to run all the jobs on Yeti using this set of scripts:
https://github.com/syclik/yeti-scripts

(let me know if you can't see it)

It broke every compile to a different qsub job (well -- all the tests as a single array job). When I ran it last, everything executed in an hour or two. (I forget the time -- I couldn't get the notification to work properly either.)




Daniel

[Bob Carpenter]

What's the single slowest test we have? Shouldn't we aim to get
the cluster time be a bit more than that?

Is there a way to schedule jobs so that CPUs take them as needed.
I'd think a random grab of tests by CPUs would have good expected
behavior. For instance, if you could use map-reduce where each
test was mapped and then the reduce just appends the outputs into a report.

- Bob

[Ben Goodrich]

On Monday, September 29, 2014 1:37:33 PM UTC-4, Daniel Lee wrote:

On Sun, Sep 28, 2014 at 7:11 PM, Ben Goodrich <goodri...@gmail.com> wrote:

On Sunday, September 28, 2014 6:45:31 PM UTC-4, Daniel Lee wrote:

See https://github.com/stan-dev/stan/wiki/Testing-Stan-using-Gnu-Make-and-Python

I forgot to notify everyone when I merged the pull request earlier today.

This looks good. Although as long as we are requiring python for developers, we might as well use python to shuffle the distribution tests when running them on a single machine.

Sure -- for building or for executing?

Building since the execution time / RAM required is pretty negligible. The main thing is to get it all to build without swapping.
 

 What needs to be done to make this work on Yeti? Basically, what I was doing before was to build on the dependencies on the submit node and then pass each individual test to a Torque job array. Ideally, it would also be best if I could send each batch of 100 distribution tests to one worker of a job array rather than sending all the distribution tests for one distribution to a worker of a job array.

What do you mean "build on the dependencies"? Can you spec out what you want to do in more concrete detail? I say we build a separate script for yeti, but I'm not exactly sure what you want.

I meant to say "build all the dependencies". So, it was

1. make clean-all > /dev/null
   
2. make test/libgtest.a 
3. make bin/libstan.a
4. make bin/libstanc.a
5. make test/dummy.cpp
6. make test/unit-distribution/generate_tests
7. find src/test/unit-distribution -type f -name "*_test.hpp" -exec ./test/unit-distribution/generate_tests {} \;

Then pass all of the tests to the job array, but actually it had to be done in two batches because there is a size limit to a job array. We might want to instead pass each folder to a worker of a job array and have it run all the tests in that folder. Then it could be run in one batch but the wall time would not be optimal if some folders take a lot longer than others to run all the tests.

Ben

 

[Ben Goodrich]

On Monday, September 29, 2014 4:07:47 PM UTC-4, Bob Carpenter wrote:

What's the single slowest test we have?  Shouldn't we aim to get
the cluster time be a bit more than that?  

I think we should be able to get it down well under an hour if the cluster is mostly idle. I think the slowest tests nowdays are some of the distribution tests, which can take a few minutes each to compile but run almost instantly. The problem is that the slowest tests tend to be for the same distributions, so if we pass a whole directory then it would take a while for the longest workers to finish.
 

Is there a way to schedule jobs so that CPUs take them as needed.
I'd think a random grab of tests by CPUs would have good expected
behavior.  For instance, if you could use map-reduce where each
test was mapped and then the reduce just appends the outputs into a report.

There is some randomness as to what order it does the tests in. It seemed to work okay before but wasn't fully optimized. I don't think Yeti has map-reduce integrated with Torque but previously we had 1 worker per test and it spit out a file for standard output and another for standard error. If any failed, I was just cat-ing the failed reports together.

Ben

[Bob Carpenter]

On Sep 29, 2014, at 6:02 PM, Ben Goodrich <goodri...@gmail.com> wrote:

> ...I don't think Yeti has map-reduce integrated with Torque but previously we had 1 worker per test and it spit out a file for standard output and another for standard error. If any failed, I was just cat-ing the failed reports together.

Cool --- you just built your own map-reduce!

- Bob