test-bugs
Ben Goodrich
make test-bugs
is intended to accomplish at the moment.
It seems to me that we should either be fixing the --seed to some constant and testing whether the results are within rounding tolerance to what the results were before or passing --test_grad with a clock seed to see if the model compiles and can iterate once with auto-diff gradients within tolerance of finite-diff gradients.
At the moment, we do neither so it just spends a lot of time drawing a lot of samples from a lot of models and ignoring the output. I can see the value in looking at the stan results and comparing them (with a human) to the BUGS / JAGS results, but that is not going to get done every time someone does make test-bugs.
Or at least the makefile should have command-line options along these lines.
Opinions?
Ben
Daniel Lee
1) that the compiler is indeed compilable (bin/stanc)
2) that each BUGS model is able to be translated through the stanc compiler
3) that each BUGS model is compilable by the compiler
4) that each BUGS model runs to completion (that is, to 2000 samples)
Once the mcmc_output is created, we can write unit tests that will check the output of the stan results.
This target was created to run on Jenkins. It runs once a day just to make sure we can do the things above.
Ben Goodrich
Right now the test-bugs target verifies:
1) that the compiler is indeed compilable (bin/stanc)
2) that each BUGS model is able to be translated through the stanc compiler
3) that each BUGS model is compilable by the compiler
4) that each BUGS model runs to completion (that is, to 2000 samples)
Once the mcmc_output is created, we can write unit tests that will check the output of the stan results.
This target was created to run on Jenkins. It runs once a day just to make sure we can do the things above.
1) through 3) make sense. And 4) makes sense too in that you would want to see whether it runs into problems after many iterations. But 4) makes more sense for Jenkins than for a human, who primarily wants to quickly know if a change to the code results in any changes to the models.
So can we create a make target that does 1) through 3), then
3.5) execute model with --seed = 0 and --test_grad, check whether lp__ matches the last known correct value of lp__ and check whether the auto-diff gradient matches the finite-diff gradient
And then Jenkins would invoke an option to do 4) that humans typically wouldn't.
Ben
Daniel Lee
On Wednesday, April 11, 2012 12:31:35 PM UTC-4, Daniel Lee wrote:Right now the test-bugs target verifies:
1) that the compiler is indeed compilable (bin/stanc)
2) that each BUGS model is able to be translated through the stanc compiler
3) that each BUGS model is compilable by the compiler
4) that each BUGS model runs to completion (that is, to 2000 samples)
Once the mcmc_output is created, we can write unit tests that will check the output of the stan results.
This target was created to run on Jenkins. It runs once a day just to make sure we can do the things above.
1) through 3) make sense. And 4) makes sense too in that you would want to see whether it runs into problems after many iterations. But 4) makes more sense for Jenkins than for a human, who primarily wants to quickly know if a change to the code results in any changes to the models.
So can we create a make target that does 1) through 3), then
make models/bugs_examples/vol1/blocker/blocker
This works for all .stan models in the src/models subdirectory.
That target takes care of steps 1 to 3. Well, they also run the model as a convenience. We could remove the running portion of that target if that makes more sense.
3.5) execute model with --seed = 0 and --test_grad, check whether lp__ matches the last known correct value of lp__ and check whether the auto-diff gradient matches the finite-diff gradient
That sounds like a decent check. How would you suggest doing that? Creating an actual unit test? This is a really fragile test, though. If there are any underlying changes in implementation, the final lp__ could be off.
Bob and I were talking about implementing unit tests comparing known values to the estimated values taking into account standard errors, which relies on an effective sample size calculation. This would be done without specifying a seed.
And then Jenkins would invoke an option to do 4) that humans typically wouldn't.
The test-bugs target was supposed to be a target that humans typically would not call. Do you use this target? If so, what are you using the target for?
Daniel
Marcus Brubaker
On Wed, Apr 11, 2012 at 2:18 PM, Ben Goodrich <goodri...@gmail.com> wrote:On Wednesday, April 11, 2012 12:31:35 PM UTC-4, Daniel Lee wrote:Right now the test-bugs target verifies:
1) that the compiler is indeed compilable (bin/stanc)
2) that each BUGS model is able to be translated through the stanc compiler
3) that each BUGS model is compilable by the compiler
4) that each BUGS model runs to completion (that is, to 2000 samples)
Once the mcmc_output is created, we can write unit tests that will check the output of the stan results.
This target was created to run on Jenkins. It runs once a day just to make sure we can do the things above.
1) through 3) make sense. And 4) makes sense too in that you would want to see whether it runs into problems after many iterations. But 4) makes more sense for Jenkins than for a human, who primarily wants to quickly know if a change to the code results in any changes to the models.
So can we create a make target that does 1) through 3), thenWe have a target for each BUGS model. If we take the vol1/blocker/blocker.stan model, I believe you are looking for the following:
make models/bugs_examples/vol1/blocker/blocker
This works for all .stan models in the src/models subdirectory.
That target takes care of steps 1 to 3. Well, they also run the model as a convenience. We could remove the running portion of that target if that makes more sense.
3.5) execute model with --seed = 0 and --test_grad, check whether lp__ matches the last known correct value of lp__ and check whether the auto-diff gradient matches the finite-diff gradient
That sounds like a decent check. How would you suggest doing that? Creating an actual unit test? This is a really fragile test, though. If there are any underlying changes in implementation, the final lp__ could be off.
Indeed. Innocuous changes, like the dot_product vari implementation, would likely result in unpredictable changes over long chains. Checking the lp__ of known states could be a good unit test though.
Bob and I were talking about implementing unit tests comparing known values to the estimated values taking into account standard errors, which relies on an effective sample size calculation. This would be done without specifying a seed.
This seems like the right solution to me.
And then Jenkins would invoke an option to do 4) that humans typically wouldn't.
The test-bugs target was supposed to be a target that humans typically would not call. Do you use this target? If so, what are you using the target for?
I think people are prone to calling make test-all if it's there.
Cheers,
Marcus
Daniel Lee
I think people are prone to calling make test-all if it's there.
Users or developers? I thought the typical Stan users wouldn't call any of the tests.
Marcus Brubaker
Probably true, and I suppose it's fair to say that if you call make test-all without knowing how much work in your for, you get what you deserve.
Cheers,
Marcus
Daniel Lee
Maybe we should add a "test" target. That is more common, right?
Marcus Brubaker
Cheers,
Marcus
Bob Carpenter
1. Unit tests -- make sure everyhting's working according
to spec. That's where Ben's reformulated tests for BUGS
models would go -- just check they're getting the gradients
right.
Having this be just "make test" is fine by me. It is what
a user who builds from source will want to do. Presumably most
users will never see it one way or the other.
2. Regression tests (sorry for the confusing name, but that's
what they're called) -- make sure the big functional pieces
still give the right answers in similar or less time.
This was the intent for the BUGS models in that place, but Ben
makes a good point for including them in the unit tests in
a limited capacity. We'd sort of confused their role.
Eventually, the BUGS models full runs will track that we still
get the right answers and also track timing.
- Bob
Ben Goodrich
We have a target for each BUGS model. If we take the vol1/blocker/blocker.stan model, I believe you are looking for the following:
make models/bugs_examples/vol1/blocker/blocker
This works for all .stan models in the src/models subdirectory.
That target takes care of steps 1 to 3. Well, they also run the model as a convenience. We could remove the running portion of that target if that makes more sense.
It is good to check 1) through 3) on every model we can get our hands on. If we could wrap all those into a foreach construct in the makefile, that would be good. But that is basically just test-bugs without all the sampling. So, it might be easier to just add the corresponding option to test-bugs.
3.5) execute model with --seed = 0 and --test_grad, check whether lp__ matches the last known correct value of lp__ and check whether the auto-diff gradient matches the finite-diff gradient
That sounds like a decent check. How would you suggest doing that? Creating an actual unit test? This is a really fragile test, though. If there are any underlying changes in implementation, the final lp__ could be off.
I think it would be good to have stan write the auto-diff gradient and the finite-diff gradient into samples.csv when --test_grad is passed, in addition to lp__. Then if we had a text file in each subdirectory that just contained the last known correct value for lp__, we could check things using R if need be.
Bob and I were talking about implementing unit tests comparing known values to the estimated values taking into account standard errors, which relies on an effective sample size calculation. This would be done without specifying a seed.
That is certainly worth implementing as part of 4)
The test-bugs target was supposed to be a target that humans typically would not call. Do you use this target? If so, what are you using the target for?
These days, I have been using it to see if the models parse and compile with various compilers. Or when I accidentally do make test-all instead of make test-unit. Either way, everything past the first sample is useless to me, and even the first sample is not as useful as it could be if we implemented step 3.5 .
Ben
Bob Carpenter
(This is just the deliverables, not an actual plan
per se.)
We will test:
1) that the compiler is indeed compilable (bin/stanc)
Yes.
This isn't technically an API test, but we might
as well test it compiles.
2) that each BUGS model is able to be translated through the stanc compiler
Yes.
We don't need stanc for this. stanc
is just the command line method to call the compiler,
which in turn calls the parser and code generator.
We can do that all programatically without invoking
the stanc executable.
3) that each BUGS model is compilable by the compiler
Yes.
This is halfway between a unit test and regression test.
Because the compiler for the models is relatively slow
and we're talking about dozens of them, I think we
should let that decide us to delegate it to the regression
tests.
3.5) that each BUGS model gives right gradients
Yes.
We can also do without stanc here. Writing the same
kind of code into the tests also lets us avoid file I/O.
We can also test random seeds.
4) that each BUGS model runs to completion (that is, to 2000 samples)
Yes.
Move from unit tests to the yet-to-be developed
regression (aka functional) tests. Run to completion
should be split R-hat on 4 chains.
4.5) that each BUGS model gets the right answer
Yes.
As Daniel pointed out, this will take some by-hand work
to get the right answers specified somehow. CSV is easy to
read and if we put it down in order or with headers like
Stan produces, it should be easy to test. We also have
the dump format reader.
5.0) that models are no slower this time than baseine
This is where we test time per effective sample size.
- Bob
Andrew Gelman
A
Daniel Lee
> 5.0) that models are no slower this time than baseine
>
> This is where we test time per effective sample size.
>
Bob Carpenter
On 4/12/12 10:41 PM, Andrew Gelman wrote:
> Just one thing . . . I don't think you mean "BUGS model" below. I think you mean "Stan model" (or whatever we want to call the Stan modeling language). Actual BUGS models won't run in Stan untranslated. I think I know what you mean, but I don't think we want to refer to Stan's modeling language as BUGS.
> A
And I thought I was the pickiest person about
terminology. I want to keep two things distinct (though
I'm apparently not doing a very good job of it):
1. The model itself, as defined by the parameters, data
and joint probability function, and
2. The implementation of such in a programming language.
Would you be happier if I replaced "BUGS model" with
"a statistical model originally implemented in BUGS, later
translated with slight changes in meaning to an implementation in
JAGS, and now translated from JAGS to Stan's domain-specific
programming language".
Stan also lets you code such things directly in C++, so
that would be
"statistical model originally coded in BUGS,
later translated with slight changes in meaning to JAGS,
and now translated from JAGS to C++ with dependencies on
Stan's C++ API".
:-)
- Bob
Jiqiang Guo
"a statistical model originally implemented in BUGS, later
translated with slight changes in meaning to an implementation in
JAGS, and now translated from JAGS to Stan's domain-specific
programming language".