Metropolis testing (and generalizing testing for other

[Daniel Lee]

[moving discussion to stan-dev]

On Wed, May 15, 2013 at 4:09 PM, Peter Li <li.pe...@gmail.com> wrote:

Why would the starting value need to be changed? 

 

The init values don't need to be changed.

 

As for changing the sampler and the number of iterations, couldn't we just implement a new method that would add "--[insert sampler call here]" to the method that actually starts the sampling?

That sounds good and should work. Want to start working on that?

 

Perhaps, we could do this by adding a new string variable in model_test_fixture called 'sampler' which would be added in the get_command method inside model_test_fixture. We could assign the sampler name or whatever the call for it is to the new string and have NUTS be the default if nothing is specified.

Yes, definitely. If you think of a way to make model_test_fixture more flexible, go ahead and do it. It was only designed with one sampler in mind and I think we've now outgrown its original intent. Did you want to work on that for a short amount of time? Or do you want to work together on it?

 

I believe we could do something similar for number of iterations. And for iterations, how would we determine the number of iterations necessary?

Trial and error.

 

On 5/15/13 3:41 PM, Daniel Lee wrote:

Hi Peter,

Testing framework:

- hasn't been sorted out yet.

- do you want to help specify the behavior of what the framework should do? Since you're adding another use-case for the testing framework, I think it's time we rethink the whole thing. Maybe even pull it out of google test. I'm not sure. Here is a small list of requirements I was thinking about when I built it:

  * cross platform

  * configurable for each model

  * each model may have its own:

     - # iterations

     - values to check

     - values to skip

     - data and init files

     - special code for dealing with unidentified models

   * have the results integrate into Jenkins -- we've used junit output for the time being, but we could look for another solution.

- that all ran the default sampler, but now you're considering adding a different sampler, so we'd have to consider a few more things. I guess we want to keep everything but the first value, but allow changes to # iterations and the sampler parameters. Is there anything else we should really consider?

 

If you want to play with the sampler testing framework, let's start discussing what needs to be in and then we can build up something that's going to be reusable and flexible enough.


Daniel

[Peter Li]

I think I've got a working framework (in feature/metropolis), but I haven't gone through all of the model files and made the appropriate changes because I'm not sure how many iterations will be needed for the metropolis samplers. Also, is it absurd for it to take somewhere around 500k to 1000k iterations for metropolis to pass the chains test for BUGS models?


Peter

--
You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Peter Li]

I've been thinking some more about the testing framework. The way I set up the new framework, all of the model tests are run for all samplers, but I'm not sure that's what we want. Will there be models that we do not want to test against certain samplers just because it would take so long?

If so, a different set-up for the framework would be to have separate test files for each sampler. (i.e. for bugs_examples/vol/blocker, we could have blocker_nuts_test.cpp, blocer_unit_metro_test.cpp, etc.). Then, if we do not want to test a certain sampler we just don't include that file. But if we do it this way, I think it would end up compiling the model as many times as we want to test it, which would significantly increase the amount of time it takes to run test-models. That's one of the main reasons why I opted to setup the tests the other way. However, another plus for this is that we can create make commands (if that's what they're called) that will test all the models against certain samplers in case we don't want to run all of the models against all the samplers (NUTS, unit/diag/dense metro and later RM-HMC, Ensemble (with a few variations)).

We could also opt to keep the current setup and just have an if/else statement inside the model_test_fixture that doesn't run the tests if the number of iterations is greater than 500k or whatever upper bound we choose. As for being able to just test certain samplers, that would not be difficult, but I don't see a way to just give a command that does that--we'd have to go into model_test_fixture and comment out the tests that call the other samplers. This still wouldn't be difficult, but it just wouldn't be systematic.

Peter

[Bob Carpenter]

You bring up a good point about what we want to test.

Ideally, we would test everything --- every sampler in
every configuration on every model. I think of this as more of a functional
or regression test than a unit test, though. And we know ahead
of time that not every model is going to be fittable with every
sampler under every configuration, so it's a bit of a tricky
business configuring it all.

What I'd really like to see is something that looks more like
unit tests for the models. Tests that exercise all of the functions
being generated by the model rather than how they run as a whole.

Then, we can back off and run the functional/regression tests
much less frequently than every pull request.

- Bob

On 5/23/13 2:23 PM, Peter Li wrote:
> I've been thinking some more about the testing framework. The way I set up the new framework, all of the model tests are
> run for all samplers, but I'm not sure that's what we want. Will there be models that we do not want to test against
> certain samplers just because it would take so long?
>
> If so, a different set-up for the framework would be to have separate test files for each sampler. (i.e. for
> bugs_examples/vol/blocker, we could have blocker_nuts_test.cpp, blocer_unit_metro_test.cpp, etc.). Then, if we do not
> want to test a certain sampler we just don't include that file. But if we do it this way, I think it would end up
> compiling the model as many times as we want to test it, which would significantly increase the amount of time it takes
> to run test-models. That's one of the main reasons why I opted to setup the tests the other way. However, another plus
> for this is that we can create make commands (if that's what they're called) that will test all the models against
> certain samplers in case we don't want to run all of the models against all the samplers (NUTS, unit/diag/dense metro
> and later RM-HMC, Ensemble (with a few variations)).
>
> We could also opt to keep the current setup and just have an if/else statement inside the model_test_fixture that
> doesn't run the tests if the number of iterations is greater than 500k or whatever upper bound we choose. As for being
> able to just test certain samplers, that would not be difficult, but I don't see a way to just give a command that does
> that--we'd have to go into model_test_fixture and comment out the tests that call the other samplers. This still
> wouldn't be difficult, but it just wouldn't be systematic.
>
>
>
> Peter
>
>
> On Wed, May 22, 2013 at 2:09 PM, Daniel Lee <bea...@alum.mit.edu <mailto:bea...@alum.mit.edu>> wrote:
>
> [moving discussion to stan-dev]
>

> stan-dev+u...@googlegroups.com <mailto:stan-dev%2Bunsu...@googlegroups.com>.

[Peter Li]

On 5/23/13 2:53 PM, Bob Carpenter wrote:
> You bring up a good point about what we want to test.
>
> Ideally, we would test everything --- every sampler in
> every configuration on every model. I think of this as more of a
> functional
> or regression test than a unit test, though. And we know ahead
> of time that not every model is going to be fittable with every
> sampler under every configuration, so it's a bit of a tricky
> business configuring it all.
>
> What I'd really like to see is something that looks more like
> unit tests for the models. Tests that exercise all of the functions
> being generated by the model rather than how they run as a whole.

I'm not sure I follow what you're saying here. What functions are you
referring to?

>
> Then, we can back off and run the functional/regression tests
> much less frequently than every pull request.

So you're saying that we should have a test for the models similar to
what we have now but for all samplers and then have a separate test
suite similar to unit-tests but for models?

[Ben Goodrich]

On Thursday, May 23, 2013 2:53:22 PM UTC-4, Bob Carpenter wrote:

What I'd really like to see is something that looks more like
unit tests for the models.  Tests that exercise all of the functions
being generated by the model rather than how they run as a whole.

I do think we need a test-grad metatarget that just compiles all the models, runs them with --test-grad, and returns 0 iff lp__ is finite and the AD gradient is close to the numerical gradient. But that is more of a unit test of the libstan and stanc than a test of a particular sample. To test a sampler, I think we need something like

-- Generate a random correlation matrix
-- Maybe scale with random standard deviations to get a covariance matrix
-- Generate lots of multivariate normal data with mean zero and that covariance matrix
-- Run a sampler
-- Verify that posterior is not too far from the population parameters

You could play around with how the population gets generated, but in general, we need a set of tests with known populations to supplement all the toy BUGS models.

Ben

[Daniel Lee]

On Thu, May 23, 2013 at 3:14 PM, Ben Goodrich <goodri...@gmail.com> wrote:

On Thursday, May 23, 2013 2:53:22 PM UTC-4, Bob Carpenter wrote:

What I'd really like to see is something that looks more like
unit tests for the models.  Tests that exercise all of the functions
being generated by the model rather than how they run as a whole.

I do think we need a test-grad metatarget that just compiles all the models, runs them with --test-grad, and returns 0 iff lp__ is finite and the AD gradient is close to the numerical gradient. But that is more of a unit test of the libstan and stanc than a test of a particular sample.

+1. Let's work on getting that up sometime soon.

 

To test a sampler, I think we need something like

-- Generate a random correlation matrix
-- Maybe scale with random standard deviations to get a covariance matrix
-- Generate lots of multivariate normal data with mean zero and that covariance matrix
-- Run a sampler
-- Verify that posterior is not too far from the population parameters

You could play around with how the population gets generated, but in general, we need a set of tests with known populations to supplement all the toy BUGS models.

That looks great start. So you're thinking of running a single model through multiple samplers? Want to write something more concrete and we can find figure out how to put it into a test?

On Thu, May 23, 2013 at 2:23 PM, Peter Li <li.pe...@gmail.com> wrote:
If so, a different set-up for the framework would be to have separate test files for each sampler. (i.e. for bugs_examples/vol/blocker, we could have blocker_nuts_test.cpp, blocer_unit_metro_test.cpp, etc.). Then, if we do not want to test a certain sampler we just don't include that file. But if we do it this way, I think it would end up compiling the model as many times as we want to test it, which would significantly increase the amount of time it takes to run test-models.

That's now how it works. We would could use the folder structure to generate the correct model dependencies. That is, if the tests are in src/test/models/bugs_examples/vol1/blocker/*_test.cpp, the dependency could be: search for src/models/bugs_examples/vol1/blocker/*.stan, replace *.stan with *$(EXE).

Then each model would be compiled once.


Daniel