Re: NUTS vs ADVI Test

[Andrew Gelman]

P.S. I'm really glad to hear that there will be more work on this NUTS vs ADVI Testing. I think it's a wonderful project, in the immediate sense (learning where ADVI is working and where it has problems), in a longer-term sense (motivating the ADVI team to figure out how to make ADVI more reliable), and also more generally as an example of validation of approxiimate Bayesian methods.


> On Jul 31, 2016, at 11:25 AM, Bob Carpenter <ca...@alias-i.com> wrote:
>
> I would very strongly recommend keeping the most recent
> version of everything pitched to GitHub. I try not to
> go longer than one day without pushing. That's why
> branches are so useful.
>
> - Bob
>
>> On Jul 30, 2016, at 1:28 PM, Rayleigh Lei <rxl...@columbia.edu> wrote:
>>
>> Hi,
>>
>> I've posted code on Github (https://github.com/rayleigh/ADVI-test), but I just checked and it's not the most up to date code. I'm on a biking trip right now, but when I get back, I can update it.
>>
>> Best,
>> Rayleigh
>>
>> On Thursday, July 28, 2016, Michael Betancourt <betan...@gmail.com> wrote:
>> I can’t remember if the tests were ever public, hence
>> sending to stan-core first.
>>
>> On Jul 28, 2016, at 4:55 PM, Bob Carpenter <ca...@alias-i.com> wrote:
>>
>>> [include Rayleigh, remove stan-core]
>>>
>>> Did you mean to send this to stan-dev? Rayleigh will know
>>> which repo it's in.
>>>
>>> - Bob
>>>
>>>> On Jul 28, 2016, at 11:38 AM, Michael Betancourt <betan...@gmail.com> wrote:
>>>>
>>>> Are the materials (scripts, data, models, results) from Rayleigh’s tests
>>>> available anywhere convenient? There has been some interest in
>>>> continuing those tests or at least using them as a baseline and I
>>>> wanted to review them first. Bonus points if they’re publicly
>>>> accessible.
>>>>
>>>> Thanks.
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups "Stan Core" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an email to stan-core+...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>
>>

[Bob Carpenter]

For that matter, we want regression (in the computer
science sense, not the stats one) tests even for MCMC.

- Bob

> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.

[Andrew Gelman]

Is "regression (in the computer science sense) tests" the same thing as fake-data checking? Cos we have talked about (but not implemented) a workflow for Cook/Gelman/Rubin using Stan. It could be worth us preparing a little document on all these sorts of checking. Maybe it's all written up, someone will do it!
A

[Jonah Gabry]

Regression in this sense means making sure that changes to code don't break anything that worked prior to the change. It doesn't necessarily tell you that what you were seeing before was correct (that should be tested too of course), just that no unintended changes were introduced.

Regarding the Cook-Gelman-Rubin procedures, we have a function in rstanarm that does that (simulate data from model, refit the model, check quantiles, etc). I'm not sure if there's anything like that in the tests on the C++ side, although I think Bob mentioned at some point that there was.

Jonah

[Bob Carpenter]

It's "regressing" in the sense of not allowing the code to regress
and reintroduce errors that were already fixed once.

I do not believe there is anything on the C++ side to do this
kind of testing. All of this will get easier when the refactor's
done and it's easier to call CmdStan-like functionality from C++.

- Bob

[Dustin Tran]

Just tuning in to also express enthusiasm in this project! (both for improving ADVI/GMO/posterior mode-finders, and validating inference in general)

fyi, there are stochastic unit tests, which operationalize cook, gelman, and robin’s original motivation. That is, rather than produce a deterministic fail/success, a probability of failure is produced, such as the probability that inference is too approximate (or bugged). I don’t suggest we run these tests routinely in every build, but it is nice that we can interpret our fake data checking experiments as actually being unit tests.

Dustin

[Rayleigh L]

I updated my branch to include the latest code and to put deprecation comments on old code.

> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "stan development mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+unsubscribe@googlegroups.com.