warmup seems broken in develop

[Sebastian Weber]

Hi there!

I was running a ~1.5 month old cmdstan snapshot without a problem. I thought updating to the current version in develop would be a good idea. I also set stan and stan-math to the most recent develop versions (cmdstan refers to older git submodules).

Now my models only give me crap! That is, most of the runs are incredibly fast, only few runs run at a normal speed. To me it looks as if warmup fails to do something sensible (my residual error is way too large). I have to say that some chains give me results which are ok (different seed and different chain_id).

To cross check I also checked out cmdstan with the snapshot versions as the git submodules are set currently, see below for the hashes.

Model and data can't be shared, sorry; but I am happy to try out any updates on them.

Best,
Sebastian

cmdstan submodules:
[weberse2@node13:~/libs/cmdstan] make stan-update
git submodule update --init --recursive
Submodule path 'stan': checked out 'fb1b76962cd8495812c87f0a0e0d8ab328444f34'
Submodule path 'stan/lib/stan_math': checked out 'fe69fed0c91ac1c9f022673d70d7d94c48296e38'

[Sebastian Weber]

Doing a bit more testing reveals that the cmdstan hashes

- 169c0fb does not work; 17 April
- 27e078f WORKS OK (before the BIG HMC refactor); 26 March

So something between the above two cmdstan hashes makes the warmup fail - at least on my model. We need some test for this, but I have no clue yet where to start.

Sebastian

[Krzysztof Sakrejda]

On Wednesday, June 1, 2016 at 8:52:27 AM UTC-4, Sebastian Weber wrote:
> Doing a bit more testing reveals that the cmdstan hashes
>
> - 169c0fb does not work; 17 April
> - 27e078f WORKS OK (before the BIG HMC refactor); 26 March

That's helpful, I think. Here's the comparison:

https://github.com/stan-dev/cmdstan/compare/27e078f...169c0fb

[Sebastian Weber]

Yep, I hope this is helpful. My guess is that this has to do with changes in stan which happened between those commits. In particular, those changes done to the HMC sub-system. From my impression so far, the bug seems to be triggered more reliably the more random effects you have.

I hope someone can create a minimal example which we can throw into a test - I don't have one ready yet, but from my perspective this should (in fact must) be adressed before we tag 2.10.

Sebastian

[Krzysztof Sakrejda]

On Wednesday, June 1, 2016 at 9:37:28 AM UTC-4, Krzysztof Sakrejda wrote:
> On Wednesday, June 1, 2016 at 8:52:27 AM UTC-4, Sebastian Weber wrote:
> > Doing a bit more testing reveals that the cmdstan hashes
> >
> > - 169c0fb does not work; 17 April
> > - 27e078f WORKS OK (before the BIG HMC refactor); 26 March
>
> That's helpful, I think. Here's the comparison:
>
> https://github.com/stan-dev/cmdstan/compare/27e078f...169c0fb

Matching Stan update: https://github.com/stan-dev/stan/compare/e1fb49da...cd8e2246

[Bob Carpenter]

Agreed --- this definitely needs to be fixed.

Sebastian --- could you create a reproducible example
for an issue? Then I'm hoping Michael can look at what's
going on as all the changes were from his updates to HMC.
Hopefully this'll be a simple fix.

- Bob

> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Sebastian Weber]

Hi!

Ok, I got an example which is reproducible, i.e. the first chain triggers the weird behavior and the second seems to calculate correctly. This is one of those fruitful PK examples and needs a couple of files (init, data and stan program)... where can I put these? Issues don't allow me to put files there.

I can try to make the example more basic if needed.

Sebastian

On Wednesday, June 1, 2016 at 6:40:41 PM UTC+2, Bob Carpenter wrote:
> Agreed --- this definitely needs to be fixed.
>
> Sebastian --- could you create a reproducible example
> for an issue? Then I'm hoping Michael can look at what's
> going on as all the changes were from his updates to HMC.
> Hopefully this'll be a simple fix.
>
> - Bob
>
> > On Jun 1, 2016, at 10:20 AM, Sebastian Weber
> >

[Sebastian Weber]

Hi!

Ok, so I have filed a case where things fail for me, the issue is this one:

https://github.com/stan-dev/stan/issues/1902

I am guessing here that stan is the right place for this. If someone could confirm me that the behavior is odd for him as well (on his system, platform) that would be great.

... and yeah, I am crossing fingers that this is an easy (but important) one.

Best,
Sebastian

[Bob Carpenter]

Thanks. Where'd that files directory come from you're using
for the zip?

And don't worry --- we need to make sure adaptation didn't get
broken in the recent merges. We won't let it go out until
it works as well as the last version. Worst case we can just
revert to the older behavior.

- Bob

[Sebastian Weber]

Ok, that's good to hear that this will be taken care of. I just checked the zip file again and I am not sure what you mean wrt to the directory. I updated the ticket to include detailed instructions on how to run each chain.

I hope that this example triggers the issue for others as well and qualifies as reproducible example of the problem.

Sebastian

On Thursday, June 2, 2016 at 5:33:44 AM UTC+2, Bob Carpenter wrote:
> Thanks. Where'd that files directory come from you're using
> for the zip?
>
> And don't worry --- we need to make sure adaptation didn't get
> broken in the recent merges. We won't let it go out until
> it works as well as the last version. Worst case we can just
> revert to the older behavior.
>
> - Bob
>
> > On Jun 1, 2016, at 3:02 PM, Sebastian Weber
> >