RStan Release Process

[Michael Betancourt]

We need to formalize the RStan release process, or at least fix
how dependencies work on CRAN. I gave a tutorial today with
about 50 attendees that was fucked because most of them
(not surprisingly) waited until the last second to install RStan
from CRAN and all got bit by the RStan/StanHeaders clash.
Besides the embarrassment it took substantial amount of time
away from the tutorial. We have to be able to do better.

[Ben Goodrich]

If I had known you were giving a tutorial today, I could have mitigated it somewhat with instructions on how to install it from Git or something. But there is not much we can do about CRAN. CRAN separately approved both StanHeaders and rstan within a few hours but getting binaries built and mirrored takes a few days and at any moment someone could get versions that are out of sync.

Ben
 

[Michael Betancourt]

I understand the limitations of CRAN, but we need to find

some way around it.  

For example, why were the dependencies updated so early?  

Shouldn’t StanHeaders have been submitted first, and then

only when it had been built and deployed RStan submitted?

If there’s no way to ensure a consistent way to propagate

changes to the dependencies then we at least need a status

alert somewhere, whether it be on Jenkins or the mailing

list or Twitter where (a) these kinds of submit windows can

be announced and (b) queried at any time to make it easy

to identify that it’s a CRAN issue.

--
You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Krzysztof Sakrejda]

On Wednesday, January 6, 2016 at 3:41:12 AM UTC-5, Michael Betancourt wrote:
> I understand the limitations of CRAN, but we need to find
> some way around it.  
>
>
> For example, why were the dependencies updated so early?  
> Shouldn’t StanHeaders have been submitted first, and then
> only when it had been built and deployed RStan submitted?

Could keep a simple drat-driven repo for Stan R packages that's always correctly in sync, then the install would still be a on-liner.

Krzysztof

[Ben Goodrich]

On Wednesday, January 6, 2016 at 3:41:12 AM UTC-5, Michael Betancourt wrote:

For example, why were the dependencies updated so early?  

Shouldn’t StanHeaders have been submitted first, and then

only when it had been built and deployed RStan submitted?

That is what happened. StanHeaders 2.9.0 started to mirror out when rstan was still on 2.8.2. Due to one of the refactors, rstan 2.8.2 couldn't compile models with StanHeaders 2.9.0.
 

If there’s no way to ensure a consistent way to propagate

changes to the dependencies then we at least need a status

alert somewhere, whether it be on Jenkins or the mailing

list or Twitter where (a) these kinds of submit windows can

be announced and (b) queried at any time to make it easy

to identify that it’s a CRAN issue.

We talked about that a little at the meeting. That is possible. I hadn't done that previously because I thought it would create more confusion to say something like "StanHeaders++ is available on CRAN but don't download it yet".

But definitely if someone is doing a workshop or something, we should avoid uploading during that time and have backup installation procedures ready.

Ben

[Daniel Lee]

On Wed, Jan 6, 2016 at 11:13 AM, Ben Goodrich <goodri...@gmail.com> wrote:

On Wednesday, January 6, 2016 at 3:41:12 AM UTC-5, Michael Betancourt wrote:

For example, why were the dependencies updated so early?  

Shouldn’t StanHeaders have been submitted first, and then

only when it had been built and deployed RStan submitted?

That is what happened. StanHeaders 2.9.0 started to mirror out when rstan was still on 2.8.2. Due to one of the refactors, rstan 2.8.2 couldn't compile models with StanHeaders 2.9.0.
 

If there’s no way to ensure a consistent way to propagate

changes to the dependencies then we at least need a status

alert somewhere, whether it be on Jenkins or the mailing

list or Twitter where (a) these kinds of submit windows can

be announced and (b) queried at any time to make it easy

to identify that it’s a CRAN issue.

We talked about that a little at the meeting. That is possible. I hadn't done that previously because I thought it would create more confusion to say something like "StanHeaders++ is available on CRAN but don't download it yet".

We should be proactive about known problems. I think we should:

- describe the problem clearly

- post a workaround on stan-users; sticky the post

- put the workaround on the rstan wiki in a prominent place: for example https://github.com/stan-dev/rstan -- update the README.md so it shows up there.

 

Once that's up, we can put it up on twitter too.

But definitely if someone is doing a workshop or something, we should avoid uploading during that time and have backup installation procedures ready.

I really don't think this should concern our release process. We have to consider that there are always new users, courses, classes, etc. and we should just post a workaround that works.

 

[Michael Betancourt]

What I don’t get is why RStan 2.8.2 pointed to the updated StanHeaders

instead of the 2.8.0 version.  Do CRAN dependencies not target specific

versions?  Is that not something that we can force?

[Daniel Lee]

+1. Can we fix dependency versions?

[Ben Goodrich]

On Wednesday, January 6, 2016 at 11:34:57 AM UTC-5, Michael Betancourt wrote:

What I don’t get is why RStan 2.8.2 pointed to the updated StanHeaders

instead of the 2.8.0 version.  Do CRAN dependencies not target specific

versions?  Is that not something that we can force?

Not with that much control. You can specify that rstan 2.8.2 only works with StanHeaders 2.8.0, but if StanHeaders 2.9.0 is available it will just refuse to install rather than fetching StanHeaders 2.8.2. To actually get stuff done, you would have to manually install StanHeaders 2.8.0.

Ben

[Daniel Lee]

I would rather take that behavior than allow it to be installed and failing when you try to compile a model.

[Ben Goodrich]

On Wednesday, January 6, 2016 at 11:28:54 AM UTC-5, Daniel Lee wrote:

We should be proactive about known problems. I think we should:

- describe the problem clearly

- post a workaround on stan-users; sticky the post

- put the workaround on the rstan wiki in a prominent place: for example https://github.com/stan-dev/rstan -- update the README.md so it shows up there.

 

Once that's up, we can put it up on twitter too.

OK.

 

[Michael Betancourt]

I agree.  At least that way we’d know what was going on instead of getting

weird compiler errors.

[Ben Goodrich]

On Wednesday, January 6, 2016 at 11:49:52 AM UTC-5, Michael Betancourt wrote:

I agree.  At least that way we’d know what was going on instead of getting

weird compiler errors.

Refusing to install would be slightly better than installing but not working. But it means that it is impossible for me to do a patch fix of StanHeaders without also updating rstan, which could be a problem with the embargo on uploads within a month. In other words, if some function in Math was returning the wrong result, I could have done a StanHeaders 2.8.0-1 while leaving rstan 2.8.2 as is. So, if we have to utilize workarounds to get it to actually work during the major transition periods, I would rather maintain the ability to update them asynchronously when they don't break.

Ben

[Bob Carpenter]

> On Jan 6, 2016, at 11:28 AM, Daniel Lee <bea...@alum.mit.edu> wrote:

...

> I really don't think this [our classes] should concern our release process.

> We have to consider that there are always new users, courses, classes, etc.
> and we should just post a workaround that works.

+1 to that.

- Bob

[Bob Carpenter]

> On Jan 6, 2016, at 12:01 PM, Ben Goodrich <goodri...@gmail.com> wrote:
>
> On Wednesday, January 6, 2016 at 11:49:52 AM UTC-5, Michael Betancourt wrote:
> I agree. At least that way we’d know what was going on instead of getting
> weird compiler errors.
>
> Refusing to install would be slightly better than installing but not working.

I think it's a whole lot better. The principle is to fail as early as possible
in any computational process with decent warning messages.

> But it means that it is impossible for me to do a patch fix of StanHeaders without also updating rstan, which could be a problem with the embargo on uploads within a month. In other words, if some function in Math was returning the wrong result, I could have done a StanHeaders 2.8.0-1 while leaving rstan 2.8.2 as is. So, if we have to utilize workarounds to get it to actually work during the major transition periods, I would rather maintain the ability to update them asynchronously when they don't break.

I'd accept that tradeoff.

Having some kind of up-to-date outside-of-CRAN way to install is
also necessary to mitigate the situation of CRAN and mirrors
being out-of-synch.

- Bob

[Joshua N Pritikin]

On Wed, Jan 06, 2016 at 01:54:16PM -0500, Bob Carpenter wrote:
> Having some kind of up-to-date outside-of-CRAN way to install is
> also necessary to mitigate the situation of CRAN and mirrors
> being out-of-synch.

Here's an example of how it can be done,

http://openmx.psyc.virginia.edu/getOpenMx.R

OpenMx isn't careful about tracking versions of dependencies, but I
suspect it would just be a matter of copying the appropriate packages to
the OpenMx hosted package repo.

--
Joshua N. Pritikin
Department of Psychology
University of Virginia
485 McCormick Rd, Gilmer Hall Room 102
Charlottesville, VA 22904
http://people.virginia.edu/~jnp3bc

[Ben Goodrich]

On Wednesday, January 6, 2016 at 1:54:19 PM UTC-5, Bob Carpenter wrote:

> On Jan 6, 2016, at 12:01 PM, Ben Goodrich <goodri...@gmail.com> wrote:
>
> On Wednesday, January 6, 2016 at 11:49:52 AM UTC-5, Michael Betancourt wrote:
> I agree.  At least that way we’d know what was going on instead of getting
> weird compiler errors.
>
> Refusing to install would be slightly better than installing but not working.

I think it's a whole lot better.  The principle is to fail as early as possible
in any computational process with decent warning messages.

It is just going to stay "rstan is not available for your platform" and people are still going to post on stan-users or maybe first read on stan-users that they need to utilize a workaround. It doesn't result in anyone expending less effort to have a working rstan installation.

Ben

[Bob Carpenter]

> On Jan 6, 2016, at 2:05 PM, Ben Goodrich <goodri...@gmail.com> wrote:
>
> On Wednesday, January 6, 2016 at 1:54:19 PM UTC-5, Bob Carpenter wrote:
>
> > On Jan 6, 2016, at 12:01 PM, Ben Goodrich <goodri...@gmail.com> wrote:

> ...

> > Refusing to install would be slightly better than installing but not working.
>
> I think it's a whole lot better. The principle is to fail as early as possible
> in any computational process with decent warning messages.
>
> It is just going to stay "rstan is not available for your platform" and people are still going to post on stan-users

That's much better than having it get part way and then break in
a way that leads users to try to figure out what went wrong on
their end.

> or maybe first read on stan-users that they need to utilize a workaround.

Could the message point them to that page? That'd be ideal.

> It doesn't result in anyone expending less effort to have a working rstan installation.

It does. People uninstall, reinstall, reinstall R, try again, reboot,
try different machines. They're very persistent when they think it's
their fault.

- Bob

[Ben Goodrich]

I think it would be better to let rstan install if possible and then at the time that the model is compiled check whether StanHeaders has the same version number up to two digits (currently, we are only checking that StanHeaders exists). We can then offer to downgrade StanHeaders.

Ben

[Ben Goodrich]

On Wednesday, January 6, 2016 at 12:01:20 PM UTC-5, Ben Goodrich wrote:

[Bob Carpenter]

If we can do that in a way that makes sense to simple users like
me or Andrew (not power users like yourself or Jonah), that'd
be ideal.

- Bob

[Ben Goodrich]

On Wednesday, January 6, 2016 at 2:25:45 PM UTC-5, Bob Carpenter wrote:

If we can do that in a way that makes sense to simple users like
me or Andrew (not power users like yourself or Jonah), that'd
be ideal.

I can't upload this for a month but rstan on GitHub now has

  if (packageVersion("StanHeaders") > packageVersion("rstan"))
    stop("StanHeaders version is ahead of rstan version; ",
         "see https://github.com/stan-dev/rstan/wiki/RStan-Transition-Periods")

I don't think the necessity of the workarounds will make sense to many people, but that doesn't mean they can't follow them.

Ben


 

[Bob Carpenter]

I agree that R's dependency management isn't going to make sense to most people.
It's a software release issue, which is even less studied than software testing.

But many of the ones we hear from will be able to follow step-by-step instructions,
so I think it'll be a huge win over just failing during compilation.

- Bob

[Michael Betancourt]

I would prefer to have the package not even install, but at the very
least the warning has to be obvious. Spending a third of a tutorial
running around tracking down include errors is not an okay
circumstance.

[Ben Goodrich]

On Wednesday, January 6, 2016 at 5:39:39 PM UTC-5, Michael Betancourt wrote:

I would prefer to have the package not even install, but at the very
least the warning has to be obvious.  Spending a third of a tutorial
running around tracking down include errors is not an okay
circumstance.

If rstan doesn't install, we have no control over the error message because the error will come from base::install.packages(). It will just say something like installation failed. If we allow rstan to install and throw an error when it tries to compile with a mismatched StanHeaders, then we can error with a customized message and a link to the wiki page.

Ben

[Michael Betancourt]

We _can_, but do we?  These installation errors keep

popping up, and if we can’t predict what errors will arise

well enough then not installing is by far the most user-friendly

option.

[Ben Goodrich]

On Wednesday, January 6, 2016 at 5:58:34 PM UTC-5, Michael Betancourt wrote:

We _can_, but do we?  

We will in rstan 2.10 but do not 2.9.0.

https://github.com/stan-dev/rstan/commit/3782791591fcabeccd9ef4b660627c566c9b36aa

Ben