More frequent, regular releases

[Roman Leventov]

Druid release pace is slowing down quickly:

2015 - 9 releases

2016 - 5 releases

2017 - 2 so far, and probably won't be more

The last 0.10.1 release was initially supposed to go out "not more than a month" after 0.10.0, but it took 4 months and 4 days.

I think the main reason for this is "PR milestones", and that we don't cut a release until all PRs targeted to that release are merged. And while they are not, other "very important" PRs appear, targeting the same release. There are 2-3 "generations" of such PRs.

Also, milestone assignment process seems largely non-sense: everybody who have rights to assign milestones target all PRs they author to the closest release, and all other people who don't have such rights, have their PRs not targeted to any release.

The proposal is to retire milestones and start a release process every 3 months, regardless of anything. What is merged into master, will be in this release, what is not, will be in a +3 months release, or later. Critical bugs could be resolved during release candidate hardening period.

[Gian Merlino]

I am strongly in favor of time based releases rather than feature based. I feel that has worked well for other communities that have moved to such a system.

It sounds like you are suggesting doing 4 per year which sounds good to me. I would be ok with anything in the 4-6 range.

I guess the other related question is what standard of QA / testing do we want to apply to releases. In the past one thing that slowed down the release cycles has been the slowness of our "classic" testing process. It depends on major organizations involved in Druid development to deploy release candidates to production and validate them there, which depends on various internal release cycles. We could do releases faster if we were less stringent here, although it may affect quality.

My org can commit to rolling out and testing releases internally on an aggressive schedule, although our internal clusters are relatively small compared to the largest ones out there. We also don't use all the features internally -- for example we don't use HDFS and we don't do much Hadoop based indexing beyond the suite of Hadoop integration tests we do run.

Gian

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CAB5L%3Dwd17w4qaFj4QYeZnO2%3DnKC_jgfuSpJW%3DcFMonHueCu5Fw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Roman Leventov]

In my opinion the current QA/testing process is acceptable. It means sometimes that release actually happens up to 2 months after the first RC or a release branch created out of master, but if those initial actions are done at the specified interval, final releases will also have the same frequency, even through intervals between them may vary by a couple of weeks.

I started this discussion because we at Metamarkets want to move to using (almost) pure official releases, rather than releases from our deeply diverging branches, as we do now. It means that we are going to be more actively involved into testing of official RC releases, that could make the testing process faster.

On Fri, Aug 25, 2017 at 1:48 PM, Gian Merlino <gi...@imply.io> wrote:

I am strongly in favor of time based releases rather than feature based. I feel that has worked well for other communities that have moved to such a system.

It sounds like you are suggesting doing 4 per year which sounds good to me. I would be ok with anything in the 4-6 range.

I guess the other related question is what standard of QA / testing do we want to apply to releases. In the past one thing that slowed down the release cycles has been the slowness of our "classic" testing process. It depends on major organizations involved in Druid development to deploy release candidates to production and validate them there, which depends on various internal release cycles. We could do releases faster if we were less stringent here, although it may affect quality.

My org can commit to rolling out and testing releases internally on an aggressive schedule, although our internal clusters are relatively small compared to the largest ones out there. We also don't use all the features internally -- for example we don't use HDFS and we don't do much Hadoop based indexing beyond the suite of Hadoop integration tests we do run.

Gian

On Thu, Aug 24, 2017 at 5:47 PM, Roman Leventov <roman.leventov@metamarkets.com> wrote:

Druid release pace is slowing down quickly:

2015 - 9 releases

2016 - 5 releases

2017 - 2 so far, and probably won't be more

The last 0.10.1 release was initially supposed to go out "not more than a month" after 0.10.0, but it took 4 months and 4 days.

I think the main reason for this is "PR milestones", and that we don't cut a release until all PRs targeted to that release are merged. And while they are not, other "very important" PRs appear, targeting the same release. There are 2-3 "generations" of such PRs.

Also, milestone assignment process seems largely non-sense: everybody who have rights to assign milestones target all PRs they author to the closest release, and all other people who don't have such rights, have their PRs not targeted to any release.

The proposal is to retire milestones and start a release process every 3 months, regardless of anything. What is merged into master, will be in this release, what is not, will be in a +3 months release, or later. Critical bugs could be resolved during release candidate hardening period.

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CAB5L%3Dwd17w4qaFj4QYeZnO2%3DnKC_jgfuSpJW%3DcFMonHueCu5Fw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CACZNdYCUFmspFmaNxAxqD14c%3D%2Baoq4c-PiqxmQV7Ruarx9R9Tg%40mail.gmail.com.

[Gian Merlino]

I hope it's not actually 2 months from release branching to actual release (1 should hopefully be doable with more people testing more rapidly) but I see your point. I guess it means we should target specific dates for release branching, and then try to do the release as soon after release branching as possible.

Given that we did a release branch for 0.10.1 on Jun 16, how about planning Sep 16 (3 months later, implying 4 releases per year) for release branching of 0.11.0? It's 3 weeks from now.

Gian

On Fri, Aug 25, 2017 at 12:02 PM, Roman Leventov <roman.l...@metamarkets.com> wrote:

In my opinion the current QA/testing process is acceptable. It means sometimes that release actually happens up to 2 months after the first RC or a release branch created out of master, but if those initial actions are done at the specified interval, final releases will also have the same frequency, even through intervals between them may vary by a couple of weeks.

I started this discussion because we at Metamarkets want to move to using (almost) pure official releases, rather than releases from our deeply diverging branches, as we do now. It means that we are going to be more actively involved into testing of official RC releases, that could make the testing process faster.

On Fri, Aug 25, 2017 at 1:48 PM, Gian Merlino <gi...@imply.io> wrote:

I am strongly in favor of time based releases rather than feature based. I feel that has worked well for other communities that have moved to such a system.

It sounds like you are suggesting doing 4 per year which sounds good to me. I would be ok with anything in the 4-6 range.

I guess the other related question is what standard of QA / testing do we want to apply to releases. In the past one thing that slowed down the release cycles has been the slowness of our "classic" testing process. It depends on major organizations involved in Druid development to deploy release candidates to production and validate them there, which depends on various internal release cycles. We could do releases faster if we were less stringent here, although it may affect quality.

My org can commit to rolling out and testing releases internally on an aggressive schedule, although our internal clusters are relatively small compared to the largest ones out there. We also don't use all the features internally -- for example we don't use HDFS and we don't do much Hadoop based indexing beyond the suite of Hadoop integration tests we do run.

Gian

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CAB5L%3DwcHv9mZPZ7M3Cjc%2Br63vNDCUUGwLSNFp-SmEdGeY1cy%3DQ%40mail.gmail.com.

[Roman Leventov]

+1 from me. However I think what we also should do is to retire milestones on github. I suggest to remove them completely, even not putting them post-factum after a PR is merged, and rather prepare release notes by walking through the commit history.

Something that we also should think about is what we do with breaking/non-breaking release cadence. Should we try to preserve it? But it would mean that some PRs may be delayed, that could be unwanted.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CACZNdYA%3Do4Eyi6FgiK7Ge_gu4GJDHur8nc99aH2cL_s%2BOH1dAg%40mail.gmail.com.

[Gian Merlino]

I agree that milestones are less important in a time based release cycle. But I think they're still useful for identifying release blockers -- how else would we track what issues must be resolved in a specific release branch before we can do that release? Such as issues that are discovered during QA of a release candidate. The commits have not landed in the branch yet, so commit history is not useful.

For the breaking/nonbreaking cadence, I think we can play it by ear, and basically schedule a breaking release as the 'next' release when it makes sense. To do that I think we still need to keep the "incompatible" label for PRs and those should not be merged until the 'next' release is understood to be a major release. One way that might happen, is the author of the incompatible PRs posts on the dev list saying they think the next release should be major, and if there is consensus, then it will be.

Gian

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CAB5L%3DwdMWGxmD9yj20puxidyd9DsZK1CXC-86W8h_bJ5yighBg%40mail.gmail.com.

[Gian Merlino]

For milestones, I think it would work to use them only for PRs/issues that are truly release blockers -- should be limited to critical bugs discovered after a release branch is cut. We could make release notes the way you suggest, by walking the commit history.

Gian

On Fri, Aug 25, 2017 at 12:44 PM, Gian Merlino <gi...@imply.io> wrote:

I agree that milestones are less important in a time based release cycle. But I think they're still useful for identifying release blockers -- how else would we track what issues must be resolved in a specific release branch before we can do that release? Such as issues that are discovered during QA of a release candidate. The commits have not landed in the branch yet, so commit history is not useful.

For the breaking/nonbreaking cadence, I think we can play it by ear, and basically schedule a breaking release as the 'next' release when it makes sense. To do that I think we still need to keep the "incompatible" label for PRs and those should not be merged until the 'next' release is understood to be a major release. One way that might happen, is the author of the incompatible PRs posts on the dev list saying they think the next release should be major, and if there is consensus, then it will be.

Gian

[Hagen Rother]

Concering the testing in production, I am now at the point that I could run druid dual stack, we do ~1m/s events (we failed with indexing, back on realtime. Love to test the new kafka 0.10 code, we are at 0.11 in production)

Ubuntu and node.js are have scheduled releases; both have the concept of LTS. While the later are a bit too hipster for me lately, they do have a nice version graph: https://github.com/nodejs/LTS#release-schedule1 - probably worth to consider adopting and adjusting.

tl;dr: +1 for timed releases

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CACZNdYDs8spBLiiqxy1CGfQUbW%3DKCfzvTADRGezOey4cftw50Q%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Hagen Rother
Lead Architect | LiquidM
LiquidM Technology GmbH Rosenthaler Str. 36 | 10178 Berlin | Germany
Phone:	+49 176 15 00 38 77
Internet:	www.liquidm.com | LinkedIn

Managing Directors | André Bräuer, Philipp Simon, Thomas Hille
Jurisdiction | Local Court Berlin-Charlottenburg HRB 152426 B

[Charles Allen]

We have had the pattern for a while now where we have one or two minor releases, then a "major" release. Would it make sense to move to a versioning scheme using timestamp, either like either coreos with their release channels, or jetty with a more traditional versioning but with timestamps in the version? 

On Sunday, August 27, 2017 at 11:57:04 AM UTC-7, Hagen Rother wrote:

Concering the testing in production, I am now at the point that I could run druid dual stack, we do ~1m/s events (we failed with indexing, back on realtime. Love to test the new kafka 0.10 code, we are at 0.11 in production)

Ubuntu and node.js are have scheduled releases; both have the concept of LTS. While the later are a bit too hipster for me lately, they do have a nice version graph: https://github.com/nodejs/LTS#release-schedule1 - probably worth to consider adopting and adjusting.

tl;dr: +1 for timed releases

On Fri, Aug 25, 2017 at 9:49 PM, Gian Merlino <gi...@imply.io> wrote:

For milestones, I think it would work to use them only for PRs/issues that are truly release blockers -- should be limited to critical bugs discovered after a release branch is cut. We could make release notes the way you suggest, by walking the commit history.

Gian

On Fri, Aug 25, 2017 at 12:44 PM, Gian Merlino <gi...@imply.io> wrote:

I agree that milestones are less important in a time based release cycle. But I think they're still useful for identifying release blockers -- how else would we track what issues must be resolved in a specific release branch before we can do that release? Such as issues that are discovered during QA of a release candidate. The commits have not landed in the branch yet, so commit history is not useful.

For the breaking/nonbreaking cadence, I think we can play it by ear, and basically schedule a breaking release as the 'next' release when it makes sense. To do that I think we still need to keep the "incompatible" label for PRs and those should not be merged until the 'next' release is understood to be a major release. One way that might happen, is the author of the incompatible PRs posts on the dev list saying they think the next release should be major, and if there is consensus, then it will be.

Gian

[Gian Merlino]

The CoreOS channel scheme and similar ones (like Debian's and Chrome's) I guess are an alternative to a release-candidate system. Instead of doing a series of release candidates you would promote "alpha" builds to "beta". And then a stable release promotes one of the "beta" builds to "stable". I suppose the main difference is that a channel based system is expected to put out betas more regularly than a release branch/candidate system would put out release candidates. Betas come out more or less continuously, but release candidates come out in a fast paced release train after a release branch is cut, followed by periods of quietness after a final release and before the next release branch.

I don't have much of an opinion one way or the other on that one.

On jetty: I always felt that jetty's versioning scheme was kind of weird, since the timestamps don't add much beyond embedding the release date in the version. They still have a "normal" incrementing version number to the left of it.

Charles are you proposing we adopt one of those approaches or just asking to see what people think?

In general is anyone _against_ time based releases? So far nobody has spoken up on that, so maybe we can adopt them at the next dev sync.

Gian

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/dca3a087-e4cf-48a6-b681-129712e68ef2%40googlegroups.com.

[Himanshu Gupta]

I think having time based releases makes sense. Also, as discussed here, we need to have labels to identify incompatible PRs for a given release, PRs with new documentation for a given release and to identify release blocker PRs etc.

-- Himanshu

[Gian Merlino]

In the dev sync today we decided to try out timed releases, with 0.11.0 release branch being cut on September 18.

Gian

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/a6cea015-f251-4866-b59f-5e0729afe6ef%40googlegroups.com.