[ACTION REQUIRED] Changes to the Kubernetes 1.19 Release Cycle
Stephen Augustus
This year marks an unprecedented time in the global community, which has sparked countless changes to our personal, business, and OSS lives.
As we all try to cope, we realize that the uncertainty of it all must lend itself to re-examining (and maybe redefining) established processes.
The Kubernetes release cycle timeframe and milestones are where we'd like to make some modifications to better accommodate contributors and consumers, and we wanted to take the time to discuss some of these proposed changes with you all.
tl;dr
Enhancements changes
Review the previous email from the Enhancements subproject
Enhancements Freeze happens a week later
Submit less content and fewer exceptions
Code Freeze and branch creation happen at the same time
More RCs (release candidates) will extend the release cycle by a few weeks
Code Thaw (master reopen) now happens post-release
Proposed schedule changes: https://github.com/kubernetes/sig-release/pull/1058
Over the past weeks, Tim, Taylor, and I have been ruminating on a few ideas, and we had an opportunity to more broadly present them to the newly-formed Kubernetes 1.19 Release Team earlier today (meeting notes here).
Originally, I had hoped to outright delay the start of the 1.19 release cycle until late April or early May, but realistically, the moment kubernetes/kubernetes master opens back up for development, we've functionally started the release. So you'll see the changes described here mostly affect Code Freeze and the milestones following Code Freeze.
Enhancements / KEP process changes
With inbound changes to the KEP format, please review my previous email from the Enhancements subproject, which includes action items.
Enhancements Freeze
We’re moving Enhancements Freeze back a week, to give some additional time for submissions/updates.
Exceptions
The Release Team will be more strict around approving enhancement exceptions.
We ask that all SIGs deeply consider if certain enhancements need to land for Kubernetes 1.19, before submitting an exception for the Release Team to consider.
If the enhancement must land within Kubernetes 1.19, please submit an exception request as early as possible.
Code Freeze and Branch cut
The initial creation of the “release-1.19” branch (which previously happened in conjunction with the beta.0 release) will now happen at Code Freeze.
Code Freeze will now mark the following:
Enforcement of milestone-targeted merges on master
The creation of the “release-1.19” branch
The release of 1.19.0-rc.1
By moving the branch creation later in the cycle, we eliminate some of the fast-forward work that needs to be done by the Branch Management team.
Merge restriction enforcement will now happen the day before Code Freeze
Last cycle, there was some confusion about when the mechanical enforcement of Code Freeze actually starts.
Moving forward:
Ensure PRs are lgtm’ed and approved AND milestoned by the requisite OWNERS by EOD (your time) on the day before Code Freeze (PRs that are not properly milestoned should not merge)
Branch Managers can apply the merge restrictions at any point on the day before Code Freeze (we have a global team, so it’s not always possible that a Branch Manager waits until EOD PT to enable these restrictions)
The Release Team will announce that Code Freeze has officially started
Upon that announcement, the Release Team will begin to scan PRs that have the milestone and work with you to ensure that they merge
More pre-releases
In the current iteration of the schedule, you'll notice that we've added more pre-releases (both betas and RCs (release candidates) to the schedule.
This should give everyone more checkpoints in the cycle to test against before the official release.
We will release three RCs (more, if needed) on a biweekly cadence.
This effectively pushes the official release date back a few weeks.
This increase in time relative to an expected decrease in human testing/adoption of betas and RCs and slowed collaboration during this cycle is not aiming to increase quality, rather it is a stop gap to possibly maintain quality. We are operating in an unprecedented environment.
Code Thaw
Code Thaw is the point that master opens back up for next release development, in this case release 1.20.
Instead of re-opening master immediately before the cherry pick deadline, we'll wait to lift milestone restrictions until soon after the official release date.
We hope this will encourage SIGs to give an increased focus on delivering just the content for Kubernetes 1.19 instead of final work on 1.19 and beginning 1.20 in parallel.
It also gives us (the community) an extended opportunity to:
Do SIG roadmap planning
Do issue/bug triage
Update KEPs and Enhancements issues
Formulate the next Release Team
Rest. Then kick off 1.20
Note that these are just some options we had in mind. The amalgam of them happened to form, in our eyes, a cohesive go-forward strategy.
These schedule updates are proposed in https://github.com/kubernetes/sig-release/pull/1058.
Ultimately, this is a community-wide decision, so it's critical that we get your feedback before proceeding.
Tomorrow (Tuesday, April 21) is our bi-weekly SIG Release meeting and we'll be dedicating it to discussing these options.
Please make an effort to attend, if you'd like to suggest changes to what we're proposing.
If you're unable to attend, please feel free to discuss on this thread. We'd ask that you direct any feedback to this thread or tomorrow's meeting (as opposed to Slack) so that it's easier to collect the various options.
We plan to make a decision in the coming days and will post to this thread with updates.
Best,
Stephen // Tim // Taylor // Bob // Jeremy (on behalf of SIG Release)
Stephen Augustus
After a lively discussion in today’s SIG Release meeting, we’d like to provide some updates to the proposed schedule!
Before getting into this (much shorter) update, we believe it’s important to understand some of our motivations for these changes:
Allow SIG Chairs & Technical Leads, reviewers/approvers more time to do architectural work, as well as review and approve issues/PRs in their purview
Allow contributors more time to deliver 1.19 content: KEPs, code, tests, docs
Minimize the complexity of consumption for Kubernetes users
Extend the support cycle of Kubernetes 1.15
The first thing we plan to do is keep Kubernetes 1.15 around, extending its support cycle until shortly after the first patch release of Kubernetes 1.19 (1.19.1).
For certain Kubernetes users, like some on managed Kubernetes, 1.15 has only recently become available, yet, we were on track to mark it as unsupported in the coming days.
By extending 1.15’s support cycle, we help delay infrastructure upgrades during this time.
This effectively extends 1.15’s total support cycle to more than a year.
This dovetails nicely with the WG LTS proposal around extending the overall support period to a year. The KEP is on track to merge soon, so please take a moment to review, if you have opinions.
Minimize consumer complexity
For the 1.19 release, we would like to minimize the burden consumers will face when they adopt the release. In previous releases, there have been disruptive changes that have made it more difficult to consume the release. We would like to ask each SIG to minimize any API removals or other changes that would otherwise require a "(No, really, you MUST read this before you upgrade) section in the release notes and to carefully consider PRs that have the release-note-action-required label.
Further extend the Kubernetes 1.19 release cycle
The updated 1.19 release schedule now includes the following additional changes:
Move Enhancements Freeze to week 6 (Tuesday, May 19)
Move Code Freeze to week 10 (Thursday, June 18)
Push Docs deadlines (Monday, June 29)
Move Cherry Pick Deadline to week 16 (Thursday, July 30)
Add an RC4 release (Tuesday, August 4)
Add a Release Blackout for the weeks surrounding KubeCon (Monday, August 10 - Friday, August 21)
Move 1.19.0 release date to week 20 (Tuesday, August 25)
Kubernetes 1.20 will be the final release for 2020
With an elongated Kubernetes 1.19 cycle, that leaves us at the end of August for 1.19.0, which means Kubernetes 1.20 work would start sometime in September.
Given the proximity to November and December holidays, it doesn’t seem feasible to try to squeeze two releases into that timeframe.
Instead, we will end 2020 with the release of Kubernetes 1.20, which will have an approximately 4-month release cycle.
During the 1.20 release cycle, we will begin to assess the release timelines for 2021.
Continue the conversation on feature branches/forks
With a longer period of merge restriction on master, we think this may be a good time to discuss feature branches/forks.
We’ll defer the discussion here, but please provide feedback on this issue, if that’s a development model you’d be interested in.
Lazy consensus
Thanks to the wonderful feedback from several Chairs/TLs/top-level approvers over the past few weeks, I think we’re close to the final state of the schedule.
That said, we’d like to move towards consensus on these proposed changes as soon as possible.
We’re setting a lazy consensus timeout on the new 1.19 release schedule for Friday, April 24, 5pm US ET.
Best,
Stephen // Tim // Taylor // Bob // Jeremy (on behalf of SIG Release)
Stephen Augustus
On Wed, 22 Apr 2020 at 03:46, Stephen Augustus <steph...@agst.us> wrote:
>
> The first thing we plan to do is keep Kubernetes 1.15 around
...
> This dovetails nicely with the WG LTS proposal around extending the overall support period to a year.
we removed the kubeadm 1.15 jobs already, because even if the LTS KEP
merged as an 1.19 "enhancement", 1.16 would have been the first
release to be supported for one year...or at least we though so.
lubomir
--
Stephen Augustus
John: "The people who are going to upgrade up to 1.19 actually aren't that much of a concern to me because they're kind of bleeding edge right like it's, it's the people who already have a difficult time keeping up with this pretty, pretty strenuous release train and now are in a situation where they don't have the resources to do it and they're gonna fall out of support."
Some time later (around 30:00)...
Dims (paraphrasing): "Can we bear the cost of tacking on an additional stable release so we don't let people fall off the wagon on the other side?"
Tim: "There's a KEP for that and we sort of said, 'Yeah, we can.' That Working Group (WG LTS) was actually probably by the end of this week saying that we have loose consensus and believe this is implementable, and put a lazy consensus on merge."
Dims: "Right, so if we do that, as in tack on an additional stable release, and then also say what Jordan was saying (no 'Action Required's). If we do those two, then we will probably serve a whole bunch of end users, I think."
John (paraphrasing): "I think that that was going to happen anyway. We already knew we had a problem and we were solving that problem by extending the support cycle, so that's necessarily provide relief because people are actually in a worse situation than when that KEP was written..."
Dims: "All I'm saying is we should just do that, for sure, and not let it slide anymore. Right?"
That section of the email was an extrapolation of this conversation.
-- Stephen
On Tue, Apr 21, 2020 at 17:46 Stephen Augustus <steph...@agst.us> wrote:
Extend the support cycle of Kubernetes 1.15
The first thing we plan to do is keep Kubernetes 1.15 around, extending its support cycle until shortly after the first patch release of Kubernetes 1.19 (1.19.1).
For certain Kubernetes users, like some on managed Kubernetes, 1.15 has only recently become available, yet, we were on track to mark it as unsupported in the coming days.
By extending 1.15’s support cycle, we help delay infrastructure upgrades during this time.
This effectively extends 1.15’s total support cycle to more than a year.
This dovetails nicely with the WG LTS proposal around extending the overall support period to a year. The KEP is on track to merge soon, so please take a moment to review, if you have opinions.
This is news to me. Where was this agreed?
John Belamaric
1.15's had a variety of things which folks on the k8s internals develeper side have specifically wanted to see finally go away. Retroactively telling developers they need to support things longer than they signed up for a year or so ago works counter to minimizing impact on everyone, while biasing to minimize impact on those who have not upgraded past 1.15.
--
You received this message because you are subscribed to the Google Groups "Kubernetes Release Team" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-releas...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-release-team/CAHuv5K%3DyZz2kLmSF5rsJKR%2BJrDPmgCwKH_3gJCG%2Bzpm5ovqSZg%40mail.gmail.com.
John Belamaric
On Wed, Apr 22, 2020 at 10:24 AM John Belamaric <jbela...@google.com> wrote:On Wed, Apr 22, 2020 at 10:01 AM Tim Pepper <tpe...@gmail.com> wrote:1.15's had a variety of things which folks on the k8s internals develeper side have specifically wanted to see finally go away. Retroactively telling developers they need to support things longer than they signed up for a year or so ago works counter to minimizing impact on everyone, while biasing to minimize impact on those who have not upgraded past 1.15.With the release schedule change, the 1.15 support pushes out the 6-7 weeks. It will require the LTS KEP to get any more than that, and it sounds like there are some things to work out still there.
1.15 is supposed to go out of support last week. Stephen's working on fixing some issues blocking what was to be its final release, prior to his proposal yesterday.Your statement would hold for 1.16 and is a modest shift. I think that's reasonable.
Tim, your statement is true, but I think that's an appropriate bias. I prefer to bias toward the end user, even in non-pandemic times, and at this time I think devs have more flexibility than end users so that is an even stronger sentiment. That said...it's a discussion for the LTS PR, I think we can pick it up there. Personally, I am up-in-the air on which release should start the 1 year schedule, but at present I think it should be something well before 1.19.
I do agree with the arguments in favor of biasing towards simplifying things for consumers of the project. That is good and is why I've been at this WG LTS thing for about two years now. But: I do not want to retroactively dramatically add as much as 5 months of support to 1.15 and add the associated demands on our developer community, especially in the current context of aiming to simplify things for everybody.Tim
Stephen Augustus
Following the feedback from our previous email…
Extend the support cycle of Kubernetes 1.16 (NOT 1.15)
This was a miscommunication on our part and we apologize for the confusion.
We’ll plan on extending the support for Kubernetes 1.16, instead of Kubernetes 1.15.
Please review the WG LTS KEP: https://github.com/kubernetes/enhancements/pull/1497
Updated Kubernetes 1.19 release cycle
We heard your feedback on the total length of the cycle, and more specifically, the length of Code Freeze.
I’ve put together an updated PR for the release schedule: https://github.com/kubernetes/sig-release/pull/1065
tl;dr for new (new) schedule:
Move Enhancements Freeze to week 6 (was week 4) (Tuesday, May 19)
Move Code Freeze to week 11 (was week 9) (Thursday, June 25)
Deviation from SIG Release meeting:
We give an extra week in the dev cycle to shorten Code Freeze to 6 weeks.
Several comments on the previous proposal said that Code Freeze was too long.RCs get released on a biweekly cadence (week 11, 13, 15)
Move Cherry Pick Deadline to week 16 (was week 11) (Thursday, July 30)
Move 1.19.0 release date to week 17 (was week 12) (Tuesday, August 4)
This is the week before KubeCon.Move retro to week 19 (was week 12) (Thursday, August 20)
This is the week after KubeCon.Push Docs deadlines
Add a Test Freeze milestone to coincide with Cherry Pick deadline
From @spiffxp on previous review:
"Specific request from SIG Arch conformance: typically code freeze is
the date by which no new conformance tests are accepted, so as not to
disrupt CI Signal and force them to chase the stability of a moving
target. Can we get a test freeze date later in the cycle, to allow
addition of new tests beforehand, but force fixing/reverting tests
afterward for CI Signal stability?"
Please comment on the PR with any additional feedback.
Best,
Stephen // Tim // Taylor // Bob // Jeremy (on behalf of SIG Release)