[ACTION REQUIRED] Reopening kubernetes/kubernetes for v1.20 development

[Stephen Augustus]

SIG Leads and Kubernetes Community,

At the end of the v1.19 release, problems with test health and CI signal got significantly worse, due to a mix of bugs, problems with tests, and problems with test environments. This impacted development velocity significantly, and prompted policies to improve Kubernetes CI, and a proposal for a reliability working group to address test gaps and use test signal as a feedback loop to focus development efforts. Those initiatives are ongoing.

At the same time, due to the extended code freeze period at the end of 1.19, there is a significant backlog of reviewed/approved PRs.

Last Thursday, Jordan drafted a proposal for reopening development while supporting the project goal to "make every release a stability release".

Over the past few days, this proposal has been reviewed and approved by the SIG Chairs/TLs of SIG Release, SIG Architecture, and SIG Testing, along with several other members of SIG leadership across the community.

For some additional context on some of the decisions/edits we made while shaping the proposal, please see this Slack thread.

At a high level, the following steps will take place after a successful release of Kubernetes v1.19.0:

1. Limit merges to PRs in the v1.20 milestone

2. Merge PRs fixing failing tests

3. Merge PRs making cleanup/non-functional changes

4. Merge low-risk bugfix PRs

5. Resume regular maintenance/development

Please review the proposal for explicit steps and timelines.

SIG Release will be coordinating much of this work and we will send regular updates to kubernetes-dev and SIG leads as we progress.

SIG leads —

Please consider this approved proposal to be the plan of record.

If you have questions, comments, or concerns, let us know by commenting on the doc or replying to this thread.

Thanks again Jordan for putting this together so quickly and to the leads that turned around reviews within a day!

-- Stephen

[Stephen Augustus]

Hey kubefolx!

An update on reopening kubernetes/kubernetes for v1.20 development...

Following the release of Kubernetes v1.19.0, we've done the following:

• Opened (and closed) a merge blocker issue to freeze merges while we pruned the 1.20 milestone
• Shifted the milestone merge restrictions from v1.19 to v1.20
• Created new (temporary) milestones to stage v1.20 PRs:
• v1.20-phase-failing-test: https://github.com/kubernetes/kubernetes/milestone/53
• v1.20-phase-doc-cleanup: https://github.com/kubernetes/kubernetes/milestone/51
• v1.20-phase-bug: https://github.com/kubernetes/kubernetes/milestone/52
• v1.20-phase-feature: https://github.com/kubernetes/kubernetes/milestone/54
• Moved PRs classified as kind/failing-test into the v1.20 milestone

This puts us in step two of the reopening plan.

We've identified a test failure in pull-kubernetes-conformance-kind-ga-only-parallel, which Ben is working on resolving right now.

Once that's done, we should start to see v1.20 PRs merge.

A few asks:

• If you see some of your PRs in one of the temporary milestones, please do not remove them.
• If you need to stage a PR for the v1.20 milestone, please instead add it to one of the temporary milestones, based on its "kind". That'll ensure that we move it into the milestone at the right time.
• Please continue to update your SIG's/area's milestone maintainers. I've approved a few PRs for this already, so thank you to the leads who have issued updates recently.

Active convo on the phased reopen is happening on a thread in the #sig-release channel.

If you have any questions, reply here or on that thread.

Thanks everyone!

Stephen 

[Stephen Augustus]

8/27 Update:

• Ben fixed the kind issue blocking pull-kubernetes-conformance-kind-ga-only-parallel
• I've moved staged PRs from the v1.20-phase-failing-test and v1.20-phase-doc-cleanup into the v1.20 milestone and closed those temporary milestones
  
• Between last night and today, 47 PRs have merged into the v1.20 milestone!

This means we've completed the following steps:

1. Limit merges to PRs in the v1.20 milestone
2. Merge PRs fixing failing tests
3. Merge PRs making cleanup/non-functional changes

As we get ready to move into the next step (merge low-risk bugfix PRs), here are some asks:

• 23 PRs labeled as kind/failing-test, kind/documentation, or kind/cleanup remain in the v1.20 milestone.
  Please review/approve these if they are merge-ready.
• I've created the v1.20-phase-bug-low-risk milestone.
  Please review PRs in the v1.20-phase-bug milestone. If you deem them to be "low-risk" bug fixes, move them into the "v1.20-phase-bug-low-risk" milestone to stage them for the next set of merges.

Right now, we are well ahead of the estimates that we initially gave for each of these steps, so let's keep up the pace!

Around 2 PM US Eastern tomorrow (8/28), I will start moving staged PRs from v1.20-phase-bug-low-risk milestone into the v1.20 milestone (so Tide can merge some more PRs over the weekend).

Please try and make an assessment on those bug fixes by then.

Thanks!

Stephen

[Stephen Augustus]

9/1 Update:

• 53 open, 149 closed PRs in the 1.20 milestone
• Of the 53 open, 24 of those are "merge-ready"
• There were some job failures related to our shift of merge-blocking jobs to K8s Infra (tl;dr: it was DNS). More to follow as signal starts clearing up.

Since the last update, we've completed step 4 (merge low-risk bugfix PRs) of the reopening plan.

As we move into step 5 (resume regular maintenance/development), here are the updated asks for SIG leads/approvers/reviewers/milestone maintainers:

• Review your SIG/component's PRs in the v1.20 milestone and get them into a merge-ready state
  
• Do the same for the v1.20-phase-bug milestone. Once the merge-ready v1.20 milestone PRs have drained, I'll shift the remaining bug fix PRs into the milestone.
• You should now add the following PR types directly to the milestone:
• kind/failing-test
• kind/documentation
• kind/cleanup
• low-risk kind/bug
• critical-urgent kind/bug
  

Daily updates to follow as PRs merge.

Thanks for doing your part in this effort!

Stephen

[Stephen Augustus]

9/2 Update:

• 26 open, 198 closed in the 1.20 milestone
• None of the 26 open are merge-ready, which means we've cleared the active queue!

Guidance remains roughly the same:

• Review your SIG/component's PRs in the v1.20 milestone and get them into a merge-ready state

• Do the same for the v1.20-phase-bug milestone. As PRs become merge-ready, you may shift them into the v1.20 milestone.

• You should now add the following PR types directly to the milestone:
• kind/failing-test
• kind/documentation
• kind/cleanup
• low-risk kind/bug
• critical-urgent kind/bug

Thanks!

Stephen

[Stephen Augustus]

Hi kubefolx,

At this point, everything that was merge-ready at the time we've enacted the milestone restriction has merged.

Overall, we've merged 348 PRs into the 1.20 milestone, and all of the staging phased milestones have been closed.

The question on a lot of people's mind is, "Okay, well when do we remove the milestone restriction?".

I'd like us to answer that as a team.

I've sent a note to the leads mailing list to solicit feedback before we make any decisions here.

If you're a SIG or component area lead, please take a moment to comment on the following thread: https://groups.google.com/a/kubernetes.io/g/leads/c/BZZk0tG2cJU

As we talk through next steps, please continue to apply the 1.20 milestone to any PR you'd like to see included in the 1.20 release.

-- Stephen

[Stephen Augustus]

FYI -- I've opened a PR to remove the 1.20 milestone restriction. We'll discuss this in tomorrow's SIG Release meeting. Please drop by if you have opinions.

-- Stephen