Kubernetes 1.2 Timeline

[T.J. Goltermann]

After discussion in today’s community meeting, I wanted to send out a proposed timeline for the remainder of the 1.2 release to this list.  I’ve also added this to the Release 1.2 wiki.

To date, k8s minor releases have been about 3 months apart, with 8 weeks of coding and 4 weeks of bug fixing and release.  1.2 has been roughly on this timeline thus far (minus some time off for holidays).  We’re coming to the close of the coding part of our milestone.

After several days of setting aside feature work for cleaning up tests, and surveying the state of several remaining 1.2 features (many are very close, or are in review), we still need a little more time to finish.

Given our three month per release target, and where we are today, the dates align well to the below:

Feb 5 (1w away) - 1.2 feature complete

• only bugfixes, not features after this date

• begin code slush on head, turn away features and large refactors

• cut Alpha release candidate

Feb 19 (+2w from feature complete) - declare Beta

• cut Beta RC and branch head into a 1.2-only branch

• end code slush on master, open for 1.3+ work

Mar 7 (+2w from Beta) - release 1.2

• this will depend on stability and bugs remaining

For Kubernetes 1.3, there are clearly a few things we need to improve on:

• propose these dates and timeline earlier, gather feedback, keep the wiki documentation updated and broadcast widely

• have a clear list of features that are important for the release, and track them in one place (milestone, label, or wiki) in GitHub

• revisit the dates in the community meeting more often to see if we think things are on track or not

--

-T.J. Goltermann

[Aaron Crickenberger]

Thanks for the additional details!  Which of these milestones should I be watching to track our progress?

- https://github.com/kubernetes/kubernetes/milestones/v1.2 (105/156 remain open)

- https://github.com/kubernetes/kubernetes/milestones/v1.2-candidate (179/231 remain open)

- aaron

[Brian Grant]

On Friday, January 29, 2016 at 10:42:09 AM UTC-8, Aaron Crickenberger wrote:

Thanks for the additional details!  Which of these milestones should I be watching to track our progress?

- https://github.com/kubernetes/kubernetes/milestones/v1.2 (105/156 remain open)

^^^^^^^^^^^ This one. But we haven't scrubbed it recently to ensure it's up to date. I'll make a quick pass over it.

The main features being worked on are enumerated here:

https://github.com/kubernetes/kubernetes/wiki/Release-1.2

[T.J. Goltermann]

After a bunch of PR merges Friday, over the weekend and yesterday, all the key 1.2 features are in, and it looks like we can declare feature complete today.

This means that the committers would start turning away new features and large refactors (entering a code slush).  We’ll pivot to fixing bugs and testing more and more, and triaging the v1.2 and v1.2-candidate milestones and driving them to zero issues.

A handful of 1.2 features still have work like moving to the v1 API, smaller refactors, adding tests, updating kubectl, etc., and we’d still let those code changes in for those features (Horizontal Pod Autoscaling, Job, Ingress, ConfigMap, Deployment, DaemonSet).

The proposed schedule from a few weeks ago is here, and we’re still roughly on track for that.  That would mean we cut a Beta release and branch head around Feb 19 (~2w).  Then release would be about 2-3w after that.  We obviously won’t release until testing is complete, documentation is updated, and the long tail of issues is fixed, but that’s a good estimate based on how 1.1 went.

If you have any concerns about:

1. Features still missing that need to be in 1.2

2. The proposed schedule

please speak up - either let me know, or bring it up at this week’s community meeting.  Otherwise, we’ll get 1.2 wrapped up and out the door soon!

-T.J. Goltermann

--

-T.J. Goltermann

[T.J. Goltermann]

After discussion in today’s community meeting, we’ve decided to enter a day-to-day slip of branching Kubernetes into a 1.2 release branch.  Our previous target was to branch tomorrow, Feb 19, ahead of our Mar 7 target release date (see release 1.2 wiki).

We did this since the number of issues we need to fix for 1.2 remain high.  We didn’t want to branch just to meet an arbitrary date estimate, or to branch early and just incur more engineering work to make sure both the head and 1.2 branches have all the right fixes.

The issues that block our release are captured as P0 or P1 issues (about 80 total - though some are docs and examples) in the v1.2 milestone.  P2 and P3 in that milestone are non-blockers we’d like to merge, and all items in the v1.2-candidate milestone are issues that need triage in or out of v1.2.

While the total count of those two milestones is around 200 issues, many are P2 and P3 non-blockers, or doc bugs that we expect to be open at this stage in the release.  There are also around 50 PRs in those milestones, and around 45 things in the submit queue right now.

If you are able to help out with fixing these issues, it would be appreciated.  To help, we suggest:

• looking in the v1.2 and v1.2-candidate milestones

• finding something that’s unassigned (assign to yourself and fix it up)

• finding something that’s assigned, but with no recent comments (ping the issue to see if the assignee is still working on it, and possibly take it)

• volunteer on Slack, kubernetes-dev or within SIGs

• working with @justinsb on some of the AWS issues

Once the total number of issues is under control, we’ll look to branch, finish up and release 1.2.  There’s nothing magical about our Mar 7 original release target, but KubeCon is Mar 10-11, so it would be nice if we can align with that.

Since we were going to cut a beta release on the day we branch, we won’t be generating that yet either.  To make up for that, we’ll be creating one more alpha release today or tomorrow.

-T.J. Goltermann

--

-T.J. Goltermann

[T.J. Goltermann]

We've been holding 3x per week milestone burndown meetings to look at each and every v1.2 milestone issue.  As of today, it looks like we've made enough progress, and that the bugs left are small enough that they can be cherrypicked.  There are still many test flakes, but key ones have been resolved and the submit queue has been running much smoother.

Thus, we're going to branch 1.2 and release a Beta build this Friday morning, Mar 4.

Any fixes needed for 1.2 (and we do expect there to still be fixes) will need to be put in the 1.2 branch and head.  Likewise, we will start enforcing that no code goes into 1.2 unless required for that release.

Historically, once branched, it's taken 1.5 - 2 weeks to get to a final release.

-T.J. Goltermann

--

-T.J. Goltermann

[David McMahon]

Just to further clarify, I will be branching at 11am PST on Friday March 4th.  

Any changes that are not in the master branch at that point will not be in the 1.2 branch.  If you have changes that are LGTM'd but not submitted by 11am PST on Friday March 4th, you will have to cherrypick those changes into the branch at some point.

David

--
You received this message because you are subscribed to the Google Groups "kubernetes-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CANsF9wr7xD_taBstF1YcWT-w5CV_CW0-5eMftC%3DzztLqb65%2B3A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Brian Grant]

Additionally, a USER DOC FREEZE will begin when we cut the branch. This will affect docs in:

docs/user-guide

docs/admin

docs/getting-started-guides

Except for generated docs (kubectl, man, api-reference, etc.).

We're moving user docs to the kubernetes.github.io repo, as discussed in the following proposal:

https://github.com/kubernetes/kubernetes/pull/20874

Any PRs that touch docs in these directories will be blocked from merging and will need to be recreated in the new repo after we copy the docs over.

On Wednesday, March 2, 2016 at 1:20:52 PM UTC-8, David McMahon wrote:

Just to further clarify, I will be branching at 11am PST on Friday March 4th.  

Any changes that are not in the master branch at that point will not be in the 1.2 branch.  If you have changes that are LGTM'd but not submitted by 11am PST on Friday March 4th, you will have to cherrypick those changes into the branch at some point.

David

On Wed, Mar 2, 2016 at 12:32 PM, 'T.J. Goltermann' via kubernetes-dev wrote:

We've been holding 3x per week milestone burndown meetings to look at each and every v1.2 milestone issue.  As of today, it looks like we've made enough progress, and that the bugs left are small enough that they can be cherrypicked.  There are still many test flakes, but key ones have been resolved and the submit queue has been running much smoother.

Thus, we're going to branch 1.2 and release a Beta build this Friday morning, Mar 4.

Any fixes needed for 1.2 (and we do expect there to still be fixes) will need to be put in the 1.2 branch and head.  Likewise, we will start enforcing that no code goes into 1.2 unless required for that release.

Historically, once branched, it's taken 1.5 - 2 weeks to get to a final release.

-T.J. Goltermann

On Thu, Feb 18, 2016 at 4:35 PM, T.J. Goltermann wrote:

After discussion in today’s community meeting, we’ve decided to enter a day-to-day slip of branching Kubernetes into a 1.2 release branch.  Our previous target was to branch tomorrow, Feb 19, ahead of our Mar 7 target release date (see release 1.2 wiki).

We did this since the number of issues we need to fix for 1.2 remain high.  We didn’t want to branch just to meet an arbitrary date estimate, or to branch early and just incur more engineering work to make sure both the head and 1.2 branches have all the right fixes.

The issues that block our release are captured as P0 or P1 issues (about 80 total - though some are docs and examples) in the v1.2 milestone.  P2 and P3 in that milestone are non-blockers we’d like to merge, and all items in the v1.2-candidate milestone are issues that need triage in or out of v1.2.

While the total count of those two milestones is around 200 issues, many are P2 and P3 non-blockers, or doc bugs that we expect to be open at this stage in the release.  There are also around 50 PRs in those milestones, and around 45 things in the submit queue right now.

If you are able to help out with fixing these issues, it would be appreciated.  To help, we suggest:

• looking in the v1.2 and v1.2-candidate milestones

• finding something that’s unassigned (assign to yourself and fix it up)

• finding something that’s assigned, but with no recent comments (ping the issue to see if the assignee is still working on it, and possibly take it)

• volunteer on Slack, kubernetes-dev or within SIGs

• working with @justinsb on some of the AWS issues

Once the total number of issues is under control, we’ll look to branch, finish up and release 1.2.  There’s nothing magical about our Mar 7 original release target, but KubeCon is Mar 10-11, so it would be nice if we can align with that.

Since we were going to cut a beta release on the day we branch, we won’t be generating that yet either.  To make up for that, we’ll be creating one more alpha release today or tomorrow.

-T.J. Goltermann

--

-T.J. Goltermann

[Brian Grant]

The v1.2-candidate milestone is now closed. We're focused on closing the P0 and P1 issues on v1.2. Any PRs merged after the branch cut today will need to be cherrypicked into the release branch.

[Brian Grant]

The doc freeze is in effect. 

All affected PRs are now labeled kind/old-docs and do-not-merge. 

The submit will not merge such PRs. Please do not merge any manually, or the content may be lost.

The branch hasn't been cut yet, but we will likely choose a cut point before now, assuming tests pass.

Once we do branch, we will merge a PR to gut the content of these docs in kubernetes/master, forcing all PRs touching the affected docs to be rebased.

https://github.com/kubernetes/kubernetes/pull/22490

When we unfreeze the docs, they'll be in a new repo: kubernetes/kubernetes.github.io. You'll need to reapply your doc changes to the files in that repo.

More documentation of the process will be forthcoming.

On Wednesday, March 2, 2016 at 6:52:15 PM UTC-8, Brian Grant wrote:

[Aaron Crickenberger]

"The branch hasn't been cut yet, but we will likely choose a cut point before now, assuming tests pass."

To be clear, what I heard during today's meeting was "all the 'critical' tests except serial are passing.  At the commit where serial passes, assuming all other tests are passing, we will branch."

To my knowledge the dashboard used to display these results is internal to google, but Brian held his laptop up to the camera and showed a dashboard with lots of green boxes.

- aaron

[David McMahon]

The cut of the 1.2 branch is awaiting a green set of Jenkins build jobs to complete.

We expect that this commit will be the branch point, but we're still awaiting final test results:

commit 4d599ea3090b36b8c8658695495e66daa5bc8d80

Merge: f9c4b3d 6a873e0

Author: Abhi Shah <abs...@google.com>

Date:   Fri Mar 4 10:55:12 2016 -0800

    Merge pull request #22261 from gmarek/kube-up

    

    kube-up for GCE chooses master size based on number of nodes

For any PRs/commits that land after this branch point, please do not cherrypick them into the branch.  

Please do mark your PRs with the cherrypick-candidate label on github so we can apply these cherrypicks to the branch at a later point.

David

--
You received this message because you are subscribed to the Google Groups "kubernetes-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/a93f238f-468d-4b80-a4c5-4423180c2022%40googlegroups.com.

[David McMahon]

The release-1.2 branch has been created at the commit I mentioned earlier (4d599ea3090b36b8c8658695495e66daa5bc8d80).

Again, for any PRs/commits that land after this branch point, please do not cherrypick them into the branch.  

Please do mark your PRs with the cherrypick-candidate label on github so we can apply these cherrypicks to the branch at a later point.

We didn't create a release entry the last time we branched, but it seems useful to collect items that landed between v1.2.0-alpha.8..v1.2.0-beta.0.

There isn't much now:

* Change default --max-pods in kubelet to 110 (#21361, @yujuhong)

* add configurable source range for loadbalancer on gce (#21431, @freehan)

If you'd like any of your PRs added to the release notes, get them labeled with the release-note label in the next couple of hours.  I'll generate an entry then.

We can also add additional notes later as needed.

David

[Brian Grant]

PLEASE DO NOT CHERRYPICK ANY PRS INTO THE 1.2 RELEASE BRANCH AT THIS TIME.

Instead, please add the cherrypick-candidate label to those PRs.

The process in past releases (1.0, 1.1) was to encourage individual contributors to cherrypick their own PRs into the release branch. However, this approach had a number of disadvantages:

• It was difficult to determine whether all necessary PRs merged in master since the branch cut had been cherrypicked.
• It was difficult to prevent PRs from being cherrypicked that shouldn't have been.
• Some PRs were merged out of order, which made subsequent PRs more difficult to cherrypick.

Additionally, we don't yet have continuous testing of the release branch. We shouldn't start merging any PRs into it until we do.

My current thinking is that we should do 2 reviews of PRs merged into master per week to review the cherrypick candidates, and then someone should be responsible for cherrypicking them in the proper order.

On Friday, March 4, 2016 at 4:32:24 PM UTC-8, David McMahon wrote:

The release-1.2 branch has been created at the commit I mentioned earlier (4d599ea3090b36b8c8658695495e66daa5bc8d80).

Again, for any PRs/commits that land after this branch point, please do not cherrypick them into the branch.  

Please do mark your PRs with the cherrypick-candidate label on github so we can apply these cherrypicks to the branch at a later point.

We didn't create a release entry the last time we branched, but it seems useful to collect items that landed between v1.2.0-alpha.8..v1.2.0-beta.0.

There isn't much now:

* Change default --max-pods in kubelet to 110 (#21361, @yujuhong)

* add configurable source range for loadbalancer on gce (#21431, @freehan)

If you'd like any of your PRs added to the release notes, get them labeled with the release-note label in the next couple of hours.  I'll generate an entry then.

We can also add additional notes later as needed.

David

On Fri, Mar 4, 2016 at 1:47 PM, David McMahon  wrote:

The cut of the 1.2 branch is awaiting a green set of Jenkins build jobs to complete.

We expect that this commit will be the branch point, but we're still awaiting final test results:

commit 4d599ea3090b36b8c8658695495e66daa5bc8d80

Merge: f9c4b3d 6a873e0

Author: Abhi Shah 

Date:   Fri Mar 4 10:55:12 2016 -0800

    Merge pull request #22261 from gmarek/kube-up

    

    kube-up for GCE chooses master size based on number of nodes

For any PRs/commits that land after this branch point, please do not cherrypick them into the branch.  

Please do mark your PRs with the cherrypick-candidate label on github so we can apply these cherrypicks to the branch at a later point.

David

[Dawn Chen]

Since we might re-cut the branch for 1.2 release, any PRs are not for
1.2 release shouldn't be merged to HEAD too.

> --
> You received this message because you are subscribed to the Google Groups
> "kubernetes-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kubernetes-de...@googlegroups.com.
> To post to this group, send email to kuberne...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/kubernetes-dev/d4509a1b-8a8b-42d5-9b95-5855d36f21ee%40googlegroups.com.

[Eric Paris]

Where is this discussion/decision being made? I'm not saying I'm
opposed, but it is extremely 'out of the blue' from where I sit...

-Eric

[Brian Grant]

Consider this part of the above cherrypick process proposal. Trying to reduce review work and merge difficulty. At minimum we're in code slush until we have 1.2.0, even according to previous policy.

[Brian Grant]

Build and unit test for 1.2 are up.

Tracking issue for e2e: https://github.com/kubernetes/kubernetes/issues/22672

[Brian Grant]

The release branch is already 222 commits behind master:

https://github.com/kubernetes/kubernetes/tree/release-1.2

[Brian Grant]

I'm reviewing all commits since the release branch was made. If there is anything you're aware of that should NOT be in 1.2, please let me know.

On Monday, March 7, 2016 at 6:16:44 PM UTC-8, Brian Grant wrote:

The release branch is already 222 commits behind master:

https://github.com/kubernetes/kubernetes/tree/release-1.2

On Mon, Mar 7, 2016 at 6:13 PM, Brian Grant wrote:

Build and unit test for 1.2 are up.

Tracking issue for e2e: https://github.com/kubernetes/kubernetes/issues/22672

On Mon, Mar 7, 2016 at 5:48 PM, Brian Grant wrote:

Consider this part of the above cherrypick process proposal. Trying to reduce review work and merge difficulty. At minimum we're in code slush until we have 1.2.0, even according to previous policy.

[Eric Paris]

It is a temporary location and I'm still playing with it, but:

http://vm-241-178-132-209.osop.rhcloud.com:8081/#/queue

should give a list of all PRs with the cherrypick-candidate label
sorted by merge date. If we unlabel all of the completed cherrypicks it
might give us reasonable visability going forward, since eventually we
are going to cherrypick instead of fast forwarding...

-Eric

On Tue, 2016-03-08 at 10:35 -0800, Brian Grant wrote:
> I'm reviewing all commits since the release branch was made. If there
> is anything you're aware of that should NOT be in 1.2, please let me
> know.
>

> > The release branch is already 222 commits behind master:
> > https://github.com/kubernetes/kubernetes/tree/release-1.2
> >
> > On Mon, Mar 7, 2016 at 6:13 PM, Brian Grant wrote:
> > > Build and unit test for 1.2 are up.
> > >
> > > Tracking issue for e2e: https://github.com/kubernetes/kubernetes/
> > > issues/22672
> > >
> > > On Mon, Mar 7, 2016 at 5:48 PM, Brian Grant wrote:
> > > > Consider this part of the above cherrypick process proposal.
> > > > Trying to reduce review work and merge difficulty. At minimum
> > > > we're in code slush until we have 1.2.0, even according to
> > > > previous policy.
> > > > On Mar 7, 2016 5:13 PM, "Eric Paris" wrote:
> > > > > Where is this discussion/decision being made? I'm not saying
> > > > > I'm
> > > > > opposed, but it is extremely 'out of the blue' from where I
> > > > > sit...
> > > > >
> > > > > -Eric
> > > > >
> > > > > On Mon, 2016-03-07 at 16:38 -0800, Dawn Chen wrote:
> > > > > > Since we might re-cut the branch for 1.2 release, any PRs
> > > > > are not for
> > > > > > 1.2 release shouldn't be merged to HEAD too.
> > > > > >
> > > > > > On Mon, Mar 7, 2016 at 1:59 PM, 'Brian Grant' via
> > > > > kubernetes-dev

[Brian Grant]

I reviewed all commits from the release branch up to a few minutes ago:

commit 43aa3d34ab76857263e7ed7025a2aabc292d8bc5

Merge pull request #18338 from gmarek/register-kubelet

I'm glad I did. cherrypick-candidate was applied very inconsistently. There were many PRs with milestone-v1.2 that didn't have the label, and vice versa. There were also a number of fairly harmless PRs -- the most risky ones already were labeled cherrypick-candidate.  I didn't find any PRs marked for cherrypicking that we obviously wouldn't want to patch over.

The only PRs I found that we might not want to cherrypick are:

https://github.com/kubernetes/kubernetes/pull/22667

which was reverted by:

https://github.com/kubernetes/kubernetes/pull/22690

(so cherrypicking would be harmless)

And one that is awaiting more testing:
https://github.com/kubernetes/kubernetes/pull/22430

If we tried to cherrypick everything in one big PR, do we think that would work?

Want to give that a try, Eric? :-)

On Tuesday, March 8, 2016 at 10:55:14 AM UTC-8, Eric Paris wrote:

It is a temporary location and I'm still playing with it, but:

http://vm-241-178-132-209.osop.rhcloud.com:8081/#/queue

should give a list of all PRs with the cherrypick-candidate label
sorted by merge date.

The sorting is what this does that looking at the PRs via github does not?

It would be useful to show which ones were NOT labeled cherrypick-candidate, also, but a different color or something.

 

If we unlabel all of the completed cherrypicks it
might give us reasonable visability going forward, since eventually we
are going to cherrypick instead of fast forwarding...

That sounds like a good idea. Or maybe add a cherrypicked label? Or do we need a cherrypicked label per release branch?

 

[Brian Grant]

Rollup "cherrypick" PR: https://github.com/kubernetes/kubernetes/pull/22710

...

[Brian Grant]

22710 was merged. Thanks to Eric Paris for generating it.

...