Annual patch support lifecycle

[Tim Pepper]

Following much discussion with SIGs and the community over the past nearly two years, WG LTS is proposing that the previously merged as "provisional" KEP:

https://github.com/kubernetes/enhancements/tree/master/keps/sig-release/1498-kubernetes-yearly-support-period

 

is indeed implementable ahead of the 1.19 enhancements freeze:

https://github.com/kubernetes/enhancements/pull/1782

 

As proposed, this is a tracked enhancement for the 1.19 milestone in keeping with an earlier intent to cover 1.19 and newer.

 

But...:

 

Additional more recent discussion on the k-dev list:

https://groups.google.com/d/msg/kubernetes-dev/IVpiIOZ4WcM/I_LGDpNkAgAJ

 

and subsequent discussion this week at the WG LTS meeting:

* video: https://youtu.be/lkDYaqPLRDM

* minutes: https://docs.google.com/document/d/1J2CJ-q9WlvCnIVkoEo9tAo19h08kOgUJAS3HxaSMsLA/edit#

 

aaand subsequent discussion this week on Slack:

https://kubernetes.slack.com/archives/CDTMPM75W/p1589381587015300

 

leads us to a place where WG LTS and SIG Release are also in support of retroactively extending planned support for 1.18, 1.17, and 1.16.  General consensus has been this is viable to attempt as a community, 1.16 marks a first point after some notable deprecation changes in favor of stable APIs, and in the context of operational complexities our project's downstream consumers are facing due to COVID-19 there is increased desire for less "must upgrade" pressure multiple times each year due to a release being end-of-support.

 

The 1.16 release would normally have reached end of patch support in June, so it may seem there isn't not an immediate pressure for a choice.  Still I would like to propose a lazy consensus period ending a noon US Pacific time on Friday May 22, 2020 to ensure a decision does not move forward without any possibly remaining major concerns having one more opportunity to be raised, and then also leaves time to communicate more broadly the decision for any operators looking at the need to move past 1.16 in the coming weeks.

 

(Note: Discussion regarding the number of releases per year can similarly continue on its own pace to eventual resolution.)

 

-- 

Tim Pepper

Orchestration Lead

VMware Open Source Technology Center

[Roger Klorese]

+1. This is very important to many of our customers. 

Roger B.A. Klorese (they/them or he/him)
Senior Product Manager
SUSE
255 King Street Suite 800

Seattle WA 98104
(P)+1 206.217.7432
(M)+1 425.444.5493
roger....@suse.com
Schedule a meeting: https://doodle.com/RogerKlorese
GPG Key: D567 F186 A6AE D244 067E  95E4 E67D 019F 0670 D9CC

On May 15, 2020, at 11:46 AM, Tim Pepper <tpe...@vmware.com> wrote:



--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/08ED552B-7EDA-46F8-9B23-4094C7397E20%40vmware.com.

[Carlos Tadeu Panato Jr]

+1

--

[Tim Pepper]

It’s been brought to my attention that there’s still a point of insufficient clarity here:

 

There are two topics for discussion so folks can express support or concern separately and still hopefully make progress on either or both without one possibly turning into a blocker for the other:

• 1.19 and newer receiving annual patch support
• Provisional KEP -> implementable (link [1])
• In discussion for 1.19 release milestone since January 2020

• lazy consensus period ending a noon US Pacific time on Friday May 22, 2020

• 1.18, 1.17, 1.16 receiving annual patch support
• k-dev discussion (link [2])
• In discussions from April/May 2020 given new COVID-19 context

• lazy consensus period ending a noon US Pacific time on Friday May 22, 2020

 

[1] https://github.com/kubernetes/enhancements/pull/1782

[2] https://groups.google.com/d/msg/kubernetes-dev/IVpiIOZ4WcM/I_LGDpNkAgAJ

 

-- 

Tim Pepper

Orchestration Lead

VMware Open Source Technology Center

 

--

[John Belamaric]

+1 to both, I /lgtm'd the KEP PR.

[Stephen Augustus]

I'm in favor of both officially starting annual support in 1.19 AND retroactively enabling it for Kubernetes 1.16 through 1.18.

Given we've set a lazy consensus timeout for Friday, which is after Enhancements Freeze, I'm also submitting this to the Release Team as an exception for the 1.19 cycle.

• Enhancement name: Kubernetes Annual Support
• Enhancement status (alpha/beta/stable): Stable -- policy-based KEP
• SIG: Release
• k/enhancements repo issue #: https://github.com/kubernetes/enhancements/issues/1498
• PR #’s: https://github.com/kubernetes/enhancements/pull/1782
• Additional time needed (in days): 3 days post-Enhancements Freeze (targeting implementable for Friday, May 22)

-- Stephen

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/7A15D6BD-B4D6-45F5-BFF5-A9ABB6EDE084%40vmware.com.

[Stephen Augustus]

To be clear, this is NOT about minting LTS releases/enabling a direct upgrade for say 1.16 --> 1.20.

This KEP is about extending the support cycle of individual releases.

Please review the KEP and let us know if there's something that could use further clarification in the verbiage.

-- Stephen

On Tue, May 19, 2020, 19:08 Roger Klorese <roger....@suse.com> wrote:

I’m confused by the term “annual support” – some might interpret that as the ability to upgrade annually and not step through intervening releases. We’re talking about a 12-month (or 4-release? Does it depend on what we do with frequency?) support policy, right?

 

-- 

 

Roger Klorese 
Senior Product Manager, SUSE CaaS Platform

 

SUSE

 

+1 206 217 7432 office
+1 425 444 5493 mobile

roger....@suse.com

 

255 King Street Suite 800

Seattle WA 98104 US

 

www.suse.com

GPG Key: 46D6 297E D349 0A68 9BB7 7362 9454 9852 704C 64CA

 

LinkedIn | Instagram | Twitter | YouTube

 

 

From: <kuberne...@googlegroups.com> on behalf of Stephen Augustus <steph...@agst.us>
Date: Tuesday, May 19, 2020 at 4:00 PM
To: Tim Pepper <tpe...@vmware.com>, Kubernetes Release Team <kubernetes-...@googlegroups.com>
Cc: Kubernetes developer/contributor discussion <kuberne...@googlegroups.com>, kubernetes-sig-release <kubernetes-...@googlegroups.com>
Subject: Re: Annual patch support lifecycle

 

I'm in favor of both officially starting annual support in 1.19 AND retroactively enabling it for Kubernetes 1.16 through 1.18.

 

Given we've set a lazy consensus timeout for Friday, which is after Enhancements Freeze, I'm also submitting this to the Release Team as an exception for the 1.19 cycle.

·         Enhancement name: Kubernetes Annual Support

·         Enhancement status (alpha/beta/stable): Stable -- policy-based KEP

·         SIG: Release

·         k/enhancements repo issue #: https://github.com/kubernetes/enhancements/issues/1498

·         PR #’s: https://github.com/kubernetes/enhancements/pull/1782

·         Additional time needed (in days): 3 days post-Enhancements Freeze (targeting implementable for Friday, May 22)

-- Stephen

 

On Tue, May 19, 2020 at 12:30 PM Tim Pepper <tpe...@vmware.com> wrote:

It’s been brought to my attention that there’s still a point of insufficient clarity here:

 

There are two topics for discussion so folks can express support or concern separately and still hopefully make progress on either or both without one possibly turning into a blocker for the other:

·      1.19 and newer receiving annual patch support

o  Provisional KEP -> implementable (link [1])

o  In discussion for 1.19 release milestone since January 2020

o  lazy consensus period ending a noon US Pacific time on Friday May 22, 2020

·      1.18, 1.17, 1.16 receiving annual patch support

o  k-dev discussion (link [2])

o  In discussions from April/May 2020 given new COVID-19 context

o  lazy consensus period ending a noon US Pacific time on Friday May 22, 2020

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAOqU-DQhc0JTDFNn10WFatAYr-0dx9_yzhv67RV-5gb1mxs-ag%40mail.gmail.com.

[Roger Klorese]

I’m confused by the term “annual support” – some might interpret that as the ability to upgrade annually and not step through intervening releases. We’re talking about a 12-month (or 4-release? Does it depend on what we do with frequency?) support policy, right?

 

-- 

 

Roger Klorese 
Senior Product Manager, SUSE CaaS Platform

 

SUSE

 

+1 206 217 7432 office
+1 425 444 5493 mobile

roger....@suse.com

 

255 King Street Suite 800
Seattle WA 98104 US

 

www.suse.com

GPG Key: 46D6 297E D349 0A68 9BB7 7362 9454 9852 704C 64CA

 

LinkedIn | Instagram | Twitter | YouTube

 

 

From: <kuberne...@googlegroups.com> on behalf of Stephen Augustus <steph...@agst.us>
Date: Tuesday, May 19, 2020 at 4:00 PM
To: Tim Pepper <tpe...@vmware.com>, Kubernetes Release Team <kubernetes-...@googlegroups.com>
Cc: Kubernetes developer/contributor discussion <kuberne...@googlegroups.com>, kubernetes-sig-release <kubernetes-...@googlegroups.com>
Subject: Re: Annual patch support lifecycle

 

I'm in favor of both officially starting annual support in 1.19 AND retroactively enabling it for Kubernetes 1.16 through 1.18.

 

Given we've set a lazy consensus timeout for Friday, which is after Enhancements Freeze, I'm also submitting this to the Release Team as an exception for the 1.19 cycle.

·         Enhancement name: Kubernetes Annual Support

·         Enhancement status (alpha/beta/stable): Stable -- policy-based KEP

·         SIG: Release

·         k/enhancements repo issue #: https://github.com/kubernetes/enhancements/issues/1498

·         PR #’s: https://github.com/kubernetes/enhancements/pull/1782

·         Additional time needed (in days): 3 days post-Enhancements Freeze (targeting implementable for Friday, May 22)

-- Stephen

 

On Tue, May 19, 2020 at 12:30 PM Tim Pepper <tpe...@vmware.com> wrote:

It’s been brought to my attention that there’s still a point of insufficient clarity here:

 

There are two topics for discussion so folks can express support or concern separately and still hopefully make progress on either or both without one possibly turning into a blocker for the other:

·      1.19 and newer receiving annual patch support

o  Provisional KEP -> implementable (link [1])

o  In discussion for 1.19 release milestone since January 2020

o  lazy consensus period ending a noon US Pacific time on Friday May 22, 2020

·      1.18, 1.17, 1.16 receiving annual patch support

o  k-dev discussion (link [2])

o  In discussions from April/May 2020 given new COVID-19 context

o  lazy consensus period ending a noon US Pacific time on Friday May 22, 2020

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAOqU-DQhc0JTDFNn10WFatAYr-0dx9_yzhv67RV-5gb1mxs-ag%40mail.gmail.com.

[Christoph Blecker]

I am still confused about how to give support on just one of these.

Moving the KEP to implementable with the goal of 1.19 forward having 12 full months of support, I'm in favour of.

The discussion around dependency policy (golang, bazel, etc) still needs to happen, and the KEP confirms that it's still an item we need to figure out. We still have time to solve it before we need to address it in 1.19. We don't have the same luxury with already released versions. I can't really support this change for 1.16-1.18 without us having a plan in place.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/7A15D6BD-B4D6-45F5-BFF5-A9ABB6EDE084%40vmware.com.

[Tim Pepper]

On Wed, May 20, 2020 at 12:01 PM Christoph Blecker <cble...@gmail.com> wrote:

I am still confused about how to give support on just one of these.

Moving the KEP to implementable with the goal of 1.19 forward having 12 full months of support, I'm in favour of.

The discussion around dependency policy (golang, bazel, etc) still needs to happen, and the KEP confirms that it's still an item we need to figure out. We still have time to solve it before we need to address it in 1.19. We don't have the same luxury with already released versions. I can't really support this change for 1.16-1.18 without us having a plan in place.

 

Thanks for bringing this up Christoph!!  An...Gah!  I didn't do all of what I was supposed to after the WG LTS meeting earlier this week.  I was supposed to also add in the PR https://github.com/kubernetes/enhancements/pull/1782 a note about our general plan/policy around dependency management.  Something like:

When we say we provide patch support for a release, what do we mean?

• We will fix security issues in our Kubernetes code
• We will pick up security fixes from dependencies as they are made available, in specific versioned patch releases of the dependency in use
• We will not maintain forks of our major dependencies
• We will not attempt our own back ports of critical fixes to dependencies which are out of support from their own communities

This is a fairly simple support policy and unchanged as a statement relative to how things actually happen currently (a few exception though to even that in our deps, see https://github.com/kubernetes/kubernetes/tree/master/third_party/forked).  But it is clear this informal policy has not been stated in sufficient detail in a prominent location.  And changes to it were considered last year in WG LTS discussions.

Operationally we take one of a few paths:

1. leave our release using an old golang, eg: no compelling content for us to move
2. leave our release using an old golang, eg: too hard to move for some reason (like trying to move our 1.15 release during its patch support lifecycle relative to all the mod changes that came in golang 1.13)
3. move our release mid-way through its patch support lifecycle to a new golang

Golang currently has what is effectively one year of patch support for their twice yearly releases, although concretely it's more that they support a release X.Y until X.(Y+2) is released.  (For folks looking for more info see https://golang.org/doc/devel/release.html and https://github.com/golang/go/wiki/Go-Release-Cycle).  Their cycle does not quite align with ours and this will always be the case across most all dependencies unless they all have slow release cadences and long support lifetimes.  The impossibility of full alignment does lead to gaps.

In the two cases above where we leave our release using an old golang, we have established a pattern for one or two months where our community does still support our code albeit with a golang that is out of support from their community.  Thus far that hasn't notably bitten us, but we might do better.  Depending on specific release dates this happens a couple times a year, most recently already for 1.15.  But:  It has been quite rare that there are actually critical toolchain fixes only discovered at the end of the cycle or security fixes that have gone unpatched in a trailing release, upon which we still depend and actually need and want to try to make a late change (eg: once in late 2018?).  Past performance is not an indicator of the future though.

So for what we think we do know of the future...what does this look like specifically for recent Kubernetes releases relative to golang (leaving out other major deps which may also be less actively managed today, but which also do need more explicit thought/decision/documentation):

Kubernetes 1.15 (June 2019 - April 2020):

• started on golang 1.12
• did not move to golang 1.13
• golang 1.12 out of support Feb. 2020
• 9mo's support: gap on order of 2 months
• 12mo's support: n/a

Kubernetes 1.16 (Sept. 2019 - July 2020 OR Sept. 2020):

• started on golang 1.12
• moved to golang 1.13 Dec. 2019
• golang 1.13 should be in support through August or Sept. 2020
• 9mo's support: no gap
• 12mo's support: gap on order of weeks

Kubernetes 1.17 (Dec. 2019 - Oct. 2020 OR Dec. 2020):

• started on golang 1.13
• golang 1.13 should be in support through August or Sept. 2020
  
• 9mo's support: gap on order of weeks
• 12mo's support: gap on order of 3 months, should move to 1.14

Kubernetes 1.18 (Mar. 2020 - Jan. 2021 OR Mar. 2021):

• started on golang 1.13
• golang 1.13 should be in support through August or Sept. 2020
  
• 9mo's support: gap on oder of 6 months, must move to 1.14
• 12mo's support: gap on order of 7 months, must move to 1.14, still leaves gap on order of weeks unless moved also to golang 1.15

[NOTE: across collating all ^^^ numbers there's surely some off-by-one's...everybody please do note any inaccuracies so the conversation can be as specific and correct as possible]

In this I see a couple patterns:

• Work actively in the early to mid months of a Kubernetes release's lifecycle to move it forward one golang release.  AND that's underway with WIP to make it easier to do toolchain updates on an ongoing basis, document how/when/why we do so, and move to go 1.14 across our current releases (go Stephen Augustus and SIG Release's Release Engineering subproject and associated friends!!)
• Communicate more clearly that we are not making an end-to-end integrated distribution where all parts are always in support.  Beyond Kubernetes, if a consumer's apps are coupled with older components that consumer is going to need to manage those as a set, and/or work with a vendor for back port support, and/or otherwise ensure sufficient risk mitigation of their deployed sets of out-of-support component levels.

Tim

[Christoph Blecker]

Thanks for this Tim. A couple more things that come to mind reading this:

- I feel like we should be be consistent in our policy. Like, we either move a golang version midcycle, or we don't.

- Moving golang versions midcycle can have the advantage of getting security fixes.. but have the downside of introducing bugs and scale issues. We fix those in PRs separate from the golang bump itself, so it might mean more complex cherry picks, and cherry picks that might otherwise be out of our normal cherry pick policy. We should probably track PRs explicitly that tie back to fixing golang problems.

- If a security issue comes up for a golang version that is out of support by golang but *is* in support for kubernetes, what do we do? Pick back the golang upgrade which might break other things, or do we put up a big banner that says "use at your own risk"?

Moving forward with this knowing that one of our key, biggest dependencies isn't aligned right now is basically a risk calculation. If we have support gaps of any kind, how likely is it that something critical will come up.

It begs the question, "What is the value that Kubernetes is adding by extending our support, if the underlying tech like golang isn't supported or we are potentially compromising our stability guarantees?"

Again, decisions we can make and questions we can answer surely for 1.19, but I still question if it's a good idea to expand support for 1.16-1.18 back mid lifecycle.

I'm curious what some folks like Jordan or Wojtek think about these statements/questions.. am I making a big deal out of these situations? Or is there a decent chance will they come back to bite us?

[Tim Pepper]

Jordan shared some good thoughts Monday in the WG LTS meeting, suggesting he felt risk balance isn't much changed relative to today and in absolute terms is fairly low (especially if we formalize that we do a regular golang update mid-patch-support-lifetime). Rather than attempt to summarize or just point at the video, maybe Jordan can share them here?  I personally have been sceptical on the extension for older branches, but he swayed me.

Regarding scale, I'm reeeally hoping that the WG K8s Infra work leads to more vendors being able to pitch in funding for the types of scale testing they see needed, eg: Running more variations for example across the in-support golang versions and even golang's HEAD, or spikes at larger scale with specific configurations). Of course that also needs to be accompanied by human power for test design and support, not just buying machine power for test runs.

Tim

[Tim Pepper]

FYI Jordan's on (much deserved!!) vacation today. The "Monday" meeting is already a week and a half ago Monday May 12, 2020 and is currently the first in the WG LTS playlist on YouTube. It's the one I linked at the start of this thread ^^^.

It's also well past end of workday if I correctly recall Wojtek's timezone.

So it'll be next week before we see comments.

Tim

[Wojciech Tyczynski]

 It seems the discussions is mostly around golang now, so let me first share my thoughts about it first:

1. I agree that we should have a consistent policy. Saying that "we upgrade it there will be a need" will put us in troubles sooner or later.

  But I think such policy is possible. Ideally we should do this upgrade when master was upgraded to the newer version at least 1-2 months ago, but it seems that

  bumping golang version 1.X k8s roughly when 1.X+2 is being released should do the job (if golang releases will remain roughly as predictable as they are now).

2. I agree that we need to start tracking any fixes that are related somehow to Go. That includes all things: security, performance and bugs (things like #75887 come to my mind, but I'm sure Jordan can point more).

  We don't have a good process for it now, and imho we need this process before we extend the support timeline.

  

  BTW - I think this is the reason I thing that extending the support period backward in time (i.e. for past releases) may not be the best idea. Though obviously it's not super strong argument, because we have this problem already.

3. Purely about performance/scale: I can't remember any issues around that that had mitigations/fixes on our side. We were always working with Golang folks to fix the problem on their side.

 Yes - we're aware of the fact that in the past it was talking a lot of time and we were releasing faulty versions being aware of it.

 And that was extremely painful not only for us, but also for Golang folks.

 And this thing, we're actually addressing as we speak - we recently invested into setting up scalability tests k8s against golang tip: https://testgrid.k8s.io/sig-scalability-golang

 They are not perfect yet and there is still work ahead of us to iterated on them and improve, but after 3 faulty releases - 1.12, 1.13, 1.14 (discovered and fixed in different phases of the release cycles),

 it seems that 1.15 will be fine (at least is seems to be fine now).

 So while I fully understand the concerns around that topic, I really believe this should become less of a concern over time (and hopefully we will forget about it completely in few quarters).

4. I don't feel competent to comment about requirements/implications around security fixes.

So TL;DR; I believe we could make it work, but my personal preference would be not to apply the new policy for already released versions.

 Now one more point that I didn't see mentioned (sorry if I missed them):

5. What about etcd? I didn't find an official support policy for it and that is our super critical dependency. Should we work with them to get official support policy with them?

 thanks

wojtek

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAHuv5KmQz%2BUKBAdUW8KHPZ_wVkyNps3mqfiojqNX%2BFuGscx_vA%40mail.gmail.com.

[Tim Pepper]

On Monday, May 25, 2020 at 6:55:21 AM UTC-7, Wojciech Tyczynski wrote:

2. I agree that we need to start tracking any fixes that are related somehow to Go. That includes all things: security, performance and bugs (things like #75887 come to my mind, but I'm sure Jordan can point more).

  We don't have a good process for it now, and imho we need this process before we extend the support timeline.

The SIG Release subproject release engineering is working to get a better handle on this in practice and in documentation.  Stephen Augustus especially has made notable progress on build tooling cleanups and image content updates, plus more docs on when/why we move images.  We aim for a place where most of this (ie: build/test/staging) can be automatic and triggered by the dependency having pushed a new release/tag, and then a human review for promotion from staging to prod.

 

 Though obviously it's not super strong argument, because we have this problem already.

Part of these discussion around how we do support has been slowing coming to consensus and decisions, then documented policy and iteratively enhanced automation to finally do a better job on this long existing concern.  We are getting a lot better, but will probably never quite be "done".

  

5. What about etcd? I didn't find an official support policy for it and that is our super critical dependency. Should we work with them to get official support policy with them?

As luck would have it there was an etcd community meeting on the calendar for today.  Josh Berkus and I attended and discussed with multiple etcd mainters generally where Kubernetes and the 1498-kubernetes-yearly-support-period KEP are trending.  The meeting minutes are at:

https://docs.google.com/document/d/16XEGyPBisZvmmoIHSZzv__LoyOeluC5a4x353CX0SIM/edit#bookmark=id.cmg5t9e1sg8i 

They acknowledged their roadmap and docs are sparse and out of date.  At this time though there is no plan or expected need for a 4.x stream with major breaking changes.  They've been supporting two or three release streams and covering a cumulative span of time which is beyond what Kubernetes has done for 9 months or would need for 1 year support lifetime.  They aim to be doing semver and we should not encounter compatibility or breaking changes across the 3.x series.  If / where that happens it's a major bug and we have a track record of successful close interaction between the projects' maintainers, eg: in https://github.com/kubernetes/kubernetes/issues/91266 to track bumping up to the latest version 3.4.9 version.  This feels like one of our smaller dependency risks!

[Jordan Liggitt]

On Mon, May 25, 2020 at 9:55 AM 'Wojciech Tyczynski' via Kubernetes developer/contributor discussion <kuberne...@googlegroups.com> wrote:

 It seems the discussions is mostly around golang now, so let me first share my thoughts about it first:

1. I agree that we should have a consistent policy. Saying that "we upgrade it there will be a need" will put us in troubles sooner or later.

  But I think such policy is possible. Ideally we should do this upgrade when master was upgraded to the newer version at least 1-2 months ago, but it seems that

  bumping golang version 1.X k8s roughly when 1.X+2 is being released should do the job (if golang releases will remain roughly as predictable as they are now).

I don't think we should plan to switch golang minor versions on release branches. Experience over the last couple years has shown that switching minor versions of go consistently requires changes I would not expect to backport, or user-visible behavior changes, or involves interactions our automated tests could easily miss. Examples:

• go1.9 - stricter certificate subjectAltName checks, impacted admission webhook certificates
• go1.10 - required bumping cfssl dependency (https://github.com/kubernetes/kubernetes/pull/60373)
• go1.12 - example interaction between stdlib bug and kubernetes components (https://github.com/kubernetes/kubernetes/pull/74755)
• go1.13 - build changes, dependency changes for modules (https://github.com/kubernetes/kubernetes/pull/82809)
• go1.14 - bbolt/etcd version updates required for stricter unsafe checking (https://github.com/kubernetes/kubernetes/pull/88638#issuecomment-634173224)

In contrast, patch versions of go have proven very reliable/non-disruptive to adopt.

2. I agree that we need to start tracking any fixes that are related somehow to Go. That includes all things: security, performance and bugs (things like #75887 come to my mind, but I'm sure Jordan can point more).

  We don't have a good process for it now, and imho we need this process before we extend the support timeline.

  

  BTW - I think this is the reason I thing that extending the support period backward in time (i.e. for past releases) may not be the best idea. Though obviously it's not super strong argument, because we have this problem already.

The only dimension I think is relevant to extended support is security-related fixes. We don't guarantee all bugfixes will be backported to all Kubernetes releases, so I don't think a bug in the stdlib is significantly different. We have a reasonable window to identify any stdlib regressions and get those resolved while we are still within the golang support window.

I think it is reasonable to ask this question: "what golang security fixes impacted Kubernetes and were fixed in supported go versions that an older Kubernetes release branch did not pick up"

Looking at the go 1.7-1.12 streams (based on release notes that indicated security-related content), here's what I found:

• go1.7 - security releases between EOL (2017/05/23) and go1.10 (2018/02/16)
• none relevant to Kubernetes artifacts (go1.9.1 and go1.9.4 had unrelated security fixes)
• go1.8 - security releases between EOL (2018/02/07) and go1.11 (2018/08/24)
• none
• go1.9 - security releases between EOL (2018/06/05) and go1.12 (2019/02/25)
• 2018/12/12 - go1.11.3 - x509 CPU DOS
• 2019/01/23 - go1.11.5 - x509 CPU DOS for specific elliptic curves
• go1.10 - security releases between EOL (2019/01/23) and go1.13 (2019/09/03)
• 2019/08/13 - go1.12.8 - http/2 DOS, URL parsing issue
• go1.11 - security releases between EOL (2019/08/13) and of go1.14 (2020/02/25)
• 2019/09/25 - go1.13.1 - http header parsing/smuggling
• 2019/10/17 - go1.13.2 - crypto/dsa fix (recoverable panic in server handling)
• 2020/01/28 - go1.13.7 - ssh tunnel panic, windows certificate bypass workaround
• go1.12 - security releases between EOL (2020/02/12) and of go1.15 (~2020/08/15)
• none as of 2020-06-01

Of the 6 go versions I looked at, go1.11 was the one missing newer security fixes I wish we could have picked up on older Kubernetes releases. The other versions either had no security fixes or were primarily DOS-oriented (those are still notable, but our biggest DOS weaknesses don't come from the stdlib).

However, in every Kubernetes minor release, several significant bugs are found/fixed and considered worthy of backporting (fixing a severe enough issue at a low enough risk). There is value in making those fixes available to consumers of one additional Kubernetes release, even if there is an occasional stdlib fix that is not available to that older release.

We should definitely be clear about what go version a particular Kubernetes releases uses, and the support lifetime of that go version, so that Kubernetes consumers can make informed decisions about what Kubernetes versions to consume, but I would not withhold fixes to Kubernetes components from older release branches based on misalignment with golang releases that is rarely impactful.