The 12-month Hypothesis: does the data support it?

[Josh Berkus]

All:

Per the meeting today, I've been digging into our survey data to see if survey responses support extending patch support to 12+ months.

The first part of this is filtering respondees for folks who are using a version in production that would have just gone EOL when the survey was released.

We had 614 responses to the survey.  Of those, 324 users were using some version of Kubernetes in production, with the average user having 1.87 versions currently in production (median of 2).

Distribution of versions used looks like this:

┌─────────┬────────────────┐
│ version │ num_responding │
├─────────┼────────────────┤
│    1.06 │             12 │
│    1.07 │             23 │
│    1.08 │             31 │
│    1.09 │             51 │
│    1.10 │            106 │

│    1.11 │            159 │
│    1.12 │            130 │
│    1.13 │             94 │
└─────────┴────────────────┘

So that's 82 "instances" of 1.8 and 1.9 in production, used by 63 users.  That's a little less than 20% of the users who responded to the survey, which is a substantial minority.

One can also argue that the users on 1.10, who were facing a forced upgrade in March, were also potential beneficiaries of a 12-month policy; those users would add 70 to the total, or 41% of users.  However, that's argueable, since most of the survey responses are from january.

Let's see who those 63 users are:

┌──────────────────────────────────────────────────────────────────────────────────────────┬───────┐
│                                         persona                                          │ count │
├──────────────────────────────────────────────────────────────────────────────────────────┼───────┤
│ Kubernetes Cluster Operator (run Kubernetes)                                             │    25 │
│ Kubernetes Distributor (create/support of a Kubernetes software distribution)            │    13 │
│ Kubernetes Hosting Provider (host Kubernetes clusters for customers)                     │     9 │
│ Kubernetes Developer/Contributor (contributor to projects in the Kubernetes GitHub organ…│     6 │

│…izations)                                                                                │       │
│ Kubernetes Vendor (provide other software and/or services associated with Kubernetes)    │     6 │
│ Kubernetes Cluster User (app developer or deployer; create app level value on top of Kub…│     4 │
│…ernetes)                                                                                 │       │
└──────────────────────────────────────────────────────────────────────────────────────────┴───────┘
(6 rows)

So 29, or 46%, list themselves as operators or users.

Another interesting question is "How many users were running 1.8, 1.9 or 1.10 and NOT running 1.11, 1.12, or 1.13 anywhere?".  These are, arguably, our users who are being "forced" into upgrading as they have no experience with later versions.  The answer is 31, or 10% of users, with 55 (or 18%) not running later versions in production.

Let's see where the original 63 get their binaries.  Take this one with 10 grams of salt, because the related questions resulted in a lot of confusion on the part of our users.

┌───────────┬───────┐
│ bins_from │ users │
├───────────┼───────┤
│ other     │    15 │
│ self      │    21 │
│ selfmod   │    25 │
│ vendor    │    28 │
└───────────┴───────┘

So, 28, or 44%, get their binaries from vendors. Also, I thought it would be interesting to check how many get their binaries ONLY from vendors, and don't have a mix, and that's 14, or 22%.

So, overall we seem to be seeing that around 1/5 of our current survey respondees would potentially immediately benefit from a 12+ month sunset period, and that around half of those, or about 1 in 10 users, are solidly in the target audience for the proposed KEP.  If we broaden that to include users who *will* need to upgrade in < 2 months, then those numbers double.

Tommorrow, we'll do some full-text searching to see who's complaining about forced upgrades.

[Josh Berkus]

Now, let's look at what people said about their upgrade schedules.

Here's how often they are *able* to upgrade:

┌─────────────────────────────────────┬───────┐
│ freq_upgrade_able │ count │
├─────────────────────────────────────┼───────┤
│ Less than quarterly │ 54 │
│ N/A (I am a not a cluster operator) │ 12 │
│ Never │ 12 │
│ Once per year │ 55 │

│ Quarterly │ 140 │
│ Twice per year │ 62 │
└─────────────────────────────────────┴───────┘

As percentages:

┌─────────────────────────────────────┬─────┐
│ freq_upgrade_able │ pct │
├─────────────────────────────────────┼─────┤
│ Less than quarterly │ 16 │
│ N/A (I am a not a cluster operator) │ 4 │
│ Never │ 4 │
│ Once per year │ 16 │
│ Quarterly │ 42 │

│ Twice per year │ 19 │
└─────────────────────────────────────┴─────┘


Here's how often they'd *prefer* to upgrade:

┌─────────────────────────────────────┬───────┐
│ freq_upgrade_prefer │ count │
├─────────────────────────────────────┼───────┤
│ Less than quarterly │ 40 │
│ N/A (I am a not a cluster operator) │ 11 │
│ Never │ 2 │

│ Once per year │ 53 │
│ Quarterly │ 144 │
│ Twice per year │ 85 │
└─────────────────────────────────────┴───────┘

As %:
┌─────────────────────────────────────┬───────┐
│ freq_upgrade_prefer │ round │
├─────────────────────────────────────┼───────┤
│ Less than quarterly │ 12 │
│ N/A (I am a not a cluster operator) │ 3 │
│ Never │ 1 │
│ Once per year │ 16 │
│ Quarterly │ 43 │

│ Twice per year │ 25 │
└─────────────────────────────────────┴───────┘


So that's 16% who can/would like to upgrade once per year.

More datamining tommorrow if I have time.


--
--
Josh Berkus
Kubernetes Community
Red Hat OSPO

[Tim Pepper]

On Tuesday, November 12, 2019, Josh Berkus <jbe...@redhat.com> wrote:

┌─────────┬────────────────┐
│ version │ num_responding │
├─────────┼────────────────┤
│    1.06 │             12 │
│    1.07 │             23 │
│    1.08 │             31 │
│    1.09 │             51 │
│    1.10 │            106 │

== IN SUPPORT vvvv ======== ^^^^ OUT OF SUPPORT =======

│    1.11 │            159 │
│    1.12 │            130 │
│    1.13 │             94 │
└─────────┴────────────────┘

So that's 82 "instances" of 1.8 and 1.9 in production, used by 63 users.  That's a little less than 20% of the users who responded to the survey, which is a substantial minority.

One can also argue that the users on 1.10, who were facing a forced upgrade in March, were also potential beneficiaries of a 12-month policy; those users would add 70 to the total, or 41% of users.  However, that's argueable, since most of the survey responses are from january.

 

 

Today we support 3 releases and the survey ran February to April, entirely after 1.13’s release.  That means 1.10 was already out of support at the survey start and 1.11 went out of support in March during the survey.

That makes 106 more out of support versus your number and 159 more close to or out of support during the survey.

[Josh Berkus]

On 11/12/19 9:09 PM, Tim Pepper wrote:
> Today we support 3 releases and the survey ran February to April,
> entirely after 1.13’s release.  That means 1.10 was already out of
> support at the survey start and 1.11 went out of support in March during
> the survey.

You sure? Dhawal was saying December to February. The data I got
contained no timestamps, so I can't tell. Changes the data rather
substantially depending.

Can someone verify?

[Dhawal Yogesh Bhanushali]

Josh,

When you asked me on chat "when does the survey stretch through?" I said "till the 1.14 release I think or ever a little more. The survey was on for 3 months"

The 1.14 did not start immediately after 1.13 ended (in December). December we took a break in the next release (1.14) cycle was from Jan-Mar (https://github.com/kubernetes/sig-release/tree/master/releases/release-1.14). I think the December break got you confused.

Regards,

Dhawal.

--
You received this message because you are subscribed to the Google Groups "kubernetes-wg-lts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-wg-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-wg-lts/d83720c1-749e-a9e0-99f0-0a20dd3e6971%40redhat.com.

[Ihor Dvoretskyi]

I went ahead and checked the Surveymonkey data for you all.

The first response to the survey has been received on Saturday, February 23, 2019; the final - Monday, April 29, 2019.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-wg-lts/CABx4%3DBbCEw6ZEp0tq%2BipTr2hUY9kTV41Xepbkw%2B88%3DzESQh27Q%40mail.gmail.com.

[Tim Pepper]

On Tue, Nov 12, 2019 at 10:14 PM Josh Berkus <jbe...@redhat.com> wrote:

On 11/12/19 9:09 PM, Tim Pepper wrote:
> Today we support 3 releases and the survey ran February to April,
> entirely after 1.13’s release.  That means 1.10 was already out of
> support at the survey start and 1.11 went out of support in March during
> the survey.

You sure?  Dhawal was saying December to February.  The data I got
contained no timestamps, so I can't tell.  Changes the data rather
substantially depending.

Can someone verify?

Yep.  End of Feb to end of April are the bookends of announcement on k-dev list. 

[Jason DeTiberus]

One question that immediately comes to mind, does the disconnect between vendor-provided binaries and other binary sources cause a disconnect in the data due to support timelines offered by said vendors vs upstream support timelines?

I'm just wondering if there is a disconnect between consumers of vendor-provided binaries and desires for longer support cycles vs consumers of upstream binaries/source and if we are potentially looking at a bimodal distribution vs a normal distribution.

---

From: kubernet...@googlegroups.com <kubernet...@googlegroups.com> on behalf of Tim Pepper <tpe...@gmail.com>
Sent: Wednesday, November 13, 2019 9:59 AM
To: Josh Berkus <jbe...@redhat.com>
Cc: kubernetes-wg-lts <kubernet...@googlegroups.com>
Subject: Re: [kubernetes-wg-lts] The 12-month Hypothesis: does the data support it?

 

--

You received this message because you are subscribed to the Google Groups "kubernetes-wg-lts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-wg-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-wg-lts/CAHuv5Kn%2B2awJJALB7jXYH_kmbOrk55bdL4C10X7Ou8By1bjaOA%40mail.gmail.com.

[Josh Berkus]

On 11/13/19 12:35 AM, Ihor Dvoretskyi wrote:
> I went ahead and checked the Surveymonkey data for you all.
>
> The first response to the survey has been received on Saturday, February
> 23, 2019; the final - Monday, April 29, 2019.

Ok, lemme rerun the charts then, because that pretty substantially
changes the target group.

[Josh Berkus]

OK, so let's redo some of those charts now that we have the right dates.
Target versions for our 12-month window are now 1.9 and 1.10.

There are 124 users in that window, or 38% of all users running
Kubernetes in production. 50 of those users, or 15% of the whole pool,
aren't running *any* supported version in production.

Here's more about that set of users.

User Type:

┌───────────────────────────────────────────────────────────────────────────────────────────────────┬───────┐
│ persona
│ users │
├───────────────────────────────────────────────────────────────────────────────────────────────────┼───────┤

│ Kubernetes Cluster Operator (run Kubernetes)

│ 48 │

│ Kubernetes Cluster User (app developer or deployer; create app level

value on top of Kubernetes) │ 14 │

│ Kubernetes Developer/Contributor (contributor to projects in the

Kubernetes GitHub organizations) │ 11 │

│ Kubernetes Distributor (create/support of a Kubernetes software

distribution) │ 20 │

│ Kubernetes Hosting Provider (host Kubernetes clusters for customers)

│ 18 │

│ Kubernetes Vendor (provide other software and/or services associated

with Kubernetes) │ 13 │
└───────────────────────────────────────────────────────────────────────────────────────────────────┴───────┘

... so about half are direct users of some kind.

Let's see where they get their Kubernetes from:

┌───────────┬───────┐
│ bins_from │ users │
├───────────┼───────┤
│ other │ 26 │

│ self │ 41 │
│ selfmod │ 39 │
│ vendor │ 54 │
└───────────┴───────┘

... but the real question is, how many report getting all their binaries
ONLY from vendors? Those could presumably be excluded from our target
pool: 26, or rougly 21% of the sunset pool (reducing the target group to
32% of users).

Now, this is at odds with how users reported their own upgrade pattern.
A plurality of users reported upgrading every quarter, yet, clearly
fewer actually are. So let's see what the users who have year-old
releases say about themselves:

┌─────────────────────────────────────┬───────┐
│ freq_upgrade_prefer │ users │
├─────────────────────────────────────┼───────┤
│ Less than quarterly │ 13 │
│ N/A (I am a not a cluster operator) │ 5 │
│ Never │ 1 │

│ Once per year │ 17 │
│ Quarterly │ 36 │
│ Twice per year │ 39 │
└─────────────────────────────────────┴───────┘

or as %:

┌─────────────────────────────────────┬─────┐
│ freq_upgrade_prefer │ pct │
├─────────────────────────────────────┼─────┤
│ Less than quarterly │ 12 │
│ N/A (I am a not a cluster operator) │ 5 │
│ Never │ 1 │

│ Once per year │ 15 │
│ Quarterly │ 32 │
│ Twice per year │ 35 │
└─────────────────────────────────────┴─────┘

... so a large majority (79%) of users with year-old deployments claim
to upgrade quarterly or bi-quarterly. Not sure what to make of that.


So depending on how we count it, users who would strongly benefit from
the 12-month KEP are between 15% and 38% of all of our users (assuming
the survey is representative). Which is a pretty darned substantial
segment.

Will attach these stats to the KEP.

[Jordan Liggitt]

Thanks for digging into the data, this is helpful.

 

... but the real question is, how many report getting all their binaries
ONLY from vendors?  Those could presumably be excluded from our target
pool: 26, or rougly 21% of the sunset pool (reducing the target group to
32% of users).

I wouldn't say we should discount those users entirely. Many vendor-provided builds still track OSS releases, and consumers of those vendors' builds would benefit from more consistent support by the OSS project.

... so a large majority (79%) of users with year-old deployments claim
to upgrade quarterly or bi-quarterly.  Not sure what to make of that.

Upgrading quarterly at the trailing edge of what is supported (read: "most stable") would explain that combination.

[Josh Berkus]

On 11/13/19 11:09 AM, Jordan Liggitt wrote:
>
> ... so a large majority (79%) of users with year-old deployments claim
> to upgrade quarterly or bi-quarterly.  Not sure what to make of that.
>
>
> Upgrading quarterly at the trailing edge of what is supported (read:
> "most stable") would explain that combination.

... but if they're upgrading every 3 months, then extending the window
to 12 months doesn't really help them, does it? I mean, on a one-time
basis, but not after that.

[Jordan Liggitt]

Sure, users that upgrade quarterly don't particularly benefit from supporting one more release. I was just responding to "Not sure what to make of that."

[Jordan Liggitt]

The ones that upgrade every six months could see benefit, if they have two windows during the year for upgrades, and could align them with an 12-month support cadence. I'd be interested to know more about the timing/reasons for those users' upgrade cadences.

[Jago Macleod]

Thanks for pulling together the data - this is great!

I'll push back on the claim that supporting one more release doesn't help those that have fallen out of a supported version. If a customer is 5 releases behind, backporting critical bug fixes and security patches to 4 versions behind (their upgrade target) does help. We see this pretty frequently for retail customers in Q1/Q2 (also when the survey was run). 

--
You received this message because you are subscribed to the Google Groups "kubernetes-wg-lts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-wg-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-wg-lts/CAMBP-pKEXKcYWgEswMGFOL-uBk0wys37b%2B6G9ymFD_nan-1Abg%40mail.gmail.com.

[John Slee]

Hi,

On Thursday, 14 November 2019 06:09:31 UTC+11, Jordan Liggitt wrote:

... so a large majority (79%) of users with year-old deployments claim
to upgrade quarterly or bi-quarterly.  Not sure what to make of that.

Upgrading quarterly at the trailing edge of what is supported (read: "most stable") would explain that combination.

Deployment tooling release timelines probably also skew the results. For example, the Kubernetes v1.12.0 => Kops v1.12.0 delay of... 229 days?

2018-09-28: https://github.com/kubernetes/kubernetes/releases/tag/v1.12.0

2019-05-15: https://github.com/kubernetes/kops/releases/tag/1.12.0

For 1.14, the Kubernetes => Kops delay was somewhat shorter at 190 days

John

[Justin Santa Barbara]

I'm sure it's a factor!  The 1.12.0 release was particular delayed because of etcd3, but in general the goal of kops is that the 1.x.0 release is production ready, including e.g. the CNI providers.  That won't be aligned to the kubernetes .0 release, and our users have said that they like this (while everyone wishes it was sooner, getting it right is more important).  We're also working on giving users choice though - having an alpha or beta available at the time of the .0 k8s release.  We're not there yet, but we're getting closer!

[Josh Berkus]

On 11/13/19 12:53 PM, 'Justin Santa Barbara' via kubernetes-wg-lts wrote:
> I'm sure it's a factor!  The 1.12.0 release was particular delayed
> because of etcd3, but in general the goal of kops is that the 1.x.0
> release is production ready, including e.g. the CNI providers.  That
> won't be aligned to the kubernetes .0 release, and our users have said
> that they like this (while everyone wishes it was sooner, getting it
> right is more important).  We're also working on giving users choice
> though - having an alpha or beta available at the time of the .0 k8s
> release.  We're not there yet, but we're getting closer!

Thirteen respondees specifically mention Kops as a reason for not
upgrading. Some samples:

"Kops / EKS not keeping up with k8s rapid release cycle.. If vendors
(EKS) and tools (kops) can't k
eep up, then maybe k8s needs to adjust it's release cadence."

"Lack of maintained tools. kops is okay but running 2 versions behind..
It is so hard to keep up. S
o many components to keep on top of, lack of nice tools to upgrade. Have
to guess what is going to break."