rpm and deb package support

[Tim Pepper]

We’ve had long standing issues in how we build and publish the rpm and deb packages for our Kubernetes releases.  This has recently caused some user facing problems for folks trying to install or upgrade to non-latest versions of a 1.X releases.  For example installing or upgrading to 1.11.6 instead of 1.11.10, or with a non-latest kubernetes-cni package.

 

There is no documented policy of how we support these package streams.  Resolving the issue at hand could be done in a couple of ways, eg:

 

1) merge a patch to the package building code, build next patch releases 1.11.10, 1.12.8, 1.13.6, 1.14.2, and remind the community we support these packages at their latest level

 

1b) optionally remove older packages so users cannot attempt to use non-preferred combinations once they’ve been superseded by newer

 

2) merge a patch to the package building code and rebuild 1.11.1-0, 1.11.2-0, 1.11.3-0, 1.11.4-0, 1.11.5-0, 1.11.6-0, 1.11.7-0, 1.11.8-0 1.11.9-0, 1.12.0-0, 1.12.1-0, 1.12.2-0, 1.12.3-0, 1.12.4-0, 1.12.5-0, 1.12.6-0, 1.12.7-0, 1.13.0-0, 1.13.1-0, 1.13.2-0, 1.13.3-0, 1.13.4-0 and 1.13.5-0 using the modified requirement specification, then publish those new 1.11.0-1, 1.11.1-1, 1.11.2-1, 1.11.3-1, 1.11.4-1, 1.11.5-1, 1.11.6-1, 1.11.7-1, 1.11.8-1 1.11.9-1, 1.12.0-1, 1.12.1-1, 1.12.2-1, 1.12.3-1, 1.12.4-1, 1.12.5-1, 1.12.6-1, 1.12.7-1, 1.13.0-1, 1.13.1-1, 1.13.2-1, 1.13.3-1, 1.13.4-1 and 1.13.5-1 builds.

 

It would be nice if we had the automation to do the latter (2) and enough humans active to offer better support to the userbase.  But given a system which is partly manual and still requires a Googler to do the sign/publish step, this option (2) is operationally problematic.

 

There is proto-KEP discussion happening toward declaring a path to a better future, but in the meantime I’d like to propose on kubernetes-dev with a timeboxed consensus limit that we institute a policy like (1) and document out to users that the community supports the rpm and deb packages in our community yum and apt repos only at the latest .y release points.  Before doing that I want to know if the rest of SIG Release and SIG Cluster Lifecycle agree with this approach.

 

-- 

Tim Pepper

Orchestration & Containers Lead

VMware Open Source Technology Center

[Fox, Kevin M]

Do you have an example of the issue going to 1.11.6 non latest?

Thanks,

Kevin

---

From: 'Tim Pepper' via kubernetes-sig-cluster-lifecycle [kubernetes-sig-c...@googlegroups.com]
Sent: Wednesday, April 03, 2019 3:20 PM
To: kubernetes-...@googlegroups.com; kubernetes-sig-c...@googlegroups.com
Subject: rpm and deb package support

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-cluster-lifecycle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-cluster...@googlegroups.com.
To post to this group, send email to kubernetes-sig-c...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-cluster-lifecycle/BB1D8404-604B-4220-8CBA-EBA98780F2C0%40vmware.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Pepper]

Eg:

 

https://github.com/kubernetes/kubernetes/issues/75701

https://github.com/kubernetes/kubernetes/issues/75683

 

-- 

Tim Pepper

Orchestration & Containers Lead

VMware Open Source Technology Center

 

[Fox, Kevin M]

Ok, so from what I gather, the issue is that:

sometimes cni dependency bumps for newer k8s versions, so the cni version was hardcoded into the older versions to prevent breakage. But, sometimes the cni version needs to be updated for security reasons, and needs to be bumped even in the older versions.

Are there any issues not covered above?

What about treating the cni version as part of the release and do "backports"? The 0.7.5 security fix could have been applied backwards and a cni 0.6.1 rpm created. The 1.11.x series rpms could depend on >= 0.6.0 and < 0.7.0 rather then specifically 0.6.0. That would allow for upgrading cni when needed without rebuilding all the older packages?

Would that work?

Thanks,

Kevin

---

From: 'Tim Pepper' via kubernetes-sig-cluster-lifecycle [kubernetes-sig-c...@googlegroups.com]

Sent: Wednesday, April 03, 2019 3:29 PM
To: Fox, Kevin M; kubernetes-...@googlegroups.com; kubernetes-sig-c...@googlegroups.com
Subject: Re: rpm and deb package support

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-cluster-lifecycle/C8A6CDFE-E7AD-412C-9238-483D2A9C5428%40vmware.com.

[Tim Pepper]

It certainly could work, but is non-trivial to enact today as a Kubernetes community process.

 

For more discussion of the complexities of that path I refer you to this week’s WG LTS meeting:

https://docs.google.com/document/d/1J2CJ-q9WlvCnIVkoEo9tAo19h08kOgUJAS3HxaSMsLA/edit#bookmark=id.6val77ctrsh7

and its discussion of support and external dependencies.

[Angus Lees]

A plea against 1b in particular: Personally, I much prefer reproducability over most other things.

Eg: One of my k8s projects has a rather sophisticated automated testing pipeline.  I use the public cloud managed kubernetes offerings, and these move to newer versions on a timescale/workflow that is outside my control.  This lack of hermeticity causes huge amounts of busywork for my team, for dubious returns since security/etc is less critical for an ephemeral test cluster.  I don't want to stand still, but I want to upgrade on _my_ schedule.

I would be super sad if the "standalone" sig-cluster-lifecycle tools similarly took control away from me, particularly if the motivation was only that I somehow can't be trusted to make my own risk assessments.

 - Gus

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-cluster-lifecycle/499A4A5A-E6DE-47FF-B53A-929C5FFD30D5%40vmware.com.

For more options, visit https://groups.google.com/d/optout.

--

 - Gus

[Richard Brown]

On Thu, 4 Apr 2019 at 00:20, 'Tim Pepper' via
kubernetes-sig-cluster-lifecycle

<kubernetes-sig-c...@googlegroups.com> wrote:

> 2) merge a patch to the package building code and rebuild 1.11.1-0, 1.11.2-0, 1.11.3-0, 1.11.4-0, 1.11.5-0, 1.11.6-0, 1.11.7-0, 1.11.8-0 1.11.9-0, 1.12.0-0, 1.12.1-0, 1.12.2-0, 1.12.3-0, 1.12.4-0, 1.12.5-0, 1.12.6-0, 1.12.7-0, 1.13.0-0, 1.13.1-0, 1.13.2-0, 1.13.3-0, 1.13.4-0 and 1.13.5-0 using the modified requirement specification, then publish those new 1.11.0-1, 1.11.1-1, 1.11.2-1, 1.11.3-1, 1.11.4-1, 1.11.5-1, 1.11.6-1, 1.11.7-1, 1.11.8-1 1.11.9-1, 1.12.0-1, 1.12.1-1, 1.12.2-1, 1.12.3-1, 1.12.4-1, 1.12.5-1, 1.12.6-1, 1.12.7-1, 1.13.0-1, 1.13.1-1, 1.13.2-1, 1.13.3-1, 1.13.4-1 and 1.13.5-1 builds.
>
>
>
> It would be nice if we had the automation to do the latter (2) and enough humans active to offer better support to the userbase. But given a system which is partly manual and still requires a Googler to do the sign/publish step, this option (2) is operationally problematic.

This sounds like a possible use case for the Open Build Service
(www.openbuildservice.org)

It's used extensively by not only SUSE and openSUSE but many others
for solving this lovely problem of "something changed in our
dependency chain, and now we have a figurative million rebuilds to do
and republish". OBS supports both rpms and debs, so it might be all we
need automative wise to help with (2)

Obviously, for something like the official k8s package we won't be
able to use the public OBS instance at build.opensuse.org, but maybe
Google would be willing/able to set one up in a trusted environment so
their own OBS could be trusted with their build key to automate the
signing and publishing?

Regards,

Richard

[Fox, Kevin M]

Mirroring is another way of solving it. I usually make a container out of the relevant subset of the kubernetes yum repo with nginx embedded and just spawn that in my own cluster. Then I have a working mirror of a particular snapshoted version and manage them with container tags. K8s is pretty good at managing that kind of service especially if your driving it with CI/CD. You can then test your container repo mirror via CI/CD as well.

I've slowly been working on releasing this code but haven't gotten a chance to get it all out yet.

Some of it can be seen here: https://github.com/pnnl-miscscripts/miscscripts/blob/master/containers/rpms-kubernetes/Dockerfile . Replace scratch with nginx and a bit of config for nginx and it should work. (i'm trying to solve that a little different in the final release)

Thanks,

Kevin

---

From: Angus Lees [g...@inodes.org]
Sent: Thursday, April 04, 2019 1:58 AM
To: Tim Pepper
Cc: Fox, Kevin M; kubernetes-...@googlegroups.com; kubernetes-sig-c...@googlegroups.com

[Calvin Hartwell]

Hi all,

One way we have solved this issue on Ubuntu is to use snap packages instead of deb packages for the k8s binaries as they automatically patch between minor releases and customers do not experience these issues: https://blog.ubuntu.com/2018/12/04/canonical-kubernetes-clusters-auto-apply-vulnerability-patches-for-cve-2018-1002105 

However, I am pretty keen to see the .deb/.rpm packaging issue resolved. Are there any more cases apart from 75683 where the user has experienced issues with the .deb  packages? 

Cheers,

    - Calvin / Кэльвин / かるびん

Calvin Hartwell Director of Field Engineering EMEA | Canonical Ltd. | 5 Blue Fin Building, 110 Southwark St, London, SE1 0SU. Mobile: +44 (0) 7534801542 | Email: calvin....@canonical.com

You received this message because you are subscribed to the Google Groups "kubernetes-sig-release" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-re...@googlegroups.com.
To post to this group, send email to kubernetes-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-release/1A3C52DFCD06494D8528644858247BF01C2EB088%40EX10MBOX03.pnnl.gov.