Proposal to drop kubectl cp in 1.16

[Maciej Szulik]

Hey,

Over the past 6 months sig-cli and security team are constantly involved in fixing security related

issues with kubectl cp. This process involves myself, Jordan Liggitt, Tim Allclair and a couple of

other people needed to manage it, and then release updates in all supported versions of kubectl.

At the same time, we are carefully reviewing several pull requests wanting to extend current cp

support to handle more edge cases (one of the recent is adding support for colon [1], or preserving

file mode [2]). I can link to several other similar PRs, but the point is that each and every single

one of them requires significant effort during review (due to the history with this many security fixes

we are extra careful) and yet almost majority expands the vector of possible security threats to

cp command.

From my experience (and others helping addressing those issues) over the past few months we've

learned that each one of those security bugs requires us to drop some kind of those edge case,

which then returns as a PR from a different side.

The risk of continuing to maintain the cp command is that we are unable to provide a command that

is both useful and secure. The costs associated with that work are significant amounts of time,

and energy consumed to maintain, review, and fix/handle security issues.

If there was no alternative to provide this functionality, those costs/risks might be worthwhile,

but there are alternative approaches that would provide the same function, see [3].

Directing users to an approach like that would be safer, provide more features, and require less

maintenance and security-release effort. It would also work with older releases of kubectl, so users

could switch to that approach today.

For those reasons, we would like to request an exception [4] from the deprecation process, and be

allowed to completely remove kubectl cp command from release 1.16 onward. The maturity of kubectl

allows users to use other commands, (see examples in [3] mentioned above) to address the gap

caused by this removal. Additionally, with the kubectl's plugins mechanism [5] we allow the community

to create their own versions of the same command and deliver that instead of the built-in.

I'm curious of your opinions,

Maciej Szulik

SIG-CLI lead

[1] https://github.com/kubernetes/kubernetes/pull/78622

[2] https://github.com/kubernetes/kubernetes/pull/73053

[3] https://gist.github.com/tallclair/9217e2694b5fdf27b55d6bd1fda01b53

[4] https://kubernetes.io/docs/reference/using-api/deprecation-policy/#exceptions

[5] https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/

[Brendan Burns]

(side note: for some reason your message is rendering as light gray on white [at least in my browser], which makes it really hard to read)

I'm strongly concerned about CLI folks taking out features that are useful, but hard to support.

That's a pretty bad message to send to our community, we're saying:

"this is hard, so why don't you do it"

this means that they will (via plugins as you suggest) do a worse, more insecure job, but in a way where CVEs are much less likely to be filed and fixed.

This seems like a losing proposition for everyone.

--brendan

---

From: kubernetes-si...@googlegroups.com <kubernetes-si...@googlegroups.com> on behalf of Maciej Szulik <masz...@redhat.com>
Sent: Monday, August 26, 2019 2:50 PM
To: kubernetes-sig-cli <kubernete...@googlegroups.com>; kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>
Subject: Proposal to drop kubectl cp in 1.16

 

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CALFFA0OL3tZTpTnT4wtbQ7pMpbsK7AnFLX8sqxag8gMm8Ua0zQ%40mail.gmail.com.

[lig...@google.com]

I'm a strong +1 on removal. Making kubectl provide a transport you can use to run other tools makes sense to me and works well.

I don't think trying to reproduce a unix toolset inside kubectl is a good trajectory, especially given there are alternative invocations to accomplish the same result as `kubectl cp` today. 

`kubectl cp` in particular has had years to resolve these issues, and has not. I don't see frequent reports and responses fixing variations of effectively identical issues as a positive thing. Delegating to a well-tested tool like standalone use of tar is both more featureful (copy preserving permissions, pod-to-pod copy, etc, all are possible today) and more secure (tar has more proven mitigations for tar bombs, symlink directory escapes, etc).

You received this message because you are subscribed to the Google Groups "kubernetes-sig-cli" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-cli/MN2PR21MB12466750C44E5BE8E58CE436DBA10%40MN2PR21MB1246.namprd21.prod.outlook.com.

[Brendan Burns]

+SIG-usability

I think everyone would be wise to consider why Docker was so successful when much of the pieces that it was built from had been in market a long time. A big part of their success was their devotion to developing a truly useful command line tool that met the needs of a broad group of users.

A user-focused and useful command line tool is a valuable thing for adoption, even (especially?) when there is an arcane piped together collection of unix command line tools that do the same thing.

I'm deeply worried that we are back-sliding on being user-focused in favor of some sort of notion of purity that ultimately makes our tools less accessible, less intuitive and therefore less useful to a wide variety of developers from different backgrounds.

Perhaps more importantly, I posit that we do not have the tools or metrics to effectively make this decision. If 99% of all `kubectl` invocations were of `kubectl cp` this wouldn't be a discussion. Likewise if it was 1%, but as we currently sit we have no idea (afaik) what the popularity of various tools are, and thus we are arguing about it using our own internal biases and pre-conceptions rather than the data that could actually be useful.

Given this lack of data, at the very least a user survey is needed to determine the user's voice in judging the importance of `kubectl cp` is a necessary pre-requisite to any decision about whether to keep or remove it.

--brendan

---

From: lig...@google.com <lig...@google.com>
Sent: Monday, August 26, 2019 3:07 PM
To: Brendan Burns <bbu...@microsoft.com>
Cc: Maciej Szulik <masz...@redhat.com>; kubernetes-sig-cli <kubernete...@googlegroups.com>; kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>
Subject: Re: Proposal to drop kubectl cp in 1.16

 

[Phillip Wittrock]

Should `kubectl cp` be the way we recommend to copy a file out of the cluster?

Why use pipe + `tar` instead `kubectl cp`:

1. tar is more transparent about the mechanics of how the file is being copied\
2. tar provides more control over the mechanics of how the file is being copied
3. tar demonstrates a pattern that may be used to solve a broader range of problems than what `kubectl` would consider supporting
4. kubectl cp has been the subject of multiple kubectl security vulnerability reports
5. ???

Why use `kubectl cp`:

1. user doesn't need to have the tar command installed locally
2. the command is discoverable from kubectl
3. the command is fewer characters to type
4. ???

Discovery seems like the biggest argument for including it in kubectl from my perspective.  Why not replace the command implementation with something that prints the `kubectl cp` prints the equivalent exec command instead?  This solves 2 & 3, and having the user install tar locally seems acceptable.

The biggest downside of this would be breaking scripts that use cp for some reason.  One possibility would be to mitigate that by initially shelling out to exec and printing a warning before fully removing the command from the cli.

Thoughts?

- Phil

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-cli/MN2PR21MB12464FABEBFC1EC5710A87FEDBA10%40MN2PR21MB1246.namprd21.prod.outlook.com.

[Brendan Burns]

All of the same arguments could be made for removing scp in favor of ssh pipe to tar, and yet I don't hear anyone clamoring for the removal of scp (nor should they)

Just because something is possible doesn't make it easy to use and the right answer when looking at something like this is: 

How can we make this better? 

Not:

How can we put more burden on our users.

---

From: Phillip Wittrock <pwit...@google.com>
Sent: Monday, August 26, 2019 4:27:52 PM
To: Brendan Burns <bbu...@microsoft.com>
Cc: lig...@google.com <lig...@google.com>; Maciej Szulik <masz...@redhat.com>; kubernetes-sig-cli <kubernete...@googlegroups.com>; kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>; kubernetes-s...@googlegroups.com <kubernetes-s...@googlegroups.com>

[Brian Grant]

On Mon, Aug 26, 2019 at 3:59 PM 'Brendan Burns' via kubernetes-sig-cli <kubernete...@googlegroups.com> wrote:

+SIG-usability

I think everyone would be wise to consider why Docker was so successful when much of the pieces that it was built from had been in market a long time. A big part of their success was their devotion to developing a truly useful command line tool that met the needs of a broad group of users.

I think there were different reasons, but it's not really relevant for this discussion.

Even Docker made a choice to draw the line somewhere on functionality.  And it became a building block for scheduling systems like Kubernetes. Docker's CLI commands aren't very relevant in that context.

Similarly, Kubernetes doesn't aspire to be an all-encompassing end-to-end solution. There is an ecosystem of tools like Helm, Telepresence, CI/CD systems, functions platforms, Operators, etc., and tools (esp. UIs, but also CLIs and others) from service and distribution providers. That makes the scope of usability of the base platform tricky.

What is kubectl cp needed for? Do we support those user journeys end to end? If not, how important are they? If very important, who would staff an effort to address them? Or are they already better served by ecosystem tools?

I agree data would be useful, but the risk of recurring security issues is high, which imposes reputational risk for the project. If scp had recurring security problems, then people probably would recommend eliminating it.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-cli/MN2PR21MB12464FABEBFC1EC5710A87FEDBA10%40MN2PR21MB1246.namprd21.prod.outlook.com.

[Tim Allclair]

I am +1 on removing kubectl cp, for obvious security reasons.

If we're going to make an argument to keep this feature for it's usability, then I'd want to see a commitment to improving that useability. IMO, pipe to tar is _more_ useable today, since it behaves as expected, except for the discovery problem Phil mentioned. Here are a few things that don't work with kubectl cp:

- cp from a scratch-based pod gives the unhelpful error message: `command terminated with exit code 126`

- the missing features that Maciej mentioned: preserve permissions, better filename handling

- cp a file to the current working directory: `kubectl cp my-pod:/foo .` --> `error: open .: is a directory`

- it's probably broken on windows due to incorrect handling of path separators (though I don't have a windows box to test on)

With an investment into securing kubectl cp and fixing some of the usability concerns I could be convinced to keep it, but so far we haven't seen that. My recommendation is that we remove it, as suggested, and anyone who's interested in improving it can start doing so through addons or as an alpha feature.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAKCBhs41r63PEBieT44d2t1RCcVF9%2BQcivFKrhgnScRoo%3DNRGw%40mail.gmail.com.

[Stephen Augustus]

(+SIG Release/Release Team)

Irrespective of the outcome of this discussion (remove vs leave in place), I'm a pretty strong -1 on making any moves on this for the 1.16 cycle.

• Code Freeze is on Thursday[1]
• No exception was filed[2]:
• unsure of the level of risk this introduces to the ecosystem
• unsure of the amount of days required to complete this
• unsure of why this is critical to this milestone
• Is this the first time this is being discussed in a larger venue? (Apologies if I haven't seen a previous conversation w/ SIG Arch or Release)

I think this is something we could consider shortening the deprecation cycle for, given its' ability to impact the security posture and reputation of the project, but I don't think it's something we should rush discussion on and try to squeeze in this close to the end of the cycle.

Release Calendar (for all releases moving forward) can be subscribed to here: https://bit.ly/k8s-release-cal

-- Stephen

[1] https://github.com/kubernetes/sig-release/blob/master/releases/release-1.16/README.md#tldr

[2] https://github.com/kubernetes/sig-release/blob/master/releases/EXCEPTIONS.md

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CADtktAV3QZOo%3Dmq7x6sMqPZRjMLJifgNhS1%3D4xnohMZ0ioRtyA%40mail.gmail.com.

[Matt Farina]

Can we consider the user experience for a moment. What an average end user, who isn't part of the community, is going to need or expect.

Let's say a new k8s user or someone needs to copy a file for the first time has to do it. They might end up in a search engine. I just performed a search in two search engines on copying a files and the only option that came up for me on the first page was `kubectl cp`. Removing it would break the experience for someone who wants to learn a method to do it.

I would suggest that any effort to remove `kubectl cp` also needs to document an alternative manner to do the same thing so that people can work it out. Otherwise it would leave people hanging. Even if it's documented it'll take a long time for someone searching to come upon that solution. There will be a time with a broken experience for folks working to figure this out. Are we ok giving people that broken experience?

Then there are the people who use `kubectl cp` today. One day they'll go to use it with a new version and it won't work. Will they go back to an old version and keep using that as long as possible? Without kubectl getting a new major version what will they think of the breaking change? Will they complain and grumble about it the way many have when docker did those things? The answer is yes but I really wonder what the impact will be.

I'm reminded that `kubectl cp` is the interface. I wonder, is there a way to do the internals in a better method while keeping the same cli interface?

If we didn't have a history with `kubectl cp` I wouldn't suggest having it. Now that we have the history, and search results, and people using it there is something about their experience we might want to consider.

- Matt

You received this message because you are subscribed to the Google Groups "kubernetes-sig-release" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-re...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-release/CAFQm5yR0%3DCD2eB8-c%2BYyMQphPP0Seqq%3DZ2F3YyYGywCtcf_Tdw%40mail.gmail.com.

--

Matt Farina

mattfarina.com

Go in Practice - A book of Recipes for the Go programming language.

Code Engineered - A blog on cloud, web, and software development.

[Brendan Burns]

Given the level of discussion on this thread and the release timeline that Stephen mentions, it seems pretty clear to me that dropping in 1.16 is off the table. Does anyone disagree?

I think we need proper time to discuss the right solution as well as possible alternatives.

As Matt correctly suggests, we can keep the same interface ('kubectl cp ...') and improve it by (for example) shelling out to tar ourselves from within the kubectl binary. That would address all of the concerns raised by people except the scratch image issue while also maintaining the pre-existing interface.

I suggest that we continue the discussion on this thread (it is the first large-scale discussion that I know of) but that we can not deprecate in the 1.16 release timeframe.

That will give us the time for the thoughtful discussion that this warrants as well as time to do a user survey to determine the potential negative user impact.

Make sense?

--brendan

---

From: Matt Farina <matt....@gmail.com>
Sent: Monday, August 26, 2019 8:13 PM
To: Stephen Augustus <Ste...@agst.us>
Cc: Tim Allclair <tall...@google.com>; Brian Grant <brian...@google.com>; Brendan Burns <bbu...@microsoft.com>; lig...@google.com <lig...@google.com>; Maciej Szulik <masz...@redhat.com>; kubernetes-sig-cli <kubernete...@googlegroups.com>; kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>; kubernetes-s...@googlegroups.com <kubernetes-s...@googlegroups.com>; kubernetes-sig-release <kubernetes-...@googlegroups.com>; Kubernetes Release Team <kubernetes-...@googlegroups.com>

[Stephen Augustus]

*whispers everyone's favorite acronym (KEP) while ducking tomatoes*

[Brendan Burns]

That is also a very good point.

Honestly, I think we're long past the tomato throwing part (and I'm definitely one of the late people to sign onto the KEP band wagon), and have collectively seen the value of having a record of our decisions and an open, community focused discussion.

This sort of change definitely deserves a KEP.

--brendan

---

From: Stephen Augustus <Ste...@agst.us>
Sent: Monday, August 26, 2019 8:22 PM
To: Brendan Burns <bbu...@microsoft.com>
Cc: Matt Farina <matt....@gmail.com>; Tim Allclair <tall...@google.com>; Brian Grant <brian...@google.com>; lig...@google.com <lig...@google.com>; Maciej Szulik <masz...@redhat.com>; kubernetes-sig-cli <kubernete...@googlegroups.com>; kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>; kubernetes-s...@googlegroups.com <kubernetes-s...@googlegroups.com>; kubernetes-sig-release <kubernetes-...@googlegroups.com>; Kubernetes Release Team <kubernetes-...@googlegroups.com>

[Derek Carr]

I am +1 on dropping the command in the future per the reasons noted.  I am supportive of the SIG ceasing further enhancements in that area pending the KEP.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/MN2PR21MB1246BE639FAA83C8157FF006DBA00%40MN2PR21MB1246.namprd21.prod.outlook.com.

[Arturo Tarin]

Hello

After reading carefully all the arguments exposed and the links attached in this thread, all of them are more than reasonable.

+1 for KEP

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAHROWxSkjvqmzakbHHGtQtM4nbh1Pvk7_5VGvNp3e31eB1nu9w%40mail.gmail.com.

[Brendan Burns]

fwiw, as a datapoint:

SCP has been vulnerable to numerous CVE (even in the past year, including a directory traversal bug)

e.g.

https://nvd.nist.gov/vuln/detail/CVE-2019-6111

https://www.cvedetails.com/cve/CVE-2018-20685/

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=234965

But there have been no (that I can find) discussions of removing SCP from Linux distros.

---

From: kubernetes-si...@googlegroups.com <kubernetes-si...@googlegroups.com> on behalf of Arturo Tarin <arturo...@gmail.com>
Sent: Monday, August 26, 2019 10:52 PM
To: Derek Carr <dec...@redhat.com>
Cc: Brendan Burns <bbu...@microsoft.com>; Brian Grant <brian...@google.com>; Kubernetes Release Team <kubernetes-...@googlegroups.com>; Maciej Szulik <masz...@redhat.com>; Matt Farina <matt....@gmail.com>; Stephen Augustus <Ste...@agst.us>; Tim Allclair <tall...@google.com>; kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>; kubernetes-sig-cli <kubernete...@googlegroups.com>; kubernetes-sig-release <kubernetes-...@googlegroups.com>; kubernetes-s...@googlegroups.com <kubernetes-s...@googlegroups.com>; lig...@google.com <lig...@google.com>

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAOvcQRzmfe3ehTUTWopAxJ0Bh34qSMquTN6H%3DARqdoZ%2Bu_X2CQ%40mail.gmail.com.

[Gareth Rushgrove]

To the point of data and usage on CLI commands. Not perfect obviously,
but here's a breakdown of occurrences of kubectl commands on GitHub.

So 6232 occurrences as of today. Of those, ~1400 are in scripts of
various forms, with the rest being documentation:
https://github.com/search?l=Shell&q=%22kubectl+cp%22&type=Code

kubectl get - 187816
kubectl create - 142078
kubectl apply - 99388
kubectl delete - 81979
kubectl describe - 44177
kubectl port-forward - 44044
kubectl exec - 43162
kubectl run - 39559
kubectl config - 38550
kubectl logs - 34607
kubectl version - 23894
kubectl proxy - 19584
kubectl scale - 19388
kubectl patch - 17638
kubectl expose - 16759
kubectl cluster-info - 15470
kubectl label - 15023
kubectl edit - 13616
kubectl rollout - 13228
kubectl set - 12010
kubectl annotate - 9183
kubectl replace - 8402
kubectl completion - 8286
kubectl taint - 6875
kubectl drain - 6534
=====> kubectl cp - 6232 <=====
kubectl top - 5379
kubectl api-versions - 5227
kubectl autoscale - 4343
kubectl attach - 4320
kubectl uncordon - 4111
kubectl explain - 3661
kubectl certificate - 3060
kubectl cordon - 2998
kubectl convert - 2329
kubectl auth - 2136
kubectl plugin - 1958
kubectl wait - 1418
kubectl api-resources - 1068
kubectl diff - 994
kubectl kustomize - 267

Script used for generating in case anyone is interested
https://gist.github.com/garethr/fd27da6da7290bf813838256369e8e66

On Tue, 27 Aug 2019 at 07:13, 'Brendan Burns' via

> To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/MN2PR21MB1246D6132363DC29FD451455DBA00%40MN2PR21MB1246.namprd21.prod.outlook.com.



--
Gareth Rushgrove
@garethr

devopsweekly.com
morethanseven.net
garethrushgrove.com

[Matt Farina]

When someone goes to draw up a KEP the deprecation policy should be taken into account. Specifically https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecating-a-flag-or-cli

Just wanted to throw that reference out there while it was on my mind.

- Matt

[Maciej Szulik]

There was a lot of points brought up in this thread that I'd like to answer.

1. Apologies for the light grey - I must have messed up something when copying between gdoc and gmail :(

2. We've spent significant amount of time arguing back and forth how much effort we can invest into fixing incoming

CVEs vs how much time we can spent into moving kubectl forward. We need to maintain a healthy balance between

the two and "sacrificing" one full-time engineer just to fix cp command seems quite wasteful. Not to mention the fact

that we don't have that many staffing to properly test and ensure working Windows support, for example.

3. It's true we miss sufficient data about command usage, but that's where [1] should help us identifying gap in the

long term. In the short term, data provided by Gareth show that kubectl cp is at the 0.6% usage across all the commands

scraped from github, with exec being almost 7x more popular. We definitely can't treat this as a effective metric but it

gives us some perspective.

4. Addressing the concern for first time users is doable through proper documentation, I don't see any objections

for kubectl cp to print how one might replace this functionality with exec counterparts for even extended period

of time. Also, I'd rather invest our time and effort into addressing pressing UX issues that sig-usability is

currently gathering, over maintaining limited functionality of cp.

5. We will be discussing this topic once again tomorrow, during SIG-CLI meeting (6pm CET/12pm EDT/9am PDT),

feel free to join and share your thoughts.

6. From the above discussion I'm reading that majority is in favour of removing cp command, with the caveat that

it will happen in 1.17, with deprecation it in 1.16, using the exception process from the deprecation document [2]

and KEP to outline all pros and cons. In the mean time cp should return to k/k repository and be expanded with

additional warning about its inherent flaws.

Maciej

[1] https://github.com/kubernetes/enhancements/blob/master/keps/sig-cli/kubectl-commands-in-headers.md

[2] https://kubernetes.io/docs/reference/using-api/deprecation-policy/#exceptions

[Jordan Liggitt]

I'm still in favor of removal, but the request for a KEP is reasonable. The section of the deprecation policy dealing with exceptions predates KEPs, but says:

No policy can cover every possible situation. This policy is a living document, and will evolve over time. In practice, there will be situations that do not fit neatly into this policy, or for which this policy becomes a serious impediment. Such situations should be discussed with SIGs and project leaders to find the best solutions for those specific cases, always bearing in mind that Kubernetes is committed to being a stable system that, as much as possible, never breaks users. Exceptions will always be announced in all relevant release notes.

A KEP seems like the natural artifact to capture that discussion.

I would avoid announcing anything user-facing (in 1.16 or otherwise) until the path forward is agreed on in sig-cli.

[Daniel Smith]

On Tue, Aug 27, 2019 at 12:36 PM Maciej Szulik <masz...@redhat.com> wrote:

There was a lot of points brought up in this thread that I'd like to answer.

1. Apologies for the light grey - I must have messed up something when copying between gdoc and gmail :(

2. We've spent significant amount of time arguing back and forth how much effort we can invest into fixing incoming

CVEs vs how much time we can spent into moving kubectl forward. We need to maintain a healthy balance between

the two and "sacrificing" one full-time engineer just to fix cp command seems quite wasteful. Not to mention the fact

that we don't have that many staffing to properly test and ensure working Windows support, for example.

3. It's true we miss sufficient data about command usage, but that's where [1] should help us identifying gap in the

long term. In the short term, data provided by Gareth show that kubectl cp is at the 0.6% usage across all the commands

scraped from github, with exec being almost 7x more popular.

That is definitely not the gloss I'd put on the usage metrics. 6k references to it is not an insignificant amount of usage, I think it's "firmly in the middle of the pack". It's more than kubectl attach, for example, or 2x kubectl cordon.

It's actually 37th percentile usage (in that list). It is not good render that as ".6%".

 

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CALFFA0PPX%3DV_CJNCFcipseNBnz1TOvQ3XN872mh%3Dn%2BF6Ah5h4A%40mail.gmail.com.

[Tim Hockin]

I have problems with how `kubectl cp` is implemented, but the problems
that it solves seem to be real. Do we really need the totality of
what `kubectl cp` does, or can we lock that down further to enable
other implementations?

For example: If the primary use case is "extract a core file or log
files", could we teach CRI to copy a file out of a running pod, so we
don't need `tar` in the container image?

On Tue, Aug 27, 2019 at 12:45 PM 'Daniel Smith' via

> To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAB_J3bbJJDsjjoJHTMu2q1YmKYxp7oFjV4N6he61rAwaXAYbDA%40mail.gmail.com.

[Tim Allclair]

To propose an alternative for 1.16, can we start omitting symlinks entirely when unpacking tars, and introduce a command-line flag along the lines of `--insecure-extract-symlinks`? I recognize it is late in the cycle for such a feature addition, but the implementation would be quite simple: https://github.com/kubernetes/kubernetes/pull/82038

[Matt Farina]

A quick policy reminder...

 

6. From the above discussion I'm reading that majority is in favour of removing cp command, with the caveat that

it will happen in 1.17, with deprecation it in 1.16, using the exception process from the deprecation document [2]

and KEP to outline all pros and cons. In the mean time cp should return to k/k repository and be expanded with

additional warning about its inherent flaws.

Per the Kubernetes deprecation policy it can't be removed in 1.17. Once a decision has been made to remove it, the command needs to kept around for 12 months or 2 releases. Which ever is longer. There are some other details such as emitting a warning in the policy. See https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecating-a-flag-or-cli

[Stephen Augustus]

This is why the exceptions clause[1] was referenced:

No policy can cover every possible situation. This policy is a living document, and will evolve over time. In practice, there will be situations that do not fit neatly into this policy, or for which this policy becomes a serious impediment. Such situations should be discussed with SIGs and project leaders to find the best solutions for those specific cases, always bearing in mind that Kubernetes is committed to being a stable system that, as much as possible, never breaks users. Exceptions will always be announced in all relevant release notes.

If we're going to exercise it though, we should come to some sort of agreement on what we're willing to modify e.g., for rule 5a, we'll concede to shorten full deprecation to two releases instead of the 12 months that would be standard for a GA CLI user-facing component.

-- Stephen 

[1] https://kubernetes.io/docs/reference/using-api/deprecation-policy/#exceptions

[Brendan Burns]

Couple of thoughts:

1) Implementation fixes that allow keeping the command:

Wouldn't shelling out to 'tar' from inside of kubectl fix everything (symlinks, etc)?

2) The outcome of the KEP is not guaranteed. e.g. Maciej said: "the majority is in favor of removing" I disagree with that assesment, but more importantly, the KEP process is designed specifically to answer the question of "should it be removed" We shouldn't presume that the outcome of the KEP will go in any direction, based on this thread or anything else. The KEP is the way we answer such questions rather than a mailing list thread.

Indeed at this point, I'd suggest that this mail thread has outlived it's usefulness and discussion should migrate to the KEP....

--brendan

---

From: Stephen Augustus <Ste...@agst.us>
Sent: Tuesday, August 27, 2019 2:21 PM
To: Matt Farina <matt....@gmail.com>
Cc: Maciej Szulik <masz...@redhat.com>; Gareth Rushgrove <gar...@morethanseven.net>; Brendan Burns <bbu...@microsoft.com>; Arturo Tarin <arturo...@gmail.com>; Derek Carr <dec...@redhat.com>; Brian Grant <brian...@google.com>; Kubernetes Release Team <kubernetes-...@googlegroups.com>; Tim Allclair <tall...@google.com>; kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>; kubernetes-sig-cli <kubernete...@googlegroups.com>; kubernetes-sig-release <kubernetes-...@googlegroups.com>; kubernetes-s...@googlegroups.com <kubernetes-s...@googlegroups.com>; lig...@google.com <lig...@google.com>

Subject: Re: Proposal to drop kubectl cp in 1.16

[Matt Farina]

When I look at the exceptions clause and I consider why we would want to exercise it I come back to priorities and motivations that may be different from my default.

To illustrate what I mean, on Helm we explicitly stated the roles Helm is designed to support and not support. Those are listed in priority order with the developers of Helm itself being last in priority. That means the developers of Helm are willing to take on more burden and be a lower priority to support those who, for example, run applications in clusters. On Helm this was an explicit decision.

When it comes to Kubernetes I get mixed signals. Which roles are different sub-projects trying to support and what is their priority relative to each other? Do the developers of those sub-projects come as a higher priority than the users of those projects. Note, in this case I am considering roles and the same person may hold multiple roles (e.g., a user and developer). Most end-users are not developers of the projects they use.

What is the motivation for dropping `kubectl cp`? I don't pretend to know the answer but it would be fantastic if the KEP for this attempted to ferret this out. It could also signal a good reason for exercising an exception to the deprecation policy.

In my personal view, the deprecation policies are in place to signal to end users and businesses that depend on these projects how we operate and give them a healthy window to make changes. I was recently reminded that a deprecation cycle I was pushing for put an end during tax season which is a bad time for a bunch of users. Right before that was the holiday shopping cycle. Change then was all around bad for end-users because of where they are in their yearly cycles. So, we extended things to be more convenient.

If `kubectl cp` is dropped in the next release than it will cause problems for people who do quick installs in their development environments using flows like `brew install kubernetes-cli`. This change would land during the holiday shopping season which isn't a good time for the end users.

I would hope that if an exception is granted it's for an exceptional reason we can trace to.

- Matt

[Benjamin Elder]

On Tue, Aug 27, 2019 at 12:54 PM 'Tim Hockin' via kubernetes-sig-architecture <kubernetes-si...@googlegroups.com> wrote:

I have problems with how `kubectl cp` is implemented, but the problems
that it solves seem to be real.  Do we really need the totality of
what `kubectl cp` does, or can we lock that down further to enable
other implementations?

For example:  If the primary use case is "extract a core file or log
files", could we teach CRI to copy a file out of a running pod, so we
don't need `tar` in the container image?

Docker seems to be able to do ~all of kubectl cp (docker cp) without tar in the container.
We have an open issue about doing exactly this (extending CRI to cover this, and using it in kubectl cp). 
https://github.com/kubernetes/kubernetes/issues/58512 

I don't know how much effort is involved, but I really like this idea, FWIW.
This also solves some issues trying to debug containers using images like distroless that do not have tar.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAO_Rewbw9eEn_%2B4HycUjUEzZyx%3D8ak-XrLGdQ0MpS%2B-XWhmOYA%40mail.gmail.com.

[Vallery Lancey]

Catching up with this, I have a few thoughts from the SIG-Usability perspective.

1. This functionality should not be removed without notice, as this is not a critical problem as I understand it. If removal is the path forward, please instead stop enhancements/non-securty-bugs, and deprecate with very clear warnings over at least 1 cycle (ideally with the full 1-year notice).

2. Not all containers contain the base Linux utils (containers from scratch, Windows containers, etc). Losing kubectl cp would be a substantial loss for a nontrivial subset of users. Is there a better way to implement the functionality, or narrow scope, to reduce the security concerns & maintenance load to something more tolerable?

3. Github stats are poor indicators of command use; kubectl command statistics would be extremely valuable.

4. Matt raises a critical point that keeps rearing its head in the project: what's the priority for various user cases/personas (including our work building Kubernetes)? This is inconsistent across the project, and results in both high friction for developers struggling with compatibility concerns, and surprises for end users. I'll bring this topic to the steering committee, but I'd like to thank Matt for bringing it up here.

SIG-CLI are obviously the experts in this issue, but I'd like to offer help from SIG-Usability if we can provide it (EG with #2),

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAOZRXm9fEHAEg-%2B%2BcciFuaPC8sBZfxK4FrfSgRS%2BjOC9S5BfSw%40mail.gmail.com.

[Brendan Burns]

Did a KEP for this get created? If so, can someone send the PR to this thread?

Given the discussion, it seems this proposal is rejected in favor of the outcome of a KEP, correct?

(and in general, it seems like this discussion will be best captured in a KEP PR rather than this mailing-list thread)

Thanks!
--brendan

---

From: Vallery Lancey <val...@zeitgeistlabs.io>
Sent: Monday, September 2, 2019 10:23 PM
To: Benjamin Elder <benth...@google.com>
Cc: Tim Hockin <tho...@google.com>; Daniel Smith <dbs...@google.com>; Maciej Szulik <masz...@redhat.com>; Matt Farina <matt....@gmail.com>; Gareth Rushgrove <gar...@morethanseven.net>; Brendan Burns <bbu...@microsoft.com>; Arturo Tarin <arturo...@gmail.com>; Derek Carr <dec...@redhat.com>; Brian Grant <brian...@google.com>; Kubernetes Release Team <kubernetes-...@googlegroups.com>; Stephen Augustus <Ste...@agst.us>; Tim Allclair <tall...@google.com>; kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>; kubernetes-sig-cli <kubernete...@googlegroups.com>; kubernetes-sig-release <kubernetes-...@googlegroups.com>; kubernetes-s...@googlegroups.com <kubernetes-s...@googlegroups.com>; lig...@google.com <lig...@google.com>

[Sally O'Malley]

I've opened a KEP to discuss the future of kubectl cp: https://github.com/kubernetes/enhancements/pull/1248

I recently joined a team where I have the pleasure of contributing more to k8s, and in particular kubectl.  I'm looking forward to continuing this discussion. 

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-cli/MN2PR21MB12461077428BAE74FB25F2FDDBB90%40MN2PR21MB1246.namprd21.prod.outlook.com.

--

Sally O'Malley

Software Engineer | OpenShift

[Phillip Wittrock]

Thanks Sally!

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-cli/CADCwYQezVMfGuWEhiq%3DTnizSmsTDu1xeHVC5Dnf-_1LnitZrfw%40mail.gmail.com.