How does "kubectl exec" work

1,387 views
Skip to first unread message

Chris Willmore

unread,
Oct 27, 2017, 9:32:19 AM10/27/17
to Kubernetes developer/contributor discussion
I'm looking for a high-level description of how "kubectl exec" works.
Basically, how is the specified command routed and executed in the running Docker container?
(Searching through docs hasn't helped thus far.)

Thanks!

Andy Goldstein

unread,
Oct 27, 2017, 9:42:44 AM10/27/17
to Chris Willmore, Kubernetes developer/contributor discussion
As of right now, when you run "kubectl exec", it does the following:

  1. Sends a POST request to the apiserver, to /v1/namespaces/<ns>/pods/<pod>/exec
    1. Query parameters are used to indicate:
      1. The command to run
      2. If stdin should be enabled
      3. If stdout should be enabled
      4. If stderr should be enabled
      5. The name of the container
  2. The request includes a protocol upgrade from normal HTTP to SPDY
    1. SPDY allows for separate stdin/stdout/stderr/spdy-error "streams" to be multiplexed over a single TCP connection
  3. The apiserver establishes a connection to the kubelet for the pod
  4. The kubelet generates a short-lived token and issues a redirect to the CRI (I'm a bit fuzzy on the details here)
  5. The CRI handles the exec request and issues a docker exec API call
    1. The SPDY stdin/stdout/stderr streams are specified as input/output/error to the exec API call

Note, we also support websockets for exec, although not via kubectl at the moment. We plan to remove SPDY and only support websockets moving forward, but the specific release when that will happen is TBD.

HTH!
Andy


--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/4d117a7b-4c84-4bad-89cc-17db3aad8a0b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Chris Willmore

unread,
Oct 27, 2017, 10:02:36 AM10/27/17
to Kubernetes developer/contributor discussion
Wonderful - thank you Andy!
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.

Clayton Coleman

unread,
Oct 27, 2017, 10:03:46 AM10/27/17
to Andy Goldstein, Chris Willmore, Kubernetes developer/contributor discussion
A pretty big gap (if you notice) is that this means any exec traffic goes through the masters.  We have occasionally discussed (but not gone very far) whether we could extend the redirection step in 4 and actually propagate that back to an end user so that the exec session could occur directly to a node.  That would mean if you tried to do a 1TB backup of a container via exec you wouldn't be sending all that through the master.  No one has had time to work through it though or the various security ramifications.

To post to this group, send email to kuberne...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CANtdwzZSN37Zd2ms2TFmLsZL0DUv0tq06vkaeCYK4Bszt%3DYsSQ%40mail.gmail.com.

Ado Tony

unread,
Jan 17, 2018, 10:37:59 PM1/17/18
to Kubernetes developer/contributor discussion
Hi Clayton, I just notice this, but I really think for exec session occur directly to node should be the right way, maybe not only for exec, how about the attach/proxy/port_forward subresoures for pod, now they all goes through the masters, and I am wondering whats the biggest problem that block us to move to the directly way?

Sincerely
Tony

在 2017年10月27日星期五 UTC+8下午10:03:46，Clayton Coleman写道：
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.

To post to this group, send email to kuberne...@googlegroups.com.

Clayton Coleman

unread,
Jan 17, 2018, 10:48:25 PM1/17/18
to Ado Tony, Kubernetes developer/contributor discussion
A couple things:

1. The plumbing to redirect a connection from the master to the kubelet
2. (optional) the plumbing to securely redirect to the shim inside the pod, so the kubelet is out of the path as well
3. Sorting out whether administrators want to open ports on all nodes in order to support this (some won’t) and having a clear fallback when they don’t want to allow it.

The last is the biggest obstacle, while the former two have some tricky security implications.

Jordan Liggitt

unread,
Jan 17, 2018, 11:07:58 PM1/17/18
to Clayton Coleman, Ado Tony, Kubernetes developer/contributor discussion
Also, it requires network visibility from the end user to the nodes, which isn't a given

Jordan Liggitt

unread,
Jan 17, 2018, 11:11:01 PM1/17/18
to Clayton Coleman, Ado Tony, Kubernetes developer/contributor discussion
And ensuring the node (or shim) has a TLS cert that can be trusted by the end user. 

Today, going through the master lets you assign a dedicated dns name to the master, set up an SNI cert for it signed by a public CA, and let end users connect to the apiserver without needing to distribute trust roots. 

If users started communicating directly with nodes, it would be almost certain to be an IP or hostname a public CA wouldn't issue a cert for. 

On Jan 17, 2018, at 10:48 PM, Clayton Coleman <ccol...@redhat.com> wrote:

Reply all
Reply to author
Forward
This conversation is locked
You cannot reply and perform actions on locked conversations.
0 new messages