Re: [kubernetes/kubernetes] Provide a first-class way to get the Kubernetes API server CA cert (#61572)

[k8s-ci-robot]

@sethvargo: Reiterating the mentions to trigger a notification:
@kubernetes/sig-cli-feature-requests, @kubernetes/sig-api-machinery-feature-requests

In response to this:

@kubernetes/sig-cli-feature-requests @kubernetes/sig-api-machinery-feature-requests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[Seth Vargo]

@kubernetes/sig-security-feature-requests for visibility

[Jordan Liggitt]

I expected a first-class API that returns the CA cert for the API server

I don't think I'd want to encourage TOFU with a dedicated API. Rather than a separate API, if the server wanted to provide it's CA, it could do so in the serving TLS handshake by including the full signing chain, including the root.

[Seth Vargo]

@liggitt I think that's a great enhancement, but it still requires the user to parse the root ca out of the chain in whatever language they use (some are easier than others) or use openssl on the CLI and somehow automate it.

[Daniel Smith]

I agree w/ @liggitt. Parsing the openssl cli output is not that bad. Also I prefer my security model to protect against MitM attacks, which this wouldn't. I recommend getting the CA out of band.

[Daniel Smith]

Closed #61572.

[Seth Vargo]

Hi @lavalamp. I disagree, please let me explain.

Parsing the openssl cli output is not that bad

1. That requires the user has openssl installed

2. There's no documentation on how to do this in the k8s docs (which should be viewed as an action item if this is truly the path you want users to take)

3. Can you give me a 1-liner automated way to get the certificate out of the chain?

   $ openssl s_client -connect IP:443 | openssl x509 -pubkey -noout
   

   includes a bunch of extra header information, and the command doesn't exit until you press ctrl+c, so you need to parse out the extra data and somehow terminate the process. It's not a good user experience and is far from "not that bad" in my opinion.

I prefer my security model to protect against MitM attacks

The current way most users get around this is to just disable TLS verification... Wouldn't a TOFU model be an improvement over MITM fo literally every request?

Pushing this downstream onto the user is a bad UX and hurts security. Please consider reopening, at the very least for additional opinions.

[Adrian Mouat]

I just stumbled on this issue, as I need to programatically retrieve the k8s CA cert. A simple command would make a lot of sense to me as well.

[Seth Vargo]

@amouat OMG hi! I'm not sure how to get this reopened for additional opinions. I agree that the dependency on openssl and sed is annoying (less mention not cross-system compatible)

[Adrian Mouat]

Hey, I deleted my previous attempt as I wasn't getting the public key. Here's the correct version:

openssl s_client -connect 23.251.141.119:443 2>&1 </dev/null | openssl x509 -pubkey -noout

Again, annoying dep on openssl, but at least sed is gone.

[Daniel Smith]

Can you give an example system which has an API to retrieve its root CA?

[Daniel Smith]

Reopened #61572.

[Daniel Smith]

Honestly I don't see how you're going to convince me a new API is necessary, but I'll reopen so you can try :)

If the openssl command line is too much headache this can certainly be written in a go program which would work on any HTTPS endpoint. This stuff is sent over every SSL connection, I'm not seeing what value a separate API would add.

[Seth Vargo]

Hey @lavalamp - thank you for reopening 😄

Here are the ones I can think of off the top of my head:

• Vault provides it's own CA and CAs for any PKI infra it's managing
• Google, specifically after the change to the Google Maps API root CAs
• K8S itself already has an API, but it's a filesystem API, not an HTTP API at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt.
• DNSimple
• Digicert and other actual public SSL providers
• Cloudflare and even blogged about why having it available via the API is important

If the openssl command line is too much headache this can certainly be written in a go program which would work on any HTTPS endpoint.

Totally agree. Or Ruby, or Python, etc. The challenge is that all of these either require the user to build it (correctly) or depend on a third-party tool that may-or-may-not be cross-OS compatible.

This stuff is sent over every SSL connection, I'm not seeing what value a separate API would add.

The value-add is that some clients which wish to connect to Kubernetes either distrust the system root certs or don't have the certificates because they are self-signed. They are designed to trust only the CA for which the cluster API server's certificate is signed. The current use case is Vault's K8S authentication backend, but there are many other use cases where you want a programatic way to get a server's published CA certificate. I'm sure it's possible that we could nitpick and find alternative workarounds for them, but they likely aren't clean or elegant.

Honestly I don't see how you're going to convince me a new API is necessary...

I think necessary is a bit too strong here. I would use extremely helpful in aiding the automated provisioning of Kubernetes clusters instead 😄

The choices (from what I gather) are:

• Store the cert out of band in something like a GCS bucket and download it on demand
• Parse the cert out of kubectl with piping and greping (assuming you have kubectl and have authed)
• Parse the cert using openssl, pipe, grep, some dev/null
• Write and manage a custom tool that gets the certificate
• Give users an API that returns the certificate

In my opinion, one of those is significantly less complex that the alternatives, especially given the broad spread of skills and backgrounds for k8s users.

[Jordan Liggitt]

Give users an API that returns the certificate

That is an API that is inherently insecure (if you need it, you don't have the information you need to make it secure), and not one I'd support.

[Seth Vargo]

A colleague brought it to my attention that the K8S API server might not know it's TLS configuration actually. It could be running behind an LB or multiple LBs doing termination.

[Adrian Mouat]

It turns out I didn't actually need it for what I was doing, but some use cases would be served by just putting it in a config-map or secret. I note that there are similar stuff in kube-system secrets. This wouldn't be insecure as it would be using kubectl which has already been configured to use the cluster, it would just be a convenience method.

[Seth Vargo]

We all use SSH and TOFU.

Very true... I can honestly say that I've verified a server's target machine signature exactly once.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[Seth Vargo]

Closed #61572.

[Peter (pir)]

$ kubectl config view --raw -o json | jq -r '.clusters[0].cluster."certificate-authority-data"' | tr -d '"' | base64 --decode

I wanted to use this rather than openssl but found that if you have multiple clusters referred to in your kubectl config this always returns the first (the [0] array reference) not necessarily the one you actually want.

Tweaked this to:

kubectl config view --raw -o json | jq -r '.clusters[] | select(.name == "'$(kubectl config current-context)'") | .cluster."certificate-authority-data"' | base64 --decode

which does what I need, giving me the ca for the current context. You can replace the call out to kubectl config current-context with whatever cluster name you need.

[James Callahan]

certificate-authority-data is not always present, for e.g. minikube, it instead has certificate-authority with a file path instead.

[Jordan Liggitt]

certificate-authority-data is not always present, for e.g. minikube, it instead has certificate-authority with a file path instead.

--flatten inlines data from referenced files, populating certificate-authority-data

[James Callahan]

Thanks @liggitt! I didn't know about --flatten

So one command for now would be this:

kubectl config view --raw --flatten -o json | jq -r '.clusters[] | select(.name == "'$(kubectl config current-context)'") | .cluster."certificate-authority-data"'

Or the nicer:

kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}'

[Peter (pir)]

Or the nicer:

kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}'

That would output the certificate-authority-data for every context, not just the current context, which isn't what I wanted. It would need a filter as well.

[Jordan Liggitt]

--minify removes all info except that used by the current context

[Michael Tewner]

Hi!
I'm using the instructions at https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/ to issue certificates inside the cluster. In order to trust these certs inside my app, I'll need to create a Java Keystore with the cluster CA in order to allow other containers to trust the certs. I can probably use the /var/run/secrets/kubernetes.io/serviceaccount/ca.crt from inside a container (at runtime), but why not add "kubectl certificate" option to make it feature-complete? (Also, who says both the client and server will be in the same K8S cluster?)

If the "kubectl certificate" command can issue a certificate, it should be able to spit out the chain it used to create it with.

[Erno Aapa]

I had case that my kubectl config did not include the CA certificate because insecure-skip-tls-verify:true were set but found a nice workaround.
Each service account token, which by default Kubernetes creates for each namespace, includes also the CA certificate.

Here's once liner to get the certificate from the service account Secret

kubectl get secret -o jsonpath="{.items[?(@.type==\"kubernetes.io/service-account-token\")].data['ca\.crt']}" | base64 --decode

[Olmo Rupert]

I had case that my kubectl config did not include the CA certificate because insecure-skip-tls-verify:true were set but found a nice workaround.
Each service account token, which by default Kubernetes creates for each namespace, includes also the CA certificate.

Here's once liner to get the certificate from the service account Secret

kubectl get secret -o jsonpath="{.items[?(@.type==\"kubernetes.io/service-account-token\")].data['ca\.crt']}" | base64 --decode

In case of multiple service accounts you'll receive multiple keys and you'll get base64 decoding issues.

kubectl get secret -o jsonpath="{.items[?(@.type==\"kubernetes.io/service-account-token\")].data['ca\.crt']}" | sed -e 's/\(.*\)\s.*$/\1/g'

bit of sed-ing would offcourse get rid of it.

—
You are receiving this because you are on a team that was mentioned.

Reply to this email directly, view it on GitHub, or unsubscribe.

[rptaylor]

If kubectl config view shows you the CA then you already have the CA locally in your kubeconfig, so it isn't really getting the CA from the cluster, right?

If you somehow do already have credentials to authenticate to the cluster, but don't have the cluster CA in your kubeconfig (it can be an empty string)

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data:

then you can do kubectl --insecure-skip-tls-verify -n kube-public describe configmap cluster-info to insecurely retrieve the CA from the cluster.

[Florian Klink]

That configmap doesn't necessarily exist. It's missing in k3s, for example.

I think the least annoying way to expose this would be to just also send the chain of trust while doing the connection set up (which is a good practice in other https servers as well).

Tooling outside the cluster can do TOFU if they want, or provide the cert by other means - and there's no additional API to implement.

[Jeff McCune]

Here's a funny one-liner that doesn't require anything except ruby. It pulls the pem encoded CA cert out of /api/v1/namespaces/kube-public/configmaps/cluster-info.

ruby -ropen-uri -ropenssl -rjson -ryaml -rbase64 -e "open(ARGV.first, {ssl_verify_mode: OpenSSL::SSL::VERIFY_NONE}) {|f| puts Base64.decode64(YAML.load(JSON.parse(f.read)['data']['kubeconfig'])['clusters'][0]['cluster']['certificate-authority-data']) }" \
  https://kube-apiserver.example.com:6443/api/v1/namespaces/kube-public/configmaps/cluster-info

—

Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/kubernetes/issues/61572/1251854846@github.com>

[Daniel Smith]

That configmap doesn't necessarily exist. It's missing in k3s, for example.

File a bug report against k3s? I'm surprised this doesn't cause conformance test failures?

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/kubernetes/issues/61572/1252562663@github.com>

[Jordan Liggitt]

fwiw, it's been checked in conformance tests since 1.21:

framework.ConformanceIt("should guarantee kube-root-ca.crt exist in any namespace", func() {

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/kubernetes/issues/61572/1252571298@github.com>

[Jeff McCune]

@liggitt FYI, I first tried to pull kube-root-ca.crt, but that config map isn't available anonymously. cluster-info is the only one I could find that's anonymously available.

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/kubernetes/issues/61572/1252573528@github.com>

[Jordan Liggitt]

oh, I missed what that comment was referring to... the kube-public configmap is only set up by some tools, not universally ... I think kubeadm uses it but other tools may not

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/kubernetes/issues/61572/1252573706@github.com>

[Jordan Liggitt]

I will note that fetching a CA from a server without verifying SSL so you can use the result to verify a connection back to the same server is sort of nonsensical

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/kubernetes/issues/61572/1252575586@github.com>

[Jeff McCune]

I will note that fetching a CA from a server without verifying SSL so you can use the result to verify SSL for a subsequent connection back to the same server is sort of nonsensical

Let's focus on the issue of getting the cert in the first place. Verifying it is a separate concern. Today, it's not possible to get the ca cert in a sensible way. I had to stand up an ingress and nginx just to copy the ca.crt file out of the config map and into /usr/share/nginx/html/ca.crt so I could serve it up, which is for sure nonsensical.

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/kubernetes/issues/61572/1252579979@github.com>

[Pas]

This might be helpful for others like me, who end up here via some search engine. :) Sorry for the noise!

Less elegant than a HTTP API, but at least one line.

echo '' | openssl s_client  -connect <ip>:6443 -showcerts    2>&1 | awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/{ if (/END CERTIFICATE/){ print ; exit } print  }' > ca.crt

(My use case was documenting kubectl config set-context kubectl config set-cluster cluster1 --server=https://<ip>:6443 --certificate-authority=ca.crt --embed-certs which was needed for a CI/CD pipeline, and simply wanted to provide a reliable way to get ca.crt.)

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/kubernetes/issues/61572/1845827049@github.com>