k8s dashboard deploy fails for x509: certificate signed by unknown authority

[Jerry Hwang]

Hello k8s experts,

Kubernetes v1.5.3

(fyi, k8s upgrade is not an option)

I can't succeed to install k8s dashboard on the master node by running this.

kubectl create -f https://git.io/kube-dashboard-no-rbac

I see it fails for x509: certificate signed by unknown authority and it's because k8s nodes are behind my company corp https proxy.

I imported the correct proxy CA certs.

This succeeds from the node that proves the OS node has a correct proxy CA cert.

curl -v https://storage.googleapis.com/artifacts.google-containers.appspot.com/containers/images/sha256:691a82db1ecd12bf573b1b9992108a48e0d1a8640564c96d4f07e18e69dd83e6

docker pull from docker hub just works fine.

However, k8s pod deployment fails (kubectl create -f https://git.io/kube-dashboard-no-rbac)

Here is the details of failure from k8s.

main error: x509: certificate signed by unknown authority

It seems k8s uses another cert store than OS (/etc/ssl/certs), does it?

Events:

  FirstSeen

LastSeen Count

From SubObjectPath

Type Reason

Message

  ---------

-------- -----

---- -------------

-------- ------

-------

  5m

5m 1

{default-scheduler }  Normal Scheduled

Successfully assigned kubernetes-dashboard-3803355946-xqd5p to c6fe9a20902748819727950f1c78e0eb.infra.caasp.local

  5m

1m 4

{kubelet c6fe9a20902748819727950f1c78e0eb.infra.caasp.local}

spec.containers{kubernetes-dashboard}

Normal Pulling

pulling image "gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.3"

  4m

1m 4

{kubelet c6fe9a20902748819727950f1c78e0eb.infra.caasp.local}

spec.containers{kubernetes-dashboard}

Warning Failed

Failed to pull image "gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.3": image pull failed for gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.3, this may be because there are no credentials on this request.  details: (error pulling image configuration: Get https://storage.googleapis.com/artifacts.google-containers.appspot.com/containers/images/sha256:691a82db1ecd12bf573b1b9992108a48e0d1a8640564c96d4f07e18e69dd83e6: x509: certificate signed by unknown authority)

  4m

1m 4

{kubelet c6fe9a20902748819727950f1c78e0eb.infra.caasp.local}

Warning FailedSync

Error syncing pod, skipping: failed to "StartContainer" for "kubernetes-dashboard" with ErrImagePull: "image pull failed for gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.3, this may be because there are no credentials on this request.  details: (error pulling image configuration: Get https://storage.googleapis.com/artifacts.google-containers.appspot.com/containers/images/sha256:691a82db1ecd12bf573b1b9992108a48e0d1a8640564c96d4f07e18e69dd83e6: x509: certificate signed by unknown authority)"

  4m

0s 9

{kubelet c6fe9a20902748819727950f1c78e0eb.infra.caasp.local}

spec.containers{kubernetes-dashboard}

Normal BackOff

Back-off pulling image "gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.3"

  4m

0s 9

{kubelet c6fe9a20902748819727950f1c78e0eb.infra.caasp.local}

Warning FailedSync

Error syncing pod, skipping: failed to "StartContainer" for "kubernetes-dashboard" with ImagePullBackOff: "Back-off pulling image \"gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.3\""

I would highly appreciate if you could help me.

Thanks,

Jerry

[Jerry Hwang]

I found the solution. I actually also needed to import the CA cert to minions..

[Andrew Ray]

Jerry how did you import that cert? was it a config setting or just putting in in /etc/pki/... on the host machines.  i am running k8s in docker and having trouble getting it installed on the master machines.

[Jerry Hwang]

Hi Andrew,

Perform this on k8s nodes.
My example below is with SUSE.

1. Put all your chained CA certificates in /etc/pki/trust/anchors

2. Run 'update-ca-certificates'

3. Check the imported certificates by 'ls /etc/ssl/certs/'

4. Restart docker service

systemctl restart docker

5. Test

Jerry

--
You received this message because you are subscribed to a topic in the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-dev/6xcGao4-5Bs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/12150a76-85cf-40ae-a8e8-a6ce9bb7cd4a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew Ray]

Got it.  That's what i thought.  I was hoping we could pass it in without host modifications.

On Mon, Nov 6, 2017, 10:54 AM Jerry Hwang <speci...@gmail.com> wrote:

Hi Andrew,

Perform this on k8s nodes.
My example below is with SUSE.

1. Put all your chained CA certificates in /etc/pki/trust/anchors

2. Run 'update-ca-certificates'

3. Check the imported certificates by 'ls /etc/ssl/certs/'

4. Restart docker service

systemctl restart docker

5. Test

Jerry

On Mon, Nov 6, 2017 at 6:51 AM, Andrew Ray <drew...@gmail.com> wrote:

Jerry how did you import that cert? was it a config setting or just putting in in /etc/pki/... on the host machines.  i am running k8s in docker and having trouble getting it installed on the master machines.

On Thursday, August 31, 2017 at 12:54:45 PM UTC-5, Jerry Hwang wrote:

I found the solution. I actually also needed to import the CA cert to minions..

--
You received this message because you are subscribed to a topic in the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-dev/6xcGao4-5Bs/unsubscribe.

To unsubscribe from this group and all its topics, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.