tls_config: insecure_skip_verify: true seems not to working

[Mark Eisenblätter]

Hello list

i am trying to scrape from my 1.4 K8S Cluster with the new Prometheus 20.

but i have problems with the certificates.

I get following error messages from Prometheus:

Nov 21 06:33:19 prometheus-srv-1 docker[4689]: level=error ts=2017-11-21T06:33:19.878110706Z caller=main.go:211 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:177: Failed to list *v1.Endpoints: Get https://master-kubernetes.example.de/api/v1/endpoints?resourceVersion=0: x509: certificate signed by unknown authority"

   - job_name: 'kubernetes-apiserver-cadvisor'

   ¦ tls_config:

   ¦ ¦ ca_file: /opt/prometheus/etc/ca.pem

   ¦ ¦ cert_file: /opt/prometheus/etc/admin.pem

   ¦ ¦ key_file: /opt/prometheus/etc/admin-key.pem

   ¦ ¦ insecure_skip_verify: true

   ¦ kubernetes_sd_configs:

   ¦ - api_server: 'https://master-kubernetes.example.de'

   ¦ ¦ role: endpoints

   ¦ scheme: https

i thought with insecure_skip_verify: true the certificate should not be verified, but the logs say other wise.

i already triple checked the pem files if i have the right one. the same files are working fine with kubectl.

any idea what the problem could be?

thanks in advance.

Mark

[Brian Brazil]

The insecure_skip_verify there applies to the scrapes, not talking to k8. You need an additional tls_config section to go with the api_server.

--

Brian Brazil

www.robustperception.io

[Mark Eisenblätter]

ahh ok, that did the trick,

unfortunally i can scrape nothing, my prometheus is not in the pod network. but that is an other problem

Thanks.

mark

[viks...@gmail.com]

Mark, I am facing somewhat similar situation, after making changes in api context, now I am not getting signed certificate error but my Prometheus is not able to scrape anything. I am running Prometheus as a docker container on a host which is not a part of cluster. 

Seems like you were also facing something like that. Did it work for you finally?

Any suggestions?