I am setting up k8s monitoring from outside (different prometheus server . but i can access private IP of k8s server). I generated the certificates (ref is
https://rootsquash.com/2016/05/10/securing-the-kubernetes-api/) for my Prometheus server on k8s and copied to Prometheus server. also made entry in prometheus.yml file as below.
- job_name: 'k8s_testing'
scrape_interval: 15s
honor_labels: true
kubernetes_sd_configs:
- role: node
api_server:
https://172.30.0.89:6443 tls_config:
cert_file: /etc/prometheus/k8s_certs/k8scl2/ip-172-30-0-90.ec2.internal.crt
key_file: /etc/prometheus/k8s_certs/k8scl2/ip-172-30-0-90.ec2.internal.key
ca_file: /etc/prometheus/k8s_certs/k8scl2/ca.pem
insecure_skip_verify: true
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- source_labels: [__address__]
regex: '(.*):10250'
replacement: '${1}:10255'
target_label: __address__
after restarting prometheus service. I am getting below error.
Apr 10 07:41:58 ip-172-30-0-11 prometheus: level=error ts=2018-04-10T07:41:58.46774488Z caller=main.go:221 component=k8s_client_runtime err="
github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:354: Failed to list *v1.Node: Get
https://172.30.0.89:6443/api/v1/nodes?resourceVersion=0: x509: certificate signed by unknown authority"
when i cross checked Issuer is kubernetes in both (apiserver's pem file and prometheus server's certificate). if issuer is same then why it is saying certificate signed by unknown authority.