Can Prometheus be configured for multiple Kubernetes clusters?

[nayab zehra]

I am trying to configure Prometheus outside of the Kubernetes cluster and I want to monitor more than one Kubernetes cluster, however in the Kubernetes sd Config (for outside cluster monitoring) api_server option is there. I have configured one cluster successfully, but api_server only takes one host.
I know one solution to achieve this is to write multiple jobs for multiple clusters, but what i need is that to have one Kubernetes sd config job and same job can scrape multiple Kubernetes clusters.
Is there any way to achieve that?

[Yong Zhang]

Hi, here's what I did (For kubernetes 1.7 and above):

scrape_configs:

- job_name: 'kubernetes-kubelet'

  kubernetes_sd_configs:

  - api_server: http://cluster01

    role: node

    bearer_token: xxxx

  - api_server: http://cluster02

    role: node

    bearer_token: xxxx

  relabel_configs:

  - action: labelmap

    regex: __meta_kubernetes_node_label_(.+)

  - source_labels: [__address__]

    regex: '(.*):10250'

    replacement: '${1}:10255'

    target_label: __address__

- job_name: 'kubernetes-cadvisor'

  kubernetes_sd_configs:

  - api_server: http://cluster01

    role: node

    bearer_token: xxxx

  - api_server: http://cluster02

    role: node

    bearer_token: xxxx

  relabel_configs:

  - action: labelmap

    regex: __meta_kubernetes_node_label_(.+)

  - source_labels: [__address__]

    regex: '(.*):10250'

    replacement: '${1}:10255'

    target_label: __address__

  - source_labels: [__meta_kubernetes_node_name]

    regex: (.+)

    target_label: __metrics_path__

    replacement: /metrics/cadvisor

[kvranj...@gmail.com]

Hi Nayab Zehra,

I am trying to configure Prometheus outside of the Kubernetes cluster and I want to monitor more than one Kubernetes cluster, however in the Kubernetes sd Config (for outside cluster monitoring) api_server option is there. 

we used api_server ,however we are getting error(msg="Error loading config" err="couldn't load configuration (--config.file=all.yml): role missing (one of: pod, service, endpoints, node)").

it will be great if you share the steps to configure  Kubernetes sd Config.. (Already we tired https://prometheus.io/docs/prometheus/latest/configuration/configuration/#<kubernetes_sd_config>) but no luck

[nayab zehra]

Hi Yong Zhang,

Thank you for your time and bundle of thanks it worked!!

I have two more question, my configurations are now

Question 1:
Currently the configs looks like this:
scrape_configs:

- job_name: 'kubernetes-cadvisor'

  kubernetes_sd_configs:

  - api_server: http://cluster01

    role: node

    basic_auth:

        username: xx

        password: 'xx'

  - api_server: http://cluster02

    role: node

    basic_auth:

        username: xx

        password: 'xx'

  relabel_configs:

    - action: labelmap

      regex: __meta_kubernetes_node_label_(.+)

    - target_label: __address__

      replacement: http://cluster01 or http://cluster02 

    - source_labels: [__meta_kubernetes_node_name]

      regex: (.+)

      target_label: __metrics_path__

      replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

as you can see that i am trying to scrape the metrics through api server rather than directly from kubelet apis, so i need to provide the ip and port of api_server in the __address__  label.
I could have done this if a label for api_server has been given, but that's not the case. Is there any way to achieve this?

Question 2:
how are you authenticating the api_server? I can see that you have used bearer token. I am using Kubeadm and in new versions there is no bearer token, so how it can be done through certificates? 

[nayab zehra]

Hi kvranj, 

you can try this example. https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml 

If you are running prometheus inside the cluster it should work.

If you are running prometheus outside the cluster then you have to access kubernetes api server for which you can use api_server that is provide in kubernetes_sd_config,

[Yong Zhang]

Question2 first:

This may resolve your issue:

tls_config:

  ca_file: /path/to/your/ca.crt

For Question1, this will have trouble if you're trying to scrape from api and outside of Kubernetes. I don't know anyone else will have a solution, but let me tell where's the trouble.

First of all, I could not find a way to define the api server as a variable to your address relabel, so for your case, you may have to define different jobs for different clusters, because you have to define different api server to your address relabels.

And then, in cluster01, __metrics_path__ will be relabeled to http://cluster01/api/v1/nodes/${1}/proxy/metrics/cadvisor, as I know, this will have additional authentication to your api server, it will not inherit from authentication token for api server discovery, it may be a bug, I'm not sure, maybe I'm wrong, you can have a verify.

So, in general, I suggest you scrape from kubelet instead.

[nayab zehra]

Multiple jobs is not an option that i can use. Only if the meta label for master api is given, it can be used in relabeling.
This metrics relabel path api/v1/nodes/${1}/proxy/metrics/cadvisor has been given in the example prometheus kubernetes yaml file, so there should be a way. you are right may be its a bug. ill post the issue on github.
Thank you fr your help and time. 

[mohini...@synerzip.com]

Hi, have you monitored kubernetes cluster and any app within it locally ? if yes can you please share links to follow,

Thanks

Mohini

On Wednesday, December 20, 2017 at 7:23:54 PM UTC+5:30, nayab zehra wrote:

[Surbhi]

hello Nayab Zehra,
      Can i know if you found any alternative solution to your problem. I m facing the same issues. It would be very helpful if i get some guidance from you regarding this issue.

On Wednesday, December 20, 2017 at 7:23:54 PM UTC+5:30, nayab zehra wrote: