Is it possible to have 2 prometheus servers which cover part of k8s services?

[Marco]

Our env is in k8s. 

To discover the endpoints automatically, our apps will expose the endpoint as a k8s service, with annotation prometheus.io/scrape: 'true'. And on proemtheus server side, it use the k8s . As a scale perspective, we want to use more than one proemetheus in our env. And for different services, they will in different namespace, or have different label/annotation, and based on that different prometheus servers will retrieve metrics from them.

Is it possible to do that? In my study, seems the services will be discovered automatically only when prometheus.io/scrape is set as true, and have not found no doc for above case.

Thanks a lot in advance.

[Ben Kochie]

One common way to scale Prometheus is to run a server per namespace.  This filter is built in to the discovery config, simply add this to the scrape config:

namespaces:
  names:

    - foo

    - bar

It is also possible to use relabel_configs to filter (action: keep) targets that you want based on the meta labels[0] provided by the discovery module.

[0]: https://prometheus.io/docs/operating/configuration/#kubernetes_sd_config

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/5643dca2-427d-4f07-83ba-a9bc419a51ea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matthias Rampke]

Also note that all handling of the annotations you mentioned is purely done in relabeling rules in the example configuration. This is meant as a starting point and you can and should adapt it to your needs.

/MR

On Tue, Aug 15, 2017, 08:39 Ben Kochie <sup...@gmail.com> wrote:

One common way to scale Prometheus is to run a server per namespace.  This filter is built in to the discovery config, simply add this to the scrape config:

namespaces:
  names:

    - foo

    - bar

It is also possible to use relabel_configs to filter (action: keep) targets that you want based on the meta labels[0] provided by the discovery module.

[0]: https://prometheus.io/docs/operating/configuration/#kubernetes_sd_config

On Tue, Aug 15, 2017 at 1:13 AM, Marco <blue...@gmail.com> wrote:

Our env is in k8s. 

To discover the endpoints automatically, our apps will expose the endpoint as a k8s service, with annotation prometheus.io/scrape: 'true'. And on proemtheus server side, it use the k8s . As a scale perspective, we want to use more than one proemetheus in our env. And for different services, they will in different namespace, or have different label/annotation, and based on that different prometheus servers will retrieve metrics from them.

Is it possible to do that? In my study, seems the services will be discovered automatically only when prometheus.io/scrape is set as true, and have not found no doc for above case.

Thanks a lot in advance.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/5643dca2-427d-4f07-83ba-a9bc419a51ea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CABbyFmozhiurv_KqqQRsDGfP3XB02H0Ug3StpQidRZuOmo_vUw%40mail.gmail.com.

[Marco]

thanks a lot for your comments.
I have tried this in my environment, but met following error "loading config: couldn't load configuration (-config.file=/etc/config/prometheus.yml): unknown fields in scrape_config: namespaces" source="main.go:160""
same problem occurred when I tried the official docker images with tag master/v1.7.1/v1.5.2

also I have reached the same result by using relabel utility like:
          - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_annotation_prometheus_io_scrape]
            action: keep
            regex: default;true

On Tuesday, August 15, 2017 at 1:39:20 AM UTC-5, Ben Kochie wrote:

One common way to scale Prometheus is to run a server per namespace.  This filter is built in to the discovery config, simply add this to the scrape config:

namespaces:
  names:

    - foo

    - bar

It is also possible to use relabel_configs to filter (action: keep) targets that you want based on the meta labels[0] provided by the discovery module.

[0]: https://prometheus.io/docs/operating/configuration/#kubernetes_sd_config

On Tue, Aug 15, 2017 at 1:13 AM, Marco <blue...@gmail.com> wrote:

Our env is in k8s. 

To discover the endpoints automatically, our apps will expose the endpoint as a k8s service, with annotation prometheus.io/scrape: 'true'. And on proemtheus server side, it use the k8s . As a scale perspective, we want to use more than one proemetheus in our env. And for different services, they will in different namespace, or have different label/annotation, and based on that different prometheus servers will retrieve metrics from them.

Is it possible to do that? In my study, seems the services will be discovered automatically only when prometheus.io/scrape is set as true, and have not found no doc for above case.

Thanks a lot in advance.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.

[Matthias Rampke]

That's a problem with your configuration (-indentation), "namespaces" doesn't go directly into `scrape_config` but one level deeper under `kubernetes_sd_configs`: https://prometheus.io/docs/operating/configuration/#scrape_config

/MR

On Sun, Aug 20, 2017 at 11:36 PM Marco <blue...@gmail.com> wrote:

thanks a lot for your comments.
I have tried this in my environment, but met following error "loading config: couldn't load configuration (-config.file=/etc/config/prometheus.yml): unknown fields in scrape_config: namespaces" source="main.go:160""
same problem occurred when I tried the official docker images with tag master/v1.7.1/v1.5.2

also I have reached the same result by using relabel utility like:
          - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_annotation_prometheus_io_scrape]
            action: keep
            regex: default;true

On Tuesday, August 15, 2017 at 1:39:20 AM UTC-5, Ben Kochie wrote:

One common way to scale Prometheus is to run a server per namespace.  This filter is built in to the discovery config, simply add this to the scrape config:

namespaces:
  names:

    - foo

    - bar

It is also possible to use relabel_configs to filter (action: keep) targets that you want based on the meta labels[0] provided by the discovery module.

[0]: https://prometheus.io/docs/operating/configuration/#kubernetes_sd_config

On Tue, Aug 15, 2017 at 1:13 AM, Marco <blue...@gmail.com> wrote:

Our env is in k8s. 

To discover the endpoints automatically, our apps will expose the endpoint as a k8s service, with annotation prometheus.io/scrape: 'true'. And on proemtheus server side, it use the k8s . As a scale perspective, we want to use more than one proemetheus in our env. And for different services, they will in different namespace, or have different label/annotation, and based on that different prometheus servers will retrieve metrics from them.

Is it possible to do that? In my study, seems the services will be discovered automatically only when prometheus.io/scrape is set as true, and have not found no doc for above case.

Thanks a lot in advance.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/5643dca2-427d-4f07-83ba-a9bc419a51ea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/3bdf7500-7113-40a8-9905-8db392262f91%40googlegroups.com.