Scrape only a particular namespace and ignore all other namespaces

[Himanshu Gude]

Hello guys,

There is a requirement in my project which states that, the client only wants alerts from "devsecops" namespace and alerts from all other namespaces should be ignored. which means alerts from namespaces other than "devsecops" should not be captured by the alertmanager and should not trigger tickets in ServiceNow.

In short, prometheus should only scrape metrics from only one namespace ie; devsecops

would really appreciate some sort of hel here.

Thanks in advance....

[Brian Candler]

That's two different things you're asking.

* alerting only from "devsecops" namespace: ensure every metric has a label with the namespace, and configure alerting rules to match only where namespace=devsecops

* scraping only from one namespace: if you can't configure the scrape URL to fetch only one namespace, then you can use metric_relabel_rules to discard all metrics except the ones of interest (again, matching on some namespace label to identify which ones to keep or drop)

[Julien Pivotto]

You can update the prometheus config:


scrape_configs:
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
namespaces:
names:
- devsecops

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e4ef8176-930d-4743-9118-90bbab5f2164n%40googlegroups.com.


--
Julien Pivotto
@roidelapluie

[Himanshu Gude]

yes, I m new to prometheus env. could you pls give an example for each case, which will help me to figure out what changes to be made in the configurations. Your first case is what I am looking for.

Thanks in advance!

[Brian Candler]

There are actually a a couple of options.

In each alerting rule (in prometheus) you can match on labels in the actual alerting expression: e.g.

        expr: node_boot_time_seconds > (node_boot_time_seconds offset 5m + 5)

becomes

        expr: node_boot_time_seconds{namespace="devsecops"} > (node_boot_time_seconds offset 5m + 5)

But I'd say it's simpler and more flexible to allow all the alerts to fire, but suppress the notifications in alertmanager routing rules:

route:
  receiver: noc

  routes:
    - receiver: discard
      matchers:
        - namespace!="devsecops"

    # more routing rules if desired go here

receivers:

  - name: discard
  - name: noc
    email_configs:
      - to: n...@example.com
        send_resolved: false

Then you can see alerts firing in other environments in the prometheus web UI (which helps you test and tune the rules), but notifications aren't sent out for anything apart from devsecops.

This does require that all the alerting expressions pass through a namespace label. Pasting each alerting expression into the PromQL browser in the Prometheus web UI will let you see exactly what timeseries values it generates (including the labels), as long as it's generating at least one alert of course.

[Himanshu Gude]

Thank you Brian. This is exactly what I was looking for.

[Himanshu Gude]

Hello Julien,

I have used the method which you have mentioned. I think this way will stop the prometheus from scraping metrics for all the other namespaces except devsecops. If this works it will also fix the issue that i m facing.

But I think supppressing the alerts from alertmanager will be a better fix for me. If you can suggest something for that it will help.

Thanks again!