Prometheus same host multiple endpoints

[Kishore Chopne]

Hi,
We have a situation where same metrics are published on two different endpoints. Is it possible to pick one endpoint and discard the other while writing a PromQL query ?
Is it possible to configure Prometheus to collect metrics from only one endpoint?

/ Kishore

[Stuart Clark]

If they are literally the same metrics available in multiple endpoints
then I'd suggest only scraping one. That's controlled via your scrape
configuration. Depending on which mechanism you are using to manage that
it could mean changes to prometheus.yaml, removing an endpoint from a
YAML/JSON file or changing AWS/Kubernetes tags.

--
Stuart Clark

[Brian Candler]

Can you give an specific example of "same metrics are published on two different endpoints" ?

You might mean:

- two different metric names

- the same metric name, but different labels

And it might be that you're scraping the same target twice, or you're scraping one target but that target is (for some reason) returning duplicates in the scrape results.  Or you might have a more complex scenario, e.g. multiple prometheus servers scraping for redundancy, and then you're combining the results together somehow.

> Is it possible to pick one endpoint and discard the other while writing a PromQL query ?

Sure.  Just filter in the PromQL query.  For example, if you have

foo{aaa="bbb",ccc="ddd"} 123.0
foo{aaa="bbb",ccc="fff"} 123.0

and you consider the one with ccc="fff" to be a "duplicate" metric, then

foo{ccc!="fff"}

might be what you want.

Otherwise, you can avoid ingesting the duplicate metrics:

- by not scraping the second set in the first place

- if they all come from the same scrape, then using metric_relabel_configs to drop the metrics that you don't want to keep

[Nitya Sree]

Hi Brian Candler

Can you please share the metrics drop syntax to define globally under remote_write instead of under each job. I tried with different syntaxes but didn't work

Thanks

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c777e2ab-0089-4fa8-8fcd-efe49b95e2een%40googlegroups.com.

[Brian Candler]

Documentation:

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config

An example (untested) to drop metrics "foo" and "bar" from remote write:

remote_write:
  - url: http://1.2.3.4/api/v1/write

    write_relabel_configs:
      - source_labels: [__name__]
        regex: '(foo|bar)'

        action: drop

If you want any further help you'll need to explain what you're trying to do, show the config you made, and explain in what way it "didn't work" (e.g. did you get an error preventing prometheus from starting - in which case show the error; or it was accepted but didn't work in the way you were expecting -  if so explain what you were expecting)

[Nitya Sree]

Hi Brian Candler, 

Thanks for your response. I tried the below syntax, no errors in the configuration but metrics are not getting dropped.

  remoteWrite:

    - url: "http://test.com/push"
      write_relabel_configs:
      - source_labels: [__name__]
        regex: "node_xfs.*"

        action: 'drop'

I also tried below syntax but getting error "parsing YAML file /etc/config/prometheus.yml: yaml: unmarshal errors:\n  line 15: field relabel_configs not found in type config.plain"

  remoteWrite:

    - url: "http://test.com/push"

relabel_configs:
- action: labeldrop
  regex: "node_context_switches_total|node_cooling.*"

Please let me know if i need to try anything different

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/08b62d4f-ab0d-4bf8-ace0-bfc502fd724fn%40googlegroups.com.

[Brian Candler]

You don't appear to have copy-pasted the real config.  It should be remote_write, not remoteWrite (and I would expect the latter to be rejected by Prometheus).  So presumably there is something wrong in your config, somewhere, but as you've not shown the actual config I can't tell.

As for the syntax error, it's telling you that you need "write_relabel_configs" not "relabel_configs" in a remote_write block. And note that labeldrop only drops a label from a metric, not the metric itself.

If you google "prometheus write_relabel_configs" you'll find more examples, e.g. the second and fifth hits for me are

https://grafana.com/docs/grafana-cloud/billing-and-usage/control-prometheus-metrics-usage/usage-reduction/

https://docs.newrelic.com/docs/infrastructure/prometheus-integrations/install-configure/remote-write-drop-data/

Good luck!

[Agarwal ,Naveen]

Hi Brian/Stuart:

The prometheus implementation has around 2000+JVMs and uses two approaches (JMX exporter or SpringBoot Actuator/Micrometer). The implementation is up to the JVM owner.

JMX exporter exposes metrics at http://host:port/metrics whereas Micrometer exposes them at http://host:port/actuator/metrics. Given we don't know where metrics might be exposed, we have both of them mentioned in prometheus.yaml.

This works fine, given we expect metrics to only come from one source. However, we have found cases where they can come through both http://host:port/metrics and http://host:port/actuator/metrics. For these cases, there are duplicate metrics stored (it is redundant data) and also gets double counted in dashboards.

It might be quite possible there are incorrect implementations where JVM is configured to use both JMX exporter and Actuator/Micrometer. Thinking of a way of how we can stop storing the duplicates as we find and correct the implementations.

The targets to be scrapped are stored in json files which are referred to by prometheus.yaml.

We will be able to help with examples of json files and prometheus.yaml if required. Let us know.

Thanks,

Naveen

 

[Brian Candler]

Probably the ideal would be to fix the problem at source: have an alerting rule which tells you if the same JVM is exposing both sets of metrics, and push it back to the JVM owner.

Otherwise your dashboard metrics will need to take into account the duplicates.

You could simplify this a bit by having recording rules which take one or the other, taking care that some labels will match and some won't. e.g.

    expr: foo{job="jmx"} on ignoring(job) foo{job="springboot")

Then you can remote write this timeseries.

[Agarwal ,Naveen]

Thanks Brian. This is helpful. 

Sent from Outlook for Android

---

From: promethe...@googlegroups.com <promethe...@googlegroups.com> on behalf of Brian Candler <b.ca...@pobox.com>
Sent: Sunday, January 29, 2023 6:46:12 PM
To: Prometheus Users <promethe...@googlegroups.com>
Subject: Re: [prometheus-users] Re: Prometheus same host multiple endpoints

 

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/56b73228-b117-4040-9500-80c6ed4c65a4n%40googlegroups.com.

[Brian Candler]

Cheers.  BTW I mistyped that, it should have been "or" not "on".

   expr: foo{job="jmx"} or ignoring(job) foo{job="springboot"}