remote_write option for specific job

[Kiran Ali]

Hi, 

I want to write specific metrics of specific job to a webhook. How can i acheive this?

[Julius Volz]

You can add relabeling rules to your remote_write configuration to send only the metrics of your desired job, like:

remote_write:
  - url: http://remote1/push
    write_relabel_configs:
    - source_labels: [job]
      regex:         myjob
      action:        keep

On Wed, Apr 29, 2020 at 1:50 PM Kiran Ali <kira...@wanclouds.net> wrote:

Hi, 

I want to write specific metrics of specific job to a webhook. How can i acheive this?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2e33afd9-c323-41af-afeb-a38148fdaa00%40googlegroups.com.

[Sohaib Omar]

You may use write_relabel config at remote write time.

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write

for example:

remote_write:
    - url: <remote_url>

      write_relabel_configs:
      - source_labels: ["__name__"]
        regex: "<my-metric1>|<my-metric2>"
        replacement: "$1"
        action: keep

On Wed, Apr 29, 2020 at 4:50 PM Kiran Ali <kira...@wanclouds.net> wrote:

Hi, 

I want to write specific metrics of specific job to a webhook. How can i acheive this?

--

[Kiran Ali]

Prometheus is scraping target after 60s and i want as soon as target gets scraped then specific metrics from that target are forwarded to webhook. Any suggestions on this aprroach. 

On Wednesday, April 29, 2020 at 5:03:32 PM UTC+5, Julius Volz wrote:

You can add relabeling rules to your remote_write configuration to send only the metrics of your desired job, like:

remote_write:
  - url: http://remote1/push
    write_relabel_configs:
    - source_labels: [job]
      regex:         myjob
      action:        keep

On Wed, Apr 29, 2020 at 1:50 PM Kiran Ali <kira...@wanclouds.net> wrote:

Hi, 

I want to write specific metrics of specific job to a webhook. How can i acheive this?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

[Kiran Ali]

I'm asking this because remote_write is supposed to write to a long term storage. But my usecase is different. I want to forward specific metrics of target, when it gets scraped, to my webhook and then store in my desired db and perform alerting. 

[Julius Volz]

Remote write sends scraped samples in near real-time (within a couple of seconds, usually) to the remote end in a protobuf-over-HTTP format defined by Prometheus. But what you do with the data on the remote end is completely up to you. So you can store it, or you can stream-process it, or do whatever you want. So it sounds like what you are looking for.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/52a6cd82-4e51-45f1-b465-1e915371271d%40googlegroups.com.

[Kiran Ali]

But I can few issues with this global remote write. Suppose my settings are: max_samples_per_send=100 and desired metrics scraped for specific target are more than 100 then i won't get all desired metrics in one go and this can result in false alarm on receiving side. So i beleive remote_write option should be specific scrape and should send all desired metrics in one. 

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/52a6cd82-4e51-45f1-b465-1e915371271d%40googlegroups.com.