Append the value from one metric to the label of another metric.

[Yagyansh S. Kumar]

Hi. I am using Blackbox exporter to monitor my Application's and LB's HealthCheck URLs.

My alert for this looks like below:

  - alert: ServiceHTTPChecks
    expr: probe_success{job=~"blackbox_Service-HealthChecks"} == 0
    for: 2m
    labels:
      severity: "CRITICAL"
    annotations:
      summary: "Healthcheck *{{ $labels.checkname }}* failed."

      description: *I NEED THE RESPONSE CODE THAT THE INSTANCE HAS AT THIS POINT OF TIME*

      identifier: "*URL:* `{{ $labels.instance }}`, *Cluster:* `{{ $labels.cluster }}`, *node:* `{{ $labels.node }}` "

Whenever the probe_success is 0, I also want the response code for my instance at that time which probe_success itself does not have and is provided by probe_http_status_code.
My question is can I append the response code from the probe_http_status_code query to the labels of probe_success using the group_left?

If no, what other way is there to get the response code printed in my alert whenever my URL is down.

Thanks in advance!

[Julius Volz]

While you could use the "query" template function to achieve this (see https://prometheus.io/docs/prometheus/latest/configuration/template_reference/ and https://prometheus.io/docs/prometheus/latest/configuration/template_examples/), it's probably easiest to just make the alerting expression itself return the value you want by only returning status codes for checks that have failed:

  expr: probe_http_status_code{job=~"blackbox_Service-HealthChecks"} and (probe_success{job=~"blackbox_Service-HealthChecks"} == 0)

Btw., you could change the "=~" to just "=" because that regex is just doing a full string equality match anyway.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2447ca8c-c2a0-4bb7-bcf4-23fed0c30654%40googlegroups.com.

[Yagyansh S. Kumar]

Thanks a lot Julius.

Btw., you could change the "=~" to just "=" because that regex is just doing a full string equality match anyway. >>

Thanks for pointing it out. I recently removed 2-3 jobs from that alert, so forgot to remove the regex matcher.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

[Julius Volz]

Btw., I read this again the next day and wondered what happened if there's no HTTP status code at all for a failed check (like if the connection couldn't be established), but luckily the BBE still exposes the status code metric in that case, but with a value of 0, so the expression still works in that case.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7928d6bd-a94c-41fe-a9cb-a02b90f63f8d%40googlegroups.com.

[Yagyansh S. Kumar]

Yes. Actually this case happened with me just about an hour back. probe_success and response code together can give a lot of insight.

On Saturday, April 18, 2020 at 1:46:48 PM UTC+5:30, Julius Volz wrote:

Btw., I read this again the next day and wondered what happened if there's no HTTP status code at all for a failed check (like if the connection couldn't be established), but luckily the BBE still exposes the status code metric in that case, but with a value of 0, so the expression still works in that case.

On Sat, Apr 18, 2020 at 9:07 AM Yagyansh S. Kumar <yagyans...@gmail.com> wrote:

Thanks a lot Julius.

Btw., you could change the "=~" to just "=" because that regex is just doing a full string equality match anyway. >>

Thanks for pointing it out. I recently removed 2-3 jobs from that alert, so forgot to remove the regex matcher.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2447ca8c-c2a0-4bb7-bcf4-23fed0c30654%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7928d6bd-a94c-41fe-a9cb-a02b90f63f8d%40googlegroups.com.

[Yagyansh S. Kumar]

Btw I am struggling with the resolved alerts. I have posted the details in below mentioned thread. Would be really helpful if you can suggest something.

Thread Link - https://groups.google.com/forum/#!topic/prometheus-users/LLsPBIvLIME