Blackbox Exporter - a way to tag which fail_if_body_not_matches_regexp is failing

[Kevin Cameron]

I am using the blackbox exporter to monitor a results file.  There are 3 different regex entries that are checked:

  http_test:
    prober: http
    timeout: 50s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2"]
      valid_status_codes: []  # Defaults to 2xx
      method: GET
      no_follow_redirects: false
      fail_if_ssl: false
      fail_if_not_ssl: false
      fail_if_body_matches_regexp:
        - "connection refused"
      fail_if_body_not_matches_regexp:
        - "test1=1"
        - "test2=1"
        - "test3=1"
      preferred_ip_protocol: "ip4"
      ip_protocol_fallback: false

This works but I would like to know if I can format the module so that I know which regex fails by adding a label some how.  I would prefer not to have 3 different modules and 3 different jobs (one for each regex)

if that is not an option, what value would I put in the alert to include the body of the target?

[Brian Candler]

I'm afraid not.  You can see from the code that the checking function simply returns "false" if any of the regexp checks fail. There's log output which you can get in the http response using `&debug=true`, but it's not formatted as prometheus metrics, nor can it be captured as an annotation for the alert AFAIK.

There is a metric probe_failed_due_to_regex, but it is currently only set to 0 or 1.

I think it would be a reasonable feature request as to set it to <N>, where N is the number of the regexp rule which failed. Note that there are four rulesets for the http prober.

  fail_if_header_matches:
  fail_if_header_not_matches:

  fail_if_body_matches_regexp:
  fail_if_body_not_matches_regexp:

The first rule which fails prevents all further checks from being done, so you only need to identify the single failing rule.

I guess the rules could all be numbered together starting from 1: e.g. in your example, fail_if_body_not_matches_regexp conditions could be counted as 2, 3 and 4.  Or they could be in blocks, say 1000-1999, 2000-2999 etc.

Perhaps a better option would be to set the metric to a hash of the rule string. This would be consistent with how IP address resolution is dealt with (probe_ip_addr_hash) and means that each rule would have a stable value. However you wouldn't explicitly know which ruleset it was in; there's also a small potential for ambiguity if you match exactly the same string in both a header rule and a body rule.

[Kevin Cameron]

Thanks very much Brian for the quick follow up.  I think I've been able to talk the development team into reformatting the response so I can ingest it as a typical Prometheus job target so we can get exactly what we want out of it.  Some of the ideas you proposed are interesting but as long as it stops its checks once it gets a failure it limits the value in my use case.

Thanks,

Kevin

--
You received this message because you are subscribed to a topic in the Google Groups "Prometheus Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-users/eLK48Rqj2ho/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3c25847a-043e-4ce6-bec0-7e7dbf9cffe5n%40googlegroups.com.

[Brian Candler]

No problem.  If you want to see individually whether each regexp passes or fails, then you'll need to write your own exporter (or scrape multiple times).

As a quick fix or prototype, have a look at exporter_exporter. It can exec a custom script which returns metrics, so a shell script with a few lines of curl and grep will do the job.