Blackbox exporter failing after adding fail_if_body_not_matches_regexp for get or post request

1,732 views
Skip to first unread message

Michał Siwik

unread,
Mar 31, 2021, 11:55:34 AM3/31/21
to Prometheus Users
My system is Debian (Linux gnu0 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64 GNU/Linux)

I'm using blackbox exporter binary 

My blackbox.yml looks like this. When i try to add fail_if_body not matches regexp. Binary give me erros "failed with result 'exit-code''". I even copied example config and still it's failing.
modules:
  http_2xx:
    prober: http
    http:
  http_post_2xx:
    prober: http
    http:
      method: POST
  tcp_connect:
    prober: tcp
  pop3s_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^+OK"
      tls: true
      tls_config:
        insecure_skip_verify: false
  ssh_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^SSH-2.0-"
  irc_banner:
    prober: tcp
    tcp:
      query_response:
      - send: "NICK prober"
      - send: "USER prober prober prober :prober"
      - expect: "PING :([^ ]+)"
        send: "PONG ${1}"
      - expect: "^:[^ ]+ 001"
  icmp:
    prober: icmp
  http_post_api_sowa:
    prober: http
    timeout: 5s
    http:
      method: POST
      headers:
        Content-Type: "application/json"
      body: '{"data_od": "2021-03-24","data_do": "2021-03-24","program_id": 73, "status": "ODRZUCONE"}'
      basic_auth:
        username: "*****"
        password: "*****"
  http_get_api_zkat:
    prober: http
    timeout: 5s
    http:
      method: GET
      headers:
        Content-Type: "application/json"
      #fail_if_body_not_matches_regexp: "check_time"
      #fail_if_body_not_matches_regexp
      # - "check_time"
      basic_auth:
        username: "***"
        password: "***"


● prometheus-blackbox-exporter.service - Blackbox exporter for Prometheus
   Loaded: loaded (/lib/systemd/system/prometheus-blackbox-exporter.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2021-03-31 17:42:03 CEST; 1s ago
  Process: 5470 ExecStart=/usr/bin/prometheus-blackbox-exporter $ARGS (code=exited, status=1/FAILURE)
 Main PID: 5470 (code=exited, status=1/FAILURE)

Mar 31 17:42:03 gnu0 systemd[1]: prometheus-blackbox-exporter.service: Service RestartSec=100ms expired, scheduling restart.
Mar 31 17:42:03 gnu0 systemd[1]: prometheus-blackbox-exporter.service: Scheduled restart job, restart counter is at 5.
Mar 31 17:42:03 gnu0 systemd[1]: Stopped Blackbox exporter for Prometheus.
Mar 31 17:42:03 gnu0 systemd[1]: prometheus-blackbox-exporter.service: Start request repeated too quickly.
Mar 31 17:42:03 gnu0 systemd[1]: prometheus-blackbox-exporter.service: Failed with result 'exit-code'.
Mar 31 17:42:03 gnu0 systemd[1]: Failed to start Blackbox exporter for Prometheus.

Michał Siwik

unread,
Mar 31, 2021, 11:59:45 AM3/31/21
to Prometheus Users

prometheus-blackbox-exporter --version
blackbox_exporter, version 0.13.0+ds (branch: debian/sid, revision: 0.13.0+ds-2+b11)
  build user:       pkg-go-ma...@lists.alioth.debian.org
  build date:       20190608-09:58:04
  go version:       go1.11.6

Commented lines are one which  cause failing
http_get_api_zkat:
    prober: http
    timeout: 5s
    http:
      method: GET
      headers:
        Content-Type: "application/json"
      #fail_if_body_not_matches_regexp: "check_time"
      #fail_if_body_not_matches_regexp
      # - "check_time"
      basic_auth:
        username: "***"
        password: "***"


Marcelo Magallón

unread,
Mar 31, 2021, 8:54:26 PM3/31/21
to Michał Siwik, Prometheus Users
fail_if_body_not_matches_regexp is a list, so the second form should work.

You can use the blackbox_exporter binary to validate the config:

$ blackbox_exporter --config.file=config.yml --config.check
...
...
level=info ts=2021-04-01T00:53:23.671Z caller=main.go:233 msg="Config file is ok exiting..."

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b3f4f8ab-7d1a-44c1-a4a6-4db10a7961edn%40googlegroups.com.


--
Marcelo Magallón

Michał Siwik

unread,
Apr 1, 2021, 2:40:59 AM4/1/21
to Prometheus Users
Thanks for reply

prometheus-blackbox-exporter --config.file blackbox.yml --config.check
level=info ts=2021-04-01T06:26:46.112348123Z caller=main.go:215 msg="Starting blackbox_exporter" version="(version=0.13.0+ds, branch=debian/sid, revision=0.13.0+ds-2+b11)"
level=error ts=2021-04-01T06:26:46.113104132Z caller=main.go:219 msg="Error loading config" err="Error parsing config file: yaml: unmarshal errors:\n  line 17: cannot unmarshal !!seq into string"


 cannot unmarshal !!seq into string


  http_get_api_zkat:
    prober: http
    timeout: 5s
    http:
      method: GET
      headers:
        Content-Type: "application/json"
        fail_if_body_not_matches_regexp: 
        - "^{heheszki}$"


And even if I copy example config from blackbox-exporter config

cat blackbox2.yml

modules:
  http_2xx_example:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: []  # Defaults to 2xx
      method: GET
      headers:
        Host: vhost.example.com
        Accept-Language: en-US
        Origin: example.com
      no_follow_redirects: false
      fail_if_ssl: false
      fail_if_not_ssl: false
      fail_if_body_matches_regexp:      <- this is line 16
        - "Could not connect to database"
      fail_if_body_not_matches_regexp:
        - "Download the latest version here"

prometheus-blackbox-exporter --config.file blackbox2.yml --config.check
level=info ts=2021-04-01T06:38:02.91294192Z caller=main.go:215 msg="Starting blackbox_exporter" version="(version=0.13.0+ds, branch=debian/sid, revision=0.13.0+ds-2+b11)"
level=error ts=2021-04-01T06:38:02.913576183Z caller=main.go:219 msg="Error loading config" err="Error parsing config file: yaml: unmarshal errors:\n  line 16: field fail_if_body_matches_regexp not found in type config.plain\n  line 18: field fail_if_body_not_matches_regexp not found in type config.plain"

Marcelo Magallón

unread,
Apr 1, 2021, 9:50:06 AM4/1/21
to Michał Siwik, Prometheus Users
I apologize for using YAML, whitespace-sensitve-anythings should have never been put into use...

The problem is that here:

On Thu, Apr 1, 2021 at 12:41 AM Michał Siwik <micha...@gmail.com> wrote:

  http_get_api_zkat:
    prober: http
    timeout: 5s
    http:
      method: GET
      headers:
        Content-Type: "application/json"
        fail_if_body_not_matches_regexp: 
        - "^{heheszki}$"

fail_if_body_not_matches_regexp is indented at the same level as Content-Type before it. If you align it under "headers" it works.

Marcelo

Michał Siwik

unread,
Apr 1, 2021, 12:28:58 PM4/1/21
to Prometheus Users
Marcelo,

In second example when I copy config from blackbox-exporter repo
cat blackbox2.yml

modules:
  http_2xx_example:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: []  # Defaults to 2xx
      method: GET
      headers:
        Host: vhost.example.com
        Accept-Language: en-US
        Origin: example.com
      no_follow_redirects: false
      fail_if_ssl: false
      fail_if_not_ssl: false
      fail_if_body_matches_regexp:      <- this is line 16
        - "Could not connect to database"
      fail_if_body_not_matches_regexp:
        - "Download the latest version here"

prometheus-blackbox-exporter --config.file blackbox2.yml --config.check
level=info ts=2021-04-01T06:38:02.91294192Z caller=main.go:215 msg="Starting blackbox_exporter" version="(version=0.13.0+ds, branch=debian/sid, revision=0.13.0+ds-2+b11)"
level=error ts=2021-04-01T06:38:02.913576183Z caller=main.go:219 msg="Error loading config" err="Error parsing config file: yaml: unmarshal errors:\n  line 16: field fail_if_body_matches_regexp not found in type config.plain\n  line 18: field fail_if_body_not_matches_regexp not found in type config.plain"

Marcelo Magallón

unread,
Apr 1, 2021, 9:05:55 PM4/1/21
to Michał Siwik, Prometheus Users
In the version you are using it's called "fail_if_matches_regexp" and "fail_if_not_matches_regexp".

Marcelo

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/57f9125a-35b4-4d32-9545-51cdab1c9dffn%40googlegroups.com.


--
Marcelo Magallón

Michał Siwik

unread,
Apr 2, 2021, 2:06:16 AM4/2/21
to Prometheus Users
Marcelo Magallón

Thank you very much, this solved my problem. I learned that i need also to check about relases and changes in app version :)

Reply all
Reply to author
Forward
0 new messages