Blackbox module timeout value is ignored

34 views
Skip to first unread message

Tony Sargent

unread,
Jun 24, 2020, 4:55:48 PM6/24/20
to Prometheus Users
Hello,

I wanted to find out why I can't update the timeout for a module. I have some slow endpoints that need 30s+ timeout but I can't seem to update the default value of 10s no matter what value I put into the timeout. I am running the latest Blackbox version ( 0.17.0)
Here is my Module:

# --------------------- Generic HTTP(GET) -----------------------------#
  http_generic_2XX:
    prober: http
    timeout: 60s
    http:
      preferred_ip_protocol: ip4
      method: GET
      valid_http_versions: ["HTTP/1.1", "HTTP/2"]
      valid_status_codes: [200]
      no_follow_redirects: false
      headers:
        Accept-Language: en-US

And the logs for that module (or any module) show the following:

level=info msg="Beginning probe" probe=http timeout_seconds=9.5



Thank you in advance.

Brian Brazil

unread,
Jun 24, 2020, 5:05:48 PM6/24/20
to Tony Sargent, Prometheus Users
I presume that your scrape timeout is 10s, which the blackbox exporter learns from a scrape header and thus (with a bit of slack) limits the timeout to that value so that it can get a result back to Prometheus in time. Try increasing your scrape timeout/interval.

Brian
 



Thank you in advance.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b05055ae-a5d7-44e8-862d-f371aa9a0242o%40googlegroups.com.


--

Tony Sargent

unread,
Jun 24, 2020, 5:16:44 PM6/24/20
to Prometheus Users
Thanks for a quick response, Brian.
There is more to the logic than I thought. You are right it was set to the default of 10s, I bumped it up to 25s and scrape interval is set to 30s. But I am still a little confused about what the timeout in the module is for. Would you mind explaining it if you have a minute? 
I really appreciate your time and expertise.

On Wednesday, June 24, 2020 at 3:05:48 PM UTC-6, Brian Brazil wrote:
On Wed, 24 Jun 2020 at 21:55, Tony Sargent <anthony...@gmail.com> wrote:
Hello,

I wanted to find out why I can't update the timeout for a module. I have some slow endpoints that need 30s+ timeout but I can't seem to update the default value of 10s no matter what value I put into the timeout. I am running the latest Blackbox version ( 0.17.0)
Here is my Module:

# --------------------- Generic HTTP(GET) -----------------------------#
  http_generic_2XX:
    prober: http
    timeout: 60s
    http:
      preferred_ip_protocol: ip4
      method: GET
      valid_http_versions: ["HTTP/1.1", "HTTP/2"]
      valid_status_codes: [200]
      no_follow_redirects: false
      headers:
        Accept-Language: en-US

And the logs for that module (or any module) show the following:

level=info msg="Beginning probe" probe=http timeout_seconds=9.5

I presume that your scrape timeout is 10s, which the blackbox exporter learns from a scrape header and thus (with a bit of slack) limits the timeout to that value so that it can get a result back to Prometheus in time. Try increasing your scrape timeout/interval.

Brian
 



Thank you in advance.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

Brian Brazil

unread,
Jun 24, 2020, 7:31:12 PM6/24/20
to Tony Sargent, Prometheus Users
On Wed, 24 Jun 2020 at 22:16, Tony Sargent <anthony....@gmail.com> wrote:
Thanks for a quick response, Brian.
There is more to the logic than I thought. You are right it was set to the default of 10s, I bumped it up to 25s and scrape interval is set to 30s. But I am still a little confused about what the timeout in the module is for. Would you mind explaining it if you have a minute? 

It's an upper bound on the timeout used, in case you want it stricter than the scrape timeout would allow for.

Brian
 
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2641f94e-7d0d-4504-9f15-5b0cbf041438o%40googlegroups.com.


--

Tony Sargent

unread,
Jun 25, 2020, 10:42:58 AM6/25/20
to Prometheus Users
Makes sense now. Thanks again Brian.

On Wednesday, June 24, 2020 at 5:31:12 PM UTC-6, Brian Brazil wrote:
Reply all
Reply to author
Forward
0 new messages