HTTP result - Returning 502 instead of 200

RBFE

unread,

Jan 12, 2021, 12:06:01 PM1/12/21

to Prometheus Users

Host operating system: output of uname -a

Linux EUWBOMON01 4.18.0-147.8.1.el8_1.x86_64 #1 SMP Thu Apr 9 13:49:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

blackbox_exporter version: output of blackbox_exporter --version

blackbox_exporter, version 0.17.0 (branch: HEAD, revision: 1bc7680)
build user: root
build date: 20200619-11:54:41
go version: go1.14.4

What is the blackbox.yml module config.

modules:
http_2xx:
prober: http
timeout: 25s
http:
valid_http_versions: ["HTTP/1.0", "HTTP/1.1", "HTTP/2.0"]
valid_status_codes: []
preferred_ip_protocol: ip4
tls_config:
insecure_skip_verify: true
http_post_2xx:
prober: http
timeout: 5s
http:
method: POST
preferred_ip_protocol: ip4
tcp_connect:
prober: tcp
timeout: 5s
tcp:
preferred_ip_protocol: ip4
icmp:
prober: icmp
timeout: 5s
icmp:
preferred_ip_protocol: ip4
http_2xx_home:
prober: http
timeout: 5s
http:
method: GET

fail_if_body_not_matches_regexp:

- "OK"

fail_if_not_ssl: true

preferred_ip_protocol: ip4

http_2xx_tls:
prober: http
timeout: 25s
http:
valid_http_versions: ["HTTP/1.0", "HTTP/1.1", "HTTP/2.0"]
valid_status_codes: []
preferred_ip_protocol: ip4

What is the prometheus.yml scrape config.

job_name: 'blackbox-Product_Template'
scrape_interval: 30s
metrics_path: /probe
params:
module: [http_2xx]
file_sd_configs:
- files: ['./blackboxtargets/blackbox_product_template_targets.yml']
relabel_configs:
- source_labels: [address]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: address
replacement: Blackbox_server_IP

What logging output did you get from adding &debug=true to the probe URL?

Logs for the probe:
ts=2021-01-07T13:29:02.789489148Z caller=main.go:304 module=http_2xx target=https://www2.trustnet.com level=info msg="Beginning probe" probe=http timeout_seconds=25
ts=2021-01-07T13:29:02.789667447Z caller=http.go:323 module=http_2xx target=https://www2.trustnet.com level=info msg="Resolving target address" ip_protocol=ip4
ts=2021-01-07T13:29:02.804001657Z caller=http.go:323 module=http_2xx target=https://www2.trustnet.com level=info msg="Resolved target address" ip=104.18.186.229
ts=2021-01-07T13:29:02.804057257Z caller=client.go:252 module=http_2xx target=https://www2.trustnet.com level=info msg="Making HTTP request" url=https://104.18.186.229 host=www2.trustnet.com
ts=2021-01-07T13:29:02.930748562Z caller=main.go:119 module=http_2xx target=https://www2.trustnet.com level=info msg="Received HTTP response" status_code=502
ts=2021-01-07T13:29:02.930772962Z caller=main.go:119 module=http_2xx target=https://www2.trustnet.com level=info msg="Invalid HTTP response status code, wanted 2xx" status_code=502
ts=2021-01-07T13:29:02.930826762Z caller=main.go:119 module=http_2xx target=https://www2.trustnet.com level=info msg="Response timings for roundtrip" roundtrip=0 start=2021-01-07T13:29:02.804120156Z dnsDone=2021-01-07T13:29:02.804120156Z connectDone=2021-01-07T13:29:02.806239843Z gotConn=2021-01-07T13:29:02.815712583Z responseStart=2021-01-07T13:29:02.930606663Z end=2021-01-07T13:29:02.930792862Z
ts=2021-01-07T13:29:02.930893061Z caller=main.go:304 module=http_2xx target=https://www2.trustnet.com level=error msg="Probe failed" duration_seconds=0.141338714

Metrics that would have been returned:

HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds

TYPE probe_dns_lookup_time_seconds gauge

probe_dns_lookup_time_seconds 0.01433891

HELP probe_duration_seconds Returns how long the probe took to complete in seconds

TYPE probe_duration_seconds gauge

probe_duration_seconds 0.141338714

HELP probe_failed_due_to_regex Indicates if probe failed due to regex

TYPE probe_failed_due_to_regex gauge

probe_failed_due_to_regex 0

HELP probe_http_content_length Length of http content response

TYPE probe_http_content_length gauge

probe_http_content_length 1477

HELP probe_http_duration_seconds Duration of http request by phase, summed over all redirects

TYPE probe_http_duration_seconds gauge

probe_http_duration_seconds{phase="connect"} 0.002119787
probe_http_duration_seconds{phase="processing"} 0.11489408
probe_http_duration_seconds{phase="resolve"} 0.01433891
probe_http_duration_seconds{phase="tls"} 0.011592527
probe_http_duration_seconds{phase="transfer"} 0.000186199

HELP probe_http_redirects The number of redirects

TYPE probe_http_redirects gauge

probe_http_redirects 0

HELP probe_http_ssl Indicates if SSL was used for the final redirect

TYPE probe_http_ssl gauge

probe_http_ssl 1

HELP probe_http_status_code Response HTTP status code

TYPE probe_http_status_code gauge

probe_http_status_code 502

HELP probe_http_uncompressed_body_length Length of uncompressed response body

TYPE probe_http_uncompressed_body_length gauge

probe_http_uncompressed_body_length 1477

HELP probe_http_version Returns the version of HTTP of the probe response

TYPE probe_http_version gauge

probe_http_version 2

HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes.

TYPE probe_ip_addr_hash gauge

probe_ip_addr_hash 3.641314862e+09

HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6

TYPE probe_ip_protocol gauge

probe_ip_protocol 4

HELP probe_ssl_earliest_cert_expiry Returns earliest SSL cert expiry in unixtime

TYPE probe_ssl_earliest_cert_expiry gauge

probe_ssl_earliest_cert_expiry 1.6131312e+09

HELP probe_ssl_last_chain_expiry_timestamp_seconds Returns last SSL chain expiry in timestamp seconds

TYPE probe_ssl_last_chain_expiry_timestamp_seconds gauge

probe_ssl_last_chain_expiry_timestamp_seconds -6.21355968e+10

HELP probe_success Displays whether or not the probe was a success

TYPE probe_success gauge

probe_success 0

HELP probe_tls_version_info Contains the TLS version used

TYPE probe_tls_version_info gauge

probe_tls_version_info{version="TLS 1.3"} 1

Module configuration:
prober: http
timeout: 25s
http:
valid_http_versions:
- HTTP/1.0
- HTTP/1.1
- HTTP/2.0
preferred_ip_protocol: ip4
ip_protocol_fallback: true
tls_config:
insecure_skip_verify: true
tcp:
ip_protocol_fallback: true
icmp:
ip_protocol_fallback: true
dns:
ip_protocol_fallback: true

What did you expect to see?

Expect to see result of 200
We have looked into the page inspection and App Insights and both are returning the correct 200 value.

What did you see instead?

Getting an incorrect result of 502

Marcelo Magallón

unread,

Jan 31, 2021, 1:10:50 PM1/31/21

to Prometheus Users

This seems to be an issue with the server. BBE is connecting to the target (https://www2.trustnet.com) and it's getting a 502. Is the server trying to enforce some TLS parameters? Headers? Something along those lines?

Marcelo

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1718646e-fc5e-49e9-9955-2ad7c54bc632n%40googlegroups.com.

--

Marcelo Magallón

venkat raman

unread,

Feb 25, 2021, 1:27:20 PM2/25/21

to Prometheus Users

Hi Marcelo,

We are not using TLS parameters.

Please let me know if you need anymore.

Thanks,

Venkatraman N

venkat raman

unread,

Mar 9, 2021, 4:13:29 AM3/9/21

to Prometheus Users

Hi Team,

Can you please help me to resolve this issue in blackbox it is not working for only these 3 urls.

Thanks,

Venkatraman N

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a7355d4f-2597-4a40-af85-5e0195226b78n%40googlegroups.com.

Reply all

Reply to author

Forward