BlackBox Exporter issue with DNS Resolution and Timeout error

[Kabir Oberoi]

Hello Gurus,

I have configured Blackbox and it seems working fine. However, it's unable to resolve the DNS of the service and shows service down.

When checked the logs, I see it says timeout error. However, inside the pod I am able to resolve this to IP address properly.

C:\Users\opc>curl -s "http://141.144.255.247/probe?target=http://example.com&module=http_2xx&debug=true"
Logs for the probe:
ts=2022-10-29T10:25:15.927180716Z caller=main.go:352 module=http_2xx target=http://example.com level=info msg="Beginning probe" probe=http timeout_seconds=5
ts=2022-10-29T10:25:15.927352861Z caller=http.go:338 module=http_2xx target=http://example.com level=info msg="Resolving target address" target=example.com ip_protocol=ip4
ts=2022-10-29T10:25:20.927401659Z caller=http.go:338 module=http_2xx target=http://example.com level=error msg="Resolution with IP protocol failed" target=example.com err="lookup example.com: i/o timeout"
ts=2022-10-29T10:25:20.927471802Z caller=main.go:144 module=http_2xx target=http://example.com level=error msg="Error resolving address" err="lookup example.com: i/o timeout"
ts=2022-10-29T10:25:20.927488393Z caller=main.go:352 module=http_2xx target=http://example.com level=error msg="Probe failed" duration_seconds=5.000273152



Metrics that would have been returned:
# HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds
# TYPE probe_dns_lookup_time_seconds gauge
probe_dns_lookup_time_seconds 5.000093824
# HELP probe_duration_seconds Returns how long the probe took to complete in seconds
# TYPE probe_duration_seconds gauge
probe_duration_seconds 5.000273152
# HELP probe_failed_due_to_regex Indicates if probe failed due to regex
# TYPE probe_failed_due_to_regex gauge
probe_failed_due_to_regex 0
# HELP probe_http_content_length Length of http content response
# TYPE probe_http_content_length gauge
probe_http_content_length 0
# HELP probe_http_duration_seconds Duration of http request by phase, summed over all redirects
# TYPE probe_http_duration_seconds gauge
probe_http_duration_seconds{phase="resolve"} 5.000093824
# HELP probe_http_redirects The number of redirects
# TYPE probe_http_redirects gauge
probe_http_redirects 0
# HELP probe_http_ssl Indicates if SSL was used for the final redirect
# TYPE probe_http_ssl gauge
probe_http_ssl 0
# HELP probe_http_status_code Response HTTP status code
# TYPE probe_http_status_code gauge
probe_http_status_code 0
# HELP probe_http_uncompressed_body_length Length of uncompressed response body
# TYPE probe_http_uncompressed_body_length gauge
probe_http_uncompressed_body_length 0
# HELP probe_http_version Returns the version of HTTP of the probe response
# TYPE probe_http_version gauge
probe_http_version 0
# HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes.
# TYPE probe_ip_addr_hash gauge
probe_ip_addr_hash 0
# HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6
# TYPE probe_ip_protocol gauge
probe_ip_protocol 0
# HELP probe_success Displays whether or not the probe was a success
# TYPE probe_success gauge
probe_success 0



Module configuration:
prober: http
timeout: 5s
http:
    valid_http_versions:
        - HTTP/1.1
        - HTTP/2.0
    preferred_ip_protocol: ip4
    ip_protocol_fallback: true
    follow_redirects: true
    enable_http2: true
tcp:
    ip_protocol_fallback: true
icmp:
    ip_protocol_fallback: true
    ttl: 64
dns:
    ip_protocol_fallback: true
    recursion_desired: true

Inside the POD am able to resolve thsi to IP:

~ $ nslookup exmaple.com
Server:         10.96.5.5
Address:        10.96.5.5:53

Non-authoritative answer:
Name:   exmaple.com
Address: 104.18.144.11
Name:   exmaple.com
Address: 104.18.145.11

*** Can't find exmaple.com: No answer

Thanks,

Kabir Oberoi

[Brian Candler]

On Saturday, 29 October 2022 at 11:27:39 UTC+1 kabir...@gmail.com wrote:

~ $ nslookup exmaple.com

Server:         10.96.5.5
Address:        10.96.5.5:53

Non-authoritative answer:
Name:   exmaple.com
Address: 104.18.144.11
Name:   exmaple.com
Address: 104.18.145.11

*** Can't find exmaple.com: No answer

That's a very mangled response: you seem to be showing a success response and a failure response at the same time.  Can you show the output of "dig exmaple.com" instead?

You can also try tcpdump'ing your DNS traffic between your pod and your DNS cache.

[b...@ritcey.com]

I'm assuming that 'exmaple.com' is meant to be 'example.com', as in that's what you're probing?  

Are those literal cut-and-pastes?

'example.com' will in fact resolve, and has a stub website, which may be useful for testing.