blackbox dns probe failed

62 views
Skip to first unread message

e huang

unread,
Aug 4, 2020, 4:54:09 AM8/4/20
to Prometheus Users

Host operating system: output of uname -a

Linux office-monitor 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

blackbox_exporter version: output of blackbox_exporter -version

blackbox_exporter, version 0.17.0 (branch: HEAD, revision: 1bc7680)
build user: root@626fb3899f41
build date: 20200619-11:54:41
go version: go1.14.4

What is the blackbox.yml module config.

modules:
http_2xx:
prober: http
http_post_2xx:
prober: http
http:
method: POST
tcp_connect:
prober: tcp
pop3s_banner:
prober: tcp
tcp:
query_response:
- expect: "^+OK"
tls: true
tls_config:
insecure_skip_verify: false
ssh_banner:
prober: tcp
tcp:
query_response:
- expect: "^SSH-2.0-"
irc_banner:
prober: tcp
tcp:
query_response:
- send: "NICK prober"
- send: "USER prober prober prober :prober"
- expect: "PING :([^ ]+)"
send: "PONG ${1}"
- expect: "^:[^ ]+ 001"
icmp:
prober: icmp

##############################
dns_awg.enmonster.com:
prober: dns
timeout: 5s
dns:
query_type: "A"
preferred_ip_protocol: "ip4"
query_name: "awg.enmonster.com"
valid_rcodes:
- NOERROR
dns_azg.enmonster.com:
prober: dns
timeout: 5s
dns:
query_type: "A"
preferred_ip_protocol: "ip4"
query_name: "azg.enmonster.com"
valid_rcodes:
- NOERROR
dns_www.baidu.com:
prober: dns
timeout: 5s
dns:
query_type: "A"
preferred_ip_protocol: "ip4"
query_name: "www.baidu.com"
dns_eboss.enmonster.com:
prober: dns
timeout: 5s
dns:
query_type: "A"
preferred_ip_protocol: "ip4"
query_name: "eboss.enmonster.com"
valid_rcodes:
- NOERROR
dns_hydra.enmonster.com:
prober: dns
timeout: 5s
dns:
query_type: "A"
preferred_ip_protocol: "ip4"
query_name: "hydra.enmonster.com"
dns_www.dingtalk.com:
prober: dns
timeout: 5s
dns:
query_type: "A"
preferred_ip_protocol: "ip4"
query_name: "www.dingtalk.com"

What is the prometheus.yml scrape config.

my global config

global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

scrape_timeout is set to the global default (10s).

Alertmanager configuration

alerting:
alertmanagers:

  • static_configs:
    • targets:

      - alertmanager:9093

Load rules once and periodically evaluate them according to the global 'evaluation_interval'.

rule_files:

- "first_rules.yml"

- "second_rules.yml"

A scrape configuration containing exactly one endpoint to scrape:

Here it's Prometheus itself.

scrape_configs:

The job name is added as a label job=<job_name> to any timeseries scraped from this config.

  • job_name: 'prometheus'

    metrics_path defaults to '/metrics'

    scheme defaults to 'http'.

    static_configs:

    • targets: ['localhost:9090']

#####################################################dns_check############

  • job_name: 'blackbox-dns_azg.enmonster.com'
    metrics_path: /probe
    params:
    module:
    • dns_azg.enmonster.com
      static_configs:
    • targets:
      • 10.208.100.9
      • 10.208.100.8
        labels:
        type: dns_resolve
        zone: office_sy
        relabel_configs:
    • source_labels: [address]
      target_label: "__param_target"
    • source_labels: [address]
      target_label: "instance"
    • source_labels: [_param_module]
      regex: 'dns      (.*)'
      target_label: 'domain_name'
    • target_label: address
      replacement: "127.0.0.1:9115"
  • job_name: 'blackbox-dns_awg.enmonster.com'
    metrics_path: /probe
    params:
    module:
    • dns_awg.enmonster.com
      static_configs:
    • targets:
      • 10.208.100.9
      • 10.208.100.8
        labels:
        type: dns_resolve
        zone: office_sy
        relabel_configs:
    • source_labels: [address]
      target_label: "__param_target"
    • source_labels: [address]
      target_label: "instance"
    • source_labels: [_param_module]
      regex: 'dns      (.*)'
      target_label: 'domain_name'
    • target_label: address
      replacement: "127.0.0.1:9115"
  • job_name: 'blackbox-dns_sso.enmonster.com'
    metrics_path: /probe
    params:
    module:
    • dns_sso.enmonster.com
      static_configs:
    • targets:
      • 10.208.100.9
      • 10.208.100.8
        labels:
        type: dns_resolve
        zone: office_sy
        relabel_configs:
    • source_labels: [address]
      target_label: "__param_target"
    • source_labels: [address]
      target_label: "instance"
    • source_labels: [_param_module]
      regex: 'dns      (.*)'
      target_label: 'domain_name'
    • target_label: address
      replacement: "127.0.0.1:9115"
  • job_name: 'blackbox-dns_hydra.enmonster.com'
    metrics_path: /probe
    params:
    module:
    • dns_hydra.enmonster.com
      static_configs:
    • targets:
      • 10.208.100.9
      • 10.208.100.8
        labels:
        type: dns_resolve
        zone: office_sy
        relabel_configs:
    • source_labels: [address]
      target_label: "__param_target"
    • source_labels: [address]
      target_label: "instance"
    • source_labels: [_param_module]
      regex: 'dns      (.*)'
      target_label: 'domain_name'
    • target_label: address
      replacement: "127.0.0.1:9115"
  • job_name: 'blackbox-dns_eboss.enmonster.com'
    metrics_path: /probe
    params:
    module:
    • dns_eboss.enmonster.com
      static_configs:
    • targets:
      • 10.208.100.9
      • 10.208.100.8
        labels:
        type: dns_resolve
        zone: office_sy
        relabel_configs:
    • source_labels: [address]
      target_label: "__param_target"
    • source_labels: [address]
      target_label: "instance"
    • source_labels: [_param_module]
      regex: 'dns      (.*)'
      target_label: 'domain_name'
    • target_label: address
      replacement: "127.0.0.1:9115"
  • job_name: 'blackbox-dns_www.dingtalk.com'
    metrics_path: /probe
    params:
    module:
    • dns_www.dingtalk.com
      static_configs:
    • targets:
      • 10.208.100.9
      • 10.208.100.8
        labels:
        type: dns_resolve
        zone: office_sy
        relabel_configs:
    • source_labels: [address]
      target_label: "__param_target"
    • source_labels: [address]
      target_label: "instance"
    • source_labels: [_param_module]
      regex: 'dns      (.*)'
      target_label: 'domain_name'
    • target_label: address
      replacement: "127.0.0.1:9115"
      ##############################################dns_check end#########

What logging output did you get from adding &debug=true to the probe URL?

ts=2020-08-04T05:41:58.646Z caller=main.go:169 module=dns_eboss.enmonster.com target=10.208.100.9 level=debug msg="Error while sending a DNS query" err="read udp4 10.208.100.
10:36709->10.208.100.9:53: i/o timeout"
ts=2020-08-04T05:41:58.646Z caller=main.go:169 module=dns_eboss.enmonster.com target=10.208.100.9 level=debug msg="Probe failed" duration_seconds=9.500409824

What did you do that produced an error?

就如上面的配置，监控dns解析

What did you expect to see?

它不应该报错

What did you see instead?

我的dns服务器是用dnsmasq做的，压测过，没有问题，查询监控失败时间的解析日志也是返回正常,奇怪是我用过coredns搭建的dns服务器没有这个问题
image

image
image

Christian Hoffmann

unread,
Aug 5, 2020, 6:23:12 AM8/5/20
to e huang, Prometheus Users
Hi,

On 8/4/20 10:54 AM, e huang wrote:
> ts=2020-08-04T05:41:58.646Z caller=main.go:169
> module=dns_eboss.enmonster.com target=10.208.100.9 level=debug
> msg="Error while sending a DNS query" err="read udp4 10.208.100.
> 10:36709->10.208.100.9:53: i/o timeout"
> ts=2020-08-04T05:41:58.646Z caller=main.go:169
> module=dns_eboss.enmonster.com target=10.208.100.9 level=debug
> msg="Probe failed" duration_seconds=9.500409824
>
>
> What did you do that produced an error?
>
> 就如上面的配置，监控dns解析
>
>
> What did you expect to see?
>
> 它不应该报错
>
>
> What did you see instead?
>
> 我的dns服务器是用dnsmasq做的，压测过，没有问题，查询监控失败时间的解析日
> 志也是返回正常,奇怪是我用过coredns搭建的dns服务器没有这个问题
> (My dns server is made with dnsmasq, and it has been pressure tested
and there is no problem. The resolution date of the query monitoring
failure time
Zhi also returned to normal. It is strange that the dns server built by
coredns did not have this problem.)


The log seems to say that there was a timeout. Is this issue
reproducible? It may help to set up tcpdump to capture the exact traffic
between blackbox_exporter and your DNS server.

Could there be any firewalls inbetween?

Kind regards,
Christian
Reply all
Reply to author
Forward
0 new messages