Mysql handshake error with blackbox tcp_connect

[Dimitris Tzampanakis]

Hello.
When blackbox exporter makes tcp connect to mysql  on port 3306 i notice handshake errors and after 100 errors the host is blocked from mysql. It was very frustrating and difficult to spot, since other success connect from same host reset the error_count from mysql and nothing was logged. This same setup is running on other environments (without problem). But in this env that big traffic didn't started yet, locks happen.
I also found this similar
https://github.com/prometheus/blackbox_exporter/issues/505
Is there anything that i missing  in the config?
Is there any other that user tpc_connect for mysql checks with similar problems?

prometheus.yml

  - job_name: 'tcp_probes'
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets: ['xxx:2181']
        labels:
          name: 'Zookeeper'
      - targets: ['xxx:6667']
        labels:
          name: 'Kafka'
      - targets: ['xxx:6379']
        labels:

          name: 'Redis'

      - targets: ['xxxx:3306'] 

        labels:

          name: 'MySQL'

    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: xxx:9115

blackbox.yml

modules:
  http_2xx:
    prober: http
    timeout: 120s
    http:
      tls_config:
        insecure_skip_verify: true
      preferred_ip_protocol: "ip4"
  http_post_2xx:
    prober: http
    http:
      method: POST
  tcp_connect:
    prober: tcp
    timeout: 120s
    http:
      preferred_ip_protocol: "ip4"
  pop3s_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^+OK"
      tls: true
      tls_config:
        insecure_skip_verify: false
  ssh_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^SSH-2.0-"
      - send: "SSH-2.0-blackbox-ssh-check"
  irc_banner:
    prober: tcp
    tcp:
      query_response:
      - send: "NICK prober"
      - send: "USER prober prober prober :prober"
      - expect: "PING :([^ ]+)"
        send: "PONG ${1}"
      - expect: "^:[^ ]+ 001"
  icmp:
    prober: icmp

blackbox-exporter version 0.19.0
mysql-server version 8.0.27 for Linux on x86_64

      

      

---------------------------------------------------------

     

Verlabs has taken all reasonable steps to ensure the security of email communications but makes no guarantees that this email will be virus/malware-proof. We would recommend that you carry out your own virus checks before opening any attachment.  This email is confidential and intended only for the named addressee. If you have received this email in error, please do not copy or use it for any purpose. Please delete it immediately and let us know by return email. You should also know that emails exchanged with Verlabs are subject to our email privacy notice.

[Stuart Clark]

On 12/10/2022 15:46, Dimitris Tzampanakis wrote:

Hello.
When blackbox exporter makes tcp connect to mysql  on port 3306 i notice handshake errors and after 100 errors the host is blocked from mysql. It was very frustrating and difficult to spot, since other success connect from same host reset the error_count from mysql and nothing was logged. This same setup is running on other environments (without problem). But in this env that big traffic didn't started yet, locks happen.
I also found this similar
https://github.com/prometheus/blackbox_exporter/issues/505
Is there anything that i missing  in the config?
Is there any other that user tpc_connect for mysql checks with similar problems?

As you are just doing a TCP connection request it will look to be some sort of failure from the MySQL server's perspective - you are just opening a connection and then not doing anything. You would need to look at the server configuration to whitelist the IP of the Blackbox Exporter so it doesn't get blocked.

Alternatively look at using the MySQL Exporter instead of using Blackbox Exporter, which will give you more insight above just availability too?

-- 
Stuart Clark