ProxySQL Crashing due to watchdog missed heartbeats
212 views
Skip to first unread message
James Janovich
Feb 15, 2021, 8:49:35 PM2/15/21
to proxysql
All,
We have 1 master and 2 slaves. ProxySQL is v 2.0.15. Mysql is 5.7.29
Here is the important part of the log. Any ideas?
2021-02-15 19:34:52 main.cpp:1865:main(): [ERROR] Watchdog: 1 threads missed a heartbeat
2021-02-15 19:34:57 MySQL_HostGroups_Manager.cpp:2867:replication_lag_action(): [WARNING] Shunning server 172.24.16.150:3306 from HG 11 with replication lag of 9 second, count number: '1'
2021-02-15 19:35:00 main.cpp:1865:main(): [ERROR] Watchdog: 8 threads missed a heartbeat
2021-02-15 19:35:00 MySQL_HostGroups_Manager.cpp:2867:replication_lag_action(): [WARNING] Shunning server 172.24.16.246:3306 from HG 11 with replication lag of 9 second, count number: '1'
2021-02-15 19:35:03 main.cpp:1865:main(): [ERROR] Watchdog: 1 threads missed a heartbeat
2021-02-15 19:35:10 main.cpp:1865:main(): [ERROR] Watchdog: 1 threads missed a heartbeat
2021-02-15 19:35:13 MySQL_Session.cpp:6600:unable_to_parse_set_statement(): [WARNING] Unable to parse unknown SET query from client 127.0.0.1:52666. Setting lock_hostgroup. Please report a bug for future enhancements:SET SESSION wait_timeout=10000
2021-02-15 19:35:17 MySQL_HostGroups_Manager.cpp:2891:replication_lag_action(): [WARNING] Re-enabling server 172.24.16.246:3306 from HG 11 with replication lag of 2 second
2021-02-15 19:35:17 MySQL_HostGroups_Manager.cpp:2891:replication_lag_action(): [WARNING] Re-enabling server 172.24.16.150:3306 from HG 11 with replication lag of 2 second
2021-02-15 19:35:19 main.cpp:1865:main(): [ERROR] Watchdog: 3 threads missed a heartbeat
mysql: [Warning] Using a password on the command line interface can be insecure.
2021-02-15 19:35:22 main.cpp:1865:main(): [ERROR] Watchdog: 4 threads missed a heartbeat
2021-02-15 19:35:25 main.cpp:1865:main(): [ERROR] Watchdog: 6 threads missed a heartbeat
mysql: [Warning] Using a password on the command line interface can be insecure.
2021-02-15 19:35:27 MySQL_HostGroups_Manager.cpp:2867:replication_lag_action(): [WARNING] Shunning server 172.24.16.246:3306 from HG 11 with replication lag of 12 second, count number: '1'
2021-02-15 19:35:27 MySQL_HostGroups_Manager.cpp:2867:replication_lag_action(): [WARNING] Shunning server 172.24.16.150:3306 from HG 11 with replication lag of 12 second, count number: '1'
2021-02-15 19:35:28 main.cpp:1865:main(): [ERROR] Watchdog: 9 threads missed a heartbeat
2021-02-15 19:35:31 main.cpp:1865:main(): [ERROR] Watchdog: 10 threads missed a heartbeat
mysql: [Warning] Using a password on the command line interface can be insecure.
2021-02-15 19:35:34 main.cpp:1865:main(): [ERROR] Watchdog: 11 threads missed a heartbeat
mysql: [Warning] Using a password on the command line interface can be insecure.
2021-02-15 19:35:37 main.cpp:1865:main(): [ERROR] Watchdog: 11 threads missed a heartbeat
2021-02-15 19:35:40 main.cpp:1865:main(): [ERROR] Watchdog: 11 threads missed a heartbeat
mysql: [Warning] Using a password on the command line interface can be insecure.
2021-02-15 19:35:43 main.cpp:1865:main(): [ERROR] Watchdog: 13 threads missed a heartbeat
2021-02-15 19:35:46 main.cpp:1865:main(): [ERROR] Watchdog: 17 threads missed a heartbeat
2021-02-15 19:35:46 main.cpp:1869:main(): [ERROR] Watchdog: reached 10 missed heartbeats. Aborting!
2021-02-15 19:35:46 main.cpp:1870:main(): [ERROR] Watchdog: see details at https://github.com/sysown/proxysql/wiki/Watchdog
proxysql: main.cpp:1871: int main(int, const char**): Assertion `0' failed.
Error: signal 6:
/usr/bin/proxysql(_Z13crash_handleri+0x1a)[0x4bda1a]
/lib64/libc.so.6(+0x36400)[0x7f4f6e67f400]
/lib64/libc.so.6(gsignal+0x37)[0x7f4f6e67f387]
/lib64/libc.so.6(abort+0x148)[0x7f4f6e680a78]
/lib64/libc.so.6(+0x2f1a6)[0x7f4f6e6781a6]
/lib64/libc.so.6(+0x2f252)[0x7f4f6e678252]
/usr/bin/proxysql(main+0xc23)[0x4b5653]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f4f6e66b555]
/usr/bin/proxysql[0x4b695e]
ERROR 2006 (HY000) at line 1: MySQL server has gone away
ERROR 2006 (HY000) at line 1: MySQL server has gone away
ERROR 2006 (HY000) at line 1: MySQL server has gone away
ERROR 2006 (HY000) at line 1: MySQL server has gone away
ERROR 2006 (HY000) at line 1: MySQL server has gone away
ERROR 2006 (HY000) at line 1: MySQL server has gone away
2021-02-15 19:37:20 main.cpp:1573:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2021-02-15 19:37:20 [INFO] ProxySQL version 2.0.15-percona-1.1
2021-02-15 19:37:21 [INFO] ProxySQL SHA1 checksum: ba20cce5bfd236bd545869499a57a95637193e76
2021-02-15 19:37:21 [INFO] Angel process started ProxySQL process 60435
Standard ProxySQL MySQL Logger rev. 2.0.0714 -- MySQL_Logger.cpp -- Fri Nov 13 11:13:22 2020
Standard ProxySQL Cluster rev. 0.4.0906 -- ProxySQL_Cluster.cpp -- Fri Nov 13 11:13:22 2020
Standard ProxySQL Statistics rev. 1.4.1027 -- ProxySQL_Statistics.cpp -- Fri Nov 13 11:13:22 2020
Standard ProxySQL HTTP Server Handler rev. 1.4.1031 -- ProxySQL_HTTP_Server.cpp -- Fri Nov 13 11:13:22 2020
2021-02-15 19:37:21 MySQL_Thread.cpp:2032:set_variable(): [WARNING] mysql-wait_timeout is set to a low value: 3600ms
2021-02-15 19:37:21 ProxySQL_Admin.cpp:5549:flush_mysql_variables___database_to_runtime(): [WARNING] Impossible to set not existing variable default_sql_mode with value "". Deleting. If the variable name is correct, this version doesn't support it
2021-02-15 19:37:21 ProxySQL_Admin.cpp:5549:flush_mysql_variables___database_to_runtime(): [WARNING] Impossible to set not existing variable default_time_zone with value "SYSTEM". Deleting. If the variable name is correct, this version doesn't support it
2021-02-15 19:37:21 ProxySQL_Admin.cpp:5549:flush_mysql_variables___database_to_runtime(): [WARNING] Impossible to set not existing variable default_isolation_level with value "READ COMMITTED". Deleting. If the variable name is correct, this version doesn't support it
2021-02-15 19:37:21 ProxySQL_Admin.cpp:5549:flush_mysql_variables___database_to_runtime(): [WARNING] Impossible to set not existing variable default_transaction_read with value "WRITE". Deleting. If the variable name is correct, this version doesn't support it
2021-02-15 19:37:21 ProxySQL_Admin.cpp:5549:flush_mysql_variables___database_to_runtime(): [WARNING] Impossible to set not existing variable default_character_set_results with value "NULL". Deleting. If the variable name is correct, this version doesn't support it
2021-02-15 19:37:21 ProxySQL_Admin.cpp:5549:flush_mysql_variables___database_to_runtime(): [WARNING] Impossible to set not existing variable default_sql_auto_is_null with value "OFF". Deleting. If the variable name is correct, this version doesn't support it
2021-02-15 19:37:21 ProxySQL_Admin.cpp:5549:flush_mysql_variables___database_to_runtime(): [WARNING] Impossible to set not existing variable default_sql_select_limit with value "DEFAULT". Deleting. If the variable name is correct, this version doesn't support it
2021-02-15 19:37:21 ProxySQL_Admin.cpp:5549:flush_mysql_variables___database_to_runtime(): [WARNING] Impossible to set not existing variable default_sql_safe_updates with value "OFF". Deleting. If the variable name is correct, this version doesn't support it
2021-02-15 19:37:21 ProxySQL_Admin.cpp:5564:flush_mysql_variables___database_to_runtime(): [WARNING] Variable default_collation_connection with value "" is being replaced with value "utf8_general_ci".
2021-02-15 19:37:21 ProxySQL_Admin.cpp:5549:flush_mysql_variables___database_to_runtime(): [WARNING] Impossible to set not existing variable default_net_write_timeout with value "60". Deleting. If the variable name is correct, this version doesn't support it
2021-02-15 19:37:21 ProxySQL_Admin.cpp:5549:flush_mysql_variables___database_to_runtime(): [WARNING] Impossible to set not existing variable default_max_join_size with value "18446744073709551615". Deleting. If the variable name is correct, this version doesn't support it
2021-02-15 19:37:21 ProxySQL_Admin.cpp:5549:flush_mysql_variables___database_to_runtime(): [WARNING] Impossible to set not existing variable default_CHARSET with value "". Deleting. If the variable name is correct, this version doesn't support it
2021-02-15 19:37:21 ProxySQL_Admin.cpp:5549:flush_mysql_variables___database_to_runtime(): [WARNING] Impossible to set not existing variable default_action with value "". Deleting. If the variable name is correct, this version doesn't support it
2021-02-15 19:37:21 ProxySQL_Admin.cpp:5549:flush_mysql_variables___database_to_runtime(): [WARNING] Impossible to set not existing variable default_names with value "". Deleting. If the variable name is correct, this version doesn't support it
2021-02-15 19:37:21 ProxySQL_Admin.cpp:5549:flush_mysql_variables___database_to_runtime(): [WARNING] Impossible to set not existing variable default_character_set_connection with value "". Deleting. If the variable name is correct, this version doesn't support it
2021-02-15 19:37:21 ProxySQL_Admin.cpp:5549:flush_mysql_variables___database_to_runtime(): [WARNING] Impossible to set not existing variable default_character_set_client with value "". Deleting. If the variable name is correct, this version doesn't support it
2021-02-15 19:37:21 ProxySQL_Admin.cpp:5549:flush_mysql_variables___database_to_runtime(): [WARNING] Impossible to set not existing variable default_character_set_database with value "". Deleting. If the variable name is correct, this version doesn't support it
2021-02-15 19:37:21 ProxySQL_Admin.cpp:5549:flush_mysql_variables___database_to_runtime(): [WARNING] Impossible to set not existing variable default_sql_log_bin with value "". Deleting. If the variable name is correct, this version doesn't support it
2021-02-15 19:37:21 ProxySQL_Admin.cpp:5549:flush_mysql_variables___database_to_runtime(): [WARNING] Impossible to set not existing variable default_wsrep_sync_wait with value "". Deleting. If the variable name is correct, this version doesn't support it
2021-02-15 19:37:21 ProxySQL_Admin.cpp:5549:flush_mysql_variables___database_to_runtime(): [WARNING] Impossible to set not existing variable default_group_concat_max_len with value "". Deleting. If the variable name is correct, this version doesn't support it
Standard ProxySQL Admin rev. 2.0.6.0805 -- ProxySQL_Admin.cpp -- Fri Nov 13 11:13:22 2020
2021-02-15 19:37:21 [INFO] ProxySQL SHA1 checksum: ba20cce5bfd236bd545869499a57a95637193e76
Standard MySQL Threads Handler rev. 0.2.0902 -- MySQL_Thread.cpp -- Fri Nov 13 11:13:22 2020
Standard MySQL Authentication rev. 0.2.0902 -- MySQL_Authentication.cpp -- Fri Nov 13 11:13:22 2020
2021-02-15 19:37:21 [INFO] Dumping mysql_servers_incoming
+--------------+---------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+---------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 11 | 172.24.16.247 | 3306 | 0 | 1000 | 0 | 0 | 4100 | 0 | 0 | 0 | |
| 11 | 172.24.16.150 | 3306 | 0 | 1000 | 0 | 0 | 4100 | 5 | 0 | 0 | |
| 10 | 172.24.16.246 | 3306 | 0 | 1 | 0 | 0 | 4100 | 0 | 0 | 0 | |
| 11 | 172.24.16.246 | 3306 | 0 | 1 | 0 | 0 | 4100 | 5 | 0 | 0 | |
+--------------+---------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2021-02-15 19:37:21 [INFO] Dumping mysql_servers LEFT JOIN mysql_servers_incoming
+-------------+--------------+----------+------+
| mem_pointer | hostgroup_id | hostname | port |
+-------------+--------------+----------+------+
+-------------+--------------+----------+------+
2021-02-15 19:37:21 [INFO] Dumping mysql_servers JOIN mysql_servers_incoming
+--------------+---------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+---------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 11 | 172.24.16.247 | 3306 | 0 | 1000 | 0 | 0 | 4100 | 0 | 0 | 0 | | 0 | 0 | 1000 | 0 | 0 | 4100 | 0 | 0 | 0 | |
| 11 | 172.24.16.150 | 3306 | 0 | 1000 | 0 | 0 | 4100 | 5 | 0 | 0 | | 0 | 0 | 1000 | 0 | 0 | 4100 | 5 | 0 | 0 | |
| 10 | 172.24.16.246 | 3306 | 0 | 1 | 0 | 0 | 4100 | 0 | 0 | 0 | | 0 | 0 | 1 | 0 | 0 | 4100 | 0 | 0 | 0 | |
| 11 | 172.24.16.246 | 3306 | 0 | 1 | 0 | 0 | 4100 | 5 | 0 | 0 | | 0 | 0 | 1 | 0 | 0 | 4100 | 5 | 0 | 0 | |
+--------------+---------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2021-02-15 19:37:21 [INFO] Creating new server in HG 11 : 172.24.16.247:3306 , gtid_port=0, weight=1000, status=0
2021-02-15 19:37:21 [INFO] Creating new server in HG 11 : 172.24.16.150:3306 , gtid_port=0, weight=1000, status=0
2021-02-15 19:37:21 [INFO] Creating new server in HG 10 : 172.24.16.246:3306 , gtid_port=0, weight=1, status=0
2021-02-15 19:37:21 [INFO] Creating new server in HG 11 : 172.24.16.246:3306 , gtid_port=0, weight=1, status=0
2021-02-15 19:37:21 [INFO] New mysql_group_replication_hostgroups table
2021-02-15 19:37:21 [INFO] New mysql_galera_hostgroups table
2021-02-15 19:37:21 [INFO] New mysql_aws_aurora_hostgroups table
2021-02-15 19:37:21 [INFO] MySQL_HostGroups_Manager::commit() locked for 2ms
Standard Query Processor rev. 2.0.6.0805 -- Query_Processor.cpp -- Fri Nov 13 11:13:22 2020
In memory Standard Query Cache (SQC) rev. 1.2.0905 -- Query_Cache.cpp -- Fri Nov 13 11:13:22 2020
Standard MySQL Monitor (StdMyMon) rev. 2.0.1226 -- MySQL_Monitor.cpp -- Fri Nov 13 11:13:22 2020
2021-02-15 19:37:24 MySQL_Session.cpp:6600:unable_to_parse_set_statement(): [WARNING] Unable to parse unknown SET query from client 127.0.0.1:52962. Setting lock_hostgroup. Please report a bug for future enhancements:SET SESSION wait_timeout=10000
2021-02-15 19:37:24 [INFO] Latest ProxySQL version available: 2.1.0-544-g17a4b4a7
Reply all
Reply to author
Forward
0 new messages