ProxySQL_Cluster - doesn't allow sync.

[Geoff Winans]

Hello.

I'm running into an issue I cannot seem to resolve.

1. ProxySQL is deployed in Kubernetes (3 pods)

2. The nodes can clearly talk to one another.

This is showing up in the logs:

proxysql-staging-2-2 proxysql 2023-11-06 22:46:28 ProxySQL_Cluster.cpp:653:set_checksums(): [WARNING] Cluster: detected a peer proxysql-1.proxysqlcluster:6032 with mysql_servers version 1, epoch 1698968621, diff_check 6450. Own version: 1, epoch: 1699304334. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every 30 checks until LOAD MYSQL SERVERS TO RUNTIME is executed on candidate master.
proxysql-staging-2-2 proxysql 2023-11-06 22:46:28 ProxySQL_Cluster.cpp:716:set_checksums(): [WARNING] Cluster: detected a peer proxysql-1.proxysqlcluster:6032 with mysql_variables version 1, epoch 1698968621, diff_check 6450. Own version: 1, epoch: 1699304334. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every 30 checks until LOAD MYSQL VARIABLES TO RUNTIME is executed on candidate master.
proxysql-staging-2-0 proxysql 2023-11-06 22:46:29 ProxySQL_Cluster.cpp:653:set_checksums(): [WARNING] Cluster: detected a peer proxysql-0.proxysqlcluster:6032 with mysql_servers version 1, epoch 1698968632, diff_check 6480. Own version: 8, epoch: 1699310776. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every 30 checks until LOAD MYSQL SERVERS TO RUNTIME is executed on candidate master.
proxysql-staging-2-0 proxysql 2023-11-06 22:46:29 ProxySQL_Cluster.cpp:716:set_checksums(): [WARNING] Cluster: detected a peer proxysql-0.proxysqlcluster:6032 with mysql_variables version 1, epoch 1698968632, diff_check 6480. Own version: 5, epoch: 1699310776. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every 30 checks until LOAD MYSQL VARIABLES TO RUNTIME is executed on candidate master.

I jumped on proxysql-staging-2-0 (which has the desired configuration) and executed:

LOAD ADMIN VARIABLES TO RUNTIME;

LOAD MYSQL VARIABLES TO RUNTIME;

LOAD MYSQL USERS TO RUNTIME;

LOAD MYSQL SERVERS TO RUNTIME;

The error continues. The documentation doesn't seem to offer any real troubleshooting steps beyond, "Execute LOAD ... on the instance with the highest version and epoch."

proxysql-staging-2-0 has the highest version and epoch of all 3 instances.

-G

[René Cannaò]

Hi Geoff,

What you executed is all you needed to perform.

"The error continues" : show it please.

You are reporting the messages before you intervened , but you aren't showing what happened when you intervened, and what happened after.

Thanks,

René

--
You received this message because you are subscribed to the Google Groups "proxysql" group.
To unsubscribe from this group and stop receiving emails from it, send an email to proxysql+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/proxysql/84e7caf8-19af-47c0-bdd7-1597ff316ea2n%40googlegroups.com.

[Geoff Winans]

The only change was incrementing proxysql-2-0's versions for 'MYSQL VARIABLES' and 'MYSQL USERS'.

Otherwise, the messages are the same.

----

proxysql-staging-2-2 proxysql 2023-11-06 23:24:02 ProxySQL_Cluster.cpp:653:set_checksums(): [WARNING] Cluster: detected a peer proxysql-1.proxysqlcluster:6032 with mysql_servers version 1, epoch 1698968621, diff_check 1170. Own version: 1, epoch: 1699304334. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every 30 checks until LOAD MYSQL SERVERS TO RUNTIME is executed on candidate master.
proxysql-staging-2-2 proxysql 2023-11-06 23:24:02 ProxySQL_Cluster.cpp:716:set_checksums(): [WARNING] Cluster: detected a peer proxysql-1.proxysqlcluster:6032 with mysql_variables version 1, epoch 1698968621, diff_check 1170. Own version: 1, epoch: 1699304334. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every 30 checks until LOAD MYSQL VARIABLES TO RUNTIME is executed on candidate master.

---

proxysql-staging-2-0 proxysql 2023-11-06 23:24:09 ProxySQL_Cluster.cpp:653:set_checksums(): [WARNING] Cluster: detected a peer proxysql-0.proxysqlcluster:6032 with mysql_servers version 1, epoch 1698968632, diff_check 1200. Own version: 11, epoch: 1699313024. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every 30 checks until LOAD MYSQL SERVERS TO RUNTIME is executed on candidate master.

proxysql-staging-2-0 proxysql 2023-11-06 23:24:09 ProxySQL_Cluster.cpp:716:set_checksums(): [WARNING] Cluster: detected a peer proxysql-0.proxysqlcluster:6032 with mysql_variables version 1, epoch 1698968632, diff_check 1200. Own version: 8, epoch: 1699313024. diff_check is increasing, but version 1 doesn't allow sync. This message will be repeated every 30 checks until LOAD MYSQL VARIABLES TO RUNTIME is executed on candidate master.

[René Cannaò]

Hi Geoff,

"but you aren't showing what happened when you intervened" : I still don't see it...

I recommend to attach all the logs from all the relevant nodes, including the logs of the other nodes : did they sync? Yes, no? Why?

[Geoff Winans]

I'm not sure what you're looking for.

proxysql-staging-2-0 proxysql 2023-11-06 23:23:44 [INFO] Received LOAD ADMIN VARIABLES TO RUNTIME command
proxysql-staging-2-0 proxysql 2023-11-06 23:23:44 [INFO] Received LOAD MYSQL VARIABLES TO RUNTIME command
proxysql-staging-2-0 proxysql 2023-11-06 23:23:44 [INFO] Received LOAD MYSQL USERS TO RUNTIME command
proxysql-staging-2-0 proxysql 2023-11-06 23:23:44 [INFO] Received LOAD MYSQL SERVERS TO RUNTIME command
proxysql-staging-2-0 proxysql 2023-11-06 23:23:44 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
proxysql-staging-2-0 proxysql HID: 10 , address: staging-2 , port: 3306 , gtid_port: 0 , weight: 1 , status: SHUNNED , max_connections: 100 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
proxysql-staging-2-0 proxysql HID: 10 , address: staging-2-replica-1 , port: 3306 , gtid_port: 0 , weight: 1 , status: SHUNNED , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
proxysql-staging-2-0 proxysql 2023-11-06 23:23:44 [INFO] Dumping mysql_servers: ALL
proxysql-staging-2-0 proxysql +-----+---------------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
proxysql-staging-2-0 proxysql | hid | hostname            | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
proxysql-staging-2-0 proxysql +-----+---------------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
proxysql-staging-2-0 proxysql | 10  | staging-2           | 3306 | 0    | 1      | 1      | 0   | 100       | 0       | 0   | 0       |         | 139801952252768 |
proxysql-staging-2-0 proxysql | 10  | staging-2-replica-1 | 3306 | 0    | 1      | 1      | 0   | 1000      | 0       | 0   | 0       |         | 139801819105216 |
proxysql-staging-2-0 proxysql +-----+---------------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
proxysql-staging-2-0 proxysql 2023-11-06 23:23:44 [INFO] Dumping mysql_servers_incoming
proxysql-staging-2-0 proxysql +--------------+---------------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
proxysql-staging-2-0 proxysql | hostgroup_id | hostname            | port | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
proxysql-staging-2-0 proxysql +--------------+---------------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
proxysql-staging-2-0 proxysql | 10           | staging-2           | 3306 | 0         | 1      | 0      | 0           | 100             | 0                   | 0       | 0              |         |
proxysql-staging-2-0 proxysql | 10           | staging-2-replica-1 | 3306 | 0         | 1      | 0      | 0           | 1000            | 0                   | 0       | 0              |         |
proxysql-staging-2-0 proxysql +--------------+---------------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
proxysql-staging-2-0 proxysql 2023-11-06 23:23:44 [INFO] Dumping mysql_servers LEFT JOIN mysql_servers_incoming
proxysql-staging-2-0 proxysql +-------------+--------------+----------+------+
proxysql-staging-2-0 proxysql | mem_pointer | hostgroup_id | hostname | port |
proxysql-staging-2-0 proxysql +-------------+--------------+----------+------+
proxysql-staging-2-0 proxysql +-------------+--------------+----------+------+
proxysql-staging-2-0 proxysql 2023-11-06 23:23:44 [INFO] Dumping mysql_servers JOIN mysql_servers_incoming
proxysql-staging-2-0 proxysql +--------------+---------------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
proxysql-staging-2-0 proxysql | hostgroup_id | hostname            | port | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer     | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
proxysql-staging-2-0 proxysql +--------------+---------------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
proxysql-staging-2-0 proxysql | 10           | staging-2           | 3306 | 0         | 1      | 1      | 0           | 100             | 0                   | 0       | 0              |         | 139801952252768 | 0         | 1      | 0      | 0           | 100             | 0                   | 0       | 0              |         |
proxysql-staging-2-0 proxysql | 10           | staging-2-replica-1 | 3306 | 0         | 1      | 1      | 0           | 1000            | 0                   | 0       | 0              |         | 139801819105216 | 0         | 1      | 0      | 0           | 1000            | 0                   | 0       | 0              |         |
proxysql-staging-2-0 proxysql +--------------+---------------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
proxysql-staging-2-0 proxysql 2023-11-06 23:23:44 [INFO] Changing status for server 10:staging-2:3306 (staging-2:3306) from 1 (1) to 0
proxysql-staging-2-0 proxysql 2023-11-06 23:23:44 [INFO] Changing status for server 10:staging-2-replica-1:3306 (staging-2-replica-1:3306) from 1 (1) to 0
proxysql-staging-2-0 proxysql 2023-11-06 23:23:44 [INFO] New mysql_replication_hostgroups table
proxysql-staging-2-0 proxysql 2023-11-06 23:23:44 [INFO] New mysql_group_replication_hostgroups table
proxysql-staging-2-0 proxysql 2023-11-06 23:23:44 [INFO] New mysql_galera_hostgroups table
proxysql-staging-2-0 proxysql 2023-11-06 23:23:44 [INFO] New mysql_aws_aurora_hostgroups table
proxysql-staging-2-0 proxysql 2023-11-06 23:23:44 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
proxysql-staging-2-0 proxysql HID: 10 , address: staging-2 , port: 3306 , gtid_port: 0 , weight: 1 , status: ONLINE , max_connections: 100 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
proxysql-staging-2-0 proxysql HID: 10 , address: staging-2-replica-1 , port: 3306 , gtid_port: 0 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
proxysql-staging-2-0 proxysql 2023-11-06 23:23:44 [INFO] Dumping mysql_servers: ALL
proxysql-staging-2-0 proxysql +-----+---------------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
proxysql-staging-2-0 proxysql | hid | hostname            | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
proxysql-staging-2-0 proxysql +-----+---------------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
proxysql-staging-2-0 proxysql | 10  | staging-2           | 3306 | 0    | 1      | 0      | 0   | 100       | 0       | 0   | 0       |         | 139801952252768 |
proxysql-staging-2-0 proxysql | 10  | staging-2-replica-1 | 3306 | 0    | 1      | 0      | 0   | 1000      | 0       | 0   | 0       |         | 139801819105216 |
proxysql-staging-2-0 proxysql +-----+---------------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
proxysql-staging-2-0 proxysql 2023-11-06 23:23:44 [INFO] Checksum for table mysql_servers is 9633745961405148940
proxysql-staging-2-0 proxysql 2023-11-06 23:23:44 [INFO] MySQL_HostGroups_Manager::commit() locked for 2ms
proxysql-staging-2-0 proxysql 2023-11-06 23:23:44 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
proxysql-staging-2-0 proxysql HID: 10 , address: staging-2 , port: 3306 , gtid_port: 0 , weight: 1 , status: ONLINE , max_connections: 100 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
proxysql-staging-2-0 proxysql HID: 10 , address: staging-2-replica-1 , port: 3306 , gtid_port: 0 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
proxysql-staging-2-0 proxysql 2023-11-06 23:23:44 [INFO] Dumping mysql_servers: ALL
proxysql-staging-2-0 proxysql +-----+---------------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
proxysql-staging-2-0 proxysql | hid | hostname            | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
proxysql-staging-2-0 proxysql +-----+---------------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
proxysql-staging-2-0 proxysql | 10  | staging-2           | 3306 | 0    | 1      | 0      | 0   | 100       | 0       | 0   | 0       |         | 139801952252768 |
proxysql-staging-2-0 proxysql | 10  | staging-2-replica-1 | 3306 | 0    | 1      | 0      | 0   | 1000      | 0       | 0   | 0       |         | 139801819105216 |
proxysql-staging-2-0 proxysql +-----+---------------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+

----

The other two nodes show no log activity when I execute "LOAD ..." on proxysql-staging-2-0. No acknowledgement other than the warning repeating with the update version value from proxysql-0

Is there a way I can determine which node(s) an instance "thinks" are candidate masters?

[Geoff Winans]

Ah hah. This was a goof on my side related to the kubernetes hostnames.

Chalk it up to me still learning K8s!

-G