ProxySQL connection loss timeout?

4,635 views
Skip to first unread message

aaron.i...@gmail.com

unread,
Oct 18, 2016, 9:52:14 AM10/18/16
to proxysql
Good day,

I am setting up a replication auto failover for MySQL 5.5.  Yes I will be updating that, but that's what I have to work with at the moment and there currently is no auto failover mechanism in place.  My solution needs to be implemented without modifying any client code, naturally.

In my case, the salve is only there for redundancy.

I have been exploring MySQL Router for it's auto switching through a pool of servers upon loss of connection, which it seems to do quite well.  It's shortcoming, for my use case, is that it requires the client to recognize the connection error and retry the connection in order for them to communicate with the next database in the pool.

I tried replacing MySQL Router with ProxySQL, but I couldn't figure out how to make it work for this use case, or determine if it is even possible.

However, ProxySQL's query caching and connection retrying would appear to be a perfect fit in combination with MySQL Router to meet my needs, as "illustrated" below.

Client -> ProxySQL -> MySQL Router -> Master -> Slave

This setup works, but when the Master goes away, there is a delay of about a minute or so depending on the ProxySQL timeout settings I've tried.

How do I minimize the delay here?

If I do not have ProxySQL in the setup, the client immediately sees the connection loss.  I have mysql-monitor_connect_interval = 2500 and can see that reflected in the MySQL Router log.  It even shows right on time after MySQL Router switches to the Slave, so there's no delay there.

Any thoughts, suggestions or opinions would be greatly welcomed.

I attached my ProxySQL settings for reference.

Thanks,

Aaron

ProxySQL_Settings.txt

René Cannaò

unread,
Oct 18, 2016, 10:56:43 AM10/18/16
to aaron.i...@gmail.com, proxysql
Hi Aaron,

Answers inline.

On 18 October 2016 at 06:52, <aaron.i...@gmail.com> wrote:
Good day,

I am setting up a replication auto failover for MySQL 5.5.  Yes I will be updating that, but that's what I have to work with at the moment and there currently is no auto failover mechanism in place.  My solution needs to be implemented without modifying any client code, naturally.

In my case, the salve is only there for redundancy.

I have been exploring MySQL Router for it's auto switching through a pool of servers upon loss of connection, which it seems to do quite well.  It's shortcoming, for my use case, is that it requires the client to recognize the connection error and retry the connection in order for them to communicate with the next database in the pool.

I tried replacing MySQL Router with ProxySQL, but I couldn't figure out how to make it work for this use case, or determine if it is even possible.


ProxySQL doesn't perform any failover on its own.
I am not very familiar on how MySQL Router, but reading a bit around (http://mysqlhighavailability.com/easy-load-balancing-and-high-availability-using-mysql-router/) it seems that MySQL Routers doesn't perform any failover as well: it "just" stop sending traffic to the main node and start sending traffic to a secondary.

ProxySQL normally should be combined with a manager software, like MHA (https://github.com/yoshinorim/mha4mysql-manager) or mysqlfialover (https://dev.mysql.com/doc/mysql-utilities/1.5/en/mysqlfailover.html) .

There are at least two feature requests in GitHub to support a similar feature in ProxySQL:
https://github.com/sysown/proxysql/issues/697
https://github.com/sysown/proxysql/issues/730
I think this feature will be finally implemented within 2 weeks.

There are 2 ways I can think that are easy to implement right now:

a) you create an 2 hostgroups, HG1 for the master and HG2 for the slave. At regular interval you run a check and if active master is not available, you move the slave from HG2 to HG1.

b)you configure both servers in the same hostgroups you insert the active master with a weight > 0, and the standby master with a weight of 0. At regular interval you run a check and if active master is not available, the weights are changed.

What I refer to "check" here is an external script to be executed by the Scheduler:

https://github.com/sysown/proxysql/blob/master/doc/scheduler.md

To be honest, I think the use of an external script executed thought ProxySQL's Scheduler is the best option, because it allows you to create your custom criteria to determine is a server is down.
 
However, ProxySQL's query caching and connection retrying would appear to be a perfect fit in combination with MySQL Router to meet my needs, as "illustrated" below.

Client -> ProxySQL -> MySQL Router -> Master -> Slave

This setup works, but when the Master goes away, there is a delay of about a minute or so depending on the ProxySQL timeout settings I've tried.

Ah, here something interesting happens!
ProxySQL is connecting to MySQL Router that, well, it never went down!
When the master goes down, MySQL Router immediately closes also the client connection, that in this case is the connection from ProxySQL and not from the client. ProxySQL detects that the backend (MySQL Router in this case) has closed the connections. Because ProxySQL has automatic retry configured, it tries to retry to send the query without notifying the client.
What I suspect after that is:
- ProxySQL fails to communicate to a backend via MySQL Router
- after 5 consecutive failures marks its backend (MySQL Router!) as shunned
- finally, the client get an error

Can you please the log file? There may be some details there.
 
How do I minimize the delay here?

If I do not have ProxySQL in the setup, the client immediately sees the connection loss.  I have mysql-monitor_connect_interval = 2500 and can see that reflected in the MySQL Router log.  It even shows right on time after MySQL Router switches to the Slave, so there's no delay there.

As described in the previous comment, without ProxySQL,  MySQL Router closes the client connection, so the client immediately sees the connection loss.

Any thoughts, suggestions or opinions would be greatly welcomed.


My recommendation is to write a custom script to run with ProxySQL .

Thanks,
René
 
I attached my ProxySQL settings for reference.

Thanks,

Aaron

--
You received this message because you are subscribed to the Google Groups "proxysql" group.
To unsubscribe from this group and stop receiving emails from it, send an email to proxysql+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

aaron.i...@gmail.com

unread,
Oct 18, 2016, 12:05:39 PM10/18/16
to proxysql, aaron.i...@gmail.com
Hi René,

Thanks for the quick reply and suggestions.  I will have a look at ProxySQL's scheduler and MHA.  I had looked at MySQL's mysqlfailover, but it works with MySQL 5.6.5+ and not 5.5.

My primary focus in this setup was to prevent me having to script anything and have something I can get into production sooner than later.  MySQL Router handles the connectivity transition from Master to Slave very well for new connections, but not existing connections.  ProxySQL handles the query cache and connection retry very well, so the combination seems like a winner to me.  In practice, this is working, but the delay of ~52 seconds is a bit long.

I have attached relevant logs for you.  In the MySQL Router log you can see when it detects a lost connection to the Master and switches to the Slave.  You can also see that ProxySQL's monitoring is never interrupted by the master going down and why the connection to MySQL Router is never shunned.

2016-10-18 10:12:33 DEBUG   [7fa692712700] [routing:failover] [10.10.30.35]:36528 - [10.10.30.25]:3306
2016-10-18 10:12:33 DEBUG   [7fa692712700] [routing:failover] Routing stopped (up:93b;down:93b)
2016-10-18 10:12:35 DEBUG   [7fa692712700] Trying server vmmysql01:3306 (index 0)
2016-10-18 10:12:35 DEBUG   [7fa692712700] [routing:failover] [10.10.30.35]:36532 - [10.10.30.25]:3306
2016-10-18 10:12:35 DEBUG   [7fa692712700] [routing:failover] Routing stopped (up:93b;down:93b)
2016-10-18 10:12:38 DEBUG   [7fa691f11700] [routing:failover] Routing stopped (up:1831b;down:907b)
2016-10-18 10:12:38 DEBUG   [7fa691f11700] Trying server vmmysql01:3306 (index 0)
2016-10-18 10:12:39 DEBUG   [7fa691f11700] Timeout reached trying to connect to MySQL Server vmmysql01:3306
2016-10-18 10:12:39 DEBUG   [7fa691f11700] Trying server vmmysql02:3306 (index 1)
2016-10-18 10:12:39 DEBUG   [7fa691f11700] [routing:failover] [10.10.30.35]:36536 - [10.10.30.19]:3306
2016-10-18 10:12:39 DEBUG   [7fa691f11700] [routing:failover] Routing stopped (up:93b;down:93b)
2016-10-18 10:12:40 DEBUG   [7fa691f11700] Trying server vmmysql02:3306 (index 1)
2016-10-18 10:12:40 DEBUG   [7fa691f11700] [routing:failover] [10.10.30.35]:36542 - [10.10.30.19]:3306
2016-10-18 10:12:40 DEBUG   [7fa692712700] Trying server vmmysql02:3306 (index 1)
2016-10-18 10:12:40 DEBUG   [7fa692712700] [routing:failover] [10.10.30.35]:36546 - [10.10.30.19]:3306

Then later you can see where ProxySQL eventually drops it's connection and reconnects.

2016-10-18 10:13:25 DEBUG   [7fa692712700] Trying server vmmysql02:3306 (index 1)
2016-10-18 10:13:25 DEBUG   [7fa692712700] [routing:failover] [10.10.30.35]:36618 - [10.10.30.19]:3306
2016-10-18 10:13:25 DEBUG   [7fa692712700] [routing:failover] Routing stopped (up:93b;down:93b)
2016-10-18 10:13:28 DEBUG   [7fa692712700] Trying server vmmysql02:3306 (index 1)
2016-10-18 10:13:28 DEBUG   [7fa692712700] [routing:failover] [10.10.30.35]:36622 - [10.10.30.19]:3306
2016-10-18 10:13:28 DEBUG   [7fa692712700] [routing:failover] Routing stopped (up:93b;down:93b)
2016-10-18 10:13:28 DEBUG   [7fa691710700] sender read failed: (104 Connection reset by peer)
2016-10-18 10:13:28 DEBUG   [7fa691710700] [routing:failover] Routing stopped (up:3140b;down:17409b) Copy server-client failed: Connection reset by peer
2016-10-18 10:13:28 DEBUG   [7fa691710700] Trying server vmmysql02:3306 (index 1)
2016-10-18 10:13:28 DEBUG   [7fa691710700] [routing:failover] [10.10.30.35]:36626 - [10.10.30.19]:3306
2016-10-18 10:13:30 DEBUG   [7fa692712700] Trying server vmmysql02:3306 (index 1)
2016-10-18 10:13:30 DEBUG   [7fa692712700] [routing:failover] [10.10.30.35]:36630 - [10.10.30.19]:3306

In the attached ProxySQL log file, there is a single occurrence of a ping timeout, which only happened due to me forgetting to run my script which add few transactions a second.  Beyond that the only real error is :
2016-10-17 15:16:18 MySQL_Session.cpp:1185:handler(): [ERROR] Detected a broken connection during query on (0,vmmysql03,3333) , FD (Conn:22 , MyDS:22) : 2013, Lost connection to MySQL server during query
2016-10-17 15:16:18 MySQL_Session.cpp:1194:handler(): [WARNING] Retrying query.

Which shows ProxySQL doing exactly what I want.  After that point everything everything continues as usual.  The only concern is the ~52 seconds it takes for ProxySQL to acknowledge that it needs to retry the query.

The client does receive the 1317 and 2006 errors if ProxySQL isn't there, but ProxySQL doesn't seem to react to that.  Perhaps this is more a result of my using ProxySQL a bit outside of it's intended purpose?

Thanks again for your insights on this.  It is very much appreciated.

Aaron
Hi Aaron,

Answers inline.

To unsubscribe from this group and stop receiving emails from it, send an email to proxysql+u...@googlegroups.com.
mysqlrouter.log
proxysql.log

René Cannaò

unread,
Oct 18, 2016, 12:21:25 PM10/18/16
to aaron.i...@gmail.com, proxysql
Aaron,

Was your application sending traffic, or you were just trying to figure out when the client would detect the issue?

I see these errors matching (in time) in MySQL Router and ProxySQL:


2016-10-18 10:13:28 DEBUG   [7fa691710700] sender read failed: (104 Connection reset by peer)
2016-10-18 10:13:28 DEBUG   [7fa691710700] [routing:failover] Routing stopped (up:3140b;down:17409b) Copy server-client failed: Connection reset by peer

2016-10-18 10:13:28 MySQL_Session.cpp:1185:handler(): [ERROR] Detected a broken connection during query on (0,vmmysql03,3333) , FD (Conn:22 , MyDS:22) : 2013, Lost connection to MySQL server during query
2016-10-18 10:13:28 MySQL_Session.cpp:1194:handler(): [WARNING] Retrying query.



It seems that only at 10:13:28 MySQL Router terminated the connection and ProxySQL immediately detected it.
If other queries were being executed, ProxySQL should have detected the failure immediately.

Can you have your application sends for example a write per second and see in how long the issue is detected?

Thanks,
René

To unsubscribe from this group and stop receiving emails from it, send an email to proxysql+unsubscribe@googlegroups.com.

aaron.i...@gmail.com

unread,
Oct 18, 2016, 2:18:27 PM10/18/16
to proxysql, aaron.i...@gmail.com
Hi René,

Those matching errors are correctly timed, but as I noted previously, MySQL Router detects the Master going down earlier to the that ( 49 seconds in this case ).  You can see that it switches from the Master (vmmysql01) to the Slave (vmmysql02) at 2016-10-18 10:12:39.

2016-10-18 10:12:38 DEBUG   [7fa691f11700] Trying server vmmysql01:3306 (index 0)
2016-10-18 10:12:39 DEBUG   [7fa691f11700] Timeout reached trying to connect to MySQL Server vmmysql01:3306
2016-10-18 10:12:39 DEBUG   [7fa691f11700] Trying server vmmysql02:3306 (index 1)

Yes, I have a script which is constantly sending traffic, but it's been doing it in a single threaded manner.  So if a query gets caught in this delay scenario, no new queries are sent.
I ran 3 instances of the script to see how things reacts to that.  I tested this twice and in both cases all instanced are delayed.  In the first test, one instance of the script was briefly delayed by 1.5 seconds, while the other 2 were each delayed by 103 seconds.  The second test was similar but the delays were shorter.  The script instance which was only briefly delayed, the delay was 1 second, while the other 2 were delayed by 52 seconds.  What is really interesting about this that ProxySQL logged all three connection timeouts right away on the first test, but only 2 on the second test.  I would wager to guess because on the script side the query took 1.005376 seconds and my mysql-connect_timeout_server value is 1000, so it probably came in just under that within ProxySQL.  That said, I have never encountered this "Connect timeout" error before either.  That would appear to be the result of having multiple connections.

ProxySQL Log from Test 1
2016-10-18 13:17:43 mysql_connection.cpp:530:handler(): [ERROR] Connect timeout on vmmysql03:3333 : 101189942469 - 101189942164 = 305
2016-10-18 13:17:44 mysql_connection.cpp:530:handler(): [ERROR] Connect timeout on vmmysql03:3333 : 101190442753 - 101190442469 = 284
2016-10-18 13:17:44 mysql_connection.cpp:530:handler(): [ERROR] Connect timeout on vmmysql03:3333 : 101190942756 - 101190942753 = 3
2016-10-18 13:33:00 MySQL_Session.cpp:974:handler(): [ERROR] Detected a broken connection during ping on (0,vmmysql03,3333) , FD (Conn:22 , MyDS:22) : 2013, Lost connection to MySQL server during query
2016-10-18 13:33:00 MySQL_Session.cpp:974:handler(): [ERROR] Detected a broken connection during ping on (0,vmmysql03,3333) , FD (Conn:25 , MyDS:25) : 2013, Lost connection to MySQL server during query
2016-10-18 13:34:28 MySQL_Session.cpp:974:handler(): [ERROR] Detected a broken connection during ping on (0,vmmysql03,3333) , FD (Conn:19 , MyDS:19) : 2013, Lost connection to MySQL server during query

ProxySQL Log from Test 2
2016-10-18 13:37:34 mysql_connection.cpp:530:handler(): [ERROR] Connect timeout on vmmysql03:3333 : 102381086164 - 102381086164 = 0
2016-10-18 13:37:35 mysql_connection.cpp:530:handler(): [ERROR] Connect timeout on vmmysql03:3333 : 102381586164 - 102381586164 = 0
2016-10-18 13:38:25 MySQL_Session.cpp:1185:handler(): [ERROR] Detected a broken connection during query on (0,vmmysql03,3333) , FD (Conn:22 , MyDS:22) : 2013, Lost connection to MySQL server during query
2016-10-18 13:38:25 MySQL_Session.cpp:1194:handler(): [WARNING] Retrying query.
2016-10-18 13:38:26 MySQL_Session.cpp:1185:handler(): [ERROR] Detected a broken connection during query on (0,vmmysql03,3333) , FD (Conn:25 , MyDS:25) : 2013, Lost connection to MySQL server during query
2016-10-18 13:38:26 MySQL_Session.cpp:1194:handler(): [WARNING] Retrying query.

From the logs, it's interesting to point out the with the first test, I didn't get the "[ERROR] Detected a broken connection during ping" errors until I was setting up do the second test several minutes later.  Whereas in the second test they appear 50 seconds later, which coincides with the test scripts reporting their query time of 52 seconds.

I am going to test this a couple more times to see how consistent the results are.

Thanks,

Aaron

René Cannaò

unread,
Oct 18, 2016, 3:01:10 PM10/18/16
to proxysql
Aaron,

ProxySQL should immediately detect if a connection goes down, so probably this doesn't happen.
From a new analysis of the log, I now suspect that my assumptions on MySQL Router are incorrect.
Therefore, to better understand what is happening here, I need to setup ProxySQL and MySQL Router myself.

Meanwhile, would you consider not using MySQL Router and implement the failover using a script for ProxySQL's Scheduler?

Thanks,
René


To unsubscribe from this group and stop receiving emails from it, send an email to proxysql+unsubscribe@googlegroups.com.

aaron.i...@gmail.com

unread,
Oct 18, 2016, 3:26:49 PM10/18/16
to proxysql
Hi René,

Cool, I am curious to find out what you learn from this.  I like MySQL Router for how quick and easy it is to get up and running, but it is not a complete solution.  It's components are plugin based, so it's possible to write a plugin for it too.  I was considering this, but I fear it will be far more involved than I have time for.  Scripting something for ProxySQL's scheduler will likely be far easier/quicker, so I will look into that more now.

Thanks,

Aaron

aaron.i...@gmail.com

unread,
Oct 20, 2016, 5:25:19 PM10/20/16
to proxysql
Hi René,

I've decided against using the scheduler and implementing a daemon instead.

My script is working well, but my queries through the ProxySQL admin interface generate logging by ProxySQL.  It seems that it's logging the result of my querying?  I'm guessing this is a debugging feature that I can disable, but I'm not sure how.  Any insights would be great.  I don't mind the information, but it makes for a very needlessly noisy/bloated log file.  My script is written in Perl and I'm using DBI with DBD::MySQL to connect.

Thanks,

Aaron

René Cannaò

unread,
Oct 20, 2016, 5:36:33 PM10/20/16
to aaron.i...@gmail.com, proxysql
Hi Aaron,

Indeed Admin logs some of the commands in the error log, mainly SET, LOAD, and SAVE commands.
It is indeed for debugging/support purpose: with a chronological order of when SET/LOAD/SAVE were executed, it is possible to debug some behavior.

There is no option to disable such logging. Would you mind to create a feature request on https://github.com/sysown/proxysql/issues ?
Thank you

To unsubscribe from this group and stop receiving emails from it, send an email to proxysql+unsubscribe@googlegroups.com.

aaron.i...@gmail.com

unread,
Oct 20, 2016, 5:53:09 PM10/20/16
to proxysql, aaron.i...@gmail.com
Hi René,

I do not think those are what I'm seeing and I am familiar with those.

This is what it looks like :
HID: 0 , address: vmmysql02 , port: 3306 , weight: 0 , status: ONLINE , max_connections: 500 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 0 , address: vmmysql01 , port: 3306 , weight: 1 , status: ONLINE , max_connections: 500 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 0 , address: vmmysql02 , port: 3306 , weight: 0 , status: ONLINE , max_connections: 500 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 0 , address: vmmysql01 , port: 3306 , weight: 1 , status: ONLINE , max_connections: 500 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:

It repeats like that every time I run my SELECT query to check the status of the servers, which is a few times a second.

Here's the query I'm running.
SELECT b.hostname, b.status, b.weight
FROM stats.stats_mysql_connection_pool a,
     main.runtime_mysql_servers b
WHERE b.hostgroup_id = 0
  AND b.hostgroup_id = a.hostgroup
  AND b.hostname = a.srv_host
  AND b.port = a.srv_port
ORDER BY b.weight ASC, a.Latency_ms ASC;

Running that in the admin console generates the output I'm talking about too.

Thanks,

Aaron

René Cannaò

unread,
Oct 20, 2016, 5:57:58 PM10/20/16
to aaron.i...@gmail.com, proxysql

Querying runtime_mysql_servers causes Admin to dump the info also in error log, true that!

Please feel free to submit a feature request.
Thanks


To unsubscribe from this group and stop receiving emails from it, send an email to proxysql+unsubscribe@googlegroups.com.

aaron.i...@gmail.com

unread,
Oct 21, 2016, 1:28:34 PM10/21/16
to proxysql, aaron.i...@gmail.com
Hi René,

I'm seeing a curious behaviour, I'm hoping you can explain for me.

As you know from my previous posts, I'm querying the runtime_mysql_servers table to get the status of my servers.  The status for the server which goes offline, becomes SHUNNED, as expected, but that only seems to last until the server which is ONLINE receives traffic, then it becomes ONLINE again.  I didn't notice this originally because I was testing the failover when the master goes down and everything on my end happens as it should.  Then I was testing what the effect of the slave going down would be and it appears as though ProxySQL thinks it is mostly ONLINE according to the status column of runtime_mysql_servers.  The error log regularly reports, "MySQL_Monitor.cpp:1102:monitor_ping(): [ERROR] Server vmmysql01:3306 missed 3 heartbeats, shunning it and killing all the connections" and the actual behaviour of ProxySQL is correct.

Is there a better table I should query to determine the status of a server or should I be creating an issue?

Thanks,

Aaron

aaron.i...@gmail.com

unread,
Oct 21, 2016, 2:02:31 PM10/21/16
to proxysql, aaron.i...@gmail.com
Hi René,

To test this more I started adjusting the value of mysql-monitor_ping_interval.  I had it at 2500 and the offline server would show up SHUNNED at 2.5 second intervals.  Changing it to 100ms, the same rate I'm currently checking the status, at it is only caught coming ONLINE about 4 times a second. 

So the monitor ping is definitely doing it's job properly.

I will file this as a bug for you confirm/debunk.

Thanks,

Aaron

René Cannaò

unread,
Oct 21, 2016, 2:05:45 PM10/21/16
to aaron.i...@gmail.com, proxysql

Hi Aaron,

mysql_servers is a configuration table, and doesn't change status automatically when a server goes offline.
runtime_mysql_servers shows what the Hostgroup Manager knows about the servers, and can be slightly more accurate than the config table (it shows SHUNNED).
Although, unless you set the server explicitly offline in mysql_servers, at regular intervals (all configurable) the Hostgroup Manager tries to bring the server back online because ... it is still configured as ONLINE.
This algorithm allows to automatically recover situations in which a server is only temporarily unavailable (crash, reboot, restart, network outage, etc).
Another useful table to understand the status of the servers as seen from the connection pool is stats.stats_mysql_connection_pool .

For monitoring purposes, the tables in monitor schema are very useful. This query will list the tables available from Monitor nodules:

SHOW TABLES FROM monitor;

Thanks,
René


To unsubscribe from this group and stop receiving emails from it, send an email to proxysql+unsubscribe@googlegroups.com.

aaron.i...@gmail.com

unread,
Oct 21, 2016, 4:37:33 PM10/21/16
to proxysql, aaron.i...@gmail.com
Hi René,

Thanks for the info.

This confirms that I'm querying stats correctly and that the behaviour I've been observing is not a bug, but the system operating as expected.  Fantastic!

This is great.  I will modify my script to be aware of the mysql-monitor_ping_interval and have a look into the other monitoring tables.

Thanks again!

Aaron

aaron.i...@gmail.com

unread,
Oct 25, 2016, 5:50:48 PM10/25/16
to proxysql, aaron.i...@gmail.com
Hi René,

Sorry to dredge this up on you again, but I am hoping you can shed some light on an issue I'm having.  I imagine it can be fixed via configuration, I just haven't found the right configuration yet.  Perhaps I messed it up too much along the way?

Up until yesterday I have been testing with a script I wrote which roughly simulates our production traffic input of a few transactions a second.  Yesterday I started trying to test with sysbench.  On an uninterrupted run sysbench is fine and the numbers look good, but when I kill the master (VM) sysbench exits due to lost connection.  My script performs the switch over from the master having weight 1 to weight 0 and the slave having weight 0 to weight 1 in under 20 milliseconds, which should be fine 99.9999% of the time.  Changing the delay in my data insert script to be 20 milliseconds between inserts proves this to work, where as if I remove the delay completely, it experiences the same connection loss as sysbench.

I found that if I upped the mysql-shun_on_failures to 50, I was able to have my data insert script not fail with a delay of 5 milliseconds.  Upping this further to 5000 still does not allow me to run it without a delay between inserts.

Any thoughts on which settings I should focus on tuning in order to be able to test this via sysbench without a query delay?

Thanks,

Aaron


For reference here is my current config, because it's entirely possible that I messed something up along the way.

admin (monitor)>select * from global_variables order by variable_name;
+----------------------------------------+-----------------------------------------+
| variable_name                          | variable_value                          |
+----------------------------------------+-----------------------------------------+
| admin-admin_credentials                | admin:admin                             |
| admin-hash_passwords                   | true                                    |
| admin-mysql_ifaces                     | 127.0.0.1:6032;/tmp/proxysql_admin.sock |
| admin-read_only                        | false                                   |
| admin-refresh_interval                 | 2000                                    |
| admin-stats_credentials                | stats:stats                             |
| admin-telnet_admin_ifaces              | (null)                                  |
| admin-telnet_stats_ifaces              | (null)                                  |
| admin-version                          | 1.2.4                                   |
| mysql-client_found_rows                | true                                    |
| mysql-commands_stats                   | true                                    |
| mysql-connect_retries_delay            | 10                                      |
| mysql-connect_retries_on_failure       | 10                                      |
| mysql-connect_timeout_server           | 1000                                    |
| mysql-connect_timeout_server_max       | 3000                                    |
| mysql-connection_max_age_ms            | 0                                       |
| mysql-default_charset                  | utf8                                    |
| mysql-default_max_latency_ms           | 2000                                    |
| mysql-default_query_delay              | 0                                       |
| mysql-default_query_timeout            | 36000000                                |
| mysql-default_reconnect                | true                                    |
| mysql-default_schema                   | information_schema                      |
| mysql-enforce_autocommit_on_reads      | false                                   |
| mysql-eventslog_filename               |                                         |
| mysql-eventslog_filesize               | 104857600                               |
| mysql-free_connections_pct             | 10                                      |
| mysql-have_compress                    | true                                    |
| mysql-init_connect                     | (null)                                  |
| mysql-interfaces                       | 0.0.0.0:3306;/tmp/proxysql.sock         |
| mysql-long_query_time                  | 1000                                    |
| mysql-max_allowed_packet               | 1073741824                              |
| mysql-max_connections                  | 500                                     |
| mysql-max_transaction_time             | 14400000                                |
| mysql-monitor_connect_interval         | 2500                                    |
| mysql-monitor_connect_timeout          | 200                                     |
| mysql-monitor_enabled                  | true                                    |
| mysql-monitor_history                  | 600000                                  |
| mysql-monitor_password                 | Pr0xySQL!                               |
| mysql-monitor_ping_interval            | 500                                     |
| mysql-monitor_ping_max_failures        | 10                                      |
| mysql-monitor_ping_timeout             | 250                                     |
| mysql-monitor_query_interval           | 60000                                   |
| mysql-monitor_query_timeout            | 100                                     |
| mysql-monitor_read_only_interval       | 1500                                    |
| mysql-monitor_read_only_timeout        | 500                                     |
| mysql-monitor_replication_lag_interval | 10000                                   |
| mysql-monitor_replication_lag_timeout  | 1000                                    |
| mysql-monitor_slave_lag_when_null      | 60                                      |
| mysql-monitor_username                 | proxysql                                |
| mysql-monitor_writer_is_also_reader    | true                                    |
| mysql-multiplexing                     | true                                    |
| mysql-ping_interval_server_msec        | 10000                                   |
| mysql-ping_timeout_server              | 200                                     |
| mysql-poll_timeout                     | 1000                                    |
| mysql-poll_timeout_on_failure          | 100                                     |
| mysql-query_cache_size_MB              | 256                                     |
| mysql-query_digests                    | true                                    |
| mysql-query_processor_iterations       | 0                                       |
| mysql-query_retries_on_failure         | 50                                      |
| mysql-server_capabilities              | 45578                                   |
| mysql-server_version                   | 5.5.52                                  |
| mysql-servers_stats                    | true                                    |
| mysql-session_debug                    | (null)                                  |
| mysql-sessions_sort                    | true                                    |
| mysql-shun_on_failures                 | 500                                     |
| mysql-shun_recovery_time_sec           | 2                                       |
| mysql-ssl_p2s_ca                       | (null)                                  |
| mysql-ssl_p2s_cert                     | (null)                                  |
| mysql-ssl_p2s_cipher                   | (null)                                  |
| mysql-ssl_p2s_key                      | (null)                                  |
| mysql-stacksize                        | 4194304                                 |
| mysql-threads                          | 4                                       |
| mysql-threshold_query_length           | 524288                                  |
| mysql-threshold_resultset_size         | 4194304                                 |
| mysql-wait_timeout                     | 28800000                                |
+----------------------------------------+-----------------------------------------+
Reply all
Reply to author
Forward
0 new messages