App -> ProxySQL -> ProxySQL -> MySQL

[Daniel Henninger]

Hi folk,

We are using a mix of ProxySQL and Signal 18's Replication Manager with our MySQL servers to provide some load balancing and failover support.  We have a network appliance based load balancer in front of that (provided by our networking group), but that's not particularly important right now as I took it out of the equation.

I have started down a path of requiring all MySQL traffic to be over SSL, and have a few apps that simply do not support it.  As such, I have set up a second "close to the app" ProxySQL to handle what amounts to unencrypted MySQL -> SSL MySQL (and optional query caching specific to the app, but we haven't gotten that far yet).

At first, this is working wonderfully.  The unencrypted traffic stays local to the app/host and everything that leaves the host is encrypted.  After a little while, I begin seeing messages in ProxySQL's logs like:

2025-08-18 21:32:34 mysql_connection.cpp:1322:handler(): [ERROR] Connect timeout on TARGETHOST:TARGETPORT : exceeded by 1182us

With a fairly large variety of times (in the current logs, anywhere from 32us to 51324us).  A few hours after that, I begin seeing site outages, very short blips.  I also noticed two of these:

2025-08-18 23:04:03 mysql_data_stream.cpp:455:check_data_flow(): [ERROR] Session=0x7fe19e8d4d80, DataStream=0x7fe1ae032900 -- Data at both ends of a MySQL data stream: IN <1 bytes 0 packets> , OUT <0 bytes 16384 packets>

I have a feeling that I have misconfigured something but I am strugging to track down what it is.  I will fill in a few details but I'm not sure what all would be useful to share:

App - decently sized PHP CodeIgniter based app, some imo poorly written statements especially revolving around an absolutely ton of temp tables.

App-ProxySQL - currently pointing directly to ONE of the two DB side proxysql servers to restrict where I have to look at logs, and eliminate the load balancer as a problem.  It is set to allow max of 100 connections to DB-ProxySQL.  I am running 3.0.2 here.  I have fast_forward enabled here due to some previous weirdness that occurred during a past upgrade.

DB-ProxySQL - Only set up to handle port based routing -- come on on port 3307, you go to the associated pool of two MySQL servers.  I am running 2.7.3 here simply because I haven't gotten around to upgrading them yet.  Many other applications connect that do NOT have the extra App-ProxySQL layer.  (however, I see occasional blips here too -- it's currently extremely difficult to tell where the problem is as I have been told the network firewall is being overzealous on some traffic and that encrypting everything can get around that issue) Max connections is stupidly high here atm -- I haven't tuned it -- perhaps I should.  However, I'm not seeing high numbers of active connections.  In fact it tends to stay what I would describe as "bored".  Similar to App-ProxySQL, I have enabled fast_forward on the user for this application.

DB (MySQL) - Servers are set up source->replica only, and replication manager handles switching them.  IE only one is read-write at a given time.  Running MySQL 8.0.35 currently.

I recognize that I'm probably not providing enough information quite yet -- but I also did not want to make this exceedingly long until I know what information would be useful.

However. a couple of starter questions:

1. Any known reason why 3.0.2 -> 2.7.3 would not play well together?

2. Are there any particular best practice configuration settings I should be using with this ProxySQL -> ProxySQL setup?

Thanks!

Daniel