Spare connections not used

[Art van Scheppingen]

Hi,

We have a multi-layer ProxySQL setup, which could be described as following:
app -> glb -> proxysql-collect -> proxysql-locfen -> database shards

Both ProxySQL layers have connection pooling and connection multiplexing enabled and are running ProxySQL 2.1.0

We have issues where we see a sudden increase of connections happening between proxysql-collect and proxysql-locfen. The incoming connections or number of queries between the app and proxysql-collect doesn't increase. The cause could be contention in our database shards causing the throughput of multiplexing to become lower which spawns new connections. But the root cause is besides the discussion in this topic.

What we observe is that on proxysql-collect we see the number of outgoing connections increase:

We use the proxysql_connection_pool_conn_used to plot this graph.

At the same time we observe the increase happening for one specific user, which directs traffic to one specific hostgroup:

But when we look at the number of free connections for that hostgroup we don't see the expected decrease of free connections happening:

We use the metric proxysql_connection_pool_conn_free to plot this.

What we would expect is that first the number of free connections would drop to near zero before ProxySQL would start spawning new connections. Especially as the rate of the increase of new incoming connections happens within 1 minute timespan. Is this assumption correct? If so, what could cause that from not happening in our configuration?

It would be welcome if anyone could shed a light on this.

Thanks!

Best,

Art