Admin Interface is getting stuck.

[Slawek Korwel]

Good morning! 

My next issue I need to struggle with is weird. It is even hard to describe...

- I have my 3-instances proxysql cluster working with 3-nodes mysql cluster. Mysql instances are using simple replication (not even a group-replication), but that is not important here. 

- I have some rules, I have two hostgroups (read (20)/write (10)) and that works just fine. 

- I'm using default ports - 6032 and 6033.  

- All communication on 6033 (mysql backend) work as normal. 

So, to monitor my connections, to have some insight into the traffic I created a few "one-liner" scripts to fetch data from ProxySQL. Like:

"select sum(sum_time) as total_time from stats_mysql_query_digest where username = 'xxx' and hostgroup = '10'

And the 'scripts' like that - around 10 already - are triggered every 2 minutes (by the third party software - Zabbix in my case)

And now there is my problem. Every day, around the same time of a day, those scripts start to getting stuck with no result from Proxy Admin Interface. It looks like Admin stopped responding and handling those requests. When it happens I cannot - of course - even connect to Admin from the command line. Every new run is added to the stuck and waits. 

Here is the list of stacked queries checked during a few seconds. It is clearly see that this number grows and now it is around 8 thousands. But, we can see that it also goes down a bit sometimes (8269 -> 8268 -> 8264 -> 8257 -> after few seconds it went up to 8323)

 

Again, during that time everything is fine with the proxy -> mysql connection. All requests are handled on the fly. 

The way for me to clear that stack is to restart entire proxysql. And then I have another day when all my scripts are handled in a second.

Anyone saw something like this before? Any clue how to troubleshoot it? 

Thanks a lot!

Sławek

[René Cannaò]

> Every day, around the same time of a day, those scripts start to getting stuck with no result from Proxy Admin Interface.

Probably you should start the investigation from here.

What happens at that time?

Maybe some other client is connecting to Admin at that time and holding a global mutex.

Maybe something else.

Yet, "around the same time of the day" is the key to start an investigation 

--
You received this message because you are subscribed to the Google Groups "proxysql" group.
To unsubscribe from this group and stop receiving emails from it, send an email to proxysql+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/proxysql/ea8171dd-9798-49ee-ac29-c92e13eb6e17n%40googlegroups.com.