slow performance with maxscale vs direct connections

[Ryan Bellows]

Hi all,

I've been attempting to setup Maxscale in a very simple configuration, to test its function/performance as a mysql proxy. Just as a start, I setup a single router, readonly (readconnroute), single listener, single server. No load balancing, filtering, etc. The problem is when I turned a portion of our production traffic through the maxscale proxy, a simple login latency test of the application went from ~.20-30ms to 3-5 seconds. I'm running it on a dedicated R620 with plenty of cpu/memory, including a 10g network interface. When running traffic through it, it looks as if only one core is being used, which would explain the latency spike. I've set threads=8, but this made very little if any difference in performance. Server is running CentOS 6.2 (2.6.32-220.el6.x86_64), with the latest 1.0.1-1 rpm from the maxscale repo. I tried building from master (pulled yesterday morning), but this made no difference. Any help would be appreciated! Below is my MaxScale.cnf

-Ryan

[maxscale]

threads=8

[MySQL Monitor]

type=monitor

module=mysqlmon

servers=server1

user=maxscale

passwd=pass

[server1 read router]

type=service

router=readconnroute

router_options=slave

servers=server1

user=maxscale

passwd=pass

[Debug Interface]

type=service

router=debugcli

[CLI]

type=service

router=cli

[CLI Listener]

type=listener

service=CLI

protocol=maxscaled

address=localhost

port=6603

[server1 read listener]

type=listener

service=server1 read router

protocol=MySQLClient

port=4006

[Debug Listener]

type=listener

service=Debug Interface

protocol=telnetd

port=4442

[server1]

type=server

address=10.123.123.58

port=3308

protocol=MySQLBackend

[Vilho Raatikka]

Hi Ryan,

I tried to reply something at least distantly similar with your test; I ran mysql client first through readconn router and then directly to MariaDB 5.5.40. First repeatedly by using single thread and then by forking concurrent processes.

for ((i=0 ; i<10 ; i++)); do time mysql -h 127.0.0.1 -P <port> -u user -ppwd -e"select 1" ; done

and

for ((i=0 ; i<100 ; i++)) ; do { sleep 1 ; time mysql -h 127.0.0.1 -P <port> -u user -ppwd  -e'select 1'; } &  done

I used 4-core machine with 32GB RAM and all processes were running on the same host. Communication was through TCP.

With single thread direct connection was 2-5% faster (method I use dis so coarse that the result really tells that they are 'almost as fast'). With multiple processes the difference remained roughly the same.
--

Conclusion is that this test is not close enough with that of yours. Can you share the test you use or sufficient information so that we can try the same and find the reason for readconn router's poor performance in it?

Regards

Vilho

--
You received this message because you are subscribed to the Google Groups "MaxScale" group.
To unsubscribe from this group and stop receiving emails from it, send an email to maxscale+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ryan Bellows]

Hi Vilho,

Thanks for your reply. Our use case is a php front end application, running the read-only queries through the maxscale router. I'm able to take a portion of our production traffic and run it through, this is about 5-7k queries/s running on the mysql db.

Is there any debugging information or profiling in maxscale that would be useful? 

-Ryan

[Vilho Raatikka]

Hi Ryan,

unfortunately we don't have much such tools. You can monitor routing to some extent but if this is about connecting or authenticating it wouldn't help.

I'd be interested to see the test load of yours. Any information you provide could be helpful.

I tried to find the bottleneck by executing php script, which connects and authenticates within a loop, and measure the time it took running through the loop. Results are below.

Regards

Vilho

Serial execution by using single process:

time for ((i=0; i<1000 ; i++)) ; do php ./login_user.php ; done

readconn router
real    0m11.003s
user    0m4.642s
sys     0m5.570s

rwsplit
real    0m11.563s
user    0m4.872s
sys     0m5.562s

direct
real    0m12.100s
user    0m5.051s
sys     0m5.888s

and parallel execution in 1000 processes

time for ((i=0; i<1000 ; i++)) ; do { php ./login_user.php ;} & done 

readconn router
real    0m2.447s
user    0m7.405s
sys     0m7.471s

rwsplit
real    0m2.798s
user    0m7.317s
sys     0m7.199s

direct
real    0m2.371s
user    0m7.487s
sys     0m7.404s

---

the content of login_user.php:

<?php

$mysqli = new mysqli("127.0.0.1", <user>, <pwd>, <db>, <port>);
if ($mysqli->connect_errno) {
    printf("Connect failed: %s\n", $mysqli->connect_error);
    exit();
}
$mysqli->close();

exit(0);

?>

[Martin Brampton]

Hi Ryan

Did you ever get a resolution of this problem?  If so, what did it involve?

Best regards, Martin

--
Martin Brampton, Principal Software Engineer

MariaDB Corporation | t: +44 1751-432935 | Skype: blacksheepresearch 

[James Wang]

Interested in knowing as well.  Thanks

On Friday, 27 November 2015 11:56:59 UTC, Martin Brampton wrote:

Hi Ryan

Did you ever get a resolution of this problem?  If so, what did it involve?

Best regards, Martin

On Wednesday, November 5, 2014 at 10:14:28 PM UTC, Ryan Bellows wrote:

Hi all,

I've been attempting to setup Maxscale in a very simple configuration, to test its function/performance as a mysql proxy. Just as a start, I 

--