Issue with Grafana using ProxySQL as it's database

[Jon Schneider]

I have a MySQL cluster setup (XtraDB ver 5.7.17) and provide access to it with ProxySQL (ver 1.4.3).  I am trying to configure Grafana to talk to ProxySQL for its database and just refuses to work.  I have verified I can connect to the ProxySQL with the exact same creds Grafana is configured with and I can select data, change schema, etc on the Grafana database, so I know the permissions are good and working.  If I simply adjust the IP address Grafana is connecting to from the ProxySQL IP to the IP of one of the cluster nodes it works fine.  XtraDB and ProxySQL are both listening on port 3306.

When Grafana is started it runs the following queries against MySQL (I captured this on the MySQL side):

SELECT @@max_allowed_packet;

SELECT TABLE_NAME from INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA=‘grafana_db’ and TABLE_NAME=‘migration_log’;

SELECT id, migration_id, sql, success, error, timestamp FROM migration_log;

SELECT ( SELECT COUNT() FROM user ) AS users, ( SELECT COUNT() FROM org ) AS orgs, ( SELECT COUNT() FROM dashboard ) AS dashboards, ( SELECT COUNT() FROM data_source ) AS datasources, ( SELECT COUNT() FROM playlist ) AS playlists, ( SELECT COUNT() FROM alert ) AS alerts, ( SELECT COUNT(*) FROM user where last_seen_at > ‘2017-11-06 15:19:25.556789’ ) as active_users;

I can run these queries from the MySQL client against ProxySQL using Grafana's creds and it works just fine.  However when Grafana tries to run them the first query does not return any data as can be seen in the following TCPDumps:

Run from the MySQL Client: (SELECT @@max_allowed_packet;)

10:19:01.479534 IP 10.166.9.57.42708 > 10.166.4.52.3306: Flags [P.], seq 157:189, ack 254, win 229, options [nop,nop,TS val 499757159 ecr 498421067], length 32 0x0000: 4508 0054 0b73 4000 4006 0c71 0aa6 0939 E..T.s@.@..q...9 0x0010: 0aa6 0434 a6d4 0cea f0ba 21c3 e12c 8247 ...4......!..,.G 0x0020: 8018 00e5 e92f 0000 0101 080a 1dc9 b067 ...../.........g 0x0030: 1db5 4d4b 1c00 0000 0353 454c 4543 5420 ..MK.....SELECT. 0x0040: 4040 6d61 785f 616c 6c6f 7765 645f 7061 @@max_allowed_pa 0x0050: 636b 6574 cket

10:19:01.480323 IP 10.166.4.52.3306 > 10.166.9.57.42708: Flags [P.], seq 254:337, ack 189, win 227, options [nop,nop,TS val 498517692 ecr 499757159], length 83 0x0000: 4500 0087 c1f0 4000 4006 55c8 0aa6 0434 E.....@.@.U....4 0x0010: 0aa6 0939 0cea a6d4 e12c 8247 f0ba 21e3 ...9.....,.G..!. 0x0020: 8018 00e3 2332 0000 0101 080a 1db6 c6bc ....#2.......... 0x0030: 1dc9 b067 0100 0001 012a 0000 0203 6465 ...g.....*....de 0x0040: 6600 0000 1440 406d 6178 5f61 6c6c 6f77 f....@@max_allow 0x0050: 6564 5f70 6163 6b65 7400 0c3f 0015 0000 ed_packet..?.... 0x0060: 0008 a080 0000 0005 0000 03fe 0000 0200 ................ 0x0070: 0a00 0004 0931 3334 3231 3737 3238 0500 .....134217728.. 0x0080: 0005 fe00 0002 00 ....... 10:19:01.480978 IP 10.166.9.57.42708 > 10.166.4.52.3306: Flags [.], ack 337, win 229, options [nop,nop,TS val 499757161 ecr 498517692], length 0 0x0000: 4508 0034 0b74 4000 4006 0c90 0aa6 0939 E..4.t@.@......9 0x0010: 0aa6 0434 a6d4 0cea f0ba 21e3 e12c 829a ...4......!..,.. 0x0020: 8010 00e5 7656 0000 0101 080a 1dc9 b069 ....vV.........i 0x0030: 1db6 c6bc

Starting Grafana:

10:19:29.956701 IP 10.166.9.57.37700 > 10.166.4.52.6033: Flags [P.], seq 104:136, ack 90, win 229, options [nop,nop,TS val 499785637 ecr 498546168], length 32 0x0000: 4500 0054 c7c2 4000 4006 5029 0aa6 0939 E..T..@.@.P)...9 0x0010: 0aa6 0434 9344 1791 fdfa d885 aa97 4344 ...4.D........CD 0x0020: 8018 00e5 4bc1 0000 0101 080a 1dca 1fa5 ....K........... 0x0030: 1db7 35f8 1c00 0000 0353 454c 4543 5420 ..5......SELECT. 0x0040: 4040 6d61 785f 616c 6c6f 7765 645f 7061 @@max_allowed_pa 0x0050: 636b 6574 cket 10:19:29.996216 IP 10.166.4.52.6033 > 10.166.9.57.37700: Flags [.], ack 136, win 227, options [nop,nop,TS val 498546208 ecr 499785637], length 0 0x0000: 4500 0034 771f 4000 4006 a0ec 0aa6 0434 E..4w.@.@......4 0x0010: 0aa6 0939 1791 9344 aa97 4344 fdfa d8a5 ...9...D..CD.... 0x0020: 8010 00e3 22df 0000 0101 080a 1db7 3620 ....".........6. 0x0030: 1dca 1fa5

The query appears to be the same, but when Grafana asks it no answer is given back, this causes Grafana to hang trying to start.

Any idea's or suggestions are appreciated, thanks.

[René Cannaò]

Jon,

If we are lucky, this is probably related to https://github.com/sysown/proxysql/issues/1219 .

What version of Grafana are you using?

--
You received this message because you are subscribed to the Google Groups "proxysql" group.
To unsubscribe from this group and stop receiving emails from it, send an email to proxysql+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jon Schneider]

Hmm, that does look very related, I am using the 4.5.2 rpm on CentOS 7.4.  Guess I'll wait for the 1.4.4 release and see if that clears up this issue.

Thanks,

Jon

To unsubscribe from this group and stop receiving emails from it, send an email to proxysql+u...@googlegroups.com.