wrong topology discovery

Yuriy Borysov

unread,

Mar 17, 2021, 1:15:23 PM3/17/21

to orchestrator-mysql

Hello!

I set up a simple lab with 3 nodes,

- orchestrator node

- MariaDB 10.1 - master node

- MariaDB 10.1 - slave node

All nodes are Ubuntu 18.04.5 LTS, MariaDB installed from ubuntu repo, orchestrator install via .deb package from orchestrator Github.

Between master and slave mysql nodes configured basic replication with enabled GTID:

master:

[mysqld]

binlog_format = MIXED

log_bin = mysql-bin

log_slave_updates = 1

bind-address = 10.10.100.30

gtid-domain-id = 30

server_id = 30

report_host=10.10.100.30

slave:

[mysqld]

binlog_format = MIXED

log_bin = mysql-bin

bind-address = 10.10.100.20

gtid-domain-id = 20

server_id = 20

report_host=10.10.100.20

log_slave_updates = 1

After topology discovery, the orchestrator detects both nodes as master and two separate clusters:

https://www.dropbox.com/s/iyo8ehk9xlmn5jw/Screenshot%202021-03-17%20at%2019.07.54.png?dl=0

And with the "GTID supported false" option.

Any idea why the orchestrator can't find the right topology?

Thanks!

Shlomi Noach

unread,

Mar 17, 2021, 1:23:25 PM3/17/21

to Yuriy Borysov, orchestrator-mysql

what happens when you SHOW SLAVE STATUS on both hosts?

Are you using multi-source replication?

Can you please provide output of /api/instances from HTTP API ?

--
You received this message because you are subscribed to the Google Groups "orchestrator-mysql" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orchestrator-my...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/orchestrator-mysql/794b85cc-4dcd-4a7e-b17d-f23d94e0d0c0n%40googlegroups.com.

Yuriy Borysov

unread,

Mar 17, 2021, 2:06:55 PM3/17/21

to orchestrator-mysql

On Wednesday, March 17, 2021 at 7:23:25 PM UTC+2 shlomi...@gmail.com wrote:

what happens when you SHOW SLAVE STATUS on both hosts?

It Is master-slave replication, on master "show slave" is empty. On slave:

https://gist.github.com/yokodzun/cc6869c7918ab2bf56c37341d6ee7f9e

Are you using multi-source replication?

No, one master and one slave.

Can you please provide output of /api/instances from HTTP API ?

https://gist.github.com/yokodzun/e929a69ba32e97b819dc5c94317d36d8

Thanks!

Shlomi Noach

unread,

Mar 17, 2021, 2:26:18 PM3/17/21

to Yuriy Borysov, orchestrator-mysql

Everything seems to be in order, and yet `orchestrator` does not identify the MasterKey while probing the replica. I'm not sure why.

To view this discussion on the web visit https://groups.google.com/d/msgid/orchestrator-mysql/3dd8fbd7-09f0-449f-9096-ec6b745bfce6n%40googlegroups.com.

Yuriy Borysov

unread,

Mar 17, 2021, 3:00:38 PM3/17/21

to orchestrator-mysql

On Wednesday, March 17, 2021 at 8:26:18 PM UTC+2 shlomi...@gmail.com wrote:

Everything seems to be in order, and yet `orchestrator` does not identify the MasterKey while probing the replica. I'm not sure why.

I trуed to reinstall MariaDB from scratch (apt purge/rm -rf config/apt install), but with no luck.

Tomorrow I am planning to update MySQL to the version from the MariaDB repo.

Yuriy Borysov

unread,

Mar 18, 2021, 11:05:54 AM3/18/21

to orchestrator-mysql

On Wednesday, March 17, 2021 at 9:00:38 PM UTC+2 Yuriy Borysov wrote:

On Wednesday, March 17, 2021 at 8:26:18 PM UTC+2 shlomi...@gmail.com wrote:
Everything seems to be in order, and yet `orchestrator` does not identify the MasterKey while probing the replica. I'm not sure why.

I trуed to reinstall MariaDB from scratch (apt purge/rm -rf config/apt install), but with no luck.
Tomorrow I am planning to update MySQL to the version from the MariaDB repo.

After installing "Server version: 10.3.28-MariaDB" nothing changes.

Any suggestions about the cause?

Shlomi Noach

unread,

Mar 18, 2021, 12:51:06 PM3/18/21

to Yuriy Borysov, orchestrator-mysql

Anything at all in the logs to suggest why orchestrator would fail to read the replication info? (try orchestrator --debug --stack)

To view this discussion on the web visit https://groups.google.com/d/msgid/orchestrator-mysql/32c6dae7-ad70-4e62-b6de-b01e5d956525n%40googlegroups.com.

Yuriy Borysov

unread,

Mar 22, 2021, 6:07:25 AM3/22/21

to orchestrator-mysql

On Thursday, March 18, 2021 at 6:51:06 PM UTC+2 shlomi...@gmail.com wrote:

Anything at all in the logs to suggest why orchestrator would fail to read the replication info? (try orchestrator --debug --stack)

I try to manually switch MySQL to GTID mode and relocate node via CLI (with --debug --stack):

https://gist.github.com/yokodzun/b7362e9f5a667f282eec84f1d45bedd7

In log only INFO and DEBUG messages:

Mar 22 10:03:26 poc-mysql-client orchestrator[15025]: 2021-03-22 10:03:26 DEBUG kv.SubmitMastersToKvStores, clusterName: , force: false: numPairs: 10

Mar 22 10:03:26 poc-mysql-client orchestrator[15025]: 2021-03-22 10:03:26 DEBUG kv.SubmitMastersToKvStores: submitKvPairs: 0

Mar 22 10:03:26 poc-mysql-client orchestrator[15025]: 2021-03-22 10:03:26 INFO auditType:review-unseen-instances instance::0 cluster: message:Operations: 0

Mar 22 10:04:26 poc-mysql-client orchestrator[15025]: 2021-03-22 10:04:26 INFO auditType:resolve-unknown-masters instance::0 cluster: message:Num resolved hos

Mar 22 10:04:26 poc-mysql-client orchestrator[15025]: 2021-03-22 10:04:26 INFO auditType:forget-unseen instance::0 cluster: message:Forgotten instances: 0

Mar 22 10:04:26 poc-mysql-client orchestrator[15025]: 2021-03-22 10:04:26 INFO auditType:review-unseen-instances instance::0 cluster: message:Operations: 0

Mar 22 10:04:26 poc-mysql-client orchestrator[15025]: 2021-03-22 10:04:26 INFO auditType:forget-clustr-aliases instance::0 cluster: message:Forgotten aliases:

Mar 22 10:04:26 poc-mysql-client orchestrator[15025]: 2021-03-22 10:04:26 INFO auditType:inject-unseen-masters instance::0 cluster: message:Operations: 0

Mar 22 10:04:26 poc-mysql-client orchestrator[15025]: 2021-03-22 10:04:26 INFO auditType:forget-unseen-differently-resolved instance::0 cluster: message:Forgo

Mar 22 10:04:26 poc-mysql-client orchestrator[15025]: 2021-03-22 10:04:26 DEBUG kv.SubmitMastersToKvStores, clusterName: , force: false: numPairs: 10

Mar 22 10:04:26 poc-mysql-client orchestrator[15025]: 2021-03-22 10:04:26 DEBUG kv.SubmitMastersToKvStores: submitKvPairs: 0

Mar 22 10:05:26 poc-mysql-client orchestrator[15025]: 2021-03-22 10:05:26 INFO auditType:forget-clustr-aliases instance::0 cluster: message:Forgotten aliases:

Mar 22 10:05:26 poc-mysql-client orchestrator[15025]: 2021-03-22 10:05:26 INFO auditType:resolve-unknown-masters instance::0 cluster: message:Num resolved hos

Mar 22 10:05:26 poc-mysql-client orchestrator[15025]: 2021-03-22 10:05:26 INFO auditType:inject-unseen-masters instance::0 cluster: message:Operations: 0

Mar 22 10:05:26 poc-mysql-client orchestrator[15025]: 2021-03-22 10:05:26 INFO auditType:review-unseen-instances instance::0 cluster: message:Operations: 0

Mar 22 10:05:26 poc-mysql-client orchestrator[15025]: 2021-03-22 10:05:26 INFO auditType:forget-unseen-differently-resolved instance::0 cluster: message:Forgo

Mar 22 10:05:26 poc-mysql-client orchestrator[15025]: 2021-03-22 10:05:26 INFO auditType:forget-unseen instance::0 cluster: message:Forgotten instances: 0

Mar 22 10:05:26 poc-mysql-client orchestrator[15025]: 2021-03-22 10:05:26 DEBUG kv.SubmitMastersToKvStores, clusterName: , force: false: numPairs: 10

Mar 22 10:05:26 poc-mysql-client orchestrator[15025]: 2021-03-22 10:05:26 DEBUG kv.SubmitMastersToKvStores: submitKvPairs: 0

Yuriy Borysov

unread,

Mar 25, 2021, 9:49:04 AM3/25/21

to orchestrator-mysql

On Thursday, March 18, 2021 at 6:51:06 PM UTC+2 shlomi...@gmail.com wrote:

Anything at all in the logs to suggest why orchestrator would fail to read the replication info? (try orchestrator --debug --stack)

Could you tell, what exactly command read the replication info?

I enable "audit plugin" and run:

/usr/local/orchestrator/orchestrator -c enable-gtid -i 10.10.100.30 --debug --stack cli

After that, I copy all commands from the audit log and run them manually via MySQL client from the orchestrator host and with orchestrator credentionals.

All work fine.

For example, orchestrators request from log:

20210325 13:22:13,poc-mysql-master,orchestrator,10.10.100.10,50,3607,QUERY,,'select @@global.hostname, ifnull(@@global.report_host, \'\'), @@global.server_id, @@global.version, @@global.version_comment, @@global.read_only, @@global.binlog_format, @@global.log_bin, @@global.log_slave_updates',0

Manual request:

MariaDB [(none)]> select @@global.hostname, ifnull(@@global.report_host, ''), @@global.server_id, @@global.version, @@global.version_comment, @@global.read_only, @@global.binlog_format, @@global.log_bin, @@global.log_slave_updates\G;

*************************** 1. row ***************************

@@global.hostname: poc-mysql-master

ifnull(@@global.report_host, ''): 10.10.100.30

@@global.server_id: 30

@@global.version: 10.3.28-MariaDB-1:10.3.28+maria~bionic-log

@@global.version_comment: mariadb.org binary distribution

@@global.read_only: 0

@@global.binlog_format: MIXED

@@global.log_bin: 1

@@global.log_slave_updates: 1

1 row in set (0.00 sec)

ERROR: No query specified

How I can "emulate" getting replication info?

Thanks!

To view this discussion on the web visit https://groups.google.com/d/msgid/orchestrator-mysql/32c6dae7-ad70-4e62-b6de-b01e5d956525n%40googlegroups.com.After

Reply all

Reply to author

Forward