After failover failed master shown in separate cluster

[srv.d...@gmail.com]

Hello All,

Trying to use orchestrator to handle failover.

I have three hosts:

db0[123]-xx-yy (where xx-yy is the cluster name).

As a part of failover action on a successor host I do mysql -e "stop slave; reset slave; reset slave all"

After it, on a dashboard, I can see 2 clusters: 

- the original one xx-yy with 2 hosts db02,db03 (I can still navigate to cluster page and see all hosts)

- the new one: db01-xx-yy. When on a dashboard web page I click on a this cluster name I got the error. Because there is no entry in cluster_alias table in orchestrator database

>> {"Code":"ERROR","Message":"No cluster found for alias dbtests03-dc-3s:3306","Details":null}

Have a question why after failover and in case if slave options will be reset on a new master the old one will appear in a new cluster instead of being on the same?

thanks

[Shlomi Noach]

> When on a dashboard web page I click on a this cluster name I got the error. Because there is no entry in cluster_alias table in orchestrator database

This should be fixed, regardless of the below. Can you open an issue please?

> Have a question why after failover and in case if slave options will be reset on a new master the old one will appear in a new cluster instead of being on the same?

This behavior is expected. orchestrator's way of grouping servers into clusters is by figuring out that they belong to the same cluster. I agree that orchestrator has (or can have) memory that "this server used to be part of cluster C". But the failed master is now disconnected from your replication graph. It's essentially not part of the same cluster anymore (only logically so, by your infrastructure's discovery service or by your own inventory).

Even if you could tell orchestrator that "this server really belongs with this other cluster", what would be your expectation? should orchestrator display the server in web UI in same page as the connected cluster's graph?

--
You received this message because you are subscribed to the Google Groups "orchestrator-mysql" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orchestrator-mysql+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/orchestrator-mysql/178a3e1e-a5c0-4fa4-9a2f-6ba7cb8936c2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[srv.d...@gmail.com]

Thanks for your reply.

> This should be fixed, regardless of the below. Can you open an issue please?

https://github.com/github/orchestrator/issues/424

> I agree that orchestrator has (or can have) memory that "this server used to be part of cluster C"
It is because of options: "DetectClusterAlias". I believe that all hosts regardless were they attached to replication topology or not should appear on the same cluster page just because they should be detected as a single group.
Moreover it would be nice to have API which can be used to show that current detached host is in the middle of the "attaching" process to certain host. In this case I will be able to see the actual state:
- dead/failed (recent failover or mysql is down);
- new (no recent failovers);
- is attaching as a slave to current topology (API should be used ) to certain host (nice to be shown also).

The main idea to give more visibility to the rest of a team.

> what would be your expectation? should orchestrator display the server in web UI in same page as the connected cluster's graph?

It would be good.

Is it feasible from your perspective?

In case if it is worth to implement is it possible to have a hook and API (plus drag-n-drop on UI) to attach new/failed hosts to replication topology (or may be you alredy have and missed something in documentation)?

thank you

суббота, 3 марта 2018 г., 18:43:59 UTC+1 пользователь Shlomi Noach написал:

[Dirk Weise]

Am Samstag, 3. März 2018 18:43:59 UTC+1 schrieb Shlomi Noach:

> When on a dashboard web page I click on a this cluster name I got the error. Because there is no entry in cluster_alias table in orchestrator database

This should be fixed, regardless of the below. Can you open an issue please?

This issue can be worked around by not using the alias link, see my comment on the issue.

 

> Have a question why after failover and in case if slave options will be reset on a new master the old one will appear in a new cluster instead of being on the same?

This behavior is expected. orchestrator's way of grouping servers into clusters is by figuring out that they belong to the same cluster. I agree that orchestrator has (or can have) memory that "this server used to be part of cluster C". But the failed master is now disconnected from your replication graph. It's essentially not part of the same cluster anymore (only logically so, by your infrastructure's discovery service or by your own inventory).

Even if you could tell orchestrator that "this server really belongs with this other cluster", what would be your expectation? should orchestrator display the server in web UI in same page as the connected cluster's graph?

Unfortunetely, I haven't found a way to add the old master as replica to the new one with orchestrator. I had to manually issue `change master to ...`. Is there a way to do it with orchestrator GUI?

Am I correct that I have to take care myself that the old master cannot interfere with my application when it becomes available after failover or is there a way to handle this with orchestrator (e.g. setting old masters as read only as soon as it's recognized)?

 

Cheers,

Dirk