Switchover / failover coordination / hooks

[John Smiley]

What are the best practices related to coordinating a switchover or failover with entities outside of the databases?  For example, let's take a basic switchover:

Pre-outage actions

Verifying the topology of the cluster being changed

Choosing a replica to become the new master (may depend on DC location, state of the instance, how much redo lag exists, etc.)

Creating blackouts in monitoring and notification systems

Sending notifications to interested parties

Outage actions

Notifying upstream elements to stop sending traffic to the master and waiting for their ACKs

Ensuring that the master being switched away from cannot accept any new  read/write traffic

Allowing the in-flight transactions to complete and/or kill them

Ensure that all logs needed by the replica that will become the new master have been received (and possibly applied)

Starting the shutdown of the master being switched away from

Enabling writes on the new master

Notifying upstream elements of the endpoint of the new master and waiting for their ACKs

Notifying upstream elements to start sending traffic to the master and waiting for their ACKs

Post-outage actions

Ensure that the old master shuts down cleanly.  Restart it and bring it down cleanly if it does not.

Reinstate the old master as a replica as appropriate

Should this be driven by Orchestrator through hooks to external scripts/binaries or is the expectation that the overall orchestration will be done outside of Orchestrator?

Regards,

John Smiley

Head of Databases

Proton Technologies AG

[John Smiley]

Is it these?  I found them in the config, but no docs or examples that I saw.

  "OnFailureDetectionProcesses": [
    "echo 'Detected {failureType} on {failureCluster}. Affected replicas: {countSlaves}' >> /tmp/recovery.log"
  ],
  "PreGracefulTakeoverProcesses": [
    "echo 'Planned takeover about to take place on {failureCluster}. Master will switch to read_only' >> /tmp/recovery.log"
  ],
  "PreFailoverProcesses": [
    "echo 'Will recover from {failureType} on {failureCluster}' >> /tmp/recovery.log"
  ],
  "PostFailoverProcesses": [
    "echo '(for all types) Recovered from {failureType} on {failureCluster}. Failed: {failedHost}:{failedPort}; Successor: {successorHost}:{successorPort}' >> /tmp/recovery.log"
  ],
  "PostUnsuccessfulFailoverProcesses": [],
  "PostMasterFailoverProcesses": [
    "echo 'Recovered from {failureType} on {failureCluster}. Failed: {failedHost}:{failedPort}; Promoted: {successorHost}:{successorPort}' >> /tmp/recovery.log"
  ],
  "PostIntermediateMasterFailoverProcesses": [
    "echo 'Recovered from {failureType} on {failureCluster}. Failed: {failedHost}:{failedPort}; Successor: {successorHost}:{successorPort}' >> /tmp/recovery.log"
  ],
  "PostGracefulTakeoverProcesses": [
    "echo 'Planned takeover complete' >> /tmp/recovery.log"
  ],

[Shlomi Noach]

I'll try to address some of the bullets, but it's a massive scale discussion and cannot cover it all.

The general theme is:

- orchestrator will take care of the topology for you

- use the hooks (the pre/post failover processes you listed above) to notify external systems

- use the pre- hooks only for critical and fast operations, don't stall the failover

- use the post- hooks for longer processes. By longer I mean "up to many seconds, maybe a minute", not beyond, because orchestrator still tracks post- hooks as part of the recovery and may block other operations while this takes place. If need be, run something asynchronously, or just ask orchestrator to set some flag somewhere (e.g. touch a file, set value in a KV system), upon which your own systems will react later.

Some more detail:

> Pre-outage actions

> Verifying the topology of the cluster being changed

that's what orchestrator does.

> Choosing a replica to become the new master (may depend on DC location, state of the instance, how much redo lag exists, etc.)

that's what orchestrator does.

> Creating blackouts in monitoring and notification systems

in pre-hooks. Not sure you want to create a blackout, it depends on your monitoring culture. How fast will your systems alert on failure? Do you want someone to wake up anyway? Your choice.

> Sending notifications to interested parties

do this as quickly as possible. @-mention in chat; open a GitHub issue; send an email. Do this asynchronously.

> Outage actions

> Notifying upstream elements to stop sending traffic to the master and waiting for their ACKs

IMO avoid this. This may delay the promotion indefinitely. You might need to communicate to thousands of nodes. It may be impractical. 

> Ensuring that the master being switched away from cannot accept any new  read/write traffic

If you must have fencing, consider semi-sync. See here: https://github.com/openark/orchestrator/issues/1275

> Allowing the in-flight transactions to complete and/or kill them

Use case: let a proxy layer take care of this implicitly and asynchronously: https://github.blog/2018-06-20-mysql-high-availability-at-github/

> Ensure that all logs needed by the replica that will become the new master have been received (and possibly applied)

use configuration: https://github.com/openark/orchestrator/blob/master/docs/configuration-recovery.md#promotion-actions

FailMasterPromotionOnLagMinutes/FailMasterPromotionIfSQLThreadNotUpToDate/DelayMasterPromotionIfSQLThreadNotUpToDate

> Starting the shutdown of the master being switched away from

Will that even be possible> What caused th eoutage? Will the server be accessible?

> Enabling writes on the new master

orchestrator sets read_only=0

> Notifying upstream elements of the endpoint of the new master and waiting for their ACKs

See linked blog post

> Notifying upstream elements to start sending traffic to the master and waiting for their ACKs

See linked blog post, implicit by proxy

> Post-outage actions

> Ensure that the old master shuts down cleanly.  Restart it and bring it down cleanly if it does not.

That's really up to you. 

> Reinstate the old master as a replica as appropriate

That's really up to you. 

Shlomi

--
You received this message because you are subscribed to the Google Groups "orchestrator-mysql" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orchestrator-my...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/orchestrator-mysql/d688c907-a325-496d-b8fe-86d7f66f7cf6n%40googlegroups.com.