Setting up slave using TokuDB and TokuBackup

[steve.r...@gmail.com]

Hi,

My slave server was setup when I was using InnoDB, and later I converted to TokuDB. Since then, my slave server has had a RAID failure, and became hopelessly out of sync. Percona is advising I rebuild the slave, and use TokuBackup instead of Xtrabackup (since TokuBackup works with TokuDB).

All good - except their is no guidance on how to setup the slave using TokuBackup. Usually, a slave is created from bin log information from the master (I'm told), but it's not clear if that is the way to do the setup with TokuBackup.

Can anyone provide some much needed guidance?

Thank you
Steve Mullen
Rover IDX

[George Lorch]

I apologize for the delay in responding, this got buried in my inbox and an associate nudged me.

TokuDB Hot Backup effectively makes a point in time snapshot of the server. That point in time is basically the state of the server at the time the backup is finishing.

In order for the binlog and TokuDB recovery log to be properly in sync at this snapshot time, you must ensure that tokudb_commit_sync=on, otherwise there may be unflushed writes to the TokuDB recovery log that have matching entries in the binary log that have been flushed. https://www.percona.com/doc/percona-server/5.7/tokudb/tokudb_variables.html#tokudb_commit_sync

To bootstrap a slave:

- Copy the backup to another server that you intend to slave to your master where you got your backup from, as if you were doing a regular server restore.

- Start this copied/second/slave server instance and allow/wait for crash recovery to run.

- Run SHOW MASTER STATUS _on_the_slave_server_instance_ to obtain the binlog position that slave instance is current to.

- Shut down the slave server instance and set it up as your actual slave (change serverid and other typical my.cnf changes).

- Restart slave server instance

- On the slave server instance, run  the CHANGE MASTER TO ... statement using the binlog info you just got from the SHOW MASTER STATUS.

Your slave should now be running and catching up to your master.

For some reason, we do not have this documented nor blogged about. I will pass this along to our documentation team to get clarified.

[Steve Mullen]

Very much appreciated - thank you !

Steve Mullen

Rover IDX

[George Lorch]

Not a problem Steve, and again apologize for the lack of clear documentation and slow response. Please post back if you have any other issues.

[Steve Mullen]

This solution has been implemented, but we're thinking there must be some tuning that has to happen.  The setup worked ok, but the slave is now badly out of sync.  I've attached a report.

Thank you

Steve Mullen

[Steve Mullen]

Hi,

Wondering if anyone has any insight into the slave being badly out of sync.

Thank you

Steve Mullen

[George Lorch]

The original procedure was designed by Tokutek support team and carried over when we (Percona) acquired them. Since then it has come up that using the SHOW MASTER STATUS on a re-fired slave is problematic since the slave will likely rotate the binlogs on startup, thus giving a position that is later than the one actually captured in the backup set.

Rather than using SHOW MASTER STATUS, mysqlbinlog could/should be used before starting the slave on the backed up data to get the last position in the last binlog file as described in this issue https://tokutek.atlassian.net/browse/BACKUP-118

We are working on the backup plugin to make it a little more usable for this use case, specifically we want to have the backup process execute a backup lock at the end of the copy, flush all logs to force a sync between binlog and TokuDB/InnoDB recovery logs and capture the binlog position into some output file, but this solution will be a little while in the making.

[st...@roveridx.com]

Hi

You mention that "We are working on the backup plugin to make it a little more usable".

Wondering if there has been enough progress to use this plugin in production?

[George Lorch]

There has been some recent work done on it to be better with replication and getting consistent binlog + recovery log backups with position info. Vlad recently blogged abut it here https://www.percona.com/blog/2017/04/14/tokudb-hotbackup-replication/