Backup cursor and checkpoint

[igor.sol...@percona.com]

Hello,

Reading backup cursors documentation here: http://source.wiredtiger.com/3.0.0/backup.html

I want to clarify one thing: is it true that backup created by the described procedure will mirror DB state at the moment of the most recent checkpoint made before backup cursor creation?

Thanks,

Igor

[Alex Gorrod]

Hi Igor,

On Fri, Sep 21, 2018 at 9:24 PM <igor.sol...@percona.com> wrote:

Reading backup cursors documentation here: http://source.wiredtiger.com/3.0.0/backup.html

I want to clarify one thing: is it true that backup created by the described procedure will mirror DB state at the moment of the most recent checkpoint made before backup cursor creation?

That's true, as long as logging is not configured. If you have logging configured in WiredTiger, and copy the log files along with the data files WiredTiger will apply all log records from the most recent checkpoint made before the backup cursor was created and when opening the new backup you will have all data that was written to the log prior to the files being copied. The subsequent "Incremental backup" section of the documentation explores that a bit more.

- Alex 

[igor.sol...@percona.com]

Hi Alex,

Thank you for the answer. Yes I have logging configured and I copy the log files. But there is also WT_SESSION::log_flush method. Now I have few questions related to this method's interaction with backup cursor:

1. What If I don't call log_flush after checkpoint and before opening backup cursor? Is there any chance that new log records created after checkpoint will made into the backup?

2. What will happen if log_flush is called after creating backup cursor but before log files copied? I guess in this case  copied log files will contain log records created after backup cursor was opened. Will those new records be applied to the DB when I open DB from the backup copy?

Igor

[Alex Gorrod]

Hi Igor,

If there are log files present in the backup copy any valid records in those log files will be applied on startup. It is not necessary to explicitly call log_flush after the checkpoint.

On Mon, Sep 24, 2018 at 11:09 PM <igor.sol...@percona.com> wrote:

1. What If I don't call log_flush after checkpoint and before opening backup cursor?

You don't need to call log_flush to ensure checkpoint is completed.

 

Is there any chance that new log records created after checkpoint will made into the backup?

Yes - if you startup the backup with recovery enabled all transactions that have written log records will be applied to the backup database.

 

2. What will happen if log_flush is called after creating backup cursor but before log files copied? I guess in this case  copied log files will contain log records created after backup cursor was opened. Will those new records be applied to the DB when I open DB from the backup copy?

It is intended that log records are applied to the backup, bringing the state of the database to the last durable logged transaction when files were copied. You can disable that behavior by starting up with logging disabled in the backup copy of the database.

- Alex

Igor

--
You received this message because you are subscribed to the Google Groups "wiredtiger-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wiredtiger-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[igor.sol...@percona.com]

Hi Alex,

Documentation says:

The database file may grow in size during the copy, and the file copy should not consider that an error. Blocks appended to the file after the copy starts can be safely ignored, that is, it is correct for the copy to determine an initial size of the file and then copy that many bytes, ignoring any bytes appended after the backup cursor was opened.

What if my copy procedure does not ignore bytes appended after the backup cursor was opened? (It just copies all files without any truncation). Will those new data blocks be ignored when I start from backup? From your explanation I know that nothing will be ignored from the log files but is this also true for regular data files? Until now I was thinking there is some metadata describing which data existed at the moment of the last checkpoint but now I am not sure.

On Tuesday, September 25, 2018 at 4:34:13 AM UTC+3, Alex wrote:

Hi Igor,

If there are log files present in the backup copy any valid records in those log files will be applied on startup. It is not necessary to explicitly call log_flush after the checkpoint.

On Mon, Sep 24, 2018 at 11:09 PM <igor.sol...@percona.com> wrote:

1. What If I don't call log_flush after checkpoint and before opening backup cursor?

You don't need to call log_flush to ensure checkpoint is completed.

 

Is there any chance that new log records created after checkpoint will made into the backup?

Yes - if you startup the backup with recovery enabled all transactions that have written log records will be applied to the backup database.

 

2. What will happen if log_flush is called after creating backup cursor but before log files copied? I guess in this case  copied log files will contain log records created after backup cursor was opened. Will those new records be applied to the DB when I open DB from the backup copy?

It is intended that log records are applied to the backup, bringing the state of the database to the last durable logged transaction when files were copied. You can disable that behavior by starting up with logging disabled in the backup copy of the database.

Can I achieve the same result if my backup procedure just won't copy log files?

 

Thank you Alex, your answers are invaluable

[Alex Gorrod]

Hi Igor,

On Tue, Sep 25, 2018 at 6:49 PM <igor.sol...@percona.com> wrote:

Documentation says:

The database file may grow in size during the copy, and the file copy should not consider that an error. Blocks appended to the file after the copy starts can be safely ignored, that is, it is correct for the copy to determine an initial size of the file and then copy that many bytes, ignoring any bytes appended after the backup cursor was opened.

What if my copy procedure does not ignore bytes appended after the backup cursor was opened? (It just copies all files without any truncation). Will those new data blocks be ignored when I start from backup? From your explanation I know that nothing will be ignored from the log files but is this also true for regular data files? Until now I was thinking there is some metadata describing which data existed at the moment of the last checkpoint but now I am not sure.

You just copy the file that is there - the necessary content from the point in time where the backup started is guaranteed not to move until the backup cursor has been released.

It is intended that log records are applied to the backup, bringing the state of the database to the last durable logged transaction when files were copied. You can disable that behavior by starting up with logging disabled in the backup copy of the database.

Can I achieve the same result if my backup procedure just won't copy log files?

Maybe, but that isn't really supported - WiredTiger remembers whether logging is enabled, and may expect to see something in a log file (or reserves the right to expect that in the future if it doesn't now).

- Alex