Autotuner Backup File

0 views

Skip to first unread message

Gano Richardson

unread,

Aug 3, 2024, 4:05:10 PM8/3/24

to criparorduc

In a few simple steps, you can extract EEPROM and Flash from a backup file separately in order to modify them and update them back together as a new backup file. And the best part? Checksum correction is also done automatically.

Thanks to backup file, you can modify the data located outside the maps area.
Any modification made to a file contained in the backup file causes the modification of its checksum.
Therefore, if the data has been modified, the user must systematically correct the checksum before writing one of the files from the backup.

Do not create a .zip archive of a folder that includes the different parts of the backup file. The different parts must be compressed together in a .zip format. When extracted, files must be located directly at the root of the directory.

The Slave file cannot be freely modified. It can only be processed by the Master to which the Slave tool is linked.
The Slave file is a file in .zip format that contains the same information as the backup file but also the data constituting the engine mapping.
The information it contains is encrypted and intended to be modified by a Master.

Any modification made to a file contained in the backup file causes the modification of its checksum.
Therefore, if the data has been modified, the user must systematically correct the checksum before writing one of the files from the backup.

Before TiDB v5.4.0, when you back up data using Backup & Restore (BR), the number of threads used for backup makes up 75% of the logical CPU cores. Without a speed limit, the backup process can consume a lot of cluster resources, which has a considerable impact on the performance of the online cluster. Although you can reduce the impact of backup by adjusting the size of the thread pool, it is a tedious task to observe the CPU load and manually adjust the thread pool size.

To reduce the impact of backup tasks on the cluster, TiDB v5.4.0 introduces the auto-tune feature, which is enabled by default. When the cluster resource utilization is high, BR automatically limits the resources used by backup tasks and thereby reduces their impact on the cluster. The auto-tune feature is enabled by default.

If you want to reduce the impact of backup tasks on the cluster, you can enable the auto-tune feature. With this feature enabled, TiDB performs backup tasks as fast as possible without excessively affecting the cluster.

Alternatively, you can limit the backup speed by using the TiKV configuration item backup.num-threads or using the parameter --ratelimit. When --ratelimit is set, to avoid too many tasks causing the speed limit to fail, the concurrency parameter of br is automatically adjusted to 1.

TiKV supports dynamically configuring the auto-tune feature. You can enable or disable the feature without restarting your cluster. To dynamically enable or disable the auto-tune feature, run the following command:

Auto-tune is a coarse-grained solution for limiting backup speed. It reduces the need for manual tuning. However, because of the lack of fine-grained control, auto-tune might not be able to completely remove the impact of backup on the cluster.

Issue 1: For write-heavy clusters, auto-tune might put the workload and backup tasks into a "positive feedback loop": the backup tasks take up too many resources, which causes the cluster to use fewer resources; at this point, auto-tune might mistakenly assume that the cluster is not under heavy workload and thus allowing backup to run faster. In such cases, auto-tune is ineffective.

The backup process includes lots of SST decoding, encoding, compression, and decompression, which consume CPU resources. In addition, previous test cases have shown that during the backup process, the CPU utilization of the thread pool used for backup is close to 100%. This means that the backup tasks take up a lot of CPU resources. By adjusting the number of threads used by the backup tasks, TiKV can limit the CPU cores used by backup tasks, thus reducing the impact of backup tasks on the cluster performance.

Issue 3: For scenarios with high traffic jitter, because auto-tune adjusts the speed limit on a fixed interval (1 minute by default), it might not be able to handle high traffic jitter. For details, see auto-tune-refresh-interval.

This feature has two related configuration items not listed in the TiKV configuration file. These two configuration items are only for internal tuning. You do not need to configure these two configuration items when you perform backup tasks.

If you use the WAIT BEFORE START and WITH CHECKPOINT LOG NO COPY clauses to complete a backup, you can start the backup copy of the database in read-only mode and validate it. By enabling validation of the backup database, you can avoid making an additional copy of the database.

When this clause is specified, the backup is delayed until there are no active transactions. All other activity on the database is prevented and a checkpoint is performed to ensure that the backup copy of the database does not require recovery. When the checkpoint is complete, other activity on the database resumes.

When you specify COPY, the backup reads the database files without applying any modified pages. The entire checkpoint log and the system dbspace are copied to the backup directory. The next time the database server is started, the database server automatically recovers the database to the state it was in as of the checkpoint at the time the backup started.

Because pages do not have to be written to the temporary file, using this option can provide better backup performance, and reduce internal server contention for other connections that are operating during a backup. However, since the checkpoint log contains original images of modified pages, it grows in the presence of database updates. With copy specified, the backed-up copy of the database files may be larger than the database files at the time the backup started. The COPY option should be used if disk space in the destination directory is not an issue.

This option results in smaller backed up database files, but the backup may proceed more slowly, and possibly decrease performance of other operations in the database server. It is useful in situations where space on the destination drive is limited.

If AUTO is specified, one output stream is created for each reader thread. The value n specifies the maximum number of output streams that can be created, up to the number of reader threads. The default value for this clause is 1. If you are backing up to tape, only one writer can be used.

The first stream, stream 0, produces files named myarchive.X, where X is a number that starts at 1 and continues incrementing to the number of files required. All of the other streams produce files named myarchive.Y.Z, where Y is the stream number (starting at 1), and Z is a number that starts at 1 and continues incrementing to the number of files required.

When you back up a strongly-encrypted database with free page elimination turned on, you must specify the encryption key when restoring the database. When you back up a strongly-encrypted database with free page elimination turned off, you do not need to specify the encryption key when restoring the database.

Each backup operation, whether image or archive, updates a history file called backup.syb. This file records the BACKUP and RESTORE operations that have been performed on a database server. For information about how the location of the backup.syb file is determined, see SALOGDIR environment variable.

To create a backup that can be started on a read-only server without having to go through recovery, you must use both the WAIT BEFORE START and WITH CHECKPOINT LOG NO COPY clauses. The WAIT BEFORE START clause ensures that the rollback log is empty, and the WITH CHECKPOINT LOG NO COPY clause ensures that the checkpoint log is empty. If either of these files is missing, then recovery is required. You can use WITH CHECKPOINT LOG RECOVER as an alternative to the WAIT BEFORE START and WITH CHECKPOINT LOG NO COPY clauses if you do not need to recover the database you backed up.

Alternatively, you can specify an empty string as a directory to rename or truncate the log without copying it first. This is useful in a replication environment where space is a concern. You can use this feature with an event handler on transaction log size to rename the transaction log when it reaches a given size, and with the delete_old_logs option to delete the transaction log when it is no longer needed.

If a RESTORE DATABASE statement references an archive file containing only a transaction log, the statement must specify a file name for the location of the restored database file, even if that file does not exist. For example, to restore from an archive that only contains a transaction log to the directory C:\MYNEWDB, the RESTORE DATABASE statement is:

Backup copies of the database and transaction log must not be changed in any way. If there were no transactions in progress during the backup, or if you specified BACKUP DATABASE WITH CHECKPOINT LOG RECOVER or WITH CHECKPOINT LOG NO COPY, you can check the validity of the backup database using read-only mode or by validating a copy of the backup database.

However, if transactions were in progress, or if you specified BACKUP DATABASE WITH CHECKPOINT LOG COPY, the database server must perform recovery on the database when you start it. Recovery modifies the backup copy, which is not desirable.

The foundation for this is Auto Backup, which ensures GRAX has the latest version of all your records. This means things like triggers and cascade deletes aren't going to unexpectedly cause data loss.

By default GRAX relies on Salesforce to delete related records, via cascade deletes. So when you request a Case to be archived, for example, you'll notice that related Events and Tasks are also going to be deleted.

That aside, the one case where we may delete additional records in an archive is to clean up ContentDocument records that would be left orphaned. This is important because while the cascade delete takes care of deleting files represented by an Attachment, the ContentDocument hierarchy is a bit different and would allow for orphaned records and files to be left in Salesforce.