Backup
Sarge
txn-protected environment. The documentation does not describe how and when
to coincide checkpoints with archives.
I would like to offer "full" and "incremental" backup options, where "full"
backs up the dbs and the logs and "incremental" only backs up the logs. It
seems that checkpoints must be run infrequently to offer incremental
backups.
Every time a checkpoint is run, the .db files are updated, and the
associated log files (as reported by db->log_archive) may then be removed.
Berkeley samples show running checkpoints frequently, but I found that
everytime a checkpoint is run, a full backup is required. If an incremental
backup contains a checkpoint marker in a log file, then a restore of that
backup does not restore all of the data if recovery is run.
If I wish to run checkpoints frequently (at least once per day), does that
mean that I can only offer full backups?
--Sarge
Chris Newcombe
> If I wish to run checkpoints frequently (at least once per day), does that
> mean that I can only offer full backups?
No, you can run checkpoints as frequently as you like. In fact it's
important to run them fairly frequently as recovery time is
proportional to the number of transactions (and amount of data
changed) since the last checkpoint.
>>If an incremental backup contains a checkpoint marker in a log file,
>>then a restore of that backup does not restore all of the data if
recovery is run.
It sounds like you might only be running 'normal' recovery. But when
doing incremental restoration from logs, you need to run
'catastrophic' recovery ("db_recover -c", or open the environment with
DB_RECOVER_FATAL and DB_CREATE). Do this with all of the logs
obtained since you last ran recovery.
The documentation does describe this, but it's easy to miss:
http://www.sleepycat.com/docs/ref/transapp/hotfail.html
(The '-c' in Step 4 is critical.)
The difference between normal recovery and catastrophic recovery is
that normal recovery finds the latest checkpoint in the log and only
applies changes to the databases from that point in the log forward to
the end of the log. i.e. It assumes that the databases are already
consistent up to the point of the latest checkpoint.
But catastrophic recovery always starts at the beginning of the oldest
log you provide (assuming a contiguous sequence of log files -- I'm
not sure what it does if there is a gap in the sequence).
So if you drop down a set of new incremental logs that happen to
contain multiple checkpoints, but then only run normal recovery, you
will indeed be missing some (probably many) updates. I'd guess that
your database will be corrupt (giving various invalid page-LSN errors
later), but I'm not really sure exactly what will happen. If you're
lucky recovery will detect the inconsistency and fail with a fatal
error, but I don't think that behavior is guaranteed (in which case
you're application will likely hit a fatal error later). Even if you
don't get a fatal error, your database is probably semantically
inconsistent from your applications point of view.
The only problem with catastrophic recovery is that it can take a
very, very long time to run (proportional to the number/size of logs).
So it can be important to identify the minimal safe set of logs that
you need before running it.
I believe that it is safe to remove all of the logs that you have
previously run catastrophic recovery on. **However, the new set of
logs should (must) include the new version of the 'latest' log in the
old set that you are removing.** i.e. The old and the new set must
overlap by at least 1 log, or you will break the chain of LSNs.
Example...
Old set (catastrophic recovery has already been run on these)
log.001 complete (say 10 MB)
log.002 complete (say 10 MB)
log.003 partial (say 7 MB)
New set (catastrophic recovery not yet run on these)
log.003 now complete (10 MB)
log.004 complete (10 MB)
log.005 partial (say 3 MB)
You can delete the old set of logs and drop down the entire new set,
*including* the now-complete log.003. Then run catastrophic recovery.
CAVEAT: I don't work for Sleepycat. I'd strongly recommend emailing
sup...@sleepycat.com to ask them to review any backup/recovery
proceedure that you are not sure about.
regards,
Chris
Sarge
I don't understand why either DB_RECOVER or DB_RECOVER_FATAL need to be run
after restoring a backup.
I do see the need to run DB_RECOVER after a system crash to bring the
database back to a stable condition. But a system crash does not
necessarily warrant a restore of a backup. That is the whole purpose of
DB_RECOVER is so you *don't* have to restore from backup everytime something
bad happens. The only time I see the need for restoring a backup is if the
the computer or HDD dies completely and a different computer or HDD must be
used.
I would like to be able to restore from backup by first copying the last
full backup, followed by copying every incremental backup run since the last
full backup. Once that is accomplished on the target computer, it seems the
database should be immediately usable without running DB_RECOVER.
The problem is that everytime a checkpoint is run, it invalidates the last
full backup, because the .db files are modified. This implies that every
backup following a checkpoint must be a full backup. If we allow
incremental backups to be run, and then a restore from backup to a different
computer, the new system would think that the .db files are up to date when
they really are not.
I will further study your reply and hopefully figure this thing out!
Thanks again,
--Sarge
"Chris Newcombe" <chr...@valvesoftware.com> wrote in message
news:a6cd7f01.03121...@posting.google.com...
Chris Newcombe
> I don't understand why either DB_RECOVER or DB_RECOVER_FATAL need to be run
> after restoring a backup.
Think of 'recovery' as just one name for the procedure of applying
updates from transaction logs to the database files, to bring both
into synchronization. True, that process has the name 'recovery'
because it is most often done to recover from an application or system
failure, but it is necessary in some other circumstances too.
One such circumstance (where 'catastrophic' recovery is necessary) is
to make most types of archived backups useable by an application
('restore from backup'). Another circumstance is the one you are
interested in -- to freshen a hot backup with a new batch of log
files.
'Recovery' is necessary in these cases because the transaction logs
contain updates that are not yet in the database files. Before
Berkeley DB can start to use a database, the database files and
transaction logs must be in sync (unless an intact, valid 'environment
region' is still present). That synchronization is required because
during normal operation (i.e. when getting/putting data on behalf of
an application), the database can only read and modify pages in the
database-files, and append data to the transaction logs. i.e. The
database cannot read application data from past transaction logs, or
modify past (existing) entries in the transaction logs.
This is a little confusing because it's also true that once the
application has started to use the database, the database-files are
almost always 'out of sync' with the transaction logs. This is due to
the caching of modified database pages in the 'environment region'
(the large block of shared-process memory that Berkeley DB uses for
inter-process communication; regions are usually backed by several
memory-mapped files).
Even if the only application using the database is shutdown cleanly
(DBENV::close successfully called), the database-files are still
likely to be out of sync with the transaction logs, because some
recent page updates will still be in the region cache, and not yet
flushed to disk. (i.e. DBENV::close merely 'detaches' a process from
the region, it does not flush the region cache, even if this happens
to be the last process that has the database open.)
BTW you can manually flush the cache with memp_sync, memp_trickle, or
txn_checkpoint methods. If you do that after finishing all writes and
closing all transactions, immediately before you call DBENV::close
(successfully), then I think that the database filesshould be in-sync
with the transaction logs, so you wouldn't have to run recovery to use
a backup copy of an environment in that state. But that's a rare
state for a backup because it is such a heavyweight and inconvenient
operation to stop all applications, flush a large cache to disk, etc.
> I would like to be able to restore from backup by first copying the last
> full backup, followed by copying every incremental backup run since the last
> full backup. Once that is accomplished on the target computer, it seems the
> database should be immediately usable without running DB_RECOVER.
I hope the above comments have show that you _do_ need to run
'catastrophic recovery' to apply the new transaction logs to your old
database files, and so bring the two into synchronization, before the
database can be used by an application.
> The problem is that everytime a checkpoint is run, it invalidates the last
> full backup, because the .db files are modified.
Warning: the database-files may actually be modified at any time, not
just during checkpoints. True, performing a checkpoint is very likely
to modify database-files (by flushing any 'dirty' cache pages to the
database-files).
But during any normal 'put' operations, the database may find that the
environment cache is already full of other 'dirty' pages, so to make
room for the newly modified page the database will select an existing
'dirty' page (usually the 'oldest') and flush that page to disk, so
modifying a database-file.
Therefore it's extremely unwise to assume that database files are only
modified during checkpoints. That depends entirely on the size of
your environment cache and application data-access patterns.
> This implies that every backup following a checkpoint must be a full backup.
No, fortunately that's not the case -- incremental hot backups do work
as advertised.
> If we allow incremental backups to be run, and then a restore from backup to a different computer, the new system would think that the .db files are up to date when they really are not.
The database files up-to date precisely because you ran catastrophic
recovery before starting your application.
I hope this has helped, not hindered :)
Chris
Philip Guenther
...
> One such circumstance (where 'catastrophic' recovery is necessary) is
> to make most types of archived backups useable by an application
> ('restore from backup').
Catastrophic recovery is required whenever the logs contain a checkpoint
that is not actually reflected in the .db files. If you can copy all
the .db file and transaction logs for a backup without a checkpoint
taking place, then you don't need to perform catastrophic recovery if
you restore that backup.
While I suppose you could design your checkpoint and backup tools so
that the latter could tell the former to pause during the backup, it's
fairly simple to do the test at backup time and then mark the backup as
only needing normal recovery instead of catastrophic recovery. You can
determine whether a checkpoint was made by calling DBENV->txn_stat()
before starting to copy the .db files and then again after the last
transaction log has been copied and comparing the st_last_ckp values in
the returned stats: if they compare as equal then no checkpoint took
place during the copies and the backup can be marked as only requiring
normal recovery when restored. If your backup utility is a shell
script, you can use db_stat's -t option to get the required
information.
> Another circumstance is the one you are
> interested in -- to freshen a hot backup with a new batch of log
> files.
The decision to perform incremental backups instead of full backups
should not be made lightly. If the database is subject to even
moderately stringent uptime requirements than the downtime cost of a
database restore that takes several hours (or days!) can easily prove to
be greater than the cost of the storage space needed for full backups.
While I haven't tried this myself, if a backup is stored online (and
not, for example, directly to tape), then it should generally be
possible to perform the database recovery on a backup any time after it
was taken instead of waiting until it is restored. You would have to
configure the environment with the correct paths and it might not be
possible to parallelize if the application has its own logging
operations (c.f. DB_ENV->set_app_dispatch()) or if txn_prepare() was
used, but properly done the restore operation would then involve a
single transaction log that ends with a checkpoint such that recovery
would be practically instantaneous.
> Even if the only application using the database is shutdown cleanly
> (DBENV::close successfully called), the database-files are still
> likely to be out of sync with the transaction logs, because some
> recent page updates will still be in the region cache, and not yet
> flushed to disk. (i.e. DBENV::close merely 'detaches' a process from
> the region, it does not flush the region cache, even if this happens
> to be the last process that has the database open.)
Unless you used the DB_NOSYNC flag when closing the actual database
tables or have performed a DBENV->dbrename() or DBENV->dbremove()
operation since then, the database files should be in sync with the
mpool when you get to the DBENV->close() operation, no?
Philip Guenther