RMAN DUPLICATE failed at recovery time.
perpl...@yahoo.es
I am trying to duplicate a 10gR2 Oracle database into a new host
using different directory structure for datafiles and log files. The
duplicated database is also 10gR2. Both Hosts use RHEL4 OS.
I had successfully duplicated the target database in the past and
therefore I assume that environment settings are correct.
The process completes the first two duplication phases with no issues:
1) Set newname and restore of datafiles
2) Creation of a new clone controlfile, switch of datafile names and
specification of online redologs based on log_file_name_convert
parameter
But fails in the third one:
3) Point in time recovery of the restored datafiles
I am using a backup from the target database using device PARALLELISM
= 4 and backupset compression is enabled. The RMAN backup script used
is:
run {
backup tag 'FULL_BACKUP' device type disk database;
backup tag 'REMAINING_ARCHIVELOG_BACKUP' device type disk archivelog
all not backed up delete all input;
delete noprompt obsolete device type disk;
}
Target DB retention policy set to redundancy 1. I double check that
all backup pieces are available to the duplicated host and presented
in the same path as the target database.
The duplication script used is:
RUN {
ALLOCATE AUXILIARY CHANNEL aux1 DEVICE TYPE DISK;
ALLOCATE AUXILIARY CHANNEL aux2 DEVICE TYPE DISK;
ALLOCATE AUXILIARY CHANNEL aux3 DEVICE TYPE DISK;
ALLOCATE AUXILIARY CHANNEL aux4 DEVICE TYPE DISK;
SET UNTIL SEQUENCE = 18250;
DUPLICATE TARGET DATABASE TO djprod2
SKIP READONLY;
}
The sequence number 18250, is set to the newest full backup archive
log sequence number + 1.
perpl...@yahoo.es
performing point in time recovery of the restored datafiles :
-----------------------------------------------------------------------------------------------
starting media recovery
RMAN-06560: WARNING: backup set with key 195101 will be read 3 times
RMAN-06560: WARNING: backup set with key 195102 will be read 3 times
RMAN-06560: WARNING: backup set with key 195103 will be read 3 times
available space of 6237788 kb needed to avoid reading the backup set
multiple times
channel aux1: starting archive log restore to default destination
channel aux1: restoring archive log
archive log thread=1 sequence=18247
channel aux1: reading from backup piece /backup/arch_dtfiles/
crkf7pg0_1_1
channel aux1: restored backup piece 1
piece handle=/backup/arch_dtfiles/crkf7pg0_1_1
tag=REMAINING_ARCHIVELOG_BACKUP
channel aux1: restore complete, elapsed time: 00:04:46
Oracle Error:
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get
error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '/data-lv/oradata/system01.dbf'
released channel: aux1
released channel: aux2
released channel: aux3
released channel: aux4
RMAN-00571:
===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS
===============
RMAN-00571:
===========================================================
RMAN-03002: failure of Duplicate Db command at 05/19/2009 10:08:01
RMAN-03015: error occurred in stored script Memory Script
RMAN-06561: available space must be larger than 2061949 kb
Recovery Manager complete.
------------------------------------------------------------------
The backup contains three compressed backup sets (1 backup piece each)
with archived logs (TAG: REMAINING_ARCHIVELOG_BACKUP), Oracle shows a
warning specifying the backup sets Keys. What is shocking to me is
that it is trying to recover the database with the newest logs and
leaves the older ones behind, hence the recovery issue.
Executing the RMAN list command on the target DB to get information in
archived logs backed up during the full target DB backup:
--------------------------------------------------------------------------------------------------------------------
RMAN> list backup tag REMAINING_ARCHIVELOG_BACKUP;
List of Backup Sets
===================
BS Key Size Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
195101 1.53G DISK 00:07:55 17-MAY-09
BP Key: 195108 Status: AVAILABLE Compressed: YES Tag:
REMAINING_ARCHIVELOG_BACKUP
Piece Name: /backup/djprod1/arch_dtfiles/crkf7pg0_1_1
List of Archived Logs in backup set 195101
Thrd Seq Low SCN Low Time Next SCN Next Time
---- ------- ---------- --------- ---------- ---------
1 18247 13753652757 17-MAY-09 13753968692 17-MAY-09
1 18248 13753968692 17-MAY-09 13754284761 17-MAY-09
1 18249 13754284761 17-MAY-09 13754519417 17-MAY-09
BS Key Size Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
195102 1.66G DISK 00:08:58 17-MAY-09
BP Key: 195109 Status: AVAILABLE Compressed: YES Tag:
REMAINING_ARCHIVELOG_BACKUP
Piece Name: /backup/djprod1/arch_dtfiles/cpkf7pg0_1_1
List of Archived Logs in backup set 195102
Thrd Seq Low SCN Low Time Next SCN Next Time
---- ------- ---------- --------- ---------- ---------
1 18244 13752701081 17-MAY-09 13753019578 17-MAY-09
1 18245 13753019578 17-MAY-09 13753336611 17-MAY-09
1 18246 13753336611 17-MAY-09 13753652757 17-MAY-09
BS Key Size Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
195103 1.71G DISK 00:09:09 17-MAY-09
BP Key: 195110 Status: AVAILABLE Compressed: YES Tag:
REMAINING_ARCHIVELOG_BACKUP
Piece Name: /backup/djprod1/arch_dtfiles/cqkf7pg0_1_1
List of Archived Logs in backup set 195103
Thrd Seq Low SCN Low Time Next SCN Next Time
---- ------- ---------- --------- ---------- ---------
1 18241 13751509331 16-MAY-09 13752221618 17-MAY-09
1 18242 13752221618 17-MAY-09 13752412707 17-MAY-09
1 18243 13752412707 17-MAY-09 13752701081 17-MAY-09
-----------------------------------------------------------------------------------------------
As you can see it is starting recovery from log sequence 18247, that
belongs to backup set key 195101, and is leaving behind older logs
contained in
195102 and 195103 backup sets, which explains why point in time
recovery is failing.
Is this normal?
Thanks in advance.
Tony Sequeira
> This is what is logged in the RMAN log when the duplication is
> performing point in time recovery of the restored datafiles :
[...]
> RMAN-06561: available space must be larger than 2061949 kb
What's this about?
I've seen the original symptoms before (9ir2) with a 2 node RAC
duplicate to a single instance from an old backup. I eventually got it
to recover by specifying archive logs manually. This was a production
clone, so I didn't have time to track down the root cause, but I believe
that it was only applying logs from the instance it had connected to,
not the other node. I may have been wrong.
But I didn't see the message above.
--
S. Anthony Sequeira
++
It's documented in The Book, somewhere...
-- Larry Wall in <10...@jpl-devvax.JPL.NASA.GOV>
++
perpl...@yahoo.es
The error RMAN-06561 is confusing to me. For this error, the doc says:
Cause
The recovery failed because it requires more disk space. One of the
following could have caused this error: 1) The MAXSIZE option is used
but is not large enough to restore files. 2) Files should be restored
to recovery area, but available disk space is not large enough to
restore files.
Action
One of the following: 1) Increase the MAXSIZE parameter and retry the
command. 2) Free up disk space in the recovery area.
In my case, none of both actions apply. I use default MAXSIZE
parameter and the duplicated host recovery area has enough space.
What I don't understand is why Oracle is apparently doing recovery
using backup set key order instead of log sequence number order.
joel garry
> Hi Anthony, Thanks for your comment.
> The error RMAN-06561 is confusing to me. For this error, the doc says:
>
> Cause
> The recovery failed because it requires more disk space. One of the
> following could have caused this error: 1) The MAXSIZE option is used
> but is not large enough to restore files. 2) Files should be restored
> to recovery area, but available disk space is not large enough to
> restore files.
>
> Action
> One of the following: 1) Increase the MAXSIZE parameter and retry the
> command. 2) Free up disk space in the recovery area.
>
> In my case, none of both actions apply. I use default MAXSIZE
> parameter and the duplicated host recovery area has enough space.
From the backup and recovery reference manual:
"DELETE ARCHIVELOG [MAXSIZE integer [K|M|G]] Deletes archived logs
restored from backups or copies that are no longer needed. RMAN does
not delete archived logs that were already on disk before the RESTORE
command started.
If you do not specify MAXSIZE, then RMAN deletes restored archived
logs as they are applied. If you specify MAXSIZE, then RMAN will not
use more than integer amount of disk space for restored archived logs.
If recovery requires the restore of a log larger than the MAXSIZE
value, then RMAN reports an error indicating that you should increase
the MAXSIZE value. If MAXSIZE is smaller than the backup set
containing the logs, then RMAN must read the backup set more than once
to extract the logs. In this situation, RMAN issues a warning that
MAXSIZE should be increased.
Note: If archived redo logs are restored to the flash recovery area
then the DELETE ARCHIVELOG option is enabled by default."
>
> What I don't understand is why Oracle is apparently doing recovery
> using backup set key order instead of log sequence number order.
I don't understand it either, but I'll take a wild guess. You haven't
specified maxsize, so rman deletes the logs as they are applied.
Unfortunately, it is doing this key order thingie, and so needs to
recover more after it has deleted them, which perhaps is what the
warning about read 3 times is about. The SCN's wind up not matching
between some other file and the system file, so it complains about the
system file needing more recovery with ora-1110, even though it may
"really" be the other file. Since you are getting everything, have
you tried it without the set until?
So, perhaps you need to try specifying maxsize in your run command.
Perhaps Anthony's way worked because he specified the arcs manually,
so they didn't get deleted.
I'm wondering if this is a bug, or maybe an unclear documentation bug
(unclear about the set key order, at least).
jg
--
@home.com is bogus.
Who pays for the database?
http://www3.signonsandiego.com/stories/2009/may/19/1m19records23224-county-now-charging-view-birth-de/?uniontrib
Tony Sequeira
[...]
> So, perhaps you need to try specifying maxsize in your run command.
> Perhaps Anthony's way worked because he specified the arcs manually,
> so they didn't get deleted.
Hi Joel,
I didn't get the exact symptoms that the OP experienced, I just got the
system file requiring recovery. It really threw me. I may go back and
revisit the alert log, no RMAN log files unfortunately. My recovery was
first a full recovery, then a point in time, and every time I got the
same symptoms.
> jg
> --
> @home.com is bogus.
> Who pays for the database?
> http://www3.signonsandiego.com/stories/2009/may/19/1m19records23224-county-now-charging-view-birth-de/?uniontrib
>
--
S. Anthony Sequeira
++
Every day it's the same thing -- variety. I want something different.
++
perpl...@yahoo.es
ARCHIVELOG context. I don't understand, however, how could I use
MAXSIZE in the duplication script. Is this what are you suggesting?
Could you please post an example?
I did tried different recoveries with no success.
Recovering to a sequence number prior to 18247 (trying to force oracle
to use cpkf7pg0_1_1 and cqkf7pg0_1_1 ) fails in the second phase. I
believe this is because it has not all recovery information needed to
complete Full backup restore and recovery.
I cannot remove the Set Until clause because the target database has
already backed up and catalogued further archived logs, but I did try
recover to a later point in time by using the sequence number of next
archive log taken after target db backup completion. The results are
the same, Oracle completes recovery from crkf7pg0_1_1 backup piece,
omitting cpkf7pg0_1_1 and cqkf7pg0_1_1 files information, and fails.
Liencres
thing applying recovery with compressed backup sets, I did proceed
this way:
1) Cold Backup of the target database
2) Open the target database and perform a log switch. This archives
the current log
3) Use cold Backup pieces and 1 archived log to duplicate the target
database.
Using the RMAN DUPLICATE command, you cannot duplicate a target
database without applying recovery hence the need to archive the
current log.
As reported by Tony, restoring archive logs from compressed backup
sets and then applying recovery from them should also be fine.