RMAN Legato and ORA-03113.
George Barbour
Normally backups of my production instance using RMAN and Legato are fine.
However now and again the backup fails.
The problem with having an error appear in Legato, is that there appears to
be no way,
to tell the calling script something bad has happened, ergo the RMAN script
just hangs
waiting for the Legato problem to clear.
I therefore call another script, normally about an hour after the RMAN
backup
should have completed, this script connects as sys and does a "shutdown
immediate, shutdown abort"
and eventually restarts the instance. Clunky I admit, but it has worked in
the past, and it brings the instance back online and into a useful state, to
be backed up another day.
Recently however, I have been getting the following RMAN error(s),
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-00601: fatal error in recovery manager
RMAN-03004: fatal error during execution of command
RMAN-10039: error encountered while polling for RPC completion on channel t1
RMAN-10006: error running sql statement: select act from x$ksusex where
sid=:1 and serial=:2
RMAN-10002: ORACLE error: ORA-03113: end-of-file on communication channel
Oracle Server Manager Release 3.1.5.0.0 - Production
.. when the abort/restart script kicks in, an hour later, it also suffers
the same fate! So the instance does not get
restarted until I return the following day, and therefore the overnight
batch jobs are just mince.
Bear in mind the script below connects as internal. (yes I know connect
internal is going out of fashion)
(c) Copyright 1997, Oracle Corporation. All Rights Reserved.
Oracle8i Enterprise Edition Release 8.1.5.0.0 - Production
With the Partitioning and Java options
PL/SQL Release 8.1.5.0.0 - Production
SVRMGR> Connected.
SVRMGR> ORA-03113: end-of-file on communication channel
SVRMGR> ORA-01031: insufficient privileges
SVRMGR> ORA-01031: insufficient privileges
SVRMGR>
Version:- Oracle 8.1.5 (Yes I know it's de-supported.)
OS:- Sun UNIX 2.8
Questions:
Has anybody actually cracked the hanging RMAN/Legato backup problem?
Oracle UK support haven't, I asked them.
Why should an instance suddenly, in the middle of an operation suffer an
ORA-03113 error?
How come a connect internal has "insufficient privileges" to shutdown an
instance?
Is there life on Mars?
thanks and oblige,
George Barbour.
Howard J. Rogers
internal...
Your output runs thus....
> SVRMGR> Connected.
> SVRMGR> ORA-03113: end-of-file on communication channel
> SVRMGR> ORA-01031: insufficient privileges
The point is that an ORA-3113 error disconnects the user. So when your
script says 'shutdown' or 'startup' after the 3113, you are actually trying
to perform a Privileged Action without a connection at all.
Regards
HJR
--
=============================!!=============================
The views expressed are my own only, and definitely NOT those of Oracle
Corporation
=============================!!=============================
"George Barbour" <george....@gecm.com> wrote in message
news:3b25...@pull.gecm.com...
George Barbour
Sometimes it's difficult to see through the red cloud of anger and
frustration.
There was a problem with the tape backup system says the UNIX administrator.
This somehow caused an ORA-600 on the instance, which would then cause the
first ORA-03113 within RMAN. Fine, I understand that.
It's the next bit that really hurts, an hour after the instance crashed
crontab woke up the abort/startup script, and that seemed to connect and
then immediately failed, as you say with no connection at all. The actual
very simple script is :-
svrmgrl << EOF
connect internal
shutdown immediate
shutdown abort
startup
EOF
The script is so simple it could hardly go wrong, but it does. I needed to
manually connect as sys this morning,
shutdown abort, startup, shutdown, and startup again.
I require to have the instance running overnight because of the batch
processes, and because users start earlier in the morning than I do, the
database needs to be ready for them. I accept that backup errors will happen
but I need a way to abort and then restart the instance using crontab that I
can rely on ...
open to ideas,
George Barbour.
"Howard J. Rogers" <howa...@www.com> wrote in message
news:3b25...@news.iprimus.com.au...
Dave Wotton
[snip]
> Has anybody actually cracked the hanging RMAN/Legato backup problem?
> Oracle UK support haven't, I asked them.
> Why should an instance suddenly, in the middle of an operation suffer an
>ORA-03113 error?
> How come a connect internal has "insufficient privileges" to shutdown an
>instance?
ORA-3113 "end-of-file on communication channel" is a catch-all error
message which simply means that the foreground oracle client process
(svrmgrl, sqlplus, RMAN, Oracle Forms etc.) has lost contact with its
background shadow process.
Unfortunately there are many causes of this. In particular, anything
which causes the shadow process to crash will cause this error to
be reported.
Try looking in your alert log and look for a trace file in your
background_dump_dest. They may help diagnose the problem.
Here are some of the causes of ORA-3113 I've accumulated so far:
(all of them on Oracle 8.1.6.0.0 on Solaris 8). All of these were
raised as TARs, and solved with the help of Oracle Support.
1) shared_pool_size too small. We had it set to 100Mb. Increasing
it to 200Mb solved the problem. (Yes, I know 200Mb is probably
still too small for good performance.)
2) timed_statistics=true. (Oracle Support could not provide
a solution to this one: they just advised me to run my
database with timed_statistics=false. Doing this did indeed
cure a lot of ORA-3113's I was getting when querying
data-dictionary tables.)
3) An insert into a table which grew into an additional extent
allocated in a datafile which was over 2Gb in size. I can't
remember whether it was the action of growing the datafile
which generated the ORA-3113, or whether it was the action of
extending the table into the area above the 2Gb limit which
caused it.
This has similar symptoms to your problem, as we then got
ORA-3113's when trying to restart the database. I think what
happened was that the database crashed when the ORA-3113 was
reported. Each time we brought it up it tried to re-apply the
redo logs to bring the database into a conistent state
before opening it. But this caused an insert above the 2Gb
limit again and another ORA-3113 crash. I can't remember how
I got round the problem. Probably "recover until cancel".
4) "Create table as select .... " where there was an invalid
procedure or function called from a trigger defined on the
table we were selecting from. I can't specify this in any
more detail: we used the utlrp.sql script to recompile all
invalid packages, procedures and functions. This solved the
problem, but I don't know which of the procedures or functions
was the original cause of the problem.
If you have a look at Metalink you'll find quite a lot of problems
related to ORA-3113, with lots of diverse causes. You might be
stuck. You probably need Oracle Support's help to work out why
your background process suddenly crashes in the middle of a
backup, but because you're not on a supported release you probably
won't get much help.
Just one thought. You haven't got any Unix shell script that runs
nightly and kills any oracle processes it finds running have you?
Sorry I can't help further.
Dave.
--
If you reply to this newsgroup posting by email, remove the "nospam"
from my email address first.
George Barbour
RMAN one.
The snake in the grass is the "shutdown immediate", it can take up to ten
(10) hours
for the instance to shut down if a fault appears in RMAN. I should have
known this.
Ergo, just do a "shutdown abort" .
>
> svrmgrl << EOF
> connect internal
> shutdown immediate ** remove this line.
> shutdown abort
> startup
> EOF
Thanks for all the advice.
George Barbour.
George Barbour
Thanks for your email which contains obvioulsy hard worked for information.
Unfortunately my efforts to return your message have failed.
I am checking my instance against your statements, maybe something wil turn
up.
Oracle say I must upgarade the instance to O8.1.7. but I can only do this
when
third parties have certified their software against that version.
.
Can I suppose Oracle9i will fix all problems? ;-)
Thanks again,
George Barbour.