Help! Instance Crashing (detailed)

Darryl G. Hickson

unread,

Sep 16, 1997, 3:00:00 AM9/16/97

to

Hello All,
I have an instance that keeps crashing and I am not sure why. I have been
working with another DBA and he can't figure it out either. Hopefully some
of you can help.

We are running an application called Clintrial which runs on top of the
Oracle database. The problem started not long after two datafiles were
added (one to the system tablespace and the other to an index tablespace).
Before that there had been no problems with the instance. The alert log
shows that the statements processed correctly and there was no other alters
performed that day. I have been assured that nothing else was changed in
the Oracle database or the OS and that the users are doing nothing out of
the ordinary. But, 30 minutes after the datafiles were added the instance
crashed and it has been doing it on a regular basis. Looking at the trace
files, we thought that there were not enough redo logs. But since this has
begun, I have added an additional 6 redo logs and it is still crashing!

Any help at all on this will be greatly appreciated. The HW/SW
particulars, Alert file info and trace files are below.

TIA,
Darryl

======================================================================
HW/SW
Machine: MicroVAX 3100-90
Operating System: VAX/VMS v5.5-2H4
Oracle Version: V6.0.33.0.0
Clintrial Version: CTV30
======================================================================
Alert Log:
Error signalled in file: DKA300:[ORACLE.DB_XX.NODE_XX]SRV_XX_004.TRC;
ORA-00604: error occurred at recursive SQL level 1
ORA-00470: LGWR process terminated with error
16-SEP-1997 14:46:01.32

======================================================================

NODE> type SRV_XX_001.TRC;331
Dump file DKA300:[ORACLE.DB_XX.NODE_XX]SRV_XX_001.trc
16-SEP-1997 13:45:02.82
ORACLE V6.0.33.0.0 - Production vsnsta=0
Proc: 0x12a ORA_XX_PMON User: [001,004] SYSTEM Term:
Image: DKA300:[ORACLE.RDBMS]SRV.EXE;1
Enqueue Quota: 512
vsnrcf=6.0.23.0.0 vsnwcf=6.0.30.3.0 vsnrdb=6.0.23.0.0 vsnwdb=6.1.27.2.0
vsnrlg=6.1.27.0.0 vsnwlg=6.1.27.3.0 vsnclp=6.0.23.0.0 vsnplp=6.0.30.3.0
vsnsql=1 vsnxtr=0
cpu 3190 13000202 vms V5.5-2H4 not clustered with 0 nodes
scsnd: NODE , ndname: NODE, sys$node NODE::
cpuid 13 rev 017E00000000000013000202 archflg 38F0
hwmdl: 382 hwnm: MicroVAX 3100-90 cpus: FFFFFFFF cpush: 0 active: 1, avail:
1
locktbl size 3447 max 65535 resource hash size 150

PMON started
error 471 detected in other oracle background process

PMON about to exit
OPIRIP: Uncaught error 447. Error stack:
ORA-00447: fatal error in background process
ORA-00471: DBWR process terminated with error
OPIRIP: Detached process complete.

======================================================================

NODE> type SRV_XX_002.TRC;331
Dump file DKA300:[ORACLE.DB_XX.NODE_XX]SRV_XX_002.trc
16-SEP-1997 13:45:13.98
ORACLE V6.0.33.0.0 - Production vsnsta=0
Proc: 0x12c ORA_XX_DBWR User: [001,004] SYSTEM Term:
Image: DKA300:[ORACLE.RDBMS]SRV.EXE;1
Enqueue Quota: 512
vsnrcf=6.0.23.0.0 vsnwcf=6.0.30.3.0 vsnrdb=6.0.23.0.0 vsnwdb=6.1.27.2.0
vsnrlg=6.1.27.0.0 vsnwlg=6.1.27.3.0 vsnclp=6.0.23.0.0 vsnplp=6.0.30.3.0
vsnsql=1 vsnxtr=0
cpu 3190 13000202 vms V5.5-2H4 not clustered with 0 nodes
scsnd: NODE , ndname: NODE, sys$node NODE::
cpuid 13 rev 017E00000000000013000202 archflg 38F0
hwmdl: 382 hwnm: MicroVAX 3100-90 cpus: FFFFFFFF cpush: 0 active: 1, avail:
1
locktbl size 3447 max 65535 resource hash size 150

DBWR started
16-SEP-1997 13:45:13.98
OPIRIP: Uncaught error 447. Error stack:
ORA-00447: fatal error in background process
ORA-00210: cannot open control file
'DISK$VAXVMS2H4055:[ORACLE.DB_XX]ORA_CONTROL
1.CON'
ORA-07530: sfofi: $open error
RMS-E-ACC, ACP file access failed - SYSTEM-F-EXBYTLM, exceeded byte count
quota
OPIRIP: Detached process complete.

======================================================================

NODE> type SRV_XX_003.TRC;331
Dump file DKA300:[ORACLE.DB_XX.NODE_XX]SRV_XX_003.trc
16-SEP-1997 13:45:23.79
ORACLE V6.0.33.0.0 - Production vsnsta=0
Proc: 0x12d ORA_XX_LGWR User: [001,004] SYSTEM Term:
Image: DKA300:[ORACLE.RDBMS]SRV.EXE;1
Enqueue Quota: 512
vsnrcf=6.0.23.0.0 vsnwcf=6.0.30.3.0 vsnrdb=6.0.23.0.0 vsnwdb=6.1.27.2.0
vsnrlg=6.1.27.0.0 vsnwlg=6.1.27.3.0 vsnclp=6.0.23.0.0 vsnplp=6.0.30.3.0
vsnsql=1 vsnxtr=0
cpu 3190 13000202 vms V5.5-2H4 not clustered with 0 nodes
scsnd: NODE , ndname: NODE, sys$node NODE::
cpuid 13 rev 017E00000000000013000202 archflg 38F0
hwmdl: 382 hwnm: MicroVAX 3100-90 cpus: FFFFFFFF cpush: 0 active: 1, avail:
1
locktbl size 3447 max 65535 resource hash size 150

LGWR started
16-SEP-1997 13:45:23.80
Opened log sequence 16283 file USER1:[ORACLE.DB_XX]ORA_LOG6.RDO
Advanced to log sequence 16284 file USER2:[ORACLE.DB_XX]ORA_LOG7.RDO
Advanced to log sequence 16285 file USER2:[ORACLE.DB_XX]ORA_LOG8.RDO
OPIRIP: Uncaught error 447. Error stack:
ORA-00447: fatal error in background process
ORA-00472: PMON process terminated with error
OPIRIP: Detached process complete.

======================================================================

NODE> type SRV_XX_004.TRC;330
Dump file DKA300:[ORACLE.DB_XX.NODE_XX]SRV_XX_004.trc
16-SEP-1997 13:45:35.14
ORACLE V6.0.33.0.0 - Production vsnsta=0
Proc: 0x12f ORA_XX_SMON User: [001,004] SYSTEM Term:

Image: DKA300:[ORACLE.RDBMS]SRV.EXE;1
Enqueue Quota: 512
vsnrcf=6.0.23.0.0 vsnwcf=6.0.30.3.0 vsnrdb=6.0.23.0.0 vsnwdb=6.1.27.2.0
vsnrlg=6.1.27.0.0 vsnwlg=6.1.27.3.0 vsnclp=6.0.23.0.0 vsnplp=6.0.30.3.0
vsnsql=1 vsnxtr=0
cpu 3190 13000202 vms V5.5-2H4 not clustered with 0 nodes
scsnd: NODE , ndname: NODE, sys$node NODE::
cpuid 13 rev 017E00000000000013000202 archflg 38F0
hwmdl: 382 hwnm: MicroVAX 3100-90 cpus: FFFFFFFF cpush: 0 active: 1, avail:
1
locktbl size 3447 max 65535 resource hash size 150

SMON: system monitor started
16-SEP-1997 14:46:01.25
SMON: following errors trapped and ignored:
ORA-00604: error occurred at recursive SQL level 1
ORA-00470: LGWR process terminated with error
16-SEP-1997 14:46:01.42
SMON: error 470 detected in other detached process
16-SEP-1997 14:46:01.45

SMON: system monitor exiting.
OPIRIP: Uncaught error 447. Error stack:
ORA-00447: fatal error in background process
ORA-00470: LGWR process terminated with error
OPIRIP: Detached process complete.
NODE>

Fritz Knack

unread,

Sep 16, 1997, 3:00:00 AM9/16/97

to

On 16 Sep 1997 16:06:28 GMT, "Darryl G. Hickson"
<hic...@voicenet.com> wrote:

<snip>

>The problem started not long after two datafiles were
>added (one to the system tablespace and the other to an index tablespace).
>Before that there had been no problems with the instance. The alert log
>shows that the statements processed correctly and there was no other alters
>performed that day. I have been assured that nothing else was changed in
>the Oracle database or the OS and that the users are doing nothing out of
>the ordinary. But, 30 minutes after the datafiles were added the instance
>crashed and it has been doing it on a regular basis. Looking at the trace
>files, we thought that there were not enough redo logs. But since this has
>begun, I have added an additional 6 redo logs and it is still crashing!

<snip>

Just a thought: Have you ruled out bad spots on the disk?

Fritz

Mary Travis

unread,

Sep 18, 1997, 3:00:00 AM9/18/97

to

"Darryl G. Hickson" <hic...@voicenet.com> wrote:

>Hello All,
> I have an instance that keeps crashing and I am not sure why. I have been
>working with another DBA and he can't figure it out either. Hopefully some
>of you can help.

> We are running an application called Clintrial which runs on top of the
>Oracle database. The problem started not long after two datafiles were
>added (one to the system tablespace and the other to an index tablespace).
>Before that there had been no problems with the instance. The alert log
>shows that the statements processed correctly and there was no other alters
>performed that day. I have been assured that nothing else was changed in
>the Oracle database or the OS and that the users are doing nothing out of
>the ordinary. But, 30 minutes after the datafiles were added the instance
>crashed and it has been doing it on a regular basis. Looking at the trace
>files, we thought that there were not enough redo logs. But since this has
>begun, I have added an additional 6 redo logs and it is still crashing!

> Any help at all on this will be greatly appreciated. The HW/SW
>particulars, Alert file info and trace files are below.

>TIA,
>Darryl

Darryl,
I found this portion of the trace files to be interesting because it
was one of the earlier messages written during the crash:

16-SEP-1997 13:45:13.98
OPIRIP: Uncaught error 447. Error stack:
ORA-00447: fatal error in background process
ORA-00210: cannot open control file
'DISK$VAXVMS2H4055:[ORACLE.DB_XX]ORA_CONTROL
1.CON'
ORA-07530: sfofi: $open error
RMS-E-ACC, ACP file access failed - SYSTEM-F-EXBYTLM, exceeded byte
count quota
OPIRIP: Detached process complete

Does your system have a limit set on the maximum number of files that
can be open at a time? And is it possible that with the addition of
two new files to your database that you are burshing against that
limit?

Just a thought,
(and good luck)
Mary

Mary Travis
BellSouth Telecommunications
email: Mary.B...@bridge.bellsouth.com

Dante A. Notari-Locher

unread,

Sep 19, 1997, 3:00:00 AM9/19/97

to Darryl G. Hickson

Hi Darryl !

May I attract your attention to this part of SRV_XX_002.TRC;331:

[snip]

> DBWR started
> 16-SEP-1997 13:45:13.98
> OPIRIP: Uncaught error 447. Error stack:
> ORA-00447: fatal error in background process
> ORA-00210: cannot open control file
> 'DISK$VAXVMS2H4055:[ORACLE.DB_XX]ORA_CONTROL
> 1.CON'
> ORA-07530: sfofi: $open error
> RMS-E-ACC, ACP file access failed - SYSTEM-F-EXBYTLM, exceeded byte
> count
> quota
> OPIRIP: Detached process complete.

All other terminations seems to be based on the termination of the
DBWR-Process.
It is a long time ago, I worked with VMS, but can you imagine that there

is sg wrong with he ACPs on that control-file ?

Regards

Dante A. Notari-Locher

Vittorio Turchetti

unread,

Sep 26, 1997, 3:00:00 AM9/26/97

to

Darryl G. Hickson wrote:
>
> Hello All,
> I have an instance that keeps crashing and I am not sure why. I have been

...

> DBWR started
> 16-SEP-1997 13:45:13.98
> OPIRIP: Uncaught error 447. Error stack:
> ORA-00447: fatal error in background process
> ORA-00210: cannot open control file
> 'DISK$VAXVMS2H4055:[ORACLE.DB_XX]ORA_CONTROL
> 1.CON'
> ORA-07530: sfofi: $open error
> RMS-E-ACC, ACP file access failed - SYSTEM-F-EXBYTLM, exceeded byte count
> quota
> OPIRIP: Detached process complete.

...

What about investigating about the O.S. error message RMS-E-ACC quoted
above?
I had a lot of trouble with Oracle on VAX/VMS due to incorrect O.S.
quotas and privileges.

Best regards.

Vittorio Turchetti

MarkP28665

unread,

Sep 28, 1997, 3:00:00 AM9/28/97

to

Darryl G. Hickson wrote: >>
'DISK$VAXVMS2H4055:[ORACLE.DB_XX]ORA_CONTROL
1.CON' ORA-07530: sfofi: $open error RMS-E-ACC, ACP file access failed
- SYSTEM-F-EXBYTLM, exceeded byte count
quota <<

I think the error messages are pretty clear. You have either a system
level resource limitation (quota) problem or your logicials are not set
correctly ( the control file open error) I would bet on the system resouce
being the problem.

Page limitation and output file size limits are common VMS errors
encountered when first setting up an Oracle instance.

Mark Powell -- The only advise that counts is the advise that you follow
so follow your own advise