"SYSTEM RUNNING OUT OF PAGING SPACE" (AIX-HACMP-Oracle)
Arne S
I've got a problem that all of a sudden came up after update of AIX to
ML09 and upgrade of Oracle from 7.4.3 to 8.1.7: The machine is sometimes
slowing down, and at the end I have to turn it off and back on again.
After the machine is up again, I find this error in the error log:
*********************************************
LABEL: PGSP_KILL
IDENTIFIER: C5C09FFA
Date/Time: fre 02-04-26 05:33:
Sequence Number: 3811
Machine Id: 0040AD3A4C00
Node Id: bsprod
Class: S
Type: PERM
Resource Name: SYSVMM
Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED
Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE
Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE
Recommended Actions
DEFINE ADDITIONAL PAGING SPACE
REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)
Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
66786
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
152
*********************************************
Could it be the the valuse of "ulimit -a"? My new Oracle 8.1.7 is owned
by the user "oracle8" instead of the old "oracle".
ulimit -a for oracle:
$ ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) unlimited
memory(kbytes) unlimited
coredump(blocks) unlimited
nofiles(descriptors) unlimited
ulimit -a for oracle8:
$ ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 131072
stack(kbytes) 32768
memory(kbytes) 32768
coredump(blocks) unlimited
nofiles(descriptors) 2000
Info about the environment:
2 * RS/6000
AIX 4.3.3-09
Hacmp 4.3.1 (cluster.base.server.rte 4.3.1.16)
SSA disks in concurrent VG
Oracle 8.1.7.0.0 (raw device for db files)
Mem 2GB
Paging 2GB
Any ideas??
TIA!
Arne S
Zongk_Tu
Oracle recommends two times the physical memory for the SWAP ...
Sounds like you need to tune your Primary & Secondary Swap for your OS ...
More importantly, once you have your Instance up & running,
you need to sum all of your SZ or VSZ coulmn from 'ps -o' command ..(Posix
ps).
and then multiply that with 4K page size..
That will be the total Virtual Set Size for your Instance ...
So you will have to boot to a single user mode & resize your swap ..
Unlesss you have tools like IGNITE-UX in the HP-UX world...
HTH
zongk Tu
Oracle Apps DBA/Unix SA
Md Irfan
easy and can decide.
Irfan
Rob Kouwenberg
> Recommended Actions
> DEFINE ADDITIONAL PAGING SPACE
> REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)
What does 'lsps -s' give as output ?
A box here says :
Total Paging Space Percent Used
2048MB 1%
Norman Levin
need is more paging space. You can add space dynamically thru smit. While you
can expand a page space, I like adding additional page lv's since I can remove
them a lot easier than shrinking them <grin>.
Next, since you get a message in errlog, you can set up a stanza in the error
notify class that can send you a message that this has occurred to you can get a
sense of the problem in real time instead of after the fact. Well, that's not
really true. You can also run "errpt -C" in a window and grep for the message
of interest and run some alert program - so errnotify is not the only way to go.
norm
---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.351 / Virus Database: 197 - Release Date: 4/19/2002
Norman Levin
>> Arne:
>>
>> Oracle recommends two times the physical memory for the SWAP ...
>>
>> Sounds like you need to tune your Primary & Secondary Swap for your OS
>> ...
>>
>> More importantly, once you have your Instance up & running,
>>
>> you need to sum all of your SZ or VSZ coulmn from 'ps -o' command
>> ..(Posix ps).
>>
>> and then multiply that with 4K page size..
>>
>> That will be the total Virtual Set Size for your Instance ...
>>
>> So you will have to boot to a single user mode & resize your swap ..
>>
** NO - swap can be expanded or additional page lvs added dynamically in AIX.
norm
Grover Davidson
Check to see what version of bos.rte.libc you are running.
lslpp -l bos.rte.libc
If the version is below 4.3.3.78, UPGRADE IT! 4.3.3.75 is what shopped as ml
09. It has a bad memory leak in it!
Grover
Arne S wrote:
--
Grover Davidson
16620 Malaga Hills Dr
Round Rock, Tx 78681
Voice - 512-248-1446
Email - devi...@telocity.com <<<--- Please note my *new* address! my old
one will
die soon!!
Unix is user friendly.
It's just picky about who it makes friends with!
And even the best of friends disagree sometime!
Arne S
Yes, some memory leak was my sugestion too at the time this happened, and thanks
for info about the bos.rte.libc.
My bos.rte.libc is at 4.3.3.75. I will upgrade it at once!
Thanks!
Arne S
dunix
We are experiencing the same problem on an S85 running AIX maint level 09.
Oracle version is 817 . This box has 24GB of Ram and is a 12 way the
performance has been horrible with the 817 database. Tests that we performed
months ago on 734 were 40 percent faster .
Here is what we have found and done in order of what had the best impact;
1. Recalc the Oracle stats 3 times a day (This has helped tremendously) Why
the stats are getting blown is beyond us and beyond Oracle at this point.In
the past stats were only calculated monthly
2. Oracle patched to level 8.1.7.3
3. IBM performance tuned based on performance (perfpmr) taken at peak times
and sent to IBM . They had us make vmstat , syncd changes.
4. We did update the bos.rte.libc to the latest level but our problem did not
seem to be a memory leak.
The re-calc of the stats in oracle seems to have the biggest impact on our
situation. Why we have to do it so often is still a mystery.
Any feedback you have found would also be appreciated ...