Why do I get out of memory errors when 10GB memory is free?
zigzagdna
I am on HP UNIX 64bit 11.23 titanium using oracle 10.2.0.3. My server
has 24GB memory and of that 10GB is free (as seen in glance). When I
doing oracle exp or rman commands, I get:
ORA-04030: out of process memory when trying to allocate 1049112 bytes
(KSFQ heap,KSFQ Buffers)
I checked both rman and exp are 64bit executables, so they should be
able to access all the memory on the system.
Mladen Gogala
Because your process limits are not set correctly. Ask your friendly
neighborhood system administrator to allow oracle processes to allocate
enough memory.
sybr...@hccnet.nl
'zigzagdna' has now developed the habit to post all his queries *both*
on OTN (using an anonymous handle) and on this forum.
On OTN he denies his ulimit settings are a problem.
Zigzagdna, instead of involving all of the globe, posting your 'I can
not be bothered to read documentation' queries all over the globe, and
involving DBAs all over the globe to do your work, can you please
*STOP* this behavior, *NOW*
--
Sybrand Bakker
Senior Oracle DBA
Mladen Gogala
> 'zigzagdna' has now developed the habit to post all his queries *both*
> on OTN (using an anonymous handle) and on this forum. On OTN he denies
> his ulimit settings are a problem.
It's not simple ulimit setting, in the first place. The parameter values
are described here:
http://docs.hp.com/en/1219/tuningwp.html#maxssiz
http://docs.hp.com/en/1219/tuningwp.html#maxdsiz
http://docs.hp.com/en/1219/tuningwp.html#maxtsiz
Of course, it's a HP-UX setup, st. John Hurley is the guy who knows
HP-UX on this group, he could help him. Zigzagdna is just being his
old, obstinate self. I believe it would be the right time for his
company to promote him to a managerial position so that he just doesn't
have to read all those dusty manuals. Peter's principle should be
followed.
John Hurley
snip
> 'zigzagdna' has now developed the habit to post all his queries *both*
> on OTN (using an anonymous handle) and on this forum.
> On OTN he denies his ulimit settings are a problem.
> Zigzagdna, instead of involving all of the globe, posting your 'I can
> not be bothered to read documentation' queries all over the globe, and
> involving DBAs all over the globe to do your work, can you please
> *STOP* this behavior, *NOW*
Just scroll on by and try to take a deep breath every once in a while.
John Hurley
snip
Have you looked at the documentation from oracle on how to set the
hpux kernel parameters ( correctly)?
It is going to take some aggressive settings to allow oracle to
allocate and use effectively that much memory.
John Hurley
snip
> Of course, it's a HP-UX setup, st. John Hurley is the guy who knows
> HP-UX on this group, he could help him.
AFAIK think I am missing some miracles although my racing times are
fairly strong for an old guy.
HPUX isn't that much different from all the other unix flavors each is
somewhat different release by release you just have to follow the
instructions and double check stuff.
Palooka
Try following the instructions.
Palooka
Mladen Gogala
> Try following the instructions.
He has to read them, first.
zigzagdna
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 1048576
stack(kbytes) 131072
memory(kbytes) unlimited
oredump(blocks)
2. Some kernel parameters relevant to oracle:
shmmax 25662429593 25662429593 Immed
shmmni 512 512 Immed
shmseg 120 120 Immed
max_thread_proc 1024 1024 Immed
maxdsiz 1073741824 Default Immed
maxdsiz_64bit 2147483648 2147483648 Immed
maxfiles 5120 5120
maxfiles_lim 6144 6144 Immed
maxrsessiz 8388608 Default
maxrsessiz_64bit 8388608 Default
maxssiz 134217728 134217728 Immed
maxssiz_64bit 1073741824 1073741824 Immed
maxtsiz 100663296 Default Immed
I do not see anything wrong (low values) for ulimit or anyting wrong
in kernel parameters.
zigzagdna
Here are some observations related to memory:
1. I can increase sga_target (that’s the only parameter I have for
SGA) by several Giga Bytes without any problem, i.e., Oracle instance
starts with any memory issues.
2. If I start a java process, by simply running java command, it gives
me out of memory error . It seems that sever does not have enough
emmory for heap (evne though 10GB of memory is free on server) because
java needs some default heap memory (I think 64M).. IT cannot be a
ulimit issue because maxdsiz is 4 GB.
3. If I reduce sga_target to a smaller value, restart oracle instance
and then run java, java does not give memory error, so Oracle freed
some memory. What I do not understand is Oracle is freeing shared
memory and not heap memory, so why it has impact on java (i.e., java
getting more heap memory). Server all anog has at least 9 to 10GB
memory free.
4. Problem does not happen with Oracle startup, oracle is running
fine. Oracle export command and rman off and on fail with ORA error
related to memory.
5. My server has 24GB memory and in all above cases, at least 40% of
24GB is free.
Hope someone can point what can be done to get rid of memory errors.
John Hurley
snip
Do you have a problem report open with oracle support?
That's a place to start.
I can take a look at what you sent next week sometime ... but you are
better off digging into this yourself and start by step by step going
thru the oracle documentation for your release of hpux on setting the
memory stuff.
What you mean exactly by "running java" ... what does java have to do
with your oracle instance exactly?
zigzagdna
>
> - Show quoted text -
Thanks. I have been looking at this memory issue for several days
along with my UNIX administrators. No one can figure out the cause,
I was giving java simply as an example that HP UNIX is running out of
some kind of memory resource. I suspect Oracle export and rman
commands are running out of same kind of memory resource.
zigzagdna
Sorry, my mazdsiz paremetrs were changed a few days ago, but memory
errors did not go away,
maxdsiz 4294963200 4294963200 Immed
maxdsiz_64bit 4294967296 4294967296 Immed
Mladen Gogala
> data(kbytes) 1048576
> stack(kbytes) 131072
How do you expect to have large address space with that?
Your data segment cannot exceed 1GB.
Park
Hi,
what's your PGA set to? something really big as well?
matt
joel garry
What is JAVA_POOL_SIZE SHARED_POOL_SIZE LARGE_POOL_SIZE
What exact version of java are you using? When you run java, the heap
size that is asked for can actually determine that it will run
different java programs.
Your maxssize may be too small (as Mladen hinted), see
http://docs.hp.com/en/JAVAPERFTUNE/Memory-Management.pdf under kernel
tunables.
It doesn't really matter if a lot of memory is free if you've limited
the size of what is asked for.
jg
--
@home.com is bogus.
http://www3.signonsandiego.com/stories/2009/jul/06/us-tec-microsoft-security-070609/?california&zIndex=127607
zigzagdna
> tunables.
>
> It doesn't really matter if a lot of memory is free if you've limited
> the size of what is asked for.
>
> jg
> --
>
> - Show quoted text -
I talked to HP engineer. Per HP engineer there is some kind of
limtation in HP arhicteure that for 32 bit processes, it needs 1GB
contigous space (even if 32GB process is not using 1GB memory). On
that server even though there is lot of free memory, there is not
contiguous 1Gb free memory. So root cause of the problem is something
different than kernel parameters.
zigzagdna
>
> > - Show quoted text -
>
> I talked to HP engineer. Per HP engineer there is some kind of
> limtation in HP arhicteure that for 32 bit processes, it needs 1GB
> contigous space (even if 32GB process is not using 1GB memory). On
> that server even though there is lot of free memory, there is not
> contiguous 1Gb free memory. So root cause of the problem is something
>
> - Show quoted text -
I am still talking to HP engineer because rman backup which fails off
and on is a 64 bit executable not 32 bit executable.
John Hurley
snip
> I am still talking to HP engineer because rman backup which fails off
> and on is a 64 bit executable not 32 bit executable.
The HP engineer should be able to work with you going thru the hp ux
oracle customization doc and checking and then double checking
everything.
Good luck and keep us posted.
zigzagdna
First HP engineer reduced kernel parameters for UNIX buffers, so
kernel occupies less memory. Hope was that it will make more memory
avaiable for application (which it did), but it did not fix the
problem. Subsequntly. swap space was increased.
Swap space increase fixed the rroblem . I am not a UNIX admin, so I
still wonder when 10GB memory is avaiable why HP OS first won't
utlize this memory instead of going to swap to get heap memory . We
did not change maxdsiz, maxssiz or maxtsiz prameters.
I am glad problem is now resolved. I can put more oracle instances on
the server and get my company money's worth.
John Hurley
snip
> > The HP engineer should be able to work with you going thru the hp ux
> > oracle customization doc and checking and then double checking
> > everything.
>
> > Good luck and keep us posted.
>
> First HP engineer reduced kernel parameters for UNIX buffers, so
> kernel occupies less memory. Hope was that it will make more memory
> avaiable for application (which it did), but it did not fix the
> problem. Subsequntly. swap space was increased.
> Swap space increase fixed the rroblem . I am not a UNIX admin, so I
> still wonder when 10GB memory is avaiable why HP OS first won't
> utlize this memory instead of going to swap to get heap memory . We
> did not change maxdsiz, maxssiz or maxtsiz prameters.
> I am glad problem is now resolved. I can put more oracle instances on
> the server and get my company money's worth.
The hpux kernel parameters dbc_max_pct and dbc_min_pct come out of the
box with values that reserve huge chunks of memory for unix buffering
that are pretty much unacceptable for any database server and pretty
much have to be checked and revised downward.
On a machine with 16 gig of memory or more I would really hope you
have them down to like 5 or so ... with 24 gig of memory like 2 or 3
would be probably a good value. You can find postings on this topic
in cdos from a while back but well not many hpux questions come into
here.
As an oracle dba running on unix or linux ... you just have to ante up
and take responsbilitity for reviewing and being at least somewhat
knowledgeable about setting and reviewing this stuff.
Why? Because no one else is likely to care enough or be competent
enough to take the time and assess what should be done.
Oracle has documented pretty well the relevant things that should be
checked and in most cases provides hints about checking related things
like dbc_***_pct ( as far as I can recall ). HP provides white papers
etc that are also related to tuning kernel stuff when running oracle.
Are you using direct IO and avoiding the unix buffer cache? Do you
even know what to look at to find out the answer to this question?
Before you think you are done here I would take a look at that area
immediately.
The whole subject of how many instances one puts onto a dedicated
server also deserves some consideration and review.
Good luck!
zigzagdna
>
> - Show quoted text -
Regraidng reducing UNIX buffers, we typically reduce them if server
does not have lot of memory.
I am still puzzled why increasing swap will fix the problem when
server alreday has 10GB memory free.
Prem
Mladen Gogala
> I am not
> a UNIX admin, so I still wonder when 10GB memory is avaiable why HP OS
> first won't utlize this memory instead of going to swap to get heap
> memory .
That's elementary my dear zigzagdna. Whenever Unix allocates memory, it
allocates the space on swap as well. It does that because it needs the
place to store the newly initialized pages, should they be thrown out
of memory later. If there is no swap to allocate, memory will not be
allocated. You should execute the system administrator who has set up
the system with swap partition smaller than the physical memory.
joel garry
In addition to what Mladen said, note there are several different ways
of allocating swap on hp-ux - even using memory. Since the notion
that swap will rarely actually be used is correct (barring extreme
configuration blunders), you might consider some filesystem based swap
on a lesser used device, if that makes sense for your configuration
(for example, if you have an extra unused internal drive). Modern
unix are less picky about swap than in the olden days, but it's still
an issue, basic enough that not doing it really is incompetent admin.
I told you to check swap back in May.
http://groups.google.com/group/comp.databases.oracle.server/browse_thread/thread/565184ddad1c731c/
jg
--
@home.com is bogus.
Just wondering how far back forums.oracle messages go.
http://forums.oracle.com/forums/message.jspa?messageID=119930
John Hurley
snip
> In addition to what Mladen said, note there are several different ways
> of allocating swap on hp-ux - even using memory. Since the notion
> that swap will rarely actually be used is correct (barring extreme
> configuration blunders), you might consider some filesystem based swap
> on a lesser used device, if that makes sense for your configuration
> (for example, if you have an extra unused internal drive). Modern
> unix are less picky about swap than in the olden days, but it's still
> an issue, basic enough that not doing it really is incompetent admin.
There just is not any viable choice an oracle dba running in a unix/
linux environment has to accept the responsibility and verify all the
basic stuff. You just cannot count on a system admin ... there are
some good ones ... an oracle dba has to pick up this challenge even if
they do not want to.
> I told you to check swap back in May.http://groups.google.com/group/comp.databases.oracle.server/browse_th...
So you did ... I did not remember that incident.
Hard to believe that someone was struggling with basic config for more
than 2 months ... well I guess easy to believe ... but sad.
joel garry
> On Jul 15, 5:26 pm, joel garry <joel-ga...@home.com> wrote:
>
> snip
>
> > In addition to what Mladen said, note there are several different ways
> > of allocating swap on hp-ux - even using memory. Since the notion
> > that swap will rarely actually be used is correct (barring extreme
> > configuration blunders), you might consider some filesystem based swap
> > on a lesser used device, if that makes sense for your configuration
> > (for example, if you have an extra unused internal drive). Modern
> > unix are less picky about swap than in the olden days, but it's still
> > an issue, basic enough that not doing it really is incompetent admin.
>
> There just is not any viable choice an oracle dba running in a unix/
> linux environment has to accept the responsibility and verify all the
> basic stuff. You just cannot count on a system admin ... there are
> some good ones ... an oracle dba has to pick up this challenge even if
> they do not want to.
>
> > I told you to check swap back in May.http://groups.google.com/group/comp.databases.oracle.server/browse_th...
>
> So you did ... I did not remember that incident.
Truth be told, I didn't either. I was looking for one of the threads
where I gave links to good explanations of how swap works, but when I
saw that, I stopped looking.
>
> Hard to believe that someone was struggling with basic config for more
> than 2 months ... well I guess easy to believe ... but sad.
Enough people have gotten on his case I thought he was perhaps wanting
to actually learn something the way he worded this thread, but now it
appears he's just learned to word things like that. But I'll continue
to be optimistic anyways, until things smack me in the face.
jg
--
@home.com is bogus.
Matthias Hoys
"Mladen Gogala" <mla...@bogus.email.invalid> wrote in message
news:h3kvj3$2ma$3...@solani.org...
Is this still the case? I thought that for larger amounts of memory it's no
longer needed to have the same amount of swap as the physical memory.
I don't know for HP-UX, but we are running Oracle without problems on RHEL
with 4GB of RAM and 2GB of swap space (if you consider Linux to be some kind
of UNIX ;-)).
Matthias
Mladen Gogala
> Is this still the case?
Apparently, it is. I haven't heard of any recent changes in the way
that SVR4.2 deals with the virtual memory. Why would that change? In a
virtual memory system, one has to write the pages down when necessary.
The backing store for the freshly allocated pages, as it's done with
malloc or calloc, is swap space. What leads you to the conclusion that
it has chaged?
>I thought that for larger amounts of memory it's
> no longer needed to have the same amount of swap as the physical memory.
There are pages that can be excluded from virtual memory handling. On
Linux, we're talking about hugepages. Solaris has something similar.
Other than that, if pages are a part of the virtual memory system, they'd
better be covered by the swap space.
> I don't know for HP-UX, but we are running Oracle without problems on
> RHEL with 4GB of RAM and 2GB of swap space (if you consider Linux to be
> some kind of UNIX ;-)).
It's a stretch. RHEL has a debilitated version of Unix memory management.
It's for desktop users (wink, wink). Desktop users are generally
considered to be blithering idiots by the OS designers, so they made
changes to the proven algorithm which adjust the memory management so
that it can be used by any idiot. It's designed with the following goals
in mind:
- It mustn't be easy to understand and flexible. It is "intelligent"
instead.
- It takes the control away from the system administrator and gives it to
the "artificial intelligence". With an Intel inside, a computer is
allegedly more intelligent than the SA.
- It must allow a person which has never read a page of manuals to utilize
the system. It has to make the knowledgeable users pay for their
arrogance.
Linux VM system achieved all of these goals very well.