help to understand kernel logs: sshd invoked oom-killer:

[Niyati Dave]

Hi

I am getting below error messages in my /var/log/messages
Swap gets out. and then processes are killed one by one.. sh,httpd2-
prefork,osiris.cgi repeating the same logs.. as below
Please somebody help why this is happening, what could be the cause of
the problem? what is oom-killer and why it gets invoked?

Jul 28 01:42:10 APP0001 kernel: sshd invoked oom-killer:
gfp_mask=0x1201d2, order=0, oomkilladj=0
Jul 28 01:42:10 APP0001 kernel: Pid: 16401, comm: sshd Not tainted
2.6.27.19-5-xen #1
Jul 28 01:42:10 APP0001 kernel:
Jul 28 01:42:10 APP0001 kernel: Call Trace:
Jul 28 01:42:10 APP0001 kernel: [<ffffffff8020c597>]
show_trace_log_lvl+0x41/0x58
Jul 28 01:42:10 APP0001 kernel: [<ffffffff8046357f>] dump_stack
+0x69/0x6f
Jul 28 01:42:10 APP0001 kernel: [<ffffffff80276608>] oom_kill_process
+0x5c/0x227
Jul 28 01:42:10 APP0001 kernel: [<ffffffff80276e54>] out_of_memory
+0xdd/0x10d
Jul 28 01:42:10 APP0001 kernel: [<ffffffff8027990d>]
__alloc_pages_internal+0x2ef/0x3f7
Jul 28 01:42:10 APP0001 kernel: [<ffffffff8027b70c>]
__do_page_cache_readahead+0x94/0x194
Jul 28 01:42:10 APP0001 kernel: [<ffffffff8027b8c4>]
force_page_cache_readahead+0x57/0x7e
Jul 28 01:42:10 APP0001 kernel: [<ffffffff80277022>]
sys_fadvise64_64+0x11e/0x190
Jul 28 01:42:10 APP0001 kernel: [<ffffffff8020b3b8>]
system_call_fastpath+0x16/0x1b
Jul 28 01:42:10 APP0001 kernel: [<00007f2ac4a576dd>] 0x7f2ac4a576dd
Jul 28 01:42:10 APP0001 kernel:
Jul 28 01:42:10 APP0001 kernel: Mem-Info:
Jul 28 01:42:10 APP0001 kernel: DMA per-cpu:
Jul 28 01:42:10 APP0001 kernel: CPU 0: hi: 0, btch: 1 usd: 0
Jul 28 01:42:10 APP0001 kernel: CPU 1: hi: 0, btch: 1 usd: 0
Jul 28 01:42:10 APP0001 kernel: CPU 2: hi: 0, btch: 1 usd: 0
Jul 28 01:42:10 APP0001 kernel: CPU 3: hi: 0, btch: 1 usd: 0
Jul 28 01:42:10 APP0001 kernel: DMA32 per-cpu:
Jul 28 01:42:10 APP0001 kernel: CPU 0: hi: 186, btch: 31 usd: 176
Jul 28 01:42:10 APP0001 kernel: CPU 1: hi: 186, btch: 31 usd: 179
Jul 28 01:42:10 APP0001 kernel: CPU 2: hi: 186, btch: 31 usd: 158
Jul 28 01:42:10 APP0001 kernel: CPU 3: hi: 186, btch: 31 usd: 94
Jul 28 01:42:10 APP0001 kernel: Active:503042 inactive:505023 dirty:0
writeback:0 unstable:0
Jul 28 01:42:10 APP0001 kernel: free:4576 slab:2988 mapped:1
pagetables:0 bounce:0
Jul 28 01:42:10 APP0001 kernel: DMA free:10340kB min:28kB low:32kB
high:40kB active:0kB inactive:0kB present:16128kB pages_scanned:0
all_unreclaimable? ye
s
Jul 28 01:42:10 APP0001 kernel: lowmem_reserve[]: 0 4016 4016 4016
Jul 28 01:42:10 APP0001 kernel: DMA32 free:7964kB min:8092kB low:
10112kB high:12136kB active:2012168kB inactive:2020092kB present:
4112640kB pages_scanned:
7556254 all_unreclaimable? yes
Jul 28 01:42:10 APP0001 kernel: lowmem_reserve[]: 0 0 0 0
Jul 28 01:42:10 APP0001 kernel: DMA: 1*4kB 2*8kB 1*16kB 0*32kB 3*64kB
1*128kB 1*256kB 1*512kB 1*1024kB 2*2048kB 1*4096kB = 10340kB
Jul 28 01:42:10 APP0001 kernel: DMA32: 37*4kB 5*8kB 4*16kB 3*32kB
1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7964kB
Jul 28 01:42:10 APP0001 kernel: 52 total pagecache pages
Jul 28 01:42:10 APP0001 kernel: 0 pages in swap cache
Jul 28 01:42:10 APP0001 kernel: Swap cache stats: add 4269236, delete
4269236, find 8801268/8935257
Jul 28 01:42:10 APP0001 kernel: Free swap = 0kB
Jul 28 01:42:10 APP0001 kernel: Total swap = 2104472kB
Jul 28 01:42:10 APP0001 kernel: 1048576 pages RAM
Jul 28 01:42:10 APP0001 kernel: 23652 pages reserved
Jul 28 01:42:10 APP0001 kernel: 1899 pages shared
Jul 28 01:42:10 APP0001 kernel: 1018508 pages non-shared
Jul 28 01:42:10 APP0001 kernel: Out of memory: kill process 8623 (sh)
score 48420 or a child
Jul 28 01:42:10 APP0001 kernel: Killed process 12844 (osiris.cgi)


Thanks
Niyati

[Alok Thaker]

Hi,

The problem with Linux is that by default it overcommits memory – basically when you do a malloc on linux it doesn’t reserve ane swap areay (memory + swap disk) and alway returns as successful. Then if you have couple of programs and all of them actually do want to use the memory the problem starts as system is running out of memory but there is basically no interface to tell it to applications as it already told all of them that there is enough of it… so it starts killing application in order to avoid complete lock so there oom-killer is invoked when linux overcommits the memory for the processes whereas in case of solaris that's not the case.

To prevent it you can do echo 2 > /proc/sys/vm/overcommit_memory

Thanks & Regards,
Alok

--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+un...@googlegroups.com
To post to this group, send email to VG...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

[Niyati Dave]

Hi

thanks for your response

just a few queries.. does this occur because process start utilising high memory or the number of processes running increase on the server?
I could see the above logs continuously: like at one instance it says :
 
kernel: httpd2-prefork invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
....

then after a few seconds

 kernel: osiris.cgi invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
....

Kernel: cron invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
....


How can i interprete which process is actully causing the memory issue and triggering oom-killer? cron, httpd2-prefork, sshd.. ?

Thanks again

[kandarp thaker]

Hi Niyati,

Yes it shows that apache i.e http2 consumes more memory and has a leakage https://bugzilla.redhat.com/show_bug.cgi?id=497077 this gives u more insight. Are you using rhel5. Then after some time it goes on killing other processes to free up the memory.

Thanks,
Kandarp
IT Admin-Consultant
Inferno Solutions
Go with the best
www.inferno.co.in
All Linux-Java-Web-Voip Solutions-Training under one Roof
M:-9978926593

[Niyati Dave]

Thanks again

bt this is not rhel , I have the server suselinux version 11
IN the logs the processes which invoked oom-killer are sshd, then top, httpd2-prefork and then osiris.cgi.
After a few minutes the server was normal so that is why I am not able to make out what actully may have caused the problem.

[Nirmal Pathak]

On 29 July 2011 12:37, Niyati Dave <niya...@gmail.com> wrote:

Thanks again

bt this is not rhel , I have the server suselinux version 11
IN the logs the processes which invoked oom-killer are sshd, then top, httpd2-prefork and then osiris.cgi.
After a few minutes the server was normal so that is why I am not able to make out what actully may have caused the problem.

From the oom-kill messages I can see that the system has exhausted Swap Memory

--

Jul 28 01:42:10 APP0001 kernel: Free swap  = 0kB

--

I would suggest to increase Swap Space on the system prior to setting any memory related tunable on the system.

It is difficult from logs to find exact process which triggered the oom-kill.

Out of Memory occurs when the system memory is exhausted on the system and kernel tries to recover the memory by killing various process on the system which is consuming more memory.

You have to monitor your system continuously to find the cause!
 

On Fri, Jul 29, 2011 at 12:04 PM, kandarp thaker <kandarp...@gmail.com> wrote:

Hi Niyati,

Yes it shows that apache i.e http2 consumes more memory and has a leakage https://bugzilla.redhat.com/show_bug.cgi?id=497077 this gives u more insight. Are you using rhel5. Then after some time it goes on killing other processes to free up the memory.

This does not seem to be httpd issue to me. Issue occurred as system had exhausted memory including the swap memory.

 

Thanks,
Kandarp
IT Admin-Consultant
Inferno Solutions
Go with the best
www.inferno.co.in
All Linux-Java-Web-Voip Solutions-Training under one Roof
M:-9978926593

On Fri, Jul 29, 2011 at 11:55 AM, Niyati Dave <niya...@gmail.com> wrote:

Hi

thanks for your response

just a few queries.. does this occur because process start utilising high memory or the number of processes running increase on the server?

It could be both! :)
 

I could see the above logs continuously: like at one instance it says :
 
kernel: httpd2-prefork invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
....

then after a few seconds

 kernel: osiris.cgi invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
....

Kernel: cron invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
....

As mentioned above when system is under heavy memory load, the kernel starts killing processes which are consuming high memory on the system.
 

Have FuN!
--
Nirmal D Pathak.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
What I want is freedom, not life, nor pleasure, nor good!
FOSS gives that.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[Niyati Dave]

Hello Sir,

thanks for the response and explaination:

1 query I have the physical memory : 4102 M
and swap space : 2055 M
Is it recommended to increase the swap more?


Thanks
Niyati

--

[Alok Thaker]

Hi,

It is always recommended to have swap memory as twice of physical memory as per linux fundatmentals , in your case it looks it has less swap memory as compared to physical memory.

Thanks & Regards,
Alok

[Nirmal Pathak]

On 29 July 2011 17:49, Niyati Dave <niya...@gmail.com> wrote:

Hello Sir,

thanks for the response and explaination:

1 query I have the physical memory : 4102 M
and swap space : 2055 M
Is it recommended to increase the swap more?

Yes, as Alok mentioned in old days when Physical Memory was less & costly we used to put swap as double of Physical memory.

Now days we have huge physical memory on servers like 64GB, 128GB, etc there we do not require swap double  of Physical Memory. Linux kernel hardly uses Swap space more than 20 GB, no matter how large Swap Space you keep on the servers.

[Alok Thaker]

Yes dear nirmal i.e very true..&  she is using suselinux what I found is that there is a leakage somewhere, niyati are you doing some kind of scanning on webserver like PCI or Audit scan...that can cause memory to be consumed also....

Thanks & Regards,

Alok 

[Shakthi Kannan]

Hi,

--- On Fri, Jul 29, 2011 at 5:52 PM, Alok Thaker <alok...@gmail.com> wrote:
| It is always recommended to have swap memory as twice of physical memory as
| per linux fundatmentals ,

\--

No. It was due to expensive RAM on the old architectures in the early
days. Linux memory management is quite proficient since 2.6, and this
false notion doesn't apply today. You can run GNU/Linux even without
any swap.

SK

--
Shakthi Kannan
http://www.shakthimaan.com

[Alok Thaker]

Hi,

Yes linux can be installed with or without swap but definiitely it impacts heavy applications like running big servers with big deployments of jboss,tomcat,apache,mod_jk,clustering things etc..definitely swap is helpful and here kernel version wasn't discussed but the problem of application leakage was discussed by niyati.

Thanks & Regards,
Alok

[Shakthi Kannan]

Hi,

--- On Fri, Jul 29, 2011 at 11:33 PM, Alok Thaker <alok...@gmail.com> wrote:
| and here kernel version wasn't discussed

\--

Linux is the kernel. You only gave (incorrect) information regarding
the Linux kernel which I quoted and replied. Now, you are saying
kernel version was not discussed. Read the replies, slowly, carefully
before replying.

=== QUOTED ===

On Fri, Jul 29, 2011 at 5:52 PM, Alok Thaker <alok...@gmail.com>
wrote:
| It is always recommended to have swap memory as twice of physical memory
| as per linux fundatmentals ,
\--

=== END ===

Please don't send HTML e-mail to the list. Please change your e-mail
client settings to send plain ASCII text. It is already mentioned in
the VGLUG mail footers:

http://www.catb.org/~esr/faqs/smart-questions.html#formats

[Alok Thaker]

Hi Shakti,

If you see the mail chain nirmal also recommended that in old linux kernels it was done like that and for your kind information I still use swap for my servers for improvising the performance and as i said still swap is used in 2.6 versions of kernel and latest 3.0 release too and definitely it helps.As such the person who asked the question in group, she has also made a swap partition if you look at the mail chains.

Thanks & Regards,

Alok 

[Shakthi Kannan]

Hi,

--- On Sat, Jul 30, 2011 at 11:04 AM, Alok Thaker <alok...@gmail.com> wrote:
| If you see the mail chain nirmal also recommended that in old linux kernels
| it was done like that and for your kind information I still use swap for my
| servers for improvising the performance and as i said still swap is used in
| 2.6 versions of kernel and latest 3.0 release too and definitely it helps.As
| such the person who asked the question in group, she has also made a swap
| partition if you look at the mail chains.

\--

These have nothing to do with my reply. Read the context correctly,
and re-read my reply. My reply was on the old thumb rule of using swap
as twice the RAM size.

Please don't top post in a mailing list.

[ElectroMech]

Hi,

On Fri, Jul 29, 2011 at 5:49 PM, Niyati Dave <niya...@gmail.com> wrote:

Hello Sir,

thanks for the response and explaination:

1 query I have the physical memory : 4102 M
and swap space : 2055 M
Is it recommended to increase the swap more?

Can you give us more idea what services are provided by server ?

Mostly 4 G Memory is good, still depends on what application you served. Still we can change some parameters within application ( for memory utilization) so OOM-KILLER should not occurs.
 

Thanks
Niyati

Thanks and Regards.
--
--
Nilesh Vaghela
(RHCSA RHCE)
ElectroMech Corporation
Redhat Channel Partner and Training Partner
404, Maulik arcade, Above Karnavati Pagarakha Bazar,
Nr. Mansi cross Road,
Satellite Rd, Ahmedabad
25, The Emperor, Fatehgunj, Baroda.
www.electromech.info

[Niyati Dave]

Hello all

I have read the above discussion, so I would like to clarify a few things.. the kernel is a latest one as provided by the suselinux enterprise server and version 11 - so the latest one. The server is a xen guest and applications running on the server is just apache, postfix. But yes it has a good usage of apache application and there are few cron scripts related to emailing, cgi. And as per the logs too after giving out of memory error related to swap then oom-killer started killing sh, httpd-prefork, top, the script.

Thanks again
Niyati

[Niyati Dave]

Hi all

I am continuously facing the same logs and problem again. I want to try out increasing the swap. What I know is creating a swap file or a new swap partition. Is there any way to increase the present swap which is already of 2G.
Waiting for resonse

Thanks
Niyati

[Nirmal Pathak]

On 1 August 2011 13:52, Niyati Dave <niya...@gmail.com> wrote:
> Hi all
>
> I am continuously facing the same logs and problem again. I want to try out
> increasing the swap. What I know is creating a swap file or a new swap
> partition. Is there any way to increase the present swap which is already of
> 2G.
> Waiting for resonse
>

If your existing swap filesystem is on LVM then you can extend the
Logical Volume.

If not, then you need to create a file or partition on the system and
create new swap space.
Check out this, http://www.thegeekstuff.com/2010/08/how-to-add-swap-space/

[ElectroMech]

Hi,

On Mon, Aug 1, 2011 at 1:52 PM, Niyati Dave <niya...@gmail.com> wrote:

Hi all

I am continuously facing the same logs and problem again. I want to try out increasing the swap. What I know is creating a swap file or a new swap partition. Is there any way to increase the present swap which is already of 2G.
Waiting for resonse

Can you send me the output of following command.

[root@nilesh ~]# top -b -u apache -n 1

Might suggest you soemthing valueable.

Thanks
Niyati

 --

[Niyati Dave]

Below is the output of top -b -u wwwrun -n 1

top - 12:58:50 up 130 days,  1:46,  1 user,  load average: 22.64, 22.90, 20.38
Tasks: 127 total,  19 running, 107 sleeping,   0 stopped,   1 zombie
Cpu(s):  9.7%us,  0.4%sy,  0.0%ni, 89.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.1%st
Mem:   4201096k total,  4061640k used,   139456k free,    12768k buffers
Swap:  2104472k total,  1126172k used,   978300k free,    69968k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
16057 wwwrun    20   0 97364  27m 6300 R   28  0.7   0:02.52 osiris.cgi
12918 wwwrun    20   0  154m  87m 6516 R   26  2.1   4:29.78 osiris.cgi
32376 wwwrun    20   0  839m 456m  860 R   26 11.1  29:21.22 osiris.cgi
 2111 wwwrun    20   0  778m 456m  860 R   24 11.1  25:33.46 osiris.cgi
 3033 wwwrun    20   0  757m 459m  860 R   24 11.2  24:19.25 osiris.cgi
11107 wwwrun    20   0  273m 206m 6516 R   24  5.0   8:17.03 osiris.cgi
11543 wwwrun    20   0  154m  87m 6516 R   24  2.1   7:15.97 osiris.cgi
12162 wwwrun    20   0  154m  87m 6516 R   24  2.1   5:43.74 osiris.cgi
12911 wwwrun    20   0  154m  87m 6516 R   24  2.1   4:29.74 osiris.cgi
16105 wwwrun    20   0 97336  17m 6172 R   24  0.4   0:01.20 osiris.cgi
12742 wwwrun    20   0  154m  87m 6516 R   22  2.1   4:36.37 osiris.cgi
10224 wwwrun    20   0  456m 389m 6516 R   20  9.5  11:18.80 osiris.cgi
12740 wwwrun    20   0  154m  87m 6516 R   20  2.1   3:59.65 osiris.cgi
14356 wwwrun    20   0  140m  73m 6516 R   20  1.8   1:57.56 osiris.cgi
16015 wwwrun    20   0  143m  76m 6348 R   20  1.9   0:06.58 osiris.cgi
 1870 wwwrun    20   0  773m 464m  860 R   18 11.3  25:29.98 osiris.cgi
 9615 wwwrun    20   0  540m 445m 6516 R   16 10.9  13:34.17 osiris.cgi
15622 wwwrun    20   0 91940 2784 1832 S    4  0.1   0:00.08 httpd2-prefork
  834 wwwrun    20   0 92168  352    4 S    0  0.0   0:00.30 httpd2-prefork
 1532 wwwrun    20   0 92196 2664 1492 S    0  0.1   0:00.10 httpd2-prefork
 1545 wwwrun    20   0 91940  352    4 S    0  0.0   0:00.08 httpd2-prefork
 2076 wwwrun    20   0 91940  352    4 S    0  0.0   0:00.00 httpd2-prefork
 2389 wwwrun    20   0 91924  352    4 S    0  0.0   0:03.66 httpd2-prefork
 2554 wwwrun    20   0 92024  352    4 S    0  0.0   0:00.04 httpd2-prefork
 4047 wwwrun    20   0 91928  352    4 S    0  0.0   0:00.08 httpd2-prefork
 4065 wwwrun    20   0 91928  352    4 S    0  0.0   0:00.04 httpd2-prefork
 4069 wwwrun    20   0 91936  352    4 S    0  0.0   0:00.06 httpd2-prefork
 4089 wwwrun    20   0 43608    8    4 S    0  0.0   0:00.00 osiris.cgi
 4167 wwwrun    20   0 91804  352    4 S    0  0.0   0:00.00 httpd2-prefork
 4176 wwwrun    20   0 91924  352    4 S    0  0.0   0:00.02 httpd2-prefork
 5449 wwwrun    20   0 94500  352    4 S    0  0.0   0:00.20 httpd2-prefork
 9022 wwwrun    20   0 94652 2432 1868 S    0  0.1   0:00.14 httpd2-prefork
 9221 wwwrun    20   0 92168 2740 1888 S    0  0.1   0:00.22 httpd2-prefork
 9733 wwwrun    20   0 94744 5720 1952 S    0  0.1   0:00.34 httpd2-prefork
11196 wwwrun    20   0 91940 2784 1824 S    0  0.1   0:00.06 httpd2-prefork
11612 wwwrun    20   0 92192 3184 1920 S    0  0.1   0:00.20 httpd2-prefork
12322 wwwrun    20   0 92988 3308 1432 S    0  0.1   0:00.96 httpd2-prefork
12388 wwwrun    20   0 92060 2880 1788 S    0  0.1   0:00.06 httpd2-prefork
12870 wwwrun    20   0 91804 2428 1588 S    0  0.1   0:00.00 httpd2-prefork
12881 wwwrun    20   0 91804 2388 1564 S    0  0.1   0:00.00 httpd2-prefork
13992 wwwrun    20   0 91960 3120 2016 S    0  0.1   0:00.30 httpd2-prefork
14202 wwwrun    20   0 91924 2692 1756 S    0  0.1   0:00.02 httpd2-prefork
15199 wwwrun    20   0 92168 3152 1864 S    0  0.1   0:00.10 httpd2-prefork
15300 wwwrun    20   0 92668 3524 1864 S    0  0.1   0:00.08 httpd2-prefork
15308 wwwrun    20   0 91948 2952 1872 S    0  0.1   0:00.08 httpd2-prefork
15444 wwwrun    20   0 91936 2936 1876 S    0  0.1   0:00.06 httpd2-prefork
15581 wwwrun    20   0 92112 3036 1868 S    0  0.1   0:00.08 httpd2-prefork
15617 wwwrun    20   0 92056 2916 1772 S    0  0.1   0:00.04 httpd2-prefork
15839 wwwrun    20   0 91968 2920 1844 S    0  0.1   0:00.04 httpd2-prefork
15864 wwwrun    20   0 91924 2544 1612 S    0  0.1   0:00.02 httpd2-prefork
15978 wwwrun    20   0 91940 2544 1624 S    0  0.1   0:00.00 httpd2-prefork
16001 wwwrun    20   0     0    0    0 Z    0  0.0   0:00.00 httpd2-prefork <defunct>
16017 wwwrun    20   0 91928 2684 1748 S    0  0.1   0:00.02 httpd2-prefork
31900 wwwrun    20   0 91924  352    4 S    0  0.0   0:00.04 httpd2-prefork

[ElectroMech]

Hi,

On Mon, Aug 1, 2011 at 4:23 PM, Niyati Dave <niya...@gmail.com> wrote:

Below is the output of top -b -u wwwrun -n 1

top - 12:58:50 up 130 days,  1:46,  1 user,  load average: 22.64, 22.90, 20.38
Tasks: 127 total,  19 running, 107 sleeping,   0 stopped,   1 zombie
Cpu(s):  9.7%us,  0.4%sy,  0.0%ni, 89.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.1%st
Mem:   4201096k total,  4061640k used,   139456k free,    12768k buffers
Swap:  2104472k total,  1126172k used,   978300k free,    69968k cached

Can you reduce

<IfModule prefork.c>
StartServers       8  
MinSpareServers    5
MaxSpareServers   20 <============== check this value try to reduce it to half from actual value
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000
</IfModule>



 

Thanks and Regards.

[Niyati Dave]

I have the value of MaxSpareServers 10. Should I still try changing.

[ElectroMech]

Hi,

On Mon, Aug 1, 2011 at 7:19 PM, Niyati Dave <niya...@gmail.com> wrote:

I have the value of MaxSpareServers 10. Should I still try changing.

Always check top and free -m.

Check it should not utilize swap Memory, if used should temporary.

Check also uptime for load average should not be greater than total logical cpu.

If you want change for performance should change following value without increasing usage of memory.

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers         2  
MaxClients         150
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  0
</IfModule>

Let us know the impact and check the log.
 

On Mon, Aug 1, 2011 at 7:01 PM, ElectroMech <elect...@electromech.info> wrote:

Hi,

On Mon, Aug 1, 2011 at 4:23 PM, Niyati Dave <niya...@gmail.com> wrote:

Below is the output of top -b -u wwwrun -n 1

Can you provide again the output of above ?

Remember to restart the apache.
 

top - 12:58:50 up 130 days,  1:46,  1 user,  load average: 22.64, 22.90, 20.38
Tasks: 127 total,  19 running, 107 sleeping,   0 stopped,   1 zombie
Cpu(s):  9.7%us,  0.4%sy,  0.0%ni, 89.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.1%st
Mem:   4201096k total,  4061640k used,   139456k free,    12768k buffers
Swap:  2104472k total,  1126172k used,   978300k free,    69968k cached

Can you reduce

[Niyati Dave]

Hii

This is the free -m

            total       used       free     shared    buffers     cached
Mem:          4102       4069         32          0          4         39
-/+ buffers/cache:       4025         76
Swap:         2055       2038         17

Today it has remained fully utilised- swap and also sending the output of top

top - 16:15:45 up 130 days,  5:03,  2 users,  load average: 12.20, 12.14, 13.94
Tasks: 115 total,  10 running, 105 sleeping,   0 stopped,   0 zombie
Cpu(s):  9.7%us,  0.4%sy,  0.0%ni, 89.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.1%st
Mem:   4201096k total,  4161232k used,    39864k free,    11604k buffers
Swap:  2104472k total,  2093824k used,    10648k free,    51360k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

10271 wwwrun    20   0 97740  28m 6052 R   40  0.7   0:02.64 osiris.cgi
10277 wwwrun    20   0 97376  24m 6296 R   40  0.6   0:02.06 osiris.cgi
10281 wwwrun    20   0 83064  12m 5284 R   40  0.3   0:01.30 osiris.cgi
 5729 wwwrun    20   0  378m 273m  860 R   38  6.7  10:10.27 osiris.cgi
21345 wwwrun    20   0  941m 434m  328 R   38 10.6  40:34.26 osiris.cgi
30444 wwwrun    20   0  807m 430m  860 R   38 10.5  27:06.21 osiris.cgi
14356 wwwrun    20   0  926m 529m  324 D   36 12.9  45:46.06 osiris.cgi
10264 wwwrun    20   0  124m  57m 6204 R   32  1.4   0:04.60 osiris.cgi
12162 wwwrun    20   0  943m 480m  324 R   32 11.7  49:07.30 osiris.cgi
25712 wwwrun    20   0  924m 603m  964 R   32 14.7  34:36.30 osiris.cgi
28097 wwwrun    20   0  893m 525m  964 D   32 12.8  31:20.03 osiris.cgi
 5578 wwwrun    20   0 92068  804    4 S    0  0.0   0:00.14 httpd2-prefork
 6915 wwwrun    20   0 92952 3548 1620 S    0  0.1   0:00.14 httpd2-prefork
 8131 wwwrun    20   0 94892 5752 1956 S    0  0.1   0:00.56 httpd2-prefork
 8371 wwwrun    20   0 92364 3388 1932 S    0  0.1   0:00.36 httpd2-prefork
 8644 wwwrun    20   0 92336 3404 1924 S    0  0.1   0:00.28 httpd2-prefork
 8683 wwwrun    20   0 92348 3420 1940 S    0  0.1   0:00.34 httpd2-prefork
 9093 wwwrun    20   0 94908 5676 1916 S    0  0.1   0:00.30 httpd2-prefork
 9094 wwwrun    20   0 92840 3852 1908 S    0  0.1   0:00.26 httpd2-prefork
 9352 wwwrun    20   0 92276 3248 1900 S    0  0.1   0:00.18 httpd2-prefork
 9866 wwwrun    20   0 92688 3172 1848 S    0  0.1   0:00.08 httpd2-prefork
 9982 wwwrun    20   0 92680 3548 1852 S    0  0.1   0:00.06 httpd2-prefork
10108 wwwrun    20   0 92068 2712 1620 S    0  0.1   0:00.00 httpd2-prefork
10121 wwwrun    20   0 92656 3292 1636 S    0  0.1   0:00.00 httpd2-prefork
10210 wwwrun    20   0 92068 2684 1632 S    0  0.1   0:00.00 httpd2-prefork
10268 wwwrun    20   0 97952  23m 5848 S    0  0.6   0:02.66 osiris.cgi
12322 wwwrun    20   0 92988   52    4 S    0  0.0   0:00.96 httpd2-prefork
14202 wwwrun    20   0 91924   56    4 S    0  0.0   0:00.02 httpd2-prefork
21059 wwwrun    20   0 91936   64    4 S    0  0.0   0:00.04 httpd2-prefork
24808 wwwrun    20   0 92364   84    4 S    0  0.0   0:00.20 httpd2-prefork
27685 wwwrun    20   0 94852  140    4 S    0  0.0   0:00.14 httpd2-prefork
30437 wwwrun    20   0 92068  652    4 S    0  0.0   0:00.00 httpd2-prefork
32242 wwwrun    20   0 92308  632    4 S    0  0.0   0:00.04 httpd2-prefork

[ElectroMech]

Hi,

On Mon, Aug 1, 2011 at 7:39 PM, Niyati Dave <niya...@gmail.com> wrote:

Hii

This is the free -m

            total       used       free     shared    buffers     cached
Mem:          4102       4069         32          0          4         39
-/+ buffers/cache:       4025         76
Swap:         2055       2038         17

Today it has remained fully utilised- swap and also sending the output of top

Still it seems you have around 22 apache processes it should be 11 only, if we refer last config.

Please check the setting MaxSpareServers 10 is ok and you have restarted apache.

One more observation

 

top - 16:15:45 up 130 days,  5:03,  2 users,  load average: 12.20, 12.14, 13.94
Tasks: 115 total,  10 running, 105 sleeping,   0 stopped,   0 zombie
Cpu(s):  9.7%us,  0.4%sy,  0.0%ni, 89.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.1%st
Mem:   4201096k total,  4161232k used,    39864k free,    11604k buffers
Swap:  2104472k total,  2093824k used,    10648k free,    51360k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

10271 wwwrun    20   0 97740  28m 6052 R   40  0.7   0:02.64 osiris.cgi
10277 wwwrun    20   0 97376  24m 6296 R   40  0.6   0:02.06 osiris.cgi
10281 wwwrun    20   0 83064  12m 5284 R   40  0.3   0:01.30 osiris.cgi
 5729 wwwrun    20   0  378m 273m  860 R   38  6.7  10:10.27 osiris.cgi
21345 wwwrun    20   0  941m 434m  328 R   38 10.6  40:34.26 osiris.cgi
30444 wwwrun    20   0  807m 430m  860 R   38 10.5  27:06.21 osiris.cgi
14356 wwwrun    20   0  926m 529m  324 D   36 12.9  45:46.06 osiris.cgi
10264 wwwrun    20   0  124m  57m 6204 R   32  1.4   0:04.60 osiris.cgi
12162 wwwrun    20   0  943m 480m  324 R   32 11.7  49:07.30 osiris.cgi
25712 wwwrun    20   0  924m 603m  964 R   32 14.7  34:36.30 osiris.cgi
28097 wwwrun    20   0  893m 525m  964 D   32 12.8  31:20.03 osiris.cgi

There are some problem in osiris.cgi almost consuming all memory and processor through apache.

What is the process ? please check or let us have some information.

Thanks and Regards.