Re: [AOLSERVER] More on lost of memory of aolserver process

[Fenton, Brian]

On Sun, 16 Oct 2005 05:21:17 -0700
Gustaf Neumann wrote:
> i have written the following script to help to find "application level leaks"

and later
> If the numbers don't change significantly, it is highly likely that
> there is a c-level memleak involved and
> it is not an application problem (oacs, dotlrn).

Hello everyone,

I hope nobody minds me bumping this old thread. Gustaf, I used your script on a production server that appears to be leaking memory. Can you please help me interpret it?

This was the total at startup
TOTAL: pid 10686 vsz 592489 namespaces 543 vars 96 var_bytes 10118 array_elements 313 cmds 266 procs 4511

then just before the crash
TOTAL: pid 10686 vsz 1112509 namespaces 543 vars 86 var_bytes 8771 array_elements 256 cmds 266 procs 4469
unable to alloc 495960171 bytes

Would you consider the less than double "vsz" figure a significant change? Is it of any significance that the other values decreased (vars var_bytes etc)?
So would this suggest a C-level or application-level problem?

The strange thing about this particular case is that the developers tell me that "nothing changed" on this server in quite some time. And yet it mysteriously started crashing last week.

many thanks for any assistance

Brian Fenton

--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <list...@listserv.aol.com> with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.

[Gustaf Neumann]

Hi Brian,

Fenton, Brian schrieb:

> I hope nobody minds me bumping this old thread. Gustaf, I used your script on a production server that appears to be leaking memory. Can you please help me interpret it?
>
> This was the total at startup
> TOTAL: pid 10686 vsz 592489 namespaces 543 vars 96 var_bytes 10118 array_elements 313 cmds 266 procs 4511
>
> then just before the crash
> TOTAL: pid 10686 vsz 1112509 namespaces 543 vars 86 var_bytes 8771 array_elements 256 cmds 266 procs 4469
> unable to alloc 495960171 bytes
>

well the script was written to detect, if someone adds variables into
namespaces, or
appends permanently to variables, which are not reclaimed during cleanup
of a
connection. This is what i called "application level leaks".

in your case, this does not seem to be the case (no additional name
spaces or vars, all figures
are rather less than at start-time).

The growth in vsz is not good (but not unusual) but the size
of the alloc is something to worry.

Growth of vsz might be due to fragmentation of memory (zippy is optimized
for minimal locks, not minimal memory footprint) or due to the number
of threads you are using (maybe there were less threads at startup time
than on crash time?)

> Would you consider the less than double "vsz" figure a significant change? Is it of any significance that the other values decreased (vars var_bytes etc)?
> So would this suggest a C-level or application-level problem?
>

The main question is: why does it try to allocate 500MB?
what was the last request?
do you use some image libraries?

> The strange thing about this particular case is that the developers tell me that "nothing changed" on this server in quite some time. And yet it mysteriously started crashing last week.
>

if the allocation of 500MB sounds for your apps unusual to you,
i would suggest to try to figure out, what happens here in the last
request in deail.

Hope this helps

-gustaf neumann

[Dossy Shiobara]

On 2007.10.05, Fenton, Brian <Brian....@QUEST.IE> wrote:
> unable to alloc 495960171 bytes

What does your application do that it tries to request 495MB of memory
in one shot?

I smell an [exec] ...

-- Dossy

--
Dossy Shiobara | do...@panoptic.com | http://dossy.org/
Panoptic Computer Network | http://panoptic.com/
"He realized the fastest way to change is to laugh at your own
folly -- then you can let go and quickly move on." (p. 70)

[Fenton, Brian]

Thanks very much for the replies. The developers reported back that the crashes were caused by a report getting repeatedly called that was returning massive amounts of data to the browser. Sounds a bit odd to me, but I don't have any more data at the moment. When I get more, I'll report back here.

Many thanks again for the assistance. It's much appreciated. It's comforting to know there's such capable support here on the mailing list.

Brian

[Gustaf Neumann]

Hi Brian,

Fenton, Brian schrieb:

> Thanks very much for the replies. The developers reported back that the crashes were caused by a report getting repeatedly called that was returning massive amounts of data to the browser. Sounds a bit odd to me, but I don't have any more data at the moment. When I get more, I'll report back here.
>

if this large amount of data is not a bug, you will run always into
such memory problems, since all aolserver threads run in the same
(limited) address space of a single process. I would suggest to
considerate the following options:
- spool data to a file and return the file
- use HTTP streaming via ns_write
- move to 64bit

[Fenton, Brian]

Hi Gustaf,

The developers have acknowledged that this is a bug, so they are changing their code. However thanks for the good suggestions! :-)

Brian

-----Original Message-----
From: AOLserver Discussion [mailto:AOLS...@LISTSERV.AOL.COM] On Behalf Of Gustaf Neumann
Sent: 08 October 2007 13:04
To: AOLS...@LISTSERV.AOL.COM
Subject: Re: [AOLSERVER] More on lost of memory of aolserver process

[Tom Jackson]

So if the issue isn't that you have only 500meg of memory, but that, for
instance, a user got tired of waiting and kept hitting 'reload', or even that
a bunch of users wanted the massive report at the same time, you can use a
threadpool with a very limited number of threads...max you want simultaneous
download. You could still get more requests at once, but they will queue up
and wait. Then if they hit reload, no cost.

If this is an autogenerated report, you could also save it as a file. That
might help by using fastpath as a cache, and mem map on unix like machines.
Even though AOLserver will stop sending if the client quits or reloads, you
may have a backend process which keeps on going.

tom jackson

On Monday 08 October 2007 06:22, Fenton, Brian wrote:
> Hi Gustaf,
>
> The developers have acknowledged that this is a bug, so they are changing
> their code. However thanks for the good suggestions! :-)

[Fenton, Brian]

Great suggestion Tom! Will pass it on to the team.

Thanks
Brian

-----Original Message-----
From: AOLserver Discussion [mailto:AOLS...@LISTSERV.AOL.COM] On Behalf Of Tom Jackson
Sent: 08 October 2007 14:59
To: AOLS...@LISTSERV.AOL.COM
Subject: Re: [AOLSERVER] More on lost of memory of aolserver process