Re: memory fragmentation
Massimo Rosen
in addition to Hamish's answer, your maximum NLM memory listed in your
segstats is enormous (almsot 2GB). This tells me there's some module
running wild temporarily, and in theory, the only possible (and
occasionally known for such issues) offender here is indeed SAV. Also,
SAV is also with a very high probability the offender causing your
memory fragmentation. Of course, simply running without it for a while
could prove that. But I'm almost certain that your "Cache memory
allocator" messages also point to SAV, right?
Lemeunier wrote:
>
> Our Netware Servers are 6 cluster nodes: Netware 6.5 SP 8, Cluster 1.8.
> We are running Symantec Antivirus Corporate Edition 10.1.9 and Tivoli
> Storage Manager 5.5.2.6.
>
> Servers are suffering from memory fragmentaton. We have to reboot them
> periodically. They show errors like
> "Cache memory allocator out of available memory" and
> "Short term memory allocator is out of memory".
> The opinion of Symantec Support is that SAV is not the cause of memory
> fragmentation.
> We already limited TSAFS in the amount of cache it requests (Load TSAFS
> /CacheMemoryThreshold=1). We also set a hard limit on the amount of RAM
> that DS.NLM uses.
>
> What can be the cause for the problem?
> Attached you will find the segstats.txt of one of the servers.
>
> Thank you for all hints!
> Ursula
>
> +----------------------------------------------------------------------+
> |Filename: cs07_segstats.txt |
> |Download: http://forums.novell.com/attachment.php?attachmentid=4588 |
> +----------------------------------------------------------------------+
>
> --
> Lemeunier
> ------------------------------------------------------------------------
> Lemeunier's Profile: http://forums.novell.com/member.php?userid=10807
> View this thread: http://forums.novell.com/showthread.php?t=415290
--
Massimo Rosen
Novell Product Support Forum Sysop
No emails please!
http://www.cfc-it.de
Haitch
>
> Our Netware Servers are 6 cluster nodes: Netware 6.5 SP 8, Cluster 1.8.
> We are running Symantec Antivirus Corporate Edition 10.1.9 and Tivoli
> Storage Manager 5.5.2.6.
>
> Servers are suffering from memory fragmentaton. We have to reboot them
> periodically. They show errors like
> "Cache memory allocator out of available memory" and
> "Short term memory allocator is out of memory".
You "File Cache Maximum Size" has been "tuned" to it's minimum possible
value. Reset it back to default by doing:
set File Cache Maximum Size = 2147483648
A reboot is required after setting it.
H.
Massimo Rosen
Lemeunier wrote:
>
> Hi,
>
> yesterday I changed the file cache maximum size to 2147483648 and put
> the statement in startup.ncf. Even without reboot total cache buffers
> shown in monitor were directly adjusted upwards.
> Today in the morning the file cache maximum size was reset to
> 1073741825 (autotuned by the server?),
Yes, and if a server autotunes itself in that way, it is an indicator of
a real problem. The "tuning" in Netware is nothing but a short-term
delay for the real problem, coupled with a permanent setting that kills
server performance once the real problem behind it is fixed.
> total cache buffers shown in
> monitor were set as before and 2 of the servers are showing the message
> "short term memory allocator is out of memory".
And the rest of the messages please?
> Do I have to turn off autotuning?
Personally, I do that (based on above comment) on all servers I support,
yes. BUt in your case, it wouldn't solve the problem.
> Should I use the server -u
> parameter?
Not really, no. Lowering it *may* (unlikely) delay the occurence of the
fragmentation by a few minutes to maybe a day. SP8 servers don't need
this type of tuning anymore.
> In NRM, "View Memory Config", "Tune Logical Address Space" the
> recommended settings are File Cache Maximum Size = 1,073,741,825 and
> server -u403000000.
That feature in NORM is outdated and useless/counter-productive in
currently patched servers.
> This is what the support of Symantec told us: "The way antivirus
> utilities operate, elements of the NLM must be loaded dynamically
> depending on file access; this will trigger precisely this type of
> "unexpected spike" (although it is a transient spike, and relatively
> small in scale). As such, modifying the User memory space is crucial,
> even when troubleshooting kernel memory fragmentation and exhaustion. "
Nonsense, as SAV doesn't use User memory space, but kernel memory. And
User memory space is entirely virtual, and doesn't affect kernel memory,
especially not when it's not used.
> Thanks for your suggestions,
Post the real, verbatim out of memory messages, including the requestor,
and the size of the request then we'll see. Of most interest is the
first occurence of the error after a server reboot.
CU,
Massimo Rosen
Lemeunier wrote:
>
> Hi Massimo,
>
> here is the rest of the message, which is shown the first time after
> the reboot (26.07.) on saturday, while SAV does its weekly scan of the
> volumes:
>
> 4.07.2010 1.04.51 : SERVER-5.70-0 [nmID=6001D]
> Cache memory allocator out of available memory.
>
> 4.07.2010 1.04.51 : SERVER-5.70-0 [nmID=2000A]
> Short term memory allocator is out of memory.
> 1 attempts to get more memory failed.
> request size in bytes 26214408 from Module RTVSCAN.NLM
SAV is the offender/problem. No doubt about it.
> Sometimes also DSMC shows this message while backup, but more rarely.
> Also size it requires is smaller (65 KB).
Right. This would never become a problem, if not SAV would fragment the
memory to hell first due to it's unreasonably sized memory requests of
single, comparably huge chunks.
At the point where mere memory requests of 64kb fail, the server is
essentially already dead, and these errors don't need to be taken into
account. They're just results of the memory fragmentation, but not the
cause. SAV is.