Why is the swap partition set to 6GB on compute nodes?

[Joseph Dowell]

 

Former LNXI employees,

 

Here at FNMOC we’re curious about the compute node swap partition size. swap on the compute nodes is 6GB and we’re

wondering why it wasn’t originally configured to a more customary  size of 2.5 x system memory (system memory is 8GB

in these compute nodes).

 

Any feedback will be appreciated.

 

Thanks,

 

Joe Dowell

 

 

[Cameron Harr]

Joe, I wasn't involved in selecting that size, but my hunch is it was the standard set in the node image on a 4GB node - giving you 1.5x (which is often sufficient). Different images generally weren't used for larger-memory nodes. If you think you need to make it larger, you can set a new size in Clusterworx, as I'm sure you know.

[nacks]

Joe,
I would add we have been thinking of trying to get rid of swap
entirely. We find that in most cases when users go into swap the
performance has dropped so much that they are just eating up resources
and not getting much work done (in fact they are more likely to keep
using more and more memory until swap is filled as well). The trick
is that if the linux OOM killer doesn't kill off a user's application
before it kills off other critical user space processes (like
gpfs,nfs,pbs,torque, etc) then you will end up resetting the node
anyway. later versions of the linux kernel do a little to improve the
OOM killer (oom_adj and such), but its still has to be controlled with
user limits as well in most cases.

Are you actually seeing swap being used up at your site? 6GB sounds
very reasonable. You really don't want to swap at all if you can.

-Nick

[Jim Dodd]

I agree that when you fill memory with running process and start to
swap, the performance can become so bad the system can become useless.
But, I am not sure there are any real rules for the swap with modern
virtual memory systems, i.e. systems that support logical swap.
Certainly, there use to be with the older swap systems where you had
to have enough memory for all the running swap processes in addition
to any "swap" processes.

But we have found swap useful anyway. I the past, with LSF, we have
used SIGSTOP as part of our preemption scheme. This would often cause
the preempted job to swap while the preempting job ran. Then, when the
running job completes, the other job can just resume running. Since we
are now using PBS, I am not sure if this is whow it is doing
premeption, but it is still a good use for swapping.

Jim Dodd

[jhatch]

FYI, the LNXI standard setting (listed on the Integration
Questionnaire) was 1.5x the system memory. I don't know which of your
systems you are referring to but 6GB for swap on nodes with 8GB RAM
does not follow that rule (12GB). As Cameron stated, that can easily
be changed using CWX.

[Joseph Dowell]

Hi Jarom and thanks for chiming in.
I was asking about both of the LNXI systems at FNMOC.