Preventing swapping (high memory use)

[Szabolcs Horvát]

Sometimes I don't realize how much memory a computation would take up
(or I simply make a mistake, and start a calculation that requires too
much memory), and my computer starts swapping.

This makes Mathematica (and often the OS too) completely unresponsive
for a long time. In this case my only hope is that in spite of the
system unresponsiveness I can manage to kill the kernel, so that I can
at least save the notebook ...

Is there any way to prevent swapping, for example by putting a cap on
the kernel's memory use? Even if the kernel just quits when it runs out
of memory, I still have the notebook. But if the computer gets so
unresponsive that I have to restart it, then I lose the notebook too,
which is a much more serious loss ...

I tried using MemoryConstrained (with $Pre), but it seems to constrain
only the memory used by current computation, not the full kernel memory use.

To summarize: I'm simply looking for a way to prevent the computer from
locking up because of excessive swapping.

[Leonid Shifrin]

Hi Szabolcs,

How about this:

In[1] =

ClearAll[totalMemoryConstrained];
SetAttributes[totalMemoryConstrained, HoldRest];
Module[{memException},
totalMemoryConstrained[max_, body_, failexpr_] :=
Catch[MemoryConstrained[body,
Evaluate[
If[# < 0, Throw[failexpr, memException], #] &@(max -
MemoryInUse[])], failexpr], memException]];

To test, I start with a fresh kernel:

In[2] = MemoryInUse[]

Out[2] = 5815968

In[3] =

n = 0;
lst = {};
totalMemoryConstrained[10000000,
For[lst = {}; n = 10000, n < 100000, n++,
lst = Join[lst, Range[n]]], Print[n]]

Out[3] = 10043 (Printed)

In[4] = MemoryInUse[]

Out[4] = 8257328

I deliberately use globals to prevent them from being garbage-collected
after the function returns.

Now take a limit smaller than the current usage:

In[5] =
n = 0;
lst = {};
totalMemoryConstrained[3000000,
For[lst = {}; n = 10000, n < 100000, n++,
lst = Join[lst, Range[n]]], Print[n]]

Out[3] = 0 (Printed)

- we just stop right away (through exception, obviously).

This of course relies on how often does MemoryConstrained[]
recompute its parameters, and how precise is MemoryInUse[].
My simple tests indicated that it works, up to plus-minus a few
Mb, but obviously more tests are needed.

Regards,
Leonid

2009/5/16 Szabolcs Horv=E1t <szho...@gmail.com>

[Dr. David Kirkby]

What OS ?

[Szabolcs]

On May 18, 9:31 am, "Dr. David Kirkby" <david.kir...@onetel.net>
wrote:

> Szabolcs Horv=E1t wrote:
> > Sometimes I don't realize how much memory a computation would take up
> > (or I simply make a mistake, and start a calculation that requires too
> > much memory), and my computer starts swapping.
>
> > This makes Mathematica (and often the OS too) completely unresponsive
> > for a long time. In this case my only hope is that in spite of the
> > system unresponsiveness I can manage to kill the kernel, so that I can
> > at least save the notebook ...
>
> > Is there any way to prevent swapping, for example by putting a cap on

> > the kernel's memory use? Even if the kernel just quits when it runs =

out
> > of memory, I still have the notebook. But if the computer gets so
> > unresponsive that I have to restart it, then I lose the notebook too,
> > which is a much more serious loss ...
>
> > I tried using MemoryConstrained (with $Pre), but it seems to constrain

> > only the memory used by current computation, not the full kernel memory=
use.
>
> > To summarize: I'm simply looking for a way to prevent the computer f=
rom

> > locking up because of excessive swapping.
>
> What OS ?

WinXP on a laptop (i.e. slow hard drive).