Preventing swapping (high memory use)
Szabolcs Horvát
Sometimes I don't realize how much memory a computation would take up
(or I simply make a mistake, and start a calculation that requires too
much memory), and my computer starts swapping.
This makes Mathematica (and often the OS too) completely unresponsive
for a long time. In this case my only hope is that in spite of the
system unresponsiveness I can manage to kill the kernel, so that I can
at least save the notebook ...
Is there any way to prevent swapping, for example by putting a cap on
the kernel's memory use? Even if the kernel just quits when it runs out
of memory, I still have the notebook. But if the computer gets so
unresponsive that I have to restart it, then I lose the notebook too,
which is a much more serious loss ...
I tried using MemoryConstrained (with $Pre), but it seems to constrain
only the memory used by current computation, not the full kernel memory use.
To summarize: I'm simply looking for a way to prevent the computer from
locking up because of excessive swapping.
Leonid Shifrin
How about this:
In[1] =
ClearAll[totalMemoryConstrained];
SetAttributes[totalMemoryConstrained, HoldRest];
Module[{memException},
totalMemoryConstrained[max_, body_, failexpr_] :=
Catch[MemoryConstrained[body,
Evaluate[
If[# < 0, Throw[failexpr, memException], #] &@(max -
MemoryInUse[])], failexpr], memException]];
To test, I start with a fresh kernel:
In[2] = MemoryInUse[]
Out[2] = 5815968
In[3] =
n = 0;
lst = {};
totalMemoryConstrained[10000000,
For[lst = {}; n = 10000, n < 100000, n++,
lst = Join[lst, Range[n]]], Print[n]]
Out[3] = 10043 (Printed)
In[4] = MemoryInUse[]
Out[4] = 8257328
I deliberately use globals to prevent them from being garbage-collected
after the function returns.
Now take a limit smaller than the current usage:
In[5] =
n = 0;
lst = {};
totalMemoryConstrained[3000000,
For[lst = {}; n = 10000, n < 100000, n++,
lst = Join[lst, Range[n]]], Print[n]]
Out[3] = 0 (Printed)
- we just stop right away (through exception, obviously).
This of course relies on how often does MemoryConstrained[]
recompute its parameters, and how precise is MemoryInUse[].
My simple tests indicated that it works, up to plus-minus a few
Mb, but obviously more tests are needed.
Regards,
Leonid
2009/5/16 Szabolcs Horv=E1t <szho...@gmail.com>
Dr. David Kirkby
What OS ?
Szabolcs
wrote:
> Szabolcs Horv=E1t wrote:
> > Sometimes I don't realize how much memory a computation would take up
> > (or I simply make a mistake, and start a calculation that requires too
> > much memory), and my computer starts swapping.
>
> > This makes Mathematica (and often the OS too) completely unresponsive
> > for a long time. In this case my only hope is that in spite of the
> > system unresponsiveness I can manage to kill the kernel, so that I can
> > at least save the notebook ...
>
> > Is there any way to prevent swapping, for example by putting a cap on
out
> > of memory, I still have the notebook. But if the computer gets so
> > unresponsive that I have to restart it, then I lose the notebook too,
> > which is a much more serious loss ...
>
> > I tried using MemoryConstrained (with $Pre), but it seems to constrain
use.
>
> > To summarize: I'm simply looking for a way to prevent the computer f=
rom
> > locking up because of excessive swapping.
>
> What OS ?
WinXP on a laptop (i.e. slow hard drive).