Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Preventing swapping (high memory use)

54 views
Skip to first unread message

Szabolcs Horvát

unread,
May 16, 2009, 5:19:33 AM5/16/09
to

Sometimes I don't realize how much memory a computation would take up
(or I simply make a mistake, and start a calculation that requires too
much memory), and my computer starts swapping.

This makes Mathematica (and often the OS too) completely unresponsive
for a long time. In this case my only hope is that in spite of the
system unresponsiveness I can manage to kill the kernel, so that I can
at least save the notebook ...

Is there any way to prevent swapping, for example by putting a cap on
the kernel's memory use? Even if the kernel just quits when it runs out
of memory, I still have the notebook. But if the computer gets so
unresponsive that I have to restart it, then I lose the notebook too,
which is a much more serious loss ...

I tried using MemoryConstrained (with $Pre), but it seems to constrain
only the memory used by current computation, not the full kernel memory use.


To summarize: I'm simply looking for a way to prevent the computer from
locking up because of excessive swapping.

Leonid Shifrin

unread,
May 16, 2009, 6:24:21 PM5/16/09
to
Hi Szabolcs,

How about this:

In[1] =

ClearAll[totalMemoryConstrained];
SetAttributes[totalMemoryConstrained, HoldRest];
Module[{memException},
totalMemoryConstrained[max_, body_, failexpr_] :=
Catch[MemoryConstrained[body,
Evaluate[
If[# < 0, Throw[failexpr, memException], #] &@(max -
MemoryInUse[])], failexpr], memException]];

To test, I start with a fresh kernel:

In[2] = MemoryInUse[]

Out[2] = 5815968

In[3] =

n = 0;
lst = {};
totalMemoryConstrained[10000000,
For[lst = {}; n = 10000, n < 100000, n++,
lst = Join[lst, Range[n]]], Print[n]]

Out[3] = 10043 (Printed)

In[4] = MemoryInUse[]

Out[4] = 8257328

I deliberately use globals to prevent them from being garbage-collected
after the function returns.

Now take a limit smaller than the current usage:

In[5] =
n = 0;
lst = {};
totalMemoryConstrained[3000000,
For[lst = {}; n = 10000, n < 100000, n++,
lst = Join[lst, Range[n]]], Print[n]]

Out[3] = 0 (Printed)

- we just stop right away (through exception, obviously).

This of course relies on how often does MemoryConstrained[]
recompute its parameters, and how precise is MemoryInUse[].
My simple tests indicated that it works, up to plus-minus a few
Mb, but obviously more tests are needed.


Regards,
Leonid


2009/5/16 Szabolcs Horv=E1t <szho...@gmail.com>

Dr. David Kirkby

unread,
May 18, 2009, 2:31:30 AM5/18/09
to

What OS ?


Szabolcs

unread,
May 18, 2009, 6:17:59 AM5/18/09
to
On May 18, 9:31 am, "Dr. David Kirkby" <david.kir...@onetel.net>
wrote:

> Szabolcs Horv=E1t wrote:
> > Sometimes I don't realize how much memory a computation would take up
> > (or I simply make a mistake, and start a calculation that requires too
> > much memory), and my computer starts swapping.
>
> > This makes Mathematica (and often the OS too) completely unresponsive
> > for a long time. In this case my only hope is that in spite of the
> > system unresponsiveness I can manage to kill the kernel, so that I can
> > at least save the notebook ...
>
> > Is there any way to prevent swapping, for example by putting a cap on
> > the kernel's memory use? Even if the kernel just quits when it runs =

out
> > of memory, I still have the notebook. But if the computer gets so
> > unresponsive that I have to restart it, then I lose the notebook too,
> > which is a much more serious loss ...
>
> > I tried using MemoryConstrained (with $Pre), but it seems to constrain
> > only the memory used by current computation, not the full kernel memory=
use.
>
> > To summarize: I'm simply looking for a way to prevent the computer f=
rom

> > locking up because of excessive swapping.
>
> What OS ?

WinXP on a laptop (i.e. slow hard drive).

0 new messages