huge virtual memory size when launching 7.3

[Thierry Dumont]

I have two computers, and sage installed on both:

1) Ubuntu 12.04 , sage-7.3 on an nfs mount,y

2) Ubuntu 16.04, sage-7.3 on a local system, on a ssd.

With the first one, sage starts slowly (as could be expected!), and I
have time to look at sage starting with "top". And during the starting
phase, before the prompt, sage uses more than 16 GB of virtual memory
(VIRT: 16.380g); the same quantity is used when stopping sage (in the
console interface).
It seems quite large, no ?

With the second machine, it's more difficult to see what happens with
top, but it does not seems to use more than 7gb.

Why such a large amount of memory ? (ok, it's virtual, but it's large,
no ?).

t.d.

[William Stein]

I think this is a hack that is used by PARI/GP, due to their
"interesting" primitive (but fast!) memory design.

-- William

>
> t.d.
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups "sage-support" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-support...@googlegroups.com.
> To post to this group, send email to sage-s...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-support.
> For more options, visit https://groups.google.com/d/optout.



--
William (http://wstein.org)

[Thierry Dumont]

Le 30/08/2016 à 19:06, William Stein a écrit :
> On Tue, Aug 30, 2016 at 9:56 AM, Thierry Dumont
> <tdu...@math.univ-lyon1.fr> wrote:
>> I have two computers, and sage installed on both:
>>
>> 1) Ubuntu 12.04 , sage-7.3 on an nfs mount,y
>>
>> 2) Ubuntu 16.04, sage-7.3 on a local system, on a ssd.
>>
>> With the first one, sage starts slowly (as could be expected!), and I
>> have time to look at sage starting with "top". And during the starting
>> phase, before the prompt, sage uses more than 16 GB of virtual memory
>> (VIRT: 16.380g); the same quantity is used when stopping sage (in the
>> console interface).
>> It seems quite large, no ?
>>
>> With the second machine, it's more difficult to see what happens with
>> top, but it does not seems to use more than 7gb.
>>
>> Why such a large amount of memory ? (ok, it's virtual, but it's large,
>> no ?).
>
> I think this is a hack that is used by PARI/GP, due to their
> "interesting" primitive (but fast!) memory design.
>
> -- William
>
>>
>> t.d.
>>

Mhhh, yes.And they certainly do something with what they allocate; I
tried this (c++):

-------------------------------
#define DO_SOMETHING
int main()
{
//allocate a 16 gb array.
const long int n=2*1000000000;
long int *x=new long int[n];
#ifdef DO_SOMETHING
for(long int i=0;i<n;i++)
x[i]=0;
#endif
}
------------------------------

With DO_SOMETHING defined, time ./a.out:
real 0m7.352s
user 0m2.611s
sys 0m4.648s

and DO_SOMETHING undefined (as just allocating has quite no cost):
real 0m0.004s
user 0m0.004s
sys 0m0.000s

t.

[leif]

William Stein wrote:
> On Tue, Aug 30, 2016 at 9:56 AM, Thierry Dumont
> <tdu...@math.univ-lyon1.fr> wrote:
>> I have two computers, and sage installed on both:
>>
>> 1) Ubuntu 12.04 , sage-7.3 on an nfs mount,y
>>
>> 2) Ubuntu 16.04, sage-7.3 on a local system, on a ssd.
>>
>> With the first one, sage starts slowly (as could be expected!), and I
>> have time to look at sage starting with "top". And during the starting
>> phase, before the prompt, sage uses more than 16 GB of virtual memory
>> (VIRT: 16.380g); the same quantity is used when stopping sage (in the
>> console interface).
>> It seems quite large, no ?
>>
>> With the second machine, it's more difficult to see what happens with
>> top, but it does not seems to use more than 7gb.
>>
>> Why such a large amount of memory ? (ok, it's virtual, but it's large,
>> no ?).

IMHO horrible (as *each* Sage subprocess is claiming that amount of
memory, here usually ~28 to 50+ GB IIRC, as it depends on the physical
memory installed) and dangerous, as this makes the usage of 'ulimit -v'
nearly impossible and in practice requires to allow vm-overcommit. With
the latter, it's presumably quite easy to effectively crash many (if not
most) machines, be it by just some bug in Sage, but also by a malicious
user of course. (I.e., at least the VM a Sage server / installation may
run in.)

It also makes it more complicated to see how much memory a Sage process
actually *uses*.

> I think this is a hack that is used by PARI/GP, due to their
> "interesting" primitive (but fast!) memory design.

And GAP / libgap. In this recent case I guess the latter is (once
again) to blame.


I was quite surprised apparently nobody complained, but perhaps not many
at all noticed.


-leif

[Volker Braun]

On Tuesday, August 30, 2016 at 8:26:29 PM UTC+2, leif wrote:

IMHO horrible (as *each* Sage subprocess is claiming that amount of
memory, here usually ~28 to 50+ GB IIRC, as it depends on the physical
memory installed) and dangerous, as this makes the usage of 'ulimit -v'
nearly impossible

ulimit -v is useless anyways

 

and in practice requires to allow vm-overcommit.

In the default mode (overcommit_memory==0) calls to mmap with MAP_NORESERVE are not checked, so you don't have to do anything special. Really, restricting virtual memory on a 64-bit system is only for specialist applications.

 

[Nils Bruin]

On Tuesday, August 30, 2016 at 1:23:47 PM UTC-7, Volker Braun wrote:

ulimit -v is useless anyways

In my experience it is quite useful on multi-user machines to limit the effects of runaway computer algebra processes. Many packages (maple, magma at least) will segfault earlier with a ulimit -v set than without. With a properly set limit, this will often prevent the machine from thrashing the swap (which tends to affect all users). My impression was that sage benefitted from similar protection.
 

 

and in practice requires to allow vm-overcommit.

In the default mode (overcommit_memory==0) calls to mmap with MAP_NORESERVE are not checked, so you don't have to do anything special. Really, restricting virtual memory on a 64-bit system is only for specialist applications.

Isn't that exactly what makes the above use of ulimit -v useful? Presumably, once pages DO get allocated, the operating system will check that the total virtual address space that is actually allocated (to swap or memory) fits within the "ulimit -v" bound. In that case we don't particularly need to care about the huge mappings that sage seems to request either.

In any case, I hope we can keep sage's memory management configured in such a way that one can set standard operating system limits in order to ensure that overeager memory-consuming processes will crash before they start affecting other processes too much.

[leif]

Nope, GAP / libgap by default "only" contribute about 10% to the (at
least!) 48 GB claimed by *every* Sage process here, while indeed the
PARI library in Sage takes /one fourth/ of

sage.misc.memory_info.MemoryInfo().virtual_memory_limit()

which is *total* (not free!) physical RAM + total (not free) swap space,
in bytes.

That's IMHO a regression, but dates back quite a couple of (beta)
releases AFAIK. (The last *commit* regarding that is from January this
year, don't know when it got merged.)


-leif

P.S.: The change regarding the PARI library was made in #19883;
according to the ticket's /milestone/ (which doesn't say much) it
presumably got merged into Sage 7.1.

[William Stein]

On Tue, Aug 30, 2016 at 2:02 PM, Nils Bruin <nbr...@sfu.ca> wrote:
> On Tuesday, August 30, 2016 at 1:23:47 PM UTC-7, Volker Braun wrote:
>>
>>
>> ulimit -v is useless anyways
>
>
> In my experience it is quite useful on multi-user machines to limit the
> effects of runaway computer algebra processes. Many packages (maple,

More modern approaches involving containers and cgroups are much, much
better for solving this sort of problem...

--
William (http://wstein.org)

[Volker Braun]

On Tuesday, August 30, 2016 at 11:03:39 PM UTC+2, leif wrote:

48 GB claimed by *every* Sage process here

Its a tiny fraction of the 128TB address space. We are not talking about used RAM here.

which is *total* (not free!) physical RAM + total (not free) swap space,
in bytes.

Really, you need to take enough address space so that any computation that you can possibly do isn't artificially limited by the address space.

[Jeroen Demeyer]

On 2016-08-30 20:26, leif wrote:
> this makes the usage of 'ulimit -v' nearly impossible

I don't see why. You can still use 'ulimit -v'.

[Thierry Dumont]

> --

Testing :
x=sage.misc.memory_info.MemoryInfo().virtual_memory_limit()/1024.^3
on very different machines:

1) My own computer, 8GB RAM + swap: x: 23.1133804321289 GB

2) a machine with, 96 GB Ram + swap: x: 153.79 GB

3) My raspberry pi3 :-) 1GB, No swap : x = 0.939544677734375 GB

On all these machines, sage starts gently, even if slowly (as expected)
on machine 2 (uses nfs), and 3.

On machine 2, sage stating process (as mesured with top) uses about 39GB
(no more...) out of 153.79 available.

t.

[Jeroen Demeyer]

On 2016-08-31 08:57, Thierry Dumont wrote:
> On machine 2, sage stating process (as mesured with top) uses about 39GB
> (no more...) out of 153.79 available.

It doesn't actually "use" that memory. It is reserved virtual memory but
it does not take up physical memory or swap space.

[Jonathan Bober]

I just ran into this problem on a machine with 512 GB of ram, where the Sage virtual memory usage is around 128 GB. Volker might consider that a tiny fraction of the possible address space, but it still seems a bit ridiculous. I noticed this when I tried to use @parallel to 64 processes. (overcommit_memory is set to 2 on this machine.) I could @parallel(2) but not @parallel(3).

I can change the overcommit settings, but not everyone is so lucky. I haven't actually tried (I'm just using an old version of sage for now) and I don't actually know what the default overcommit policy would do with this. 8 TB is not such a small fraction of the possible address space anymore. Also, there could be situations where I want to fork significantly more processes than the number of cpus. Also, 2 TB of ram is not so unreasonable these days (is PARI going allocate 500 GB?) and neither is > 64 cores, so this seems like it could be problematic for some users, and, in any case, it is annoying.

--
You received this message because you are subscribed to the Google Groups "sage-support" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sage-support+unsubscribe@googlegroups.com.

[Jeroen Demeyer]

Why don't you use ulimit -v to limit the per-process available memory?
That would make sense when starting lots of processes even without the
PARI non-issue.

[Jonathan Bober]

On Mon, Sep 19, 2016 at 8:29 AM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:

Why don't you use ulimit -v to limit the per-process available memory? That would make sense when starting lots of processes even without the PARI non-issue.

Because I was being stupid. That seems like it should work ok.

But in that case the memory does effectively get "used", if it is the case that a quarter of the machine's memory is only available from the PARI memory allocater. If I'm not using the PARI parts of Sage, is that memory completely wasted?

[Jeroen Demeyer]

On 2016-09-19 11:54, Jonathan Bober wrote:
> But in that case the memory does effectively get "used", if it is the
> case that a quarter of the machine's memory is only available from the
> PARI memory allocater. If I'm not using the PARI parts of Sage, is that
> memory completely wasted?

There should not be any consequences apart from the artificially high
value of virtual memory used. It certainly does not affect the physical
memory or swap space which is effectively available.

So the question becomes: what uses the "virtual memory used" value?
Apparently, the Linux overcommit thing does use that value (which is
surprising to me, I never really understood how this overcommit stuff
works).


Jeroen.

[Jonathan Bober]

If I ulimit -v 8 GB, say, (which is 512/64), and the PARI allocater immediately grabs 2 GB of the virtual address space for itself, then that seems like it leaves only 6 GB for malloc/sage_malloc/whatever else, which would be effectively limiting the physical memory available in some situations.

This specific machine has overcommit_memory set to 2, which means that the total amount of non-file-backed writable virtual memory that the kernel will allocate is fixed (probably to somewhere around the physical size of RAM). This is not the default, and you probably wouldn't want to run your laptop this way, but is not so unreasonable for a shared computer with a lot of RAM, where it usually doesn't cause problems. (It's a setting that might get changed from the default after the first or second time someone crashes the machine by using too much memory, especially if this happens on a Friday afternoon and leaves the machine unavailable all weekend...) 

When it does cause problems, it's usually because a process with a lot of writable virtual memory allocated to it forks, as is happening here.

[Jeroen Demeyer]

On 2016-09-19 13:02, Jonathan Bober wrote:
> If I ulimit -v 8 GB, say, (which is 512/64), and the PARI allocater
> immediately grabs 2 GB of the virtual address space for itself, then
> that seems like it leaves only 6 GB for malloc/sage_malloc/whatever
> else, which would be effectively limiting the physical memory available
> in some situations.

Then use a larger value for "ulimit -v" or add some swap space.

> This is not the default, and you probably wouldn't want to
> run your laptop this way

But it is an advanced setting. Given that, I assume that you are clever
enough to do the right thing.

[Jonathan Bober]

On Mon, Sep 19, 2016 at 12:13 PM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:

On 2016-09-19 13:02, Jonathan Bober wrote:

If I ulimit -v 8 GB, say, (which is 512/64), and the PARI allocater
immediately grabs 2 GB of the virtual address space for itself, then
that seems like it leaves only 6 GB for malloc/sage_malloc/whatever
else, which would be effectively limiting the physical memory available
in some situations.

Then use a larger value for "ulimit -v" or add some swap space.

That doesn't really solve the problem, which I phrased poorly. The machine has an (antiquated) setup where, when a process requests X bytes of memory, the kernel reserves X bytes of physical memory for it. This is usually fine because users usually use the memory they request, and when they don't, well, there are 512 GB of it. But it does mean that every virtual memory allocation has a physical memory cost. (It also has benefits, like when someone screws up they usually notice immediately, instead of when the OOM killer kills sshd or the login process.)

 

This is not the default, and you probably wouldn't want to
run your laptop this way

But it is an advanced setting. Given that, I assume that you are clever enough to do the right thing.

With overcommit_memory set to 2, I'm not sure that there is a right thing to do. If I'm the only person in the world with this problem, then I should keep my mouth shut and get with the times, but I don't think this setup is so unreasonable.

Anyway, where exactly is this allocation coming from? Is it a default PARI setting, or does it come from the way that Sage uses PARI? (I mean, to whom should I address my "hate mail"? :)

[William Stein]

Anyway, where exactly is this allocation coming from? Is it a default PARI setting, or does it come from the way that Sage uses PARI? (I mean, to whom should I address my "hate mail"? :)

I think it is a problem in the way that Sage uses PARI.  I just tried "sage -sh" with sage-7.3, then ran gp from the command line, and it uses very little memory.    Please continue to send "hate mail"...

We should revert whatever trac ticket did this, or at least make it an option to not allocate such a large address space. This huge vm allocation is clearly an even worse hack dealing with PARI's very hackish "memory management".

Even back in 2005 when I was writing the original libpari hackish code, I didn't seriously consider this "allocate huge vm" hack to deal with how pari works, since that was even too hackish for me.  Making it the default is surprising.

 

--

William (http://wstein.org)

[Jeroen Demeyer]

On 2016-09-19 14:04, Jonathan Bober wrote:
> That doesn't really solve the problem, which I phrased poorly. The
> machine has an (antiquated) setup where, when a process requests X bytes
> of memory, the kernel reserves X bytes of physical memory for it.

Well, in that case you would indeed "lose" 1/4 of your physical memory
using the defaults that Sage uses for the PARI virtual stack. If you are
unwilling to change your OS settings, then you can still manually change
the PARi stack size with

from sage.libs.pari.all import pari
pari.allocatemem(4*10^6, 4*10^6)

> Anyway, where exactly is this allocation coming from? Is it a default
> PARI setting, or does it come from the way that Sage uses PARI?

It is because of the way how Sage uses PARI.

> I mean, to whom should I address my "hate mail"? :)

That would be me...

Jeroen.


PS: The Sage <-> GAP interface has exactly the same problem, but with a
smaller default memory allocation.

[Jeroen Demeyer]

On 2016-09-19 19:01, William Stein wrote:
> We should revert whatever trac ticket did this

The first version of this is from
https://trac.sagemath.org/ticket/19883

> or at least make it an option to not allocate such a large address space.

That's easy to do. How would you see the user interface for this?

> This huge vm
> allocation is clearly an even worse hack dealing with PARI's very
> hackish "memory management".

It is really not so bad as it looks. Let me recall that this is only
about *virtual memory*. Normally, there are no consequences to
allocating a huge amount of virtual memory. Apparently, there are
unusual Linux settings in which it does cause problems. Unfortunately, I
don't see an easy solution for this.

> Making it the default is surprising.

I am open for suggestions, what would you make the default then?

[Jeroen Demeyer]

On 2016-09-19 14:04, Jonathan Bober wrote:

> With overcommit_memory set to 2, I'm not sure that there is a right
> thing to do. If I'm the only person in the world with this problem, then
> I should keep my mouth shut and get with the times, but I don't think
> this setup is so unreasonable.

Some pointers indicating that the Linux implementation of
overcommit_memory=2 has issues:
* https://lkml.org/lkml/2005/8/8/46
* https://lwn.net/Articles/627557/

[Jeroen Demeyer]

On 2016-09-19 19:58, Jeroen Demeyer wrote:
> * https://lwn.net/Articles/627557/

This last page indicates that one should use PROT_NONE to work around
the issue you are having.

Note this nice quote:

> Sadly, the commit charge implications of MAP_NORESERVE are documented
but silently broken, but the commit charge implications of PROT_NONE are
undocumented and in theory mutable in future releases.

[Jeroen Demeyer]

After spending some time reading on the subject, I think I might have a
solution to this "problem". It involves calling mmap() with PROT_NONE,
which will require a patch to upstream PARI.

However, before implementing this, I would like a *strong commitment*
from somebody to review my patch when I'm finished. Otherwise I'm just
wasting my time.

[Jonathan Bober]

From reading what you've sent, I guess that what you have in mind is calling mmap with PROT_NONE and then calling mprotect() to change that to read/write whenever growing the stack? That seems like it might be a reasonable thing to do (though I'm only basing that on spending a few minutes reading what you sent, not from any actual knowledge that I had before).

I'm willing to devote some time (not today) to figuring out what the right thing to do is (maybe the above is already the right thing) / implementing this / reviewing this.

On Mon, Sep 19, 2016 at 7:31 PM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:

After spending some time reading on the subject, I think I might have a solution to this "problem". It involves calling mmap() with PROT_NONE, which will require a patch to upstream PARI.

However, before implementing this, I would like a *strong commitment* from somebody to review my patch when I'm finished. Otherwise I'm just wasting my time.

[Jeroen Demeyer]

On 2016-09-20 12:54, Jonathan Bober wrote:
> From reading what you've sent, I guess that what you have in mind is
> calling mmap with PROT_NONE and then calling mprotect() to change that
> to read/write whenever growing the stack? That seems like it might be a
> reasonable thing to do (though I'm only basing that on spending a few
> minutes reading what you sent, not from any actual knowledge that I had
> before).

Yes, that is my idea.

> I'm willing to devote some time (not today) to figuring out what the
> right thing to do is (maybe the above is already the right thing) /
> implementing this / reviewing this.

I don't mind implementing it. What I *do* mind is that I implement it
and that the patch rots away on Sage Trac in needs_review state (highly
specialized patches like these have a higher chance of that). That's why
I asked for a commitment to review.

[Jonathan Bober]

I moved discussion to sage-devel, but wanted to add for anyone who comes across this and has problems: my simple temporary workaround is to change the line

paristack_setsize(size, mem.virtual_memory_limit() // 4)

to

paristack_setsize(size, min(mem.virtual_memory_limit() // 4, 1000000000))

in the file src/sage/libs/pari/pari_instance.pyx. (And then run sage -b.)

[Jeroen Demeyer]

On 2016-09-21 16:19, Jonathan Bober wrote:
> I moved discussion to sage-devel, but wanted to add for anyone who comes
> across this and has problems: my simple temporary workaround is to
> change the line
>
> paristack_setsize(size, mem.virtual_memory_limit() // 4)
>
> to
>
> paristack_setsize(size, min(mem.virtual_memory_limit() // 4, 1000000000))
>
> in the file src/sage/libs/pari/pari_instance.pyx. (And then run sage -b.)

As I already mentioned earlier in this thread, even more easy is to
change the memory at runtime, when starting up Sage:

pari.allocatemem(10^7, 10^7)