Re: [sage-support] huge virtual memory size when launching 7.3

[Jonathan Bober]

(I've swtiched from sage-support to to sage-devel.)

I can test and review. I think I know what to do and I would have just tried to implement this myself, except that it would take me a while to figure out how to make the change in such a way that it fits into the Sage build process.

I spent some time trying to understand the issue and what this PARI stack is all about. What I think I understand is something like:

- PARI uses it's own internal stack for quick memory allocation

- At the "top level", if I'm using PARI as a C-library, or through the GP interpreter, the stack is empty between function calls, so it can be resized.

- While in use, though, the stack can't really be resized, because references to memory allocated on the stack don't use the stack pointer, and resizing might require moving the stack

- If the stack runs out of space, PARI throws some error, and once upon a time Sage would notice this error, increase the stack size, and repeat

- Now instead Sage just sets the stack size to be really big, which probably is generally fine because modern systems generally don't allocation memory until it is actually touched

(Is that all correct?)

Maybe this change in Sage corresponds somewhat to a change in the way that PARI manages its stack. But from reading the PARI source code I can't actually see that PARI is managing the stack in any way that is actually distinguishable from just calling malloc() to allocate the stack and then using it until it is full. Probably the use of MAP_NORESERVE is intended to have some effect, but on Linux MAP_NORESERVE appears to do nothing. Meanwhile, the paristack_resize() function just does some arithmetic, and doesn't actually touch the allocated stack space at all. Luckily, the paristack_resize() function does exist, though, and probably gets properly used even though it doesn't really do anything, so a call to mprotect can be added there, and meanwhile the mmap call can use PROT_NONE. Maybe those are the only spots where the PARI source code needs to be changed.

(I'm not completely sure I'm correct about all of that.)

I'm probably going to try to modify a clean copy of PARI to do this, or just write some completely separate test code to check that an mmap call with PROT_NONE will work like we think it will work.

On Tue, Sep 20, 2016 at 12:02 PM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:

On 2016-09-20 12:54, Jonathan Bober wrote:

 From reading what you've sent, I guess that what you have in mind is
calling mmap with PROT_NONE and then calling mprotect() to change that
to read/write whenever growing the stack? That seems like it might be a
reasonable thing to do (though I'm only basing that on spending a few
minutes reading what you sent, not from any actual knowledge that I had
before).

Yes, that is my idea.

I'm willing to devote some time (not today) to figuring out what the
right thing to do is (maybe the above is already the right thing) /
implementing this / reviewing this.

I don't mind implementing it. What I *do* mind is that I implement it and that the patch rots away on Sage Trac in needs_review state (highly specialized patches like these have a higher chance of that). That's why I asked for a commitment to review.

--
You received this message because you are subscribed to the Google Groups "sage-support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-support+unsubscribe@googlegroups.com.
To post to this group, send email to sage-s...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-support.
For more options, visit https://groups.google.com/d/optout.

[Jeroen Demeyer]

On 2016-09-21 16:15, Jonathan Bober wrote:
> (I've swtiched from sage-support to to sage-devel.)
>
> I can test and review. I think I know what to do and I would have just
> tried to implement this myself, except that it would take me a while to
> figure out how to make the change in such a way that it fits into the
> Sage build process.

OK, I will do it then.

> - PARI uses it's own internal stack for quick memory allocation
> - At the "top level", if I'm using PARI as a C-library, or through the
> GP interpreter, the stack is empty between function calls, so it can be
> resized.
> - While in use, though, the stack can't really be resized, because
> references to memory allocated on the stack don't use the stack pointer,
> and resizing might require moving the stack
> - If the stack runs out of space, PARI throws some error, and once upon
> a time Sage would notice this error, increase the stack size, and repeat
> - Now instead Sage just sets the stack size to be really big, which
> probably is generally fine because modern systems generally don't
> allocation memory until it is actually touched
>
> (Is that all correct?)

It is more complicated than that... PARI has two stack sizes: a real
stack size and a virtual stack size. In Sage, the real stack size is
relatively small (a few megabytes) while the virtual stack size can be
huge. The virtual stack size is what is mmap()ed, so it counts towards
virtual memory.

If possible, PARI uses only the real stack size. It has a garbage
collection mechanism to clean up the stack when it is getting full. Only
when this is not sufficient, PARI increases the real stack size within
the virtual stack. Important point: this resizing is in-place, unlike
realloc(). So all existing pointers remain valid.

The virtual stack size can only be changed when the stack is empty. In
GP, this means that it can happen only at the user prompt. In Sage, this
means any time between 2 calls through the PARI interface.

In GP, the real stack size is automatically reset to the original real
stack size whenever it gets back to the user prompt. In Sage, the real
stack size is never automatically decreased.

> I'm probably going to try to modify a clean copy of PARI to do this, or
> just write some completely separate test code to check that an mmap call
> with PROT_NONE will work like we think it will work.

I tested this with a small stand-alone C program: the flags
PROT_NONE | MAP_PRIVATE | MAP_ANONYMOUS
allow to allocate huge amounts of virtual memory, even with overcommit=2.


Jeroen.

[Jonathan Bober]

On Wed, Sep 21, 2016 at 4:41 PM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:

I tested this with a small stand-alone C program: the flags
PROT_NONE | MAP_PRIVATE | MAP_ANONYMOUS
allow to allocate huge amounts of virtual memory, even with overcommit=2.

I also tried this. And I alsoo found that the memory does count against the commit limit once mprotect() is called with PROT_READ | PROT_WRITE. Also, with PROT_NONE I tried allocating 100 TB (with PROT_NONE, again) to 200 different processes at the same time, and I couldn't even notice any effect of this in /proc/meminfo. I haven't actually found this behavior documented anywhere, and it would be good to know that it works similarly on OS X.

There is still a small cost: the memory does count against the process as far as ulimit -v is concerned. But that doesn't seem like much of an issue to me.

[Jeroen Demeyer]

On 2016-09-21 16:15, Jonathan Bober wrote:

> (I've swtiched from sage-support to to sage-devel.)
>
> I can test and review.

See https://trac.sagemath.org/ticket/21582