manual memory management for GAP?
155 views
Skip to first unread message
AlexGhitza
Jun 20, 2022, 8:12:27 PM6/20/22
to sage-devel
Hi,
I'm trying to help out a friend who is computing automorphism groups of lattices in Sage (using GAP under the hood). The computation is fairly long and memory-intensive and eventually results in
sage.libs.gap.util.GAPError: Error, reached the pre-set memory limit
(change it with the -o command line option)
(change it with the -o command line option)
Apparently, until late last year, achieving the desired -o command line option effect from within Sage was possible via something like
from sage.interfaces.gap import
get_gap_memory_pool_size, set_gap_memory_pool_size
memory_gap = get_gap_memory_pool_size()
set_gap_memory_pool_size(9048*memory_gap)
memory_gap = get_gap_memory_pool_size()
set_gap_memory_pool_size(9048*memory_gap)
Now these functions are not available anymore, due to https://trac.sagemath.org/ticket/32656
Is it possible to achieve the same effect via some alternative method?
Best,
Alex
John H Palmieri
Jun 20, 2022, 8:59:38 PM6/20/22
to sage-devel
According to https://ask.sagemath.org/question/61829/how-to-deal-with-gaperror-error-reached-the-pre-set-memory-limit/, try "sage.interfaces.gap.gap_cmd = 'gap -r -o 4G '".
AlexGhitza
Jun 20, 2022, 9:30:04 PM6/20/22
to sage-devel
Thanks, John.
But, I should have mentioned that we tried that and it does not seem to work (unless I'm missing something):
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 9.6, Release Date: 2022-05-15 │
│ Using Python 3.10.3. Type "help()" for help. │
└────────────────────────────────────────────────────────────────────┘
sage: gap("GAPInfo.CommandLineOptions;")
rec(
[... redacted ...]
o := "2g",
[... redacted ...]
│ SageMath version 9.6, Release Date: 2022-05-15 │
│ Using Python 3.10.3. Type "help()" for help. │
└────────────────────────────────────────────────────────────────────┘
sage: gap("GAPInfo.CommandLineOptions;")
rec(
[... redacted ...]
o := "2g",
[... redacted ...]
)
sage: sage.interfaces.gap.gap_cmd = 'gap -r -o 128G'
sage: gap("GAPInfo.CommandLineOptions;")
rec(
[... redacted ...]
o := "2g",
[... redacted ...]
)
sage: sage.interfaces.gap.gap_cmd = 'gap -r -o 128G'
sage: gap("GAPInfo.CommandLineOptions;")
rec(
[... redacted ...]
o := "2g",
[... redacted ...]
)
Nils Bruin
Jun 20, 2022, 11:06:15 PM6/20/22
to sage-devel
Perhaps this message (see the thread it's in for context) applies to your situation?
AlexGhitza
Jun 21, 2022, 2:06:11 AM6/21/22
to sage-devel
As far as I can tell the actionable part of that thread is Dima's suggestion to use the function set_gap_memory_pool_size() from interfaces/gap.py but as I pointed out above, that function has disappeared from sage-9.5 onwards.
Best,
Alex
Dima Pasechnik
Jun 21, 2022, 3:20:26 AM6/21/22
to sage-devel
On Tue, 21 Jun 2022, 01:59 John H Palmieri, <jhpalm...@gmail.com> wrote:
According to https://ask.sagemath.org/question/61829/how-to-deal-with-gaperror-error-reached-the-pre-set-memory-limit/, try "sage.interfaces.gap.gap_cmd = 'gap -r -o 4G '".
please see
commit 1ea7b697e812302dfca03a1c7fe41c9a8dc3c98e
Author: Michael Orlitzky <mic...@orlitzky.com>
Date: Wed Oct 6 20:48:22 2021 -0400
Trac #32656: eliminate manual GAP memory management.
Our GAP interface provides a few functions to get/set the GAP memory
pool size. That size used when initializing the GAP interface, being
passed as the argument to GAP's "-o" and "-s" command-line flags:
https://www.gap-system.org/Manuals/doc/ref/chap3.html
In preparation for the removal of psutil and sage.misc.getusage, we
remove all of this custom memory management code. The "-o" flag
specifies an upper bound, and can simply be omitted.
We also omit the "-s" flag in this commit, but that flag is a bit more
delicate, since it may affect users with custom GAP modules. However,
if the omission of "-s" causes a problem down the line, and if the GAP
documentation's assumptions about mmap et al. are correct, then we
can simply add back the "-s" flag with a large, static value.
Author: Michael Orlitzky <mic...@orlitzky.com>
Date: Wed Oct 6 20:48:22 2021 -0400
Trac #32656: eliminate manual GAP memory management.
Our GAP interface provides a few functions to get/set the GAP memory
pool size. That size used when initializing the GAP interface, being
passed as the argument to GAP's "-o" and "-s" command-line flags:
https://www.gap-system.org/Manuals/doc/ref/chap3.html
In preparation for the removal of psutil and sage.misc.getusage, we
remove all of this custom memory management code. The "-o" flag
specifies an upper bound, and can simply be omitted.
We also omit the "-s" flag in this commit, but that flag is a bit more
delicate, since it may affect users with custom GAP modules. However,
if the omission of "-s" causes a problem down the line, and if the GAP
documentation's assumptions about mmap et al. are correct, then we
can simply add back the "-s" flag with a large, static value.
"-o" is not doing the trick. One needs "-s", and also there are two instances of GAP in Sage -
pexpect GAP and libgap, they are instantiated differently, IIRC.
Dima
PS. Given it's an often repeated request, we should document the GAP memory management settings somewhere.
On Monday, June 20, 2022 at 5:12:27 PM UTC-7 AlexGhitza wrote:Hi,I'm trying to help out a friend who is computing automorphism groups of lattices in Sage (using GAP under the hood). The computation is fairly long and memory-intensive and eventually results insage.libs.gap.util.GAPError: Error, reached the pre-set memory limit
(change it with the -o command line option)Apparently, until late last year, achieving the desired -o command line option effect from within Sage was possible via something likefrom sage.interfaces.gap import get_gap_memory_pool_size, set_gap_memory_pool_size
memory_gap = get_gap_memory_pool_size()
set_gap_memory_pool_size(9048*memory_gap)Now these functions are not available anymore, due to https://trac.sagemath.org/ticket/32656Is it possible to achieve the same effect via some alternative method?Best,Alex
--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/7f1922b7-afab-47cc-8cad-7a74708c5962n%40googlegroups.com.
Dima Pasechnik
Jun 21, 2022, 5:19:39 AM6/21/22
to sage-devel
On Tue, Jun 21, 2022 at 4:06 AM Nils Bruin <nbr...@sfu.ca> wrote:
>
> Perhaps this message (see the thread it's in for context) applies to your situation?
>
> https://groups.google.com/g/sage-devel/c/ntuqDmruI8w/m/vJOv6mgqCgAJ
this is outdated. However, it seems that the very old doc suggestion in
>
> Perhaps this message (see the thread it's in for context) applies to your situation?
>
> https://groups.google.com/g/sage-devel/c/ntuqDmruI8w/m/vJOv6mgqCgAJ
sage/interfaces/gap.py
import sage.interfaces.gap
sage.interfaces.gap.gap_cmd = '<whatever> '
could never have worked, as gap_cmd is hardcoded to "gap -r" (or whatever)
and it cannot be overwritten like this.
This is the question of the correct class interface design here, and
I'm a bit lost.
I think the value of gap_cmd can be set somewhere early (in sagerc?),
before the GAP pexcept is initialised.
Then one could conditionally set gap_cmd to either the default value,
or to the value in that initialisation parameter, if it was set at all.
But where?
Dima
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/ce56865b-55b2-4efc-b37e-f5e81875d7c1n%40googlegroups.com.
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
Dima Pasechnik
Jun 21, 2022, 5:24:21 AM6/21/22
to sage-devel
On Tue, Jun 21, 2022 at 10:19 AM Dima Pasechnik <dim...@gmail.com> wrote:
>
> On Tue, Jun 21, 2022 at 4:06 AM Nils Bruin <nbr...@sfu.ca> wrote:
> >
> > Perhaps this message (see the thread it's in for context) applies to your situation?
> >
> > https://groups.google.com/g/sage-devel/c/ntuqDmruI8w/m/vJOv6mgqCgAJ
>
> this is outdated. However, it seems that the very old doc suggestion in
> sage/interfaces/gap.py
>
> import sage.interfaces.gap
> sage.interfaces.gap.gap_cmd = '<whatever> '
>
> could never have worked, as gap_cmd is hardcoded to "gap -r" (or whatever)
> and it cannot be overwritten like this.
>
> This is the question of the correct class interface design here, and
> I'm a bit lost.
> I think the value of gap_cmd can be set somewhere early (in sagerc?),
> before the GAP pexcept is initialised.
> Then one could conditionally set gap_cmd to either the default value,
> or to the value in that initialisation parameter, if it was set at all.
>
> But where?
I've opened https://trac.sagemath.org/ticket/34041 to deal with these issues.
>
> On Tue, Jun 21, 2022 at 4:06 AM Nils Bruin <nbr...@sfu.ca> wrote:
> >
> > Perhaps this message (see the thread it's in for context) applies to your situation?
> >
> > https://groups.google.com/g/sage-devel/c/ntuqDmruI8w/m/vJOv6mgqCgAJ
>
> this is outdated. However, it seems that the very old doc suggestion in
> sage/interfaces/gap.py
>
> import sage.interfaces.gap
> sage.interfaces.gap.gap_cmd = '<whatever> '
>
> could never have worked, as gap_cmd is hardcoded to "gap -r" (or whatever)
> and it cannot be overwritten like this.
>
> This is the question of the correct class interface design here, and
> I'm a bit lost.
> I think the value of gap_cmd can be set somewhere early (in sagerc?),
> before the GAP pexcept is initialised.
> Then one could conditionally set gap_cmd to either the default value,
> or to the value in that initialisation parameter, if it was set at all.
>
> But where?
David Joyner
Jun 21, 2022, 5:45:16 AM6/21/22
to sage-devel
On Tue, Jun 21, 2022 at 5:19 AM Dima Pasechnik <dim...@gmail.com> wrote:
>
> On Tue, Jun 21, 2022 at 4:06 AM Nils Bruin <nbr...@sfu.ca> wrote:
> >
> > Perhaps this message (see the thread it's in for context) applies to your situation?
> >
> > https://groups.google.com/g/sage-devel/c/ntuqDmruI8w/m/vJOv6mgqCgAJ
>
> this is outdated. However, it seems that the very old doc suggestion in
> sage/interfaces/gap.py
>
> import sage.interfaces.gap
> sage.interfaces.gap.gap_cmd = '<whatever> '
>
> could never have worked, as gap_cmd is hardcoded to "gap -r" (or whatever)
> and it cannot be overwritten like this.
>
> This is the question of the correct class interface design here, and
> I'm a bit lost.
> I think the value of gap_cmd can be set somewhere early (in sagerc?),
Can one just manually edit gap.sh in local/share/gap?
>
> On Tue, Jun 21, 2022 at 4:06 AM Nils Bruin <nbr...@sfu.ca> wrote:
> >
> > Perhaps this message (see the thread it's in for context) applies to your situation?
> >
> > https://groups.google.com/g/sage-devel/c/ntuqDmruI8w/m/vJOv6mgqCgAJ
>
> this is outdated. However, it seems that the very old doc suggestion in
> sage/interfaces/gap.py
>
> import sage.interfaces.gap
> sage.interfaces.gap.gap_cmd = '<whatever> '
>
> could never have worked, as gap_cmd is hardcoded to "gap -r" (or whatever)
> and it cannot be overwritten like this.
>
> This is the question of the correct class interface design here, and
> I'm a bit lost.
> I think the value of gap_cmd can be set somewhere early (in sagerc?),
For example, will
replacing
exec "$GAP_EXE/gap" -l "$GAP_DIR" "$@"
by
exec "$GAP_EXE/gap -m 4G" -l "$GAP_DIR" "$@"
work?
Dima Pasechnik
Jun 21, 2022, 6:10:09 AM6/21/22
to sage-devel
On Tue, 21 Jun 2022, 10:45 David Joyner, <wdjo...@gmail.com> wrote:
On Tue, Jun 21, 2022 at 5:19 AM Dima Pasechnik <dim...@gmail.com> wrote:
>
> On Tue, Jun 21, 2022 at 4:06 AM Nils Bruin <nbr...@sfu.ca> wrote:
> >
> > Perhaps this message (see the thread it's in for context) applies to your situation?
> >
> > https://groups.google.com/g/sage-devel/c/ntuqDmruI8w/m/vJOv6mgqCgAJ
>
> this is outdated. However, it seems that the very old doc suggestion in
> sage/interfaces/gap.py
>
> import sage.interfaces.gap
> sage.interfaces.gap.gap_cmd = '<whatever> '
>
> could never have worked, as gap_cmd is hardcoded to "gap -r" (or whatever)
> and it cannot be overwritten like this.
>
> This is the question of the correct class interface design here, and
> I'm a bit lost.
> I think the value of gap_cmd can be set somewhere early (in sagerc?),
Can one just manually edit gap.sh in local/share/gap?
do we use gap.sh at all?
(maybe for gap_console())?
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAEQuuAUN5w5dmVWtYzs%2BZJRP1s4gYqTxeC4o6kRiFE_-b8Y-Fw%40mail.gmail.com.
Dima Pasechnik
Jun 21, 2022, 9:35:43 AM6/21/22
to sage-devel
On Tue, Jun 21, 2022 at 10:19 AM Dima Pasechnik <dim...@gmail.com> wrote:
>
> On Tue, Jun 21, 2022 at 4:06 AM Nils Bruin <nbr...@sfu.ca> wrote:
> >
> > Perhaps this message (see the thread it's in for context) applies to your situation?
> >
> > https://groups.google.com/g/sage-devel/c/ntuqDmruI8w/m/vJOv6mgqCgAJ
>
> this is outdated. However, it seems that the very old doc suggestion in
> sage/interfaces/gap.py
>
> import sage.interfaces.gap
> sage.interfaces.gap.gap_cmd = '<whatever> '
>
> could never have worked, as gap_cmd is hardcoded to "gap -r" (or whatever)
> and it cannot be overwritten like this.
indeed:
>
> On Tue, Jun 21, 2022 at 4:06 AM Nils Bruin <nbr...@sfu.ca> wrote:
> >
> > Perhaps this message (see the thread it's in for context) applies to your situation?
> >
> > https://groups.google.com/g/sage-devel/c/ntuqDmruI8w/m/vJOv6mgqCgAJ
>
> this is outdated. However, it seems that the very old doc suggestion in
> sage/interfaces/gap.py
>
> import sage.interfaces.gap
> sage.interfaces.gap.gap_cmd = '<whatever> '
>
> could never have worked, as gap_cmd is hardcoded to "gap -r" (or whatever)
> and it cannot be overwritten like this.
sage: import sage.interfaces.gap
sage: sage.interfaces.gap.gap_cmd = "gap -r -o 3G -s 5G"
sage: gap.eval('GAPInfo.CommandLineOptions.s')
'"4g"'
so changing gap_cmd has no effect.
TB
Jun 21, 2022, 10:06:34 AM6/21/22
to sage-...@googlegroups.com
On 21/06/2022 16:35, Dima Pasechnik wrote:
> On Tue, Jun 21, 2022 at 10:19 AM Dima Pasechnik <dim...@gmail.com> wrote:
>>
>> On Tue, Jun 21, 2022 at 4:06 AM Nils Bruin <nbr...@sfu.ca> wrote:
>>>
>>> Perhaps this message (see the thread it's in for context) applies to your situation?
>>>
>>> https://groups.google.com/g/sage-devel/c/ntuqDmruI8w/m/vJOv6mgqCgAJ
>>
>> this is outdated. However, it seems that the very old doc suggestion in
>> sage/interfaces/gap.py
>>
>> import sage.interfaces.gap
>> sage.interfaces.gap.gap_cmd = '<whatever> '
>>
>> could never have worked, as gap_cmd is hardcoded to "gap -r" (or whatever)
>> and it cannot be overwritten like this.
>
> indeed:
>
> sage: import sage.interfaces.gap
> sage: sage.interfaces.gap.gap_cmd = "gap -r -o 3G -s 5G"
> sage: gap.eval('GAPInfo.CommandLineOptions.s')
> '"4g"'
>
> so changing gap_cmd has no effect.
>
Maybe this works:
> On Tue, Jun 21, 2022 at 10:19 AM Dima Pasechnik <dim...@gmail.com> wrote:
>>
>> On Tue, Jun 21, 2022 at 4:06 AM Nils Bruin <nbr...@sfu.ca> wrote:
>>>
>>> Perhaps this message (see the thread it's in for context) applies to your situation?
>>>
>>> https://groups.google.com/g/sage-devel/c/ntuqDmruI8w/m/vJOv6mgqCgAJ
>>
>> this is outdated. However, it seems that the very old doc suggestion in
>> sage/interfaces/gap.py
>>
>> import sage.interfaces.gap
>> sage.interfaces.gap.gap_cmd = '<whatever> '
>>
>> could never have worked, as gap_cmd is hardcoded to "gap -r" (or whatever)
>> and it cannot be overwritten like this.
>
> indeed:
>
> sage: import sage.interfaces.gap
> sage: sage.interfaces.gap.gap_cmd = "gap -r -o 3G -s 5G"
> sage: gap.eval('GAPInfo.CommandLineOptions.s')
> '"4g"'
>
> so changing gap_cmd has no effect.
>
sage: import sage.interfaces.gap
sage: sage.interfaces.gap.gap_cmd = "gap -r -o 3G -s 5G"
sage: sage.interfaces.gap.gap = Gap() # New instance
sage: sage.interfaces.gap.gap_cmd = "gap -r -o 3G -s 5G"
sage: gap.eval('GAPInfo.CommandLineOptions.s')
'"4g"'
sage: sage.interfaces.gap.gap.eval('GAPInfo.CommandLineOptions.s')
'"4g"'
'"5G"'
sage: gap = sage.interfaces.gap.gap # For the global namespace
sage: gap.eval('GAPInfo.CommandLineOptions.s')
'"5G"'
Regards,
TB
Dima Pasechnik
Jun 21, 2022, 11:00:43 AM6/21/22
to sage-devel
hmm, and what happened to the old instance of GAP, and to Sage objects associated with it?
Sure, you can monkey-patch Python classes, but this looks wrong in this case.
Regards,
TB
--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/550315df-fd50-b6c6-5202-9b2cfcee33a9%40gmail.com.
Dima Pasechnik
Jun 22, 2022, 6:30:00 PM6/22/22
to sage-devel
On Tue, Jun 21, 2022 at 10:24 AM Dima Pasechnik <dim...@gmail.com> wrote:
>
> On Tue, Jun 21, 2022 at 10:19 AM Dima Pasechnik <dim...@gmail.com> wrote:
> >
> > On Tue, Jun 21, 2022 at 4:06 AM Nils Bruin <nbr...@sfu.ca> wrote:
> > >
> > > Perhaps this message (see the thread it's in for context) applies to your situation?
> > >
> > > https://groups.google.com/g/sage-devel/c/ntuqDmruI8w/m/vJOv6mgqCgAJ
> >
> > this is outdated. However, it seems that the very old doc suggestion in
> > sage/interfaces/gap.py
> >
> > import sage.interfaces.gap
> > sage.interfaces.gap.gap_cmd = '<whatever> '
> >
> > could never have worked, as gap_cmd is hardcoded to "gap -r" (or whatever)
> > and it cannot be overwritten like this.
> >
> > This is the question of the correct class interface design here, and
> > I'm a bit lost.
> > I think the value of gap_cmd can be set somewhere early (in sagerc?),
> > before the GAP pexcept is initialised.
> > Then one could conditionally set gap_cmd to either the default value,
> > or to the value in that initialisation parameter, if it was set at all.
> >
> > But where?
>
> I've opened https://trac.sagemath.org/ticket/34041 to deal with these issues.
a fix is ready for review. I went for instroducing env.vars to control this.
>
> On Tue, Jun 21, 2022 at 10:19 AM Dima Pasechnik <dim...@gmail.com> wrote:
> >
> > On Tue, Jun 21, 2022 at 4:06 AM Nils Bruin <nbr...@sfu.ca> wrote:
> > >
> > > Perhaps this message (see the thread it's in for context) applies to your situation?
> > >
> > > https://groups.google.com/g/sage-devel/c/ntuqDmruI8w/m/vJOv6mgqCgAJ
> >
> > this is outdated. However, it seems that the very old doc suggestion in
> > sage/interfaces/gap.py
> >
> > import sage.interfaces.gap
> > sage.interfaces.gap.gap_cmd = '<whatever> '
> >
> > could never have worked, as gap_cmd is hardcoded to "gap -r" (or whatever)
> > and it cannot be overwritten like this.
> >
> > This is the question of the correct class interface design here, and
> > I'm a bit lost.
> > I think the value of gap_cmd can be set somewhere early (in sagerc?),
> > before the GAP pexcept is initialised.
> > Then one could conditionally set gap_cmd to either the default value,
> > or to the value in that initialisation parameter, if it was set at all.
> >
> > But where?
>
> I've opened https://trac.sagemath.org/ticket/34041 to deal with these issues.
Reply all
Reply to author
Forward
0 new messages