Memory leak with matrices and finite fields

[Frederick Manners]

Dear all

[I attempted to post this a few days ago but seemingly failed, so this is a repost; apologies if duplicate.]

The following program appears to consume all the memory on my machine:

n = 1000
for i in range(0, 1000000):
    _ = identity_matrix(n).change_ring(GF(101))
    print(get_memory_usage())

I am running Sage 9.2 on a 64-bit gentoo linux box (kernel 5.4.97-gentoo, gcc version 10.2.0).

I couldn't find any related open bugs.  I have worked around the issue (solution: don't create intermediate integer matrices and then coerce them to finite fields; obviously it's less explicit in my original code) but either way this seems to be a new bug.

Best

Freddie

[jonatha...@googlemail.com]

Dear Freddie,

thank you for the report.

Fortunatly or unfortunately, I can't reproduce this on sage 9.2 or on the develop branch (on ubuntu focal with libflint-2.5.2 and debian buster with sage's flint 2.6.3).

This might be a bug with flint. What flint are you using?

You can go into detail, to see, what exactly is causing the problem and instead of

_ = identity_matrix(n).change_ring(GF(101))

run the underlying functions

M = identity_matrix(n)

L = M.list()

MS = sage.matrix.matrix_space.MatrixSpace(GF(101), 1000, 1000, sparse=False)

mat = MS(L, coerce=True, copy=False)

Maybe repeatedly running the first line causes leakage already?

Jonathan

[Dima Pasechnik]

we've fixed a memory leak in matrices recently (this fix will be in
9.3), not sure if this is related
https://trac.sagemath.org/ticket/31340

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/6fd93c63-3fad-4d20-b624-3d3208d4ec4fn%40googlegroups.com.

[Freddie Manners]

Dear all,

Sorry for the delay getting back to you -- this was waiting in the moderation queue for a bit (me being new) and then I dropped the ball.

> run the underlying functions

> 

> M = identity_matrix(n)

> L = M.list()

> MS = sage.matrix.matrix_space.MatrixSpace(GF(101), 1000, 1000, sparse=False)

> mat = MS(L, coerce=True, copy=False)

> 

> Maybe repeatedly running the first line causes leakage already?

It appears that the first *two* lines cause the leak, whereas the first line by itself does not.  Not sure if that narrows it down, but possibly strengthens the 31340 hypothesis below.

> This might be a bug with flint. What flint are you using?

To the best of my knowledge I don't have a system copy of flint, so presumably I am using whatever is bundled in sage-9.2/upstream which seems to be 2.5.2.

> https://trac.sagemath.org/ticket/31340

I can confirm that the original code on that trac also memory leaks for my system (31MB) so it could very plausibly be the same issue.  I have not yet read that bug in detail.

Thanks for the feedback.

Best
Freddie

[jonatha...@googlemail.com]

I think I tracked it down.

https://github.com/BrianGladman/flint/blob/trunk/gmpcompat.h

In there you see that `flint_mpz_set_si` is modified to work with GMP 6.2 I'm assuming this one is picked up from your system?

`sage --standard | grep gmp`

This code is not present yet in flint 2.5.2, which is installed from sage in your case, I guess:

`sage --standard | grep flint`

Hopefully this should resolve itself if you update to sage 9.3 (flint 2.6.3), because this flint does account for the changes made in GMP 6.2

Jonathan

[Freddie Manners]

Awesome, many thanks for running this down.

`sage --standard | grep gmp`

>> gmp.....................................6.1.2 (not_installed)

`equery --quiet list gmp`

>> dev-libs/gmp-6.2.1-r1

`sage --standard | grep flint`

>> flint...................................2.5.2.p5 (2.5.2.p5)

So yes, that checks out, I am using system GMP 6.2 with Sage's flint 2.5.2.

Not sure if something should have complained somewhere or whatever, but I look forward to seeing the problem disappear at the next upgrade.

Best

Freddie

[Jonathan Kliem]

Thanks Freddie for reporting this.

I'm glad it resolves so easily. Yes, that is a mistake in our 9.2 configure. One can fix this by passing `--with-system-gmp=no` to configure, but it should have rejected gmp 6.2 as long as flint 2.5.2 is the install candidate.

So you can update sage or rebuilt (large) parts of it by reconfiguring. Either way the issue goes hopefully away.

Jonathan

You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/p7dYOW29PCI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/324cea70-6662-4d08-ab10-fe2471f9d8b0n%40googlegroups.com.

[Dima Pasechnik]

On Fri, May 14, 2021 at 4:03 PM 'jonatha...@googlemail.com' via
sage-devel <sage-...@googlegroups.com> wrote:
>
> I think I tracked it down.
>
> https://github.com/BrianGladman/flint/blob/trunk/gmpcompat.h

this is a fork, dealing with Windows support of Flint.
We should be working with
https://github.com/wbhart/flint2
which is the mainline of Flint.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/94fd8faa-9f94-4300-968c-2801a885528en%40googlegroups.com.

[Jonathan Kliem]

Its the same code in the Main line. Sorry for the confusion.

I googled it and that is how I found the exact place where this is defined.

> You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/p7dYOW29PCI/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAAWYfq3ShqkUyRgg2bCBL8%2BMM13%2BXPLJnJQcTSbwob%2BOpHRFeQ%40mail.gmail.com.