accumulating many defunct child processes

[John Cremona]

On a linux (ubuntu 16.04) machine I am running one instance of Sage
version 7.6. In a loop I am calling a function of my own which
interfaces to Magma; that function starts with

mag = Magma()

then there are a whole lot of mag.eval() statements and af ew others
with which I collect the content of Magma variable back into Sage, and
the last line before the function returns is

mag.quit()

Once this has been running for a while, I have --as expected-- exactly
one magma process running. But there are many defunct python
processes (right now, 1375 lines like
jec 59385 0.0 0.0 0 0 pts/11 Z+ 12:30 0:00
[python] <defunct>
in the output of ps -ux | grep python) whose times stamps show that
many per second were created. These are all child processes of the
controlling Sage process. They seem harmless since they are not using
any resources, but I had 5 such Sage sessions running overnight and
this morning could not log into the machine for a while and I suspect
(without proof) that some maximum number of processes was exceeded --
when I did log in, I saw that all 5 of the Sage processes had been
killed and showed an error message about not being able to fork().
The computer has 512g of RAM and 72 cores, and was not heavily loaded.

Help please!

I am attaching the function in question in case that helps.

John

[Dima Pasechnik]

I guess these zombie processes come from the function quitting Magma,

something that does not destroy a Python child process used to communicate with it.

Why won't you create a Magma instance just once, and do a cleanup after your function is done?

(IIRC, Magma does not have a "reset" facility, but you can still "delete" variables.) (Well, this would be tedious,

perhaps these deletes are called by Sage interface?)

On the other hand, there is also stuff in sage.interfaces.quit.* you can probably use to clean up,

as well as sage.interfaces.cleaner.* - which is even less documented, and it's hard to say exactly what to do

with is. A documentation bug?

[William Stein]

On Mon, Jun 5, 2017 at 11:21 AM, Dima Pasechnik <dim...@gmail.com> wrote:
> I guess these zombie processes come from the function quitting Magma,
> something that does not destroy a Python child process used to communicate
> with it.
>
> Why won't you create a Magma instance just once, and do a cleanup after your
> function is done?
> (IIRC, Magma does not have a "reset" facility, but you can still "delete"
> variables.) (Well, this would be tedious,
> perhaps these deletes are called by Sage interface?)

Another hack/workaround *might* be to use Sage's @fork or @parallel
decorators...

> --
> You received this message because you are subscribed to the Google Groups
> "sage-support" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-support...@googlegroups.com.
> To post to this group, send email to sage-s...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-support.
> For more options, visit https://groups.google.com/d/optout.



--
William (http://wstein.org)

[John Cremona]

Thanks for the suggestions. I was implementing something like Dima's
suggestion and noticed that the function has 3 different return
statements, one of which is not normal (some runtime error in Magma)
but two are normal and only one of those had the mag.quit() in it.
However I don't think that this was the cause of the problem since I
did not see any idle Magma processes lying about, and also I know from
my output that the rogue return route was not used (I am looking for
elliptic curves and these runs had 100% success).

So I just started a run with all calls to the function using the same
Magma() instance, and after a few minutes when the function had been
called about 50 times there were already hundreds of these python
processes, all clocking between 0 and 4s and asleep. Bizarre.
Perhaps I'll try the @fork route.

John

[John Cremona]

On 5 June 2017 at 20:04, John Cremona <john.c...@gmail.com> wrote:
> Thanks for the suggestions. I was implementing something like Dima's
> suggestion and noticed that the function has 3 different return
> statements, one of which is not normal (some runtime error in Magma)
> but two are normal and only one of those had the mag.quit() in it.
> However I don't think that this was the cause of the problem since I
> did not see any idle Magma processes lying about, and also I know from
> my output that the rogue return route was not used (I am looking for
> elliptic curves and these runs had 100% success).
>
> So I just started a run with all calls to the function using the same
> Magma() instance, and after a few minutes when the function had been
> called about 50 times there were already hundreds of these python
> processes, all clocking between 0 and 4s and asleep. Bizarre.

Correction -- with the new version in which I only start up one Magma
process and pass it down to the function so that each call uses the
same one, all is well.

So there is a bug -- this is a workaround.