Parallel tests, maximum recursion depth ignored and #10963; help!

[Nicolas M. Thiery]

Hi!

The functorial construction ticket #10963 is almost done. However,
since very recently (Sage 5.11?), some errors started to appear when
running the tests in parallel:

sage -tp 8 sage/categories/*.py
...
Exception RuntimeError: 'maximum recursion depth exceeded' in
<function remove at 0x398c2a8> ignored
...

They seem to appear always at the same spot
(e.g. sage/categories/pushout.py). Things go fine if the tests are run
separately on each file.

Would anyone have any clue on what might be the cause or how to debug
this?

Thanks much in advance!

Cheers,
Nicolas
--
Nicolas M. Thi�ry "Isil" <nth...@users.sf.net>
http://Nicolas.Thiery.name/

[Volker Braun]

On Monday, August 19, 2013 5:07:32 PM UTC+1, Nicolas M. Thiéry wrote:

    Exception RuntimeError: 'maximum recursion depth exceeded' in
    <function remove at 0x398c2a8> ignored

Would anyone have any clue on what might be the cause or how to debug
this?

Obviously you are deleting something in the remove callback, causing another remove callback to be called. If you only have enough objects pointing to each other then you'll eventually hit the recursion limit. Every recursive function can be rewritten without recursion... (maintain a list of dead objects that you append to instead of delete while running remove())

[Simon King]

Hi!

On 2013-08-19, Volker Braun <vbrau...@gmail.com> wrote:
> Obviously you are deleting something in the remove callback, causing=20

> another remove callback to be called.

Meanwhile we made progress (or even found a solution).

One indeed obtained the "recursion depth exceeded ... ignored", when one
has a MonoDict M and objects a1, a2, a3,..., putting M[a1]=a2,
M[a2]=a3,..., keeping a reference to a1. And if one then does "del a1",
then the callback for a1 deletes a2, which immediately triggers a
callback that deletes a3, and so on.

The point is: The next callback is invoked *before* the previous
callback has returned. So, the callback for a1 should better keep a2 alive
until it has returned. I tested that it is not enough to keep it alive
by having a local variable inside of the callback pointing to it. So,
the solution that I suggest at #15069: Let the callback *return* a2. In
that way, a2 will be alive until the return is completed, and
immediately afterwards it will be deleted, causing the next callback.
Hence, the callbacks are executed one after the other, without nesting.

Note that Python's weakref.WeakKeyDictionary has the same problems with
nested callbacks. Hence, if you think you can give #15069 a positive
review (hint-hint), then Sage's MonoDict and TripleDict will have yet
another advantage over WeakKeyDicitonary :)

Best regards,
Simon