tp_new atrocities in 2.8.14

[mabshoff]

Hello,

compiling python with the configure option "--without-pymalloc" makes
the noise regarding invalid memory access and so on goes away. But the
following issue remains:

==465== Invalid free() / delete / delete[]
==465== at 0x4A1B74A: free (vg_replace_malloc.c:320)
==465== by 0x43FE9A: dict_dealloc (dictobject.c:847)
==465== by 0x43FE9A: dict_dealloc (dictobject.c:847)
==465== by 0x499E5B: _PyImport_Fini (import.c:229)
==465== by 0x4A5D66: Py_Finalize (pythonrun.c:419)
==465== by 0x4A58AA: handle_system_exit (pythonrun.c:1616)
==465== by 0x4A5AA8: PyErr_PrintEx (pythonrun.c:1062)
==465== by 0x4A62B6: PyRun_SimpleFileExFlags (pythonrun.c:976)
==465== by 0x4120FF: Py_Main (main.c:523)
==465== by 0x4FD94C9: (below main) (in /lib/libc-2.3.6.so)
==465== Address 0x530de28 is 32 bytes inside a block of size 80
alloc'd
==465== at 0x4A1BB35: malloc (vg_replace_malloc.c:207)
==465== by 0x4B0079: _PyObject_GC_Malloc (gcmodule.c:1321)
==465== by 0x454C29: PyType_GenericAlloc (typeobject.c:454)
==465== by 0x97622E0:
__pyx_tp_new_4sage_9structure_7element_Element (element.c:22299)
==465== by 0xAC9C9DA: __pyx_tp_new_4sage_5rings_7integer_Integer
(integer.c:19673)
==465== by 0x458D92: type_call (typeobject.c:422)
==465== by 0x415542: PyObject_Call (abstract.c:1860)
==465== by 0x47C480: PyEval_CallObjectWithKeywords (ceval.c:3433)
==465== by 0xAC978B3: initinteger (integer.c:20610)
==465== by 0x49E17D: _PyImport_LoadDynamicModule (importdl.c:53)
==465== by 0x49C08D: import_submodule (import.c:2394)
==465== by 0x49C550: load_next (import.c:2214)

I tracked this down to the integer code which clearly warns about
certain hacks, so I am hoping that somebody with more knowledge in
that area will give some explanation of what might go wrong and if we
can fix this.

Cheers,

Michael

[Robert Bradshaw]

I've been looking into this and I can't seem to find where the issue
is. The thing I don't understand is that dictobject calls Py_XDECREF
on line 847, which is a macro that essentially boils down to
_Py_Dealloc in object.c. Even if this were inlined, it calls op-
>ob_type->tp_dealloc which should be fast_tp_dealloc (integer.c:
3258) if indeed the integer created on at integer.c:19673 is the
thing being collected.

Also, an integer object is 28 bytes:

{{{
%cython
from sage.rings.integer cimport Integer
print sizeof(Integer)
///
28
}}}

so I can't figure out why 80 bytes is getting allocated at that spot.
Also strange, in my version of sage, element.c:22299 is the closing
brace of __pyx_tp_new_4sage_9structure_7element_Element, so the
traceback for the allocation seems strange as well.

It would be useful if we could figure out what module is being
collected that is throwing this error.

- Robert

[mabshoff]

On Nov 29, 5:04 am, Robert Bradshaw <rober...@math.washington.edu>
wrote:

Hello Robert,

> I've been looking into this and I can't seem to find where the issue
> is. The thing I don't understand is that dictobject calls Py_XDECREF
> on line 847, which is a macro that essentially boils down to
> _Py_Dealloc in object.c. Even if this were inlined, it calls op-
> >ob_type->tp_dealloc which should be fast_tp_dealloc (integer.c:
> 3258) if indeed the integer created on at integer.c:19673 is the
> thing being collected.
>
> Also, an integer object is 28 bytes:
>
> {{{
> %cython
> from sage.rings.integer cimport Integer
> print sizeof(Integer)
> ///
> 28
>
> }}}
>
> so I can't figure out why 80 bytes is getting allocated at that spot.
> Also strange, in my version of sage, element.c:22299 is the closing
> brace of __pyx_tp_new_4sage_9structure_7element_Element, so the
> traceback for the allocation seems strange as well.
>
> It would be useful if we could figure out what module is being
> collected that is throwing this error.
>

Yep. You once send me a patch for some "magic" integer not to be
deallocated with high garbage collection level with Cython - I am
going to see if applying that patch fixes it.

> - Robert

This is now #1337. On Linux PPC the above leads to a crash on exit for
every Sage session started. It didn't used to do that with 2.8.7, but
that really doesn't indicate a whole lot.

Cheers,

Michael

<SNIP>

[mabshoff]

On Nov 29, 10:50 am, mabshoff <Michael.Absh...@fsmath.mathematik.uni-

dortmund.de> wrote:
> On Nov 29, 5:04 am, Robert Bradshaw <rober...@math.washington.edu>
> wrote:
>
> Hello Robert,
>

Hi Robert,

I commented on #1337: Your no-collect-integer.patch prevents the issue
from happening by incrementing the ref count on global_integer_dummy.
It isn't really a fix, but an appropriate short term solution.

I plan to apply the patch to 2.8.15, unless you object somehow :)

<SNIP>

Cheers,

Michael

[Robert Bradshaw]

I'm glad the fix was so simple. I think this is OK permanently, as
bad things can happen if this integer ever goes away before the
session dies completely.

- Robert

[mabshoff]

On Nov 30, 10:18 am, Robert Bradshaw <rober...@math.washington.edu>

Arrg, it is back. I spoke too soon - not sure where I screwed up. I
still think the patch should be applied.

> - Robert

Sorry,

Michael