How to find out where an AttributeError is ignored
Simon King
Doing Sage development, I am currently writing a patch that results in
a doctest error. The problem is: So far I am unable to find out where
the error exactly occurs. I see the following warning
Exception AttributeError: 'PolynomialRing_field_with_category' object
has no attribute '_modulus' in ignored
Exception AttributeError: 'PolynomialRing_field_with_category' object
has no attribute '_modulus' in ignored
and some other test really fails - but when I try the same test in an
interactive Sage session then it works.
Question: In the warning message above, it is not told *where* the
error occurs. It just says "... in ignored", seemingly in a failing
attempt to insert the object's string representation between the words
"in" and "ignored".
Suggestion: If an object's string representation is not available (for
example during initialisation or deallocation of the object), then
name the type of the object instead.
Question: Is there some debug mode that I can use in order to locate
the problem? So far, I don't even know in what file the error is
ignored.
Best regards,
Simon
Simon King
By some guess-work, I was able to fix the doctest. Nevertheless, I'd
appreciate to learn a debugging technique that might help me in
future. And of course I still suggest to at least insert the type and
perhaps the id of an object in the warning, if the string
representation of the object is not available.
Cheers,
Simon
Robert Bradshaw
We are trying to be compatible with Python's attribute errors here.
The real bug seems to be not printing out where the exception was
suppressed (e.g. due to a cdef method without an except clause needed
to propagate the exception up the call stack). Could you point me to a
ticket number so I can investigate?
- Robert
Simon King
On 5 Jan., 21:12, Robert Bradshaw <rober...@math.washington.edu>
wrote:
> suppressed
> (e.g. due to a cdef method without an except clause needed
> to propagate the exception up the call stack). Could you point me to a
> ticket number so I can investigate?
there is only one dependency, namely #715.
But note that the error will not occur with the latest version of the
patch trac11521_triple_homset.patch.
When you delete the last hunk of the patch (which introduces a
"gc.collect()" to a doctest of sage/rings/polynomial/
multi_polynomial_libsingular.pyx), then you will hopefully see the
error by
sage -t "devel/sage/sage/rings/polynomial/
multi_polynomial_libsingular.pyx"
Thank you very much,
Simon
Simon King
> suppressed (e.g. due to a cdef method without an except clause needed
> to propagate the exception up the call stack). Could you point me to a
> ticket number so I can investigate?
error on that ticket disappear by inserting a garbage collection, but
this is not an actual solution of the problem.
Actually, today, the same problem (an attribute error for '_modulus'
was ignored, but Cython does not reveal the location) hit me again,
but this time I can't point you to a ticket.
Cheers,
Simon
Simon King
I am still *very* frustrated about the fact that Cython does not tell
where the error occurs. Since about one week, I am adding lots and
lots of lines into Sage that write a log into some file, so that I get
at least some idea where the error occurs. But still: Even these
extensive logs do not provide a hint on what exactly is happening.
How can I patch Cython such that some more information on the location
of the error is printed? I unpacked Sage's Cython spkg, and did "grep -
R ignored .", but the code lines containing the word "ignored" did not
seem to be the lines that are responsible for printing the warning
message
object has no attribute '_modulus' in ignored
responsible for printing the warning?
Best regards,
Simon
mark florisson
These messages are written by PyErr_WriteUnraisable, which is a
CPython C API function that writes unraisable exceptions. There are
typically two reasons for unraisable exceptions:
1) as Robert mentioned, a function that does not allow propagation
of exceptions, e.g.
cdef int func():
raise Exception
Here there is no way to propagate the raised exception, so
instead one should write something like
cdef int func() except -1: ...
Alternatively one may use 'except *' in case there is no error
indicator and Cython should always check, or "except ? -1" which means
"-1 may or may not indicate an error".
2) in deallocators or finalizers (e.g. __dealloc__ or __del__)
For functions the right thing is to add an except clause, for
finalizers and destructors one could use the traceback module, e.g.
try:
...
except:
traceback.print_exc()
If this all still doesn't help, try setting a (deferred) breakpoint on
__Pyx_WriteUnraisable or PyErr_WriteUnraisable.
mark florisson
Actually, I don't see why the default is to write unraisable
exceptions. Instead Cython could detect that exceptions may propagate
and have callers do the check (i.e. make it implicitly "except *").
Was this not implemented because Cython only knows whether functions
may propagate exceptions at code generation time by looking at the
presence of an error label?
Maybe it could keep code insertion points around for every call to
such a potential function and if the function uses the error label
have the caller perform the check? Although I do forsee problems for
external such functions... maybe Cython could have it's own
threadstate regardless of the GIL which would indicate whether an
error has occurred? e.g. CyErr_Occurred()?
Simon King
On 27 Jan., 17:22, mark florisson <markflorisso...@gmail.com> wrote:
> These messages are written by PyErr_WriteUnraisable, which is a
> CPython C API function that writes unraisable exceptions.
of the error, I need to patch the Python sources?
> There are
> typically two reasons for unraisable exceptions:
> For functions the right thing is to add an except clause, ...
That is exactly what I'd love to do -- but the point is that I have
absolutely no information whatsoever on the location of the error.
WHERE should I add an except clause? WHAT cdef function/method needs
an "except -1:", "except ?:" or whatever is appropriate?
Exception AttributeError: 'PolynomialRing_field_with_category'
object has no attribute '_modulus' in ignored
"... in ignored" -- but where???
It does not tell whether it is a finaliser/destructor, or a cdef
function, or a cdef method, not the module in which the error occurs,
not the name of the function -- simply nothing!
Is there a way to get additional information without patching CPython?
Cheers,
Simon
mark florisson
Yes, I just told you, add a breakpoint to PyErr_WriteUnraisable. So basically:
$ gdb --args python -c 'import mymodule'
(gdb) break PyErr_WriteUnraisable
(gdb) run
Simon King
On 27 Jan., 18:12, mark florisson <markflorisso...@gmail.com> wrote:
> Yes, I just told you, add a breakpoint to PyErr_WriteUnraisable. So basically:
>
> $ gdb --args python -c 'import mymodule'
> (gdb) break PyErr_WriteUnraisable
> (gdb) run
with gdb than to inspect a core dump resulting from a segmentation
fault), that's why I did not understand the concluding line of your
message above: I thought "setting a breakpoint to
PyErr_WriteUnraisable" involves to add something to the source code of
PyErr_WriteUnraisable.
Thank you very much!
Simon
Stefan Behnel
> On 27 Jan., 17:22, mark florisson wrote:
>> These messages are written by PyErr_WriteUnraisable, which is a
>> CPython C API function that writes unraisable exceptions.
>
> I see. Does that mean, in order to get any information on the location
> of the error, I need to patch the Python sources?
No, Mark already pointed you to gdb.
>> There are typically two reasons for unraisable exceptions:
>> ...
>> cdef int func() except -1: ...
>> ...
>> For functions the right thing is to add an except clause, ...
>
> That is exactly what I'd love to do -- but the point is that I have
> absolutely no information whatsoever on the location of the error.
> WHERE should I add an except clause? WHAT cdef function/method needs
> an "except -1:", "except ?:" or whatever is appropriate?
>
> Exception AttributeError: 'PolynomialRing_field_with_category'
> object has no attribute '_modulus' in ignored
> is all what I get.
> "... in ignored" -- but where???
>
> It does not tell whether it is a finaliser/destructor, or a cdef
> function, or a cdef method, not the module in which the error occurs,
> not the name of the function -- simply nothing!
There's clearly something wrong here. The missing name may hint at a
destructor, as you mentioned, but it may just as well be something else.
gdb is your best bet. The "bt" command in gdb gives you a stack trace once
you've reached the breakpoint, that will tell you where the function gets
called.
Stefan
Simon King
>
> $ gdb --args python -c 'import mymodule'
> (gdb) break PyErr_WriteUnraisable
> (gdb) run
> gdb --args ./sage /home/simon/SAGE/work/memleak/debug.sage
GNU gdb (GDB) SUSE (7.3-41.1.2)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/
gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
and "show warranty" for details.
This GDB was configured as "x86_64-suse-linux".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
"/home/simon/SAGE/sage-5.0.prealpha0/sage": not in executable format:
File format not recognized
I continued with providing the symbols:
(gdb) file local/bin/python
Reading symbols from /home/simon/SAGE/sage-5.0.prealpha0/local/bin/
python...done.
Fine! But now:
(gdb) break PyErr_WriteUnraisable
Function "PyErr_WriteUnraisable" not defined.
What next? I don't know if this is more of a gdb question, a Sage
question or still a Cython question (probably not), but I'd appreciate
directions.
Best regards,
Simon
Stefan Behnel
That's ok, it hasn't seen it yet because the shared library hasn't been
loaded yet. Just let it run through once, then set the breakpoint, then run
it again.
Stefan
mark florisson
Apparently 'sage' is a bash script (and gdb does not understand
scripts). I'm not sure how well following child processes will work
here, but sage has a -gdb switch. If you use that you can trap to gdb
using ^C, set a breakpoint and 'continue'.
Usually you use pending breakpoints, so
$ gdb --args bash `which sage`
(gdb) set breakpoint pending on
(gdb) break PyErr_WriteUnraisable
(gdb) run
I'm not sure if you would need 'set follow-fork-mode child' for that.
In fact, I'm not sure if this works when executing different
executables (bash executing python). In any case, I think sage -gdb
does what you want.
Simon King
On 29 Jan., 10:54, mark florisson <markflorisso...@gmail.com> wrote:
> I'm not sure if you would need 'set follow-fork-mode child' for that.
> In fact, I'm not sure if this works when executing different
> executables (bash executing python). In any case, I think sage -gdb
> does what you want.
It doesn't mean that I can already fix my problem, but indeed "sage -
gdb" breaks at the right moment, so that
(gdb) bt
gives me something to study further.
Cheers,
Simon
Simon King
The error occurs in a cdef function that looks like this:
cdef cparent get_cparent(parent) except? NULL:
if parent is None:
return NULL
cdef ntl_ZZ_pEContext_class c
c = parent._modulus # Here is the error raised
return &(c.x)
If I understand correctly, the function is defined so that the error
will be propagated.
But this function is called during deallocation of a polynomial p:
p.__dealloc__() calls get_cparent(p.parent()).
So, this is why the error is ignored, and I can certainly add a "try -
except" clause to p.__dealloc__ to solve the problem.
Question to me: Why is the attribute "_modulus" of the polynomial ring
deleted before all of its elements are deleted?
Question to you: Do you think that the error occuring in a cdef
function (namely: occuring in get_cparent) and being ignored by some
__dealloc__ calling the cdef function can explain why the location of
the error is not shown?
I am starting with a patched version of sage-5.0.prealpha0: I think
the relevant trac tickets are #715, #11521, #12313 (they fix some
memory leaks by making polynomial rings garbage collectable). The
problem can then be triggered as follows:
sage: K.<z> = GF(4)
sage: P.<x> = K[]
sage: del P
sage: del x
sage: import gc
sage: gc.collect()
object has no attribute '_modulus' in ignored
Thank you for your help!
Cheers,
Simon
Simon King
Here is a small example, that may help you to track the problem down
on the side of Cython.
I've put the following code in some file hidden_error.pyx:
class P:
def __init__(self, m):
self._modulus = m
cdef int get_modulus(P) except? -1:
return <int>P._modulus
cdef class E:
cdef object P
def __init__(self, P):
self.P = P
def __dealloc__(self):
print get_modulus(self.P)
def __repr__(self):
return "element of P(%d)"%self.P._modulus
And then (attaching the file to a Sage session):
sage: attach /home/simon/SAGE/work/memleak/hidden_error.pyx
Compiling /home/simon/SAGE/work/memleak/hidden_error.pyx...
sage: p = P(6)
sage: e = E(p)
sage: del p._modulus
sage: del e
Exception AttributeError: "P instance has no attribute '_modulus'" in
ignored
So, if I am not mistaken, the problem is that the error occurs during
deallocation, under circumstances making it impossible to obtain the
string representation of the object.
I believe, in that situation, it would help to print something like
Exception AttributeError: "P instance has no attribute '_modulus'" in
<_home_simon_SAGE_work_memleak_hidden_error_pyx_1.E object at
0x45520c0> ignored
instead.
That's to say: Use type and id of the object, if no string
representation is available - I think I've already suggested it in
some early post of this thread.
Cheers,
Simon
Simon King
On 29 Jan., 22:04, Simon King <simon.k...@uni-jena.de> wrote:
> That's to say: Use type and id of the object, if no string
> representation is available - I think I've already suggested it in
> some early post of this thread.
same example is written in Python then the error looks exactly the
same.
Of course, one may argue that Cython could still try to provide a more
helpful error message than Python.
Cheers,
Simon
Robert Bradshaw
Nice job hunting this down.
The "exception ignored" message is actually generated by CPython, not
in Cython. That being said, CPython's default behavior is not the best
here, so perhaps we could mark __dealloc__ as not able to raise
exceptions and generate our own (incompatible, but only in terms of
its stdout side affect (? - we'd have to check this)) message.
- Robert
Stefan Behnel
Well, I would suspect that Cython knows that it can't raise an exception
from within __dealloc__() because it generates a call to
PyErr_WriteUnraisable().
> and generate our own (incompatible, but only in terms of
> its stdout side affect (? - we'd have to check this)) message.
Yes, I think a reimplementation of PyErr_WriteUnraisable() that provides
more helpful context information would be better. It could even print a
full stack trace. After all, the mere fact that PyErr_WriteUnraisable() is
getting called at all is a very good indicator for a bug in the code. Any
additional information that we can give the users may help in fixing it
more quickly.
Stefan
Robert Bradshaw
I was talking about the actual __dealloc__ implementation called by tp_dealloc.
>> and generate our own (incompatible, but only in terms of
>> its stdout side affect (? - we'd have to check this)) message.
>
> Yes, I think a reimplementation of PyErr_WriteUnraisable() that provides
> more helpful context information would be better. It could even print a
> full stack trace. After all, the mere fact that PyErr_WriteUnraisable() is
> getting called at all is a very good indicator for a bug in the code. Any
> additional information that we can give the users may help in fixing it
> more quickly.
Sounds good to me.