sage and python 2.7: migration strategies?
Francois Bissey
Following an initial post by Paulo from mandriva I started
to look at using sage with python-2.7 (2.7.1 in fact).
The news is two fold:
1) sage seems to work ok. It starts.
2) a good deal of the test suite is shot.
As a consequence of 2 it is difficult to work out which parts
need to be checked.
The main issue is the migration to unittest2 back from python-3.
In unittest2 the behavior of assertEqual as changed and by default
doesn't know anything about derived type leading to a pile of stuff
like this:
sage -t -force_lib "devel/sage/sage/algebras/affine_nil_temperley_lieb.py"
^[[?
1034h**********************************************************************
File
"/Users/frb15/Desktop/Gentoo/usr/share/sage/devel/sage/sage/algebras/affine_nil_temperley_lieb.py",
line 57:
sage: TestSuite(A).run()
Expected nothing
Got:
Failure in _test_an_element:
Traceback (most recent call last):
File "/Users/frb15/Desktop/Gentoo/usr/lib/python2.7/site-
packages/sage/misc/sage_unittest.py", line 275, in run
test_method(tester = tester)
File "/Users/frb15/Desktop/Gentoo/usr/lib/python2.7/site-
packages/sage/categories/sets_cat.py", line 388, in _test_an_element
tester.assertEqual(self(an_element), an_element, "element construction
is not idempotent")
File "/Users/frb15/Desktop/Gentoo/usr/lib/python2.7/unittest/case.py",
line 493, in assertEqual
assertion_func = self._getAssertEqualityFunc(first, second)
File "/Users/frb15/Desktop/Gentoo/usr/lib/python2.7/unittest/case.py",
line 476, in _getAssertEqualityFunc
asserter = self._type_equality_funcs.get(type(first))
AttributeError: 'InstanceTester' object has no attribute
'_type_equality_funcs'
------------------------------------------------------------
Failure in _test_eq:
Traceback (most recent call last):
File "/Users/frb15/Desktop/Gentoo/usr/lib/python2.7/site-
packages/sage/misc/sage_unittest.py", line 275, in run
test_method(tester = tester)
File "element.pyx", line 533, in
sage.structure.element.Element._test_eq (sage/structure/element.c:3896)
File "/Users/frb15/Desktop/Gentoo/usr/lib/python2.7/unittest/case.py",
line 493, in assertEqual
assertion_func = self._getAssertEqualityFunc(first, second)
File "/Users/frb15/Desktop/Gentoo/usr/lib/python2.7/unittest/case.py",
line 476, in _getAssertEqualityFunc
asserter = self._type_equality_funcs.get(type(first))
AttributeError: 'InstanceTester' object has no attribute
'_type_equality_funcs'
------------------------------------------------------------
Failure in _test_pickling:
Traceback (most recent call last):
File "/Users/frb15/Desktop/Gentoo/usr/lib/python2.7/site-
packages/sage/misc/sage_unittest.py", line 275, in run
test_method(tester = tester)
File "sage_object.pyx", line 396, in
sage.structure.sage_object.SageObject._test_pickling
(sage/structure/sage_object.c:312$
File "/Users/frb15/Desktop/Gentoo/usr/lib/python2.7/unittest/case.py",
line 493, in assertEqual
assertion_func = self._getAssertEqualityFunc(first, second)
File "/Users/frb15/Desktop/Gentoo/usr/lib/python2.7/unittest/case.py",
line 476, in _getAssertEqualityFunc
asserter = self._type_equality_funcs.get(type(first))
AttributeError: 'InstanceTester' object has no attribute
'_type_equality_funcs'
There are a few issues of numerical and formatting noise here and there
but because of the number of failures of assertEqual the signal to noise ratio
is very high.
So if we want to migrate to python-2.7 (or even 3) one day we need a
strategy to fix these tests and document it so that current ongoing development
doesn't lead to further problems.
Bonus point if we can do that while preserving backward compatibility.
Francois
This email may be confidential and subject to legal privilege, it may
not reflect the views of the University of Canterbury, and it is not
guaranteed to be virus free. If you are not an intended recipient,
please notify the sender immediately and erase all copies of the message
and any attachments.
Please refer to http://www.canterbury.ac.nz/emaildisclaimer for more
information.
Nicolas M. Thiery
On Fri, Apr 08, 2011 at 02:08:57PM +1200, Francois Bissey wrote:
> Following an initial post by Paulo from mandriva I started
> to look at using sage with python-2.7 (2.7.1 in fact).
> The news is two fold:
> 1) sage seems to work ok. It starts.
> 2) a good deal of the test suite is shot.
Thanks for the report!
> As a consequence of 2 it is difficult to work out which parts
> need to be checked.
>
> The main issue is the migration to unittest2 back from python-3.
> In unittest2 the behavior of assertEqual as changed and by default
> doesn't know anything about derived type leading to a pile of stuff
> like this:
>
> sage -t -force_lib "devel/sage/sage/algebras/affine_nil_temperley_lieb.py"
> ^[[?
> 1034h**********************************************************************
> File
> "/Users/frb15/Desktop/Gentoo/usr/share/sage/devel/sage/sage/algebras/affine_nil_temperley_lieb.py",
> line 57:
> sage: TestSuite(A).run()
> Expected nothing
> Got:
> ...
> AttributeError: 'InstanceTester' object has no attribute '_type_equality_funcs'
Ok, this sounds very localized and should be easy to fix. I could
reproduce the problem on Python 2.6 by installing the backport of the
new version of unittest of 2.7 with:
> sage -sh
> easy_install unittest2
and replacing in sage/misc/sage_unittest.py:
import unittest
by
import unittest2 as unittest
The problem comes from the fact that InstanceTester.__init__ does not
call its base's __init__ as requested by the documentation of
unittest.TestCase (I take the blame for that). Adding it seems to fix
the problem, and so far does not cause trouble; let's see what the
buildbot will say. I just created #11156 for this and set you as
reviewer :-)
This does not fix everything since, with the new unittest, the error
reporting messages have changed a bit. But at least this should remove
most of the noise and give a better idea of the other issues that
could come up with Python 2.7.
Cheers,
Nicolas
--
Nicolas M. Thi�ry "Isil" <nth...@users.sf.net>
http://Nicolas.Thiery.name/
Francois Bissey
> Hi François,
Merci Nicolas,
there is a more comprehensive test log in #9958. This particular one was the
first stop. I can't wait to see what your patch will produce.
Francois
Francois Bissey
Thanks to Nicolas there is a lot of progress there. Initial test by my friend
Steve has shown that test failures is way down and the problems with
assertEqual seems to be mostly solved.
Next we have a number of warning messages that have changed and the number
of significant figures displayed by default has dropped by one.
The next worry are the deprecation warnings issued by sage which have
completely disappeared:
sage -t -long -force_lib devel/sage-main/sage/misc/misc.py
**********************************************************************
File "/storage/strogdon/gentoo/usr/share/sage/devel/sage-
main/sage/misc/misc.py", line 2097:
sage: foo()
Expected:
doctest:...: DeprecationWarning: The function foo is replaced by bar.
Got nothing
**********************************************************************
File "/storage/strogdon/gentoo/usr/share/sage/devel/sage-
main/sage/misc/misc.py", line 2103:
sage: bar()
Expected:
doctest:...: DeprecationWarning: (Since Sage Version 4.2) The function bar
is removed.
Got nothing
**********************************************************************
File "/storage/strogdon/gentoo/usr/share/sage/devel/sage-
main/sage/misc/misc.py", line 2168:
sage: blo()
Expected:
doctest:1: DeprecationWarning: (Since Sage Version 42.132) blo is
deprecated. Please use bla instead.
42
Got:
42
**********************************************************************
File "/storage/strogdon/gentoo/usr/share/sage/devel/sage-
main/sage/misc/misc.py", line 2217:
sage: g(5)
Expected:
doctest:...: DeprecationWarning: (Since Sage Version 42.132) g is
deprecated. Please use number_of_partitions instead.
7
Got:
7
**********************************************************************
File "/storage/strogdon/gentoo/usr/share/sage/devel/sage-
main/sage/misc/misc.py", line 2227:
sage: cls().old_meth()
Expected:
doctest:...: DeprecationWarning: (Since Sage Version 42.132) old_meth is
deprecated. Please use new_meth instead.
42
Got:
42
**********************************************************************
File "/storage/strogdon/gentoo/usr/share/sage/devel/sage-
main/sage/misc/misc.py", line 2270:
sage: is_prime(3)
Expected:
doctest:...: DeprecationWarning:
Using is_prime from here is deprecated.
True
Got:
True
**********************************************************************
File "/storage/strogdon/gentoo/usr/share/sage/devel/sage-
main/sage/misc/misc.py", line 2276:
sage: is_prime(3)
Expected:
doctest:...: DeprecationWarning:
Using is_prime from here is deprecated. If you need to use it, please
import it directly from sage.rings.arith.
True
Got:
True
**********************************************************************
There are more. It suggest that the mechanism issuing the warning is "broken".
Any ideas for that next one?
Francois
Francois Bissey
It turns out that warnings are not visible to the end users anymore.
This is according to the python documentation:
http://docs.python.org/library/warnings.html
Section 27.6.5 to be precise.
Actually more explicit in http://docs.python.org/using/cmdline.html :
Starting from Python 2.7, DeprecationWarning and its descendants are ignored
by default. The -Wd option can be used to re-enable them.
Passing -Wd to python I get to see the warnings again but to see the warnings
globally we'll need to add:
PYTHONWARNINGS=default
to sage-env this is I hope harmless for 2.6 but useful for 2.7.
Francois
Paulo César Pereira de Andrade
> It turns out that warnings are not visible to the end users anymore.
> This is according to the python documentation:
> http://docs.python.org/library/warnings.html
> Section 27.6.5 to be precise.
> Actually more explicit in http://docs.python.org/using/cmdline.html :
> Starting from Python 2.7, DeprecationWarning and its descendants are ignored
> by default. The -Wd option can be used to re-enable them.
>
> Passing -Wd to python I get to see the warnings again but to see the
> warnings
> globally we'll need to add:
> PYTHONWARNINGS=default
> to sage-env this is I hope harmless for 2.6 but useful for 2.7.
I am just ignoring doctest failures due to it expecting a warning
but no warning generated.
But I am almost ashamed of myself by adding this to
sage/all.py:
import gc
gc.set_debug(gc.DEBUG_SAVEALL)
I still need to run a full sage -testall with that to see if
there are any remaining crashes. Last one I was debugging
was rooted in
sage/libs/singular/groebner_strategy.pyx:GroebnerStrategy:__dealloc__
where the call id_Delete(&self._strat.Shdl, self._parent._ring)
would pass "garbage" to the related singular function.
Maybe for the people more familiar with sage internals
it would be easier to figure out why python thinks that
memory is unreachable, and it can garbage collect it.
Disabling gc has a very bad side effect, e.g. I am
testing a sage rebuild, and sphynx is already using
almost 3Gb of memory while generating documentation.
> Francois
Paulo
Francois Bissey
> 2011/4/9 Francois Bissey <francoi...@canterbury.ac.nz>:
> > It turns out that warnings are not visible to the end users anymore.
> > This is according to the python documentation:
> > http://docs.python.org/library/warnings.html
> > Section 27.6.5 to be precise.
> > Actually more explicit in http://docs.python.org/using/cmdline.html :
> > Starting from Python 2.7, DeprecationWarning and its descendants are
> > ignored by default. The -Wd option can be used to re-enable them.
> >
> > Passing -Wd to python I get to see the warnings again but to see the
> > warnings
> > globally we'll need to add:
> > PYTHONWARNINGS=default
> > to sage-env this is I hope harmless for 2.6 but useful for 2.7.
>
> I am just ignoring doctest failures due to it expecting a warning
> but no warning generated.
>
Hi Paulo,
we have move a bit from this hack now:
http://trac.sagemath.org/sage_trac/ticket/11244
> But I am almost ashamed of myself by adding this to
> sage/all.py:
>
> import gc
> gc.set_debug(gc.DEBUG_SAVEALL)
>
> I still need to run a full sage -testall with that to see if
> there are any remaining crashes. Last one I was debugging
> was rooted in
> sage/libs/singular/groebner_strategy.pyx:GroebnerStrategy:__dealloc__
> where the call id_Delete(&self._strat.Shdl, self._parent._ring)
> would pass "garbage" to the related singular function.
>
> Maybe for the people more familiar with sage internals
> it would be easier to figure out why python thinks that
> memory is unreachable, and it can garbage collect it.
>
> Disabling gc has a very bad side effect, e.g. I am
> testing a sage rebuild, and sphynx is already using
> almost 3Gb of memory while generating documentation.
>
Martin (our "chief sage-on-gentoo" debugger of glibc fame) has
tracked down that one to cython:
https://github.com/cschwan/sage-on-gentoo/issues/51#issuecomment-1181259
and we now have a trac ticket for it:
http://trac.sagemath.org/sage_trac/ticket/11339
so far it is not encouraging.
I consider this particular issue to be a "canary in a coal mine" case.
It is revealed by the upgrade to python-2.7 but it is just waiting to
bite people on 2.6, in fact there are indication in the code that
similar issues have been worked around band aid style in the code
before.
Francois
Paulo César Pereira de Andrade
I needed to revert that change, because a computer with 4GB
or ram would not be able to rebuild sage from source; could
patch after rebuilding documentation, but would just leave
the problem for the user...
> Martin (our "chief sage-on-gentoo" debugger of glibc fame) has
> tracked down that one to cython:
> https://github.com/cschwan/sage-on-gentoo/issues/51#issuecomment-1181259
> and we now have a trac ticket for it:
> http://trac.sagemath.org/sage_trac/ticket/11339
> so far it is not encouraging.
>
> I consider this particular issue to be a "canary in a coal mine" case.
> It is revealed by the upgrade to python-2.7 but it is just waiting to
> bite people on 2.6, in fact there are indication in the code that
> similar issues have been worked around band aid style in the code
> before.
If I understand it correctly, at least for this particular case:
sage/libs/singular/groebner_strategy.pyx:GroebnerStrategy(SageObject)._parent =>
sage/rings/polynomial/multi_polynomial_libsingular.pyx:MPolynomialRing_libsingular(MPolynomialRing_generic)
and the _parent field has a __dealloc__ as:
singular_ring_delete(self._ring) while the GroebnerStrategy
__dealloc__ deferences _parent._ring.
So, again if I understand correctly, the crash is happening because
MPolynomialRing_libsingular:__deallocate__ should be called
after pyx:GroebnerStrategy:__deallocate__, but in python 2.7
this is not happening, as the order is not guaranteed, and
apparently just happens to be in the correct order with
python 2.6.
Maybe python's Modules/gcmodule.c is the one at fault here
then? I mean, it should not just release a list of objects, but
should make some kind of topological sort before releasing
objects? But that probably would still find cycles in other
cases, so, it would more likely need to have an api to, and use
it to construct some list of pointers to be released, but that
assuming stuff is allocated with malloc...
Damn, this indeed does not look much encouraging :-)
> Francois
Paulo
Paulo César Pereira de Andrade
<paulo.cesar.per...@gmail.com> escreveu:
> If I understand it correctly, at least for this particular case:
>
> sage/libs/singular/groebner_strategy.pyx:GroebnerStrategy(SageObject)._parent =>
> sage/rings/polynomial/multi_polynomial_libsingular.pyx:MPolynomialRing_libsingular(MPolynomialRing_generic)
>
> and the _parent field has a __dealloc__ as:
> singular_ring_delete(self._ring) while the GroebnerStrategy
> __dealloc__ deferences _parent._ring.
>
> So, again if I understand correctly, the crash is happening because
> MPolynomialRing_libsingular:__deallocate__ should be called
> after pyx:GroebnerStrategy:__deallocate__, but in python 2.7
> this is not happening, as the order is not guaranteed, and
> apparently just happens to be in the correct order with
> python 2.6.
Sorry for replying to myself, but actually, it should be quite
simple to have it functional, just that it must be done in a case
by case approach. Something like:
GroebnerStrategy():
__new__:
...
the_ring_handle = ...
incref(the_ring_handle)
__del__:
...
decref(the_ring_handle)
del the_ring_handle ## requirement of this depends on semantics
MPolynomialRing_libsingular():
__new__:
...
the_ring_ptr = ...
__del__:
if (refcnt(self) <= 0) ## how to do this in python? check if
== 1? or do not use python reference counter?
free(the_ring_ptr)
This should be easy for python internal experts to
implement (I can try to experiment with that, but I do
not know much of python and only do mark&sweep :-)
Paulo
Francois Bissey
Quoting Volker:
Just to clarify, right now Cython extension classes do not call the Python
destructor __del__ upon finalization, so Sage can't rely on this mechanism. I'm
not quite sure if it is an intentional thing or if the Cython devs just
haven't gotten around to implementing it.
So we need cython to catch up with that one.
Francois
Paulo César Pereira de Andrade
I started doing some experiments mostly for the sake of testing :-)
But my python/cython kung-fu is too weak, as well as general
knowledge of the sage "object model" and my experiments
ended in the same backtrace, so I did not work on the correct
point or something, and for now I give up.
> So we need cython to catch up with that one.
Cython probably will also not understand the context of
memory being allocated and released outside its control.
The proper approach should be to ensure that python does
not see reachable sage objects as a looping cycle with 0
references, and then "breaks" the cycle in unpredictable
ways.
In my basic understanding of sage classes, it should
release things somewhat like:
>>> del a
===>
list = <set of objects reachable from a>
while length(list):
foreach item in list:
if item._parent == item:
<blame the user>
item._parent = None
test = map(list, _._parent == item)
if length(test) == 0:
item.__dealloc__()
remove(list, item)
> Francois
Paulo
Paulo César Pereira de Andrade
Err, ignore it, this is bogus for some "complex" cycles,
and should be the case here, where somewhere down in
MPolynomialRing_libsingular it may reference again
GroebnerStrategy.
I will become silent now :-) Hopefully at least this
brought the problem to attention of someone that
knows what is going on.
>> Francois
Paulo