python 3 support update
Ondrej Certik
I fixed the import problems, so things should now work in python3,
except that the 2to3 tool breaks. :(
checkout my py3-prepare branch:
http://github.com/certik/sympy/tree/py3-prepare
and run the 2to3 tool:
$ python3.1 ~/ext/Python-3.1/Tools/scripts/2to3 . > p
RefactoringTool: Skipping implicit fixer: buffer
RefactoringTool: Skipping implicit fixer: idioms
RefactoringTool: Skipping implicit fixer: set_literal
RefactoringTool: Skipping implicit fixer: ws_comma
RefactoringTool: Can't parse ./data/IPython/ipythonrc-sympy:
ParseError: bad input: type=1, value='ipythonrc', context=(' ', (25,
8))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/BasicGAtest.py: ParseError: bad input:
type=5, value=' ', context=('', (1, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/conformalgeometryGAtest.py: ParseError:
bad input: type=5, value=' ', context=('', (1, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/headerGAtest.py: ParseError: bad input:
type=0, value='', context=('\n', (26, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/hyperbolicGAtest.py: ParseError: bad
input: type=5, value=' ', context=('', (1, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/reciprocalframeGAtest.py: ParseError:
bad input: type=5, value=' ', context=('', (1, 0))
Traceback (most recent call last):
File "/home/ondrej/ext/Python-3.1/Tools/scripts/2to3", line 6, in <module>
sys.exit(main("lib2to3.fixes"))
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/main.py", line 132, in main
options.processes)
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
544, in refactor
items, write, doctests_only)
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
207, in refactor
self.refactor_dir(dir_or_file, write, doctests_only)
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
225, in refactor_dir
self.refactor_file(fullname, write, doctests_only)
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
584, in refactor_file
*args, **kwargs)
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
264, in refactor_file
write=write, encoding=encoding)
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
363, in processed_file
self.print_output(diff_texts(old_text, new_text, filename))
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/main.py", line 47, in
print_output
print(line)
UnicodeEncodeError: 'ascii' codec can't encode character '\u03b1' in
position 32: ordinal not in range(128)
The tool from python2.6 works, but fails to convert the imports, so it
doesn't work in python3. (they fixed that in later 2to3 tool that is
in python3.1, but it fails with the above unicode problem). If someone
could look into that, it'd be awesome. I think we are very close to
gettting sympy run in python3.
Ondrej
Fabian Pedregosa
Using the workaround it all gets converted, and I pushed my changes to
http://fseoane.net/git/sympy.git, branch py3
However, I am getting some problems with string.strip() when importing
sympy:
>>> import sympy
File "sympy/__init__.py", line 23, in <module>
from sympy.core import *
File "sympy/core/__init__.py", line 4, in <module>
from .basic import Basic, S, C, sympify
File "sympy/core/basic.py", line 3, in <module>
import sympy.mpmath as mpmath
File "sympy/mpmath/__init__.py", line 3, in <module>
from .mptypes import (
File "sympy/mpmath/mptypes.py", line 40, in <module>
from string import strip
ImportError: cannot import name strip
I suppose we should be using string.format or something similar,
http://docs.python.org/3.1/library/string.html
>
> Ondrej
>
> >
>
--
http://fseoane.net/blog/
Fabian Pedregosa
update, I got this to work, just needed to use the classmethod strip()
instead of the one in the string module. Did some other fixes (it's in
my repo), but now I am blocked on this:
>>> import sympy
Traceback (most recent call last):
File "sympy/core/decorators.py", line 73, in __sympifyit_wrapper
return func(a, b)
File "sympy/core/basic.py", line 662, in __rmul__
return Mul(other, self)
File "sympy/core/cache.py", line 82, in wrapper
return func_cache_it_cache[k]
TypeError: unhashable type: 'Integer'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "sympy/__init__.py", line 25, in <module>
from .polys import *
File "sympy/polys/__init__.py", line 2, in <module>
from .monomial import monomials, monomial_count
File "sympy/polys/monomial.py", line 6, in <module>
from sympy.functions import factorial
File "sympy/functions/__init__.py", line 11, in <module>
from . import special
File "sympy/functions/special/__init__.py", line 8, in <module>
from . import polynomials
File "sympy/functions/special/polynomials.py", line 102, in <module>
class chebyshevu(PolynomialSequence):
File "sympy/functions/special/polynomials.py", line 121, in chebyshevu
@recurrence_memo([S.One, 2*_x])
File "sympy/core/decorators.py", line 100, in __sympifyit_wrapper
return func(a, b)
File "sympy/core/basic.py", line 662, in __rmul__
return Mul(other, self)
File "sympy/core/cache.py", line 82, in wrapper
return func_cache_it_cache[k]
TypeError: unhashable type: 'Integer'
>
>> Ondrej
>>
>
>
Ondrej Certik
This seems more serious, we will have to investigate it. In my branch
I also had to apply couple cmp() fixes, see the python 3.0 issue.
Ondrej
Ondrej Certik
Ok, I am now at the same error with my repo too (py3-3 branch).
I investigated this a bit and it's a bit weird --- hash(Rational(3,
2)) works, but not hash(Integer(3)). Anyway, a workaround for the time
being is not to use cache, then the error is:
$ SYMPY_USE_CACHE=no python3.1 -c "import sympy"
Traceback (most recent call last):
File "sympy/core/decorators.py", line 73, in __sympifyit_wrapper
return func(a, b)
File "sympy/core/basic.py", line 663, in __rmul__
return Mul(other, self)
File "sympy/core/operations.py", line 32, in __new__
c_part, nc_part, order_symbols = cls.flatten(list(map(_sympify, args)))
File "sympy/core/mul.py", line 303, in flatten
c_part.sort(Basic.compare)
TypeError: must use keyword argument for key function
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "sympy/__init__.py", line 25, in <module>
from .polys import *
File "sympy/polys/__init__.py", line 2, in <module>
from .monomial import monomials, monomial_count
File "sympy/polys/monomial.py", line 6, in <module>
from sympy.functions import factorial
File "sympy/functions/__init__.py", line 11, in <module>
from . import special
File "sympy/functions/special/__init__.py", line 8, in <module>
from . import polynomials
File "sympy/functions/special/polynomials.py", line 102, in <module>
class chebyshevu(PolynomialSequence):
File "sympy/functions/special/polynomials.py", line 121, in chebyshevu
@recurrence_memo([S.One, 2*_x])
File "sympy/core/decorators.py", line 100, in __sympifyit_wrapper
return func(a, b)
File "sympy/core/basic.py", line 663, in __rmul__
return Mul(other, self)
File "sympy/core/operations.py", line 32, in __new__
c_part, nc_part, order_symbols = cls.flatten(list(map(_sympify, args)))
File "sympy/core/mul.py", line 303, in flatten
c_part.sort(Basic.compare)
TypeError: must use keyword argument for key function
so some more fixes with regards to .sort() and cmp are needed, I'll
give it a shot soon, I think all those are some small things that
should be easy to fix.
Ondrej
Ondrej Certik
$ SYMPY_USE_CACHE=no python3.1 -c "import sympy"
$
so sympy at least imports. However, to give you some taste of what is
still in front of us:
============================= test process starts ==============================
executable: /home/ondrej/usr/bin/python3.1 (3.1.0-final-0)
sympy/core/tests/test_arit.py[42] EEEffEEEEEEEE..EEE....EEE.E...EEEEE...E.EE
[FAIL]
sympy/core/tests/test_assumptions.py[27] ...f......................E [FAIL]
sympy/core/tests/test_basic.py[39] EE.EEEEEEEEE....E.EEEEEEEEEEE.E.EEEEEEE
[FAIL]
sympy/core/tests/test_cache.py[1] . [OK]
sympy/core/tests/test_complex.py[10] EEEEEEEEEE [FAIL]
sympy/core/tests/test_count_ops.py[1] E [FAIL]
sympy/core/tests/test_diff.py[5] EEE.E [FAIL]
sympy/core/tests/test_equal.py[5] E.... [FAIL]
sympy/core/tests/test_eval.py[8] EEEfE..E [FAIL]
sympy/core/tests/test_eval_power.py[9] EE.EE.E.E [FAIL]
sympy/core/tests/test_evalf.py[19] .EEEEEEEEEEEEEEEEE. [FAIL]
sympy/core/tests/test_facts.py[11] ........... [OK]
sympy/core/tests/test_functions.py[?] Failed to import [FAIL]
sympy/core/tests/test_logic.py[10] ..E....... [FAIL]
sympy/core/tests/test_match.py[?] Failed to import [FAIL]
sympy/core/tests/test_multidimensional.py[4] EEEE [FAIL]
sympy/core/tests/test_numbers.py[29] ...EE...E..EE.....E..EEE...EE [FAIL]
sympy/core/tests/test_relational.py[5] ..E.. [FAIL]
sympy/core/tests/test_subs.py[?] Failed to import [FAIL]
sympy/core/tests/test_symbol.py[5] .E... [FAIL]
sympy/core/tests/test_sympify.py[?] Failed to import [FAIL]
sympy/core/tests/test_truediv.py[3] .EE [FAIL]
sympy/core/tests/test_var.py[4] .... [OK]
[...]
==== tests finished: 111 passed, 4 xfailed, 126 exceptions in 0.70 seconds =====
DO *NOT* COMMIT!
and this is just the core...
I must say I am disappointed by the 2to3 tool, which is still not quite ready:
1) it breaks on unicode (http://bugs.python.org/issue5093)
2) and it doesn't convert all imports, so still some hand work is needed.
3) it mistakes repr.py from sympy/printing and changes the import to
reprlib (fix by hand is needed)
Besides those improvements to the 2to3 tool, we need to investigate
why hash() stopped working in python 3.0, and then walk test by test
and debug it why it fails in python3.
the .sort() method now doesn't accept cmp() like callback, so we need
to emulate this, fortunately this should be easy. The sys.version_info
is backwards incompatible in python3.1 (it breaks compatibility with
python3.0!), but fortunately a fix is just to add tuple() around it.
In general there are dozens of those little things that just don't
work. I will start by reporting the bugs to the 2to3 tool, so that we
don't have to edit stuff by hand so much at least.
Ondrej
Ondrej Certik
> In general there are dozens of those little things that just don't
> work. I will start by reporting the bugs to the 2to3 tool, so that we
> don't have to edit stuff by hand so much at least.
I reported bugs that I can easily reproduce here:
http://mail.python.org/pipermail/python-porting/2009-July/000079.html
I am now looking at the hash() problem, that seems to be a cause of
lots of failures.
Ondrej
Ondrej Certik
Here is my latest state:
http://mail.python.org/pipermail/python-porting/2009-July/000082.html
Unfortunately I am afraid sympy is currently totally broken in
python3.0. If you got in touch with python3.0 already, please try to
click on the link above and reproduce the problem with __hash__, as
described in the email and think how this could be fixed.
I have absolutely no clue what is happening there.
Ondrej
Ondrej Certik
Amaury helped me with that on the python-porting list:
http://mail.python.org/pipermail/python-porting/2009-July/000083.html
so we are moving forward:
http://mail.python.org/pipermail/python-porting/2009-July/000084.html
there are a lot of other problems too, but we are not stuck anymore.
Ondrej
Aaron S. Meurer
they be backwards compatible with Python 2? I think that we should
put whatever is backwards compatible in the main release. I know a
lot of it is cleaning up imports, which will be good for both
versions. For example, I cannot currently import all() from within
mul.py because __init__.py in utilities has a "from sympy import *"
line in it that doesn't play so nice at that level. Issue 1454 is
also related to this.
Aaron Meurer
Ondrej Certik
>
> Will these changes have to go in a separate sympy-py3k release or will
> they be backwards compatible with Python 2? I think that we should
> put whatever is backwards compatible in the main release. I know a
> lot of it is cleaning up imports, which will be good for both
> versions. For example, I cannot currently import all() from within
> mul.py because __init__.py in utilities has a "from sympy import *"
> line in it that doesn't play so nice at that level. Issue 1454 is
> also related to this.
Yes, I plan to put as many fixes into our main branch. To release
python3 version would ideally amount to:
1) download our main branch
2) run the 2to3 tool
3) ./setup.py sdist
And in fact ./setup.py can get some parameter to run the 2to3 tool
automatically.
This is the ideal, in practise the 2to3 tool needs improvements
upstream to handle sympy well. The other changes should be possible to
backport.
In any case, I will do my best not to maintain two branches of sympy.
The same as with cython, I'll do my best not to maintain a cython
branch and a sympy branch. Currently unfortunately cython also needs
improvements upstream.
Ondrej
Ondrej Certik
sympy imports and some tests run (in the core), but lots of still
fails. Fortunately it's usually pretty easy to debug it using pdb and
just commit lots of little fixes. Once we fix everything, we'll try to
backport most of the changes to 2.x.
Ondrej
Aaron S. Meurer
change ntheorey/__init__.py so that it doesn't do from module import
*. This will fix issue 1454. I thought that that wasn't allowed
anymore in Python 3.0, but maybe I read it wrong.
Aaron Meurer
Ondrej Certik
>
> As long as you are changing import statements, you might as well
> change ntheorey/__init__.py so that it doesn't do from module import
> *. This will fix issue 1454. I thought that that wasn't allowed
> anymore in Python 3.0, but maybe I read it wrong.
Please go ahead and do it.
As to from something import *, that is still allowed in python3.
Ondrej
Aaron S. Meurer
issue 1454. I maintained the use of from .module import syntax, so it
should work in py3k (I don't have it to check, though. That is just
based on what I saw the 2to3 tool did in your branch).
This is what I meant. From http://docs.python.org/3.1/whatsnew/3.0.html#removed-syntax
"The from module import * syntax is only allowed at the module level,
no longer inside functions."
I guess that doesn't apply to __init__.py files.
>
> Ondrej
Also, what about Python 3.1? When we get SymPy to work in 3.0, will
it work in 3.1?
Aaron Meurer