sympy tests: jython vs cpython incompatibilities

[Ondrej Certik]

Hi,

I'd be interested in getting sympy[0] run on top of jython. It only
depends on pure python and has an extensive test suite. I noticed
that django recently started to work on top of jython, so if you have
nothing to port now, below you can find a lot of problems. :)

Here is how to play with it (I tried jython latest alpha 3):

$ git clone git://git.sympy.org/sympy.git
$ cd sympy
$ ~/ext/jython2.5a3/jython -c "import sympy"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "sympy/__init__.py", line 16, in <module>
from sympy.core import *
File "sympy/core/__init__.py", line 4, in <module>
from basic import Basic, S, C, sympify
File "sympy/core/basic.py", line 236, in <module>
class Basic(AssumeMeths):
TypeError: Error when calling the metaclass bases
mro() returned base with unsuitable layout ('Basic')

This works in python2.4, 2.5, 2.6 and pypy. It seems like some
problems with metaclasses, because this "patch" gets rid of it:

$ git diff
diff --git a/sympy/core/basic.py b/sympy/core/basic.py
index d5e4262..d369ec2 100644
--- a/sympy/core/basic.py
+++ b/sympy/core/basic.py
@@ -268,7 +268,7 @@ class Basic(AssumeMeths):

"""

- __metaclass__ = BasicMeta
+ #__metaclass__ = BasicMeta

__slots__ = ['_mhash', # hash value
'_args', # arguments

Unfortunately, the next problem is java.lang.NullPointerException,
which I have no idea how to fix:

$ ~/ext/jython2.5a3/jython -c "import sympy"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "sympy/__init__.py", line 16, in <module>
from sympy.core import *
File "sympy/core/__init__.py", line 6, in <module>
from numbers import Number, Real, Rational, Integer, igcd, ilcm,
RealNumber, \
java.lang.NullPointerException
at org.python.compiler.CodeCompiler.visitLambda(CodeCompiler.java:1798)
at org.python.antlr.ast.Lambda.accept(Lambda.java:53)
at org.python.antlr.Visitor.visit(Visitor.java:26)
at org.python.compiler.CodeCompiler.makeArray(CodeCompiler.java:342)
at org.python.compiler.CodeCompiler.visitCall(CodeCompiler.java:1503)
at org.python.antlr.ast.Call.accept(Call.java:114)
at org.python.antlr.Visitor.visit(Visitor.java:26)
at org.python.compiler.CodeCompiler.visitCall(CodeCompiler.java:1487)
at org.python.antlr.ast.Call.accept(Call.java:114)
at org.python.antlr.Visitor.visit(Visitor.java:26)
at org.python.compiler.CodeCompiler.doDecorators(CodeCompiler.java:430)
at org.python.compiler.CodeCompiler.visitFunctionDef(CodeCompiler.java:419)
at org.python.antlr.ast.FunctionDef.accept(FunctionDef.java:98)
at org.python.antlr.ast.Module.traverse(Module.java:65)
at org.python.antlr.Visitor.traverse(Visitor.java:12)
at org.python.compiler.CodeCompiler.visitModule(CodeCompiler.java:310)
at org.python.antlr.ast.Module.accept(Module.java:58)
at org.python.antlr.Visitor.visit(Visitor.java:26)
at org.python.compiler.CodeCompiler.parse(CodeCompiler.java:269)
at org.python.compiler.Module.PyCode(Module.java:467)
at org.python.compiler.Module.compile(Module.java:646)
at org.python.core.imp.compileSource(imp.java:203)
at org.python.core.imp.createFromSource(imp.java:224)
at org.python.core.imp.loadFromSource(imp.java:478)
at org.python.core.imp.find_module(imp.java:379)
at org.python.core.PyModule.impAttr(PyModule.java:109)
at org.python.core.imp.import_next(imp.java:566)
at org.python.core.imp.import_name(imp.java:664)
at org.python.core.imp.importName(imp.java:724)
at org.python.core.ImportFunction.load(__builtin__.java:1268)
at org.python.core.ImportFunction.__call__(__builtin__.java:1264)
at org.python.core.PyObject.__call__(PyObject.java:251)
at org.python.core.__builtin__.__import__(__builtin__.java:1225)
at org.python.core.imp.importFromAs(imp.java:775)
at org.python.core.imp.importFrom(imp.java:761)
at sympy.core$py.f$0(sympy/core/__init__.py)
at sympy.core$py.call_function(sympy/core/__init__.py)
at org.python.core.PyTableCode.call(PyTableCode.java:187)
at org.python.core.PyCode.call(PyCode.java:14)
at org.python.core.imp.createFromCode(imp.java:258)
at org.python.core.imp.createFromPyClass(imp.java:107)
at org.python.core.imp.loadFromSource(imp.java:471)
at org.python.core.imp.find_module(imp.java:379)
at org.python.core.PyModule.impAttr(PyModule.java:109)
at org.python.core.imp.import_next(imp.java:566)
at org.python.core.imp.import_logic(imp.java:623)
at org.python.core.imp.import_name(imp.java:679)
at org.python.core.imp.importName(imp.java:724)
at org.python.core.ImportFunction.load(__builtin__.java:1268)
at org.python.core.ImportFunction.__call__(__builtin__.java:1264)
at org.python.core.PyObject.__call__(PyObject.java:251)
at org.python.core.__builtin__.__import__(__builtin__.java:1225)
at org.python.core.imp.importAll(imp.java:802)
at sympy$py.f$0(sympy/__init__.py)
at sympy$py.call_function(sympy/__init__.py)
at org.python.core.PyTableCode.call(PyTableCode.java:187)
at org.python.core.PyCode.call(PyCode.java:14)
at org.python.core.imp.createFromCode(imp.java:258)
at org.python.core.imp.createFromPyClass(imp.java:107)
at org.python.core.imp.loadFromSource(imp.java:471)
at org.python.core.imp.find_module(imp.java:379)
at org.python.core.imp.import_next(imp.java:564)
at org.python.core.imp.import_name(imp.java:664)
at org.python.core.imp.importName(imp.java:724)
at org.python.core.ImportFunction.load(__builtin__.java:1268)
at org.python.core.ImportFunction.__call__(__builtin__.java:1264)
at org.python.core.PyObject.__call__(PyObject.java:251)
at org.python.core.__builtin__.__import__(__builtin__.java:1225)
at org.python.core.imp.importOne(imp.java:735)
at org.python.pycode._pyx0.f$0(<string>)
at org.python.pycode._pyx0.call_function(<string>)
at org.python.core.PyTableCode.call(PyTableCode.java:187)
at org.python.core.PyCode.call(PyCode.java:14)
at org.python.core.Py.runCode(Py.java:1200)
at org.python.core.Py.exec(Py.java:1227)
at org.python.util.PythonInterpreter.exec(PythonInterpreter.java:122)
at org.python.util.jython.run(jython.java:258)
at org.python.util.jython.main(jython.java:114)

java.lang.NullPointerException: java.lang.NullPointerException

Is there some way to get jython print some more useful error? Anyway,
by trial and error, the problem is somewhere in the
sympy/core/numbers.py file, so when commenting all places that import
it, one gets this error:

$ ~/ext/jython2.5a3/jython -c "import sympy"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "sympy/__init__.py", line 16, in <module>
from sympy.core import *
File "sympy/core/__init__.py", line 14, in <module>
from function import Lambda, WildFunction, Derivative, diff,
FunctionClass, \
File "sympy/core/function.py", line 76, in <module>
class Function(Basic):
File "sympy/core/function.py", line 71, in __new__
return type.__new__(cls, name, bases, attrdict)
TypeError: Error when calling the metaclass bases
mro() returned base with unsuitable layout ('Function')

So one again needs to comment out the lines with:

- __metaclass__ = FunctionClass
+ #__metaclass__ = FunctionClass

in couple more classes and then also:

-class WildFunction(Function, Atom):
+class WildFunction(Function):

Until one finally gets:

$ ~/ext/jython2.5a3/jython -c "import sympy"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "sympy/__init__.py", line 16, in <module>
from sympy.core import *
File "sympy/core/__init__.py", line 20, in <module>
for _n, _cls in Basic.singleton.items():
AttributeError: type object 'Basic' has no attribute 'singleton'

Which is due to the fact that we commented out of the metaclasses. So
to sum up, there are two problems:

1) jython's metaclasses are different to cpython, so either jython or
sympy needs to be investigated and fixed
2) there is some problem in sympy/core/numbers.py, that importing it
throws java.lang.NullPointerException

if some jython developer has any ideas to move forward, it'd be awesome.

Ondrej

[0] http://code.google.com/p/sympy/

[Alex_Gaynor]

Jython definately supports metaclasses(and by all accounts fully
supports them), django uses them extensively, and they are show to
work fine there. So my guess is the problem is in sympy for the
metaclass issue. I don't know enough about either to be useful beyond
that.

[Ondrej Certik]

On Fri, Oct 10, 2008 at 12:12 AM, Philip Jenvey <pje...@underboss.org> wrote:
>
> On Oct 8, 2008, at 7:29 PM, Philip Jenvey wrote:

>
>>
>> On Oct 8, 2008, at 5:58 PM, Ondrej Certik wrote:
>>
>>> Here is how to play with it (I tried jython latest alpha 3):
>>>
>>> $ git clone git://git.sympy.org/sympy.git
>>> $ cd sympy
>>> $ ~/ext/jython2.5a3/jython -c "import sympy"
>>> Traceback (most recent call last):
>>> File "<string>", line 1, in <module>
>>> File "sympy/__init__.py", line 16, in <module>
>>> from sympy.core import *
>>> File "sympy/core/__init__.py", line 4, in <module>
>>> from basic import Basic, S, C, sympify
>>> File "sympy/core/basic.py", line 236, in <module>
>>> class Basic(AssumeMeths):
>>> TypeError: Error when calling the metaclass bases
>>> mro() returned base with unsuitable layout ('Basic')
>>>
>>

>> Definitely log a ticket for this one.
>
> Actually, don't bother logging these. I've got this TypeError figured out.
>
> The NPE you posted is also still present in trunk, but I've about got a fix
> for that one. The compiler isn't handling lambda within a decorator's
> arglist correctly, the offending line was in numbers.py:
>
> @Memoizer((int, long), return_value_converter = lambda d: d.copy())
>
> With those two squashed I think I've unearthed some more odd TypeErrors.

Thanks for all the feedback. Let me know when it's fixed, I'll retry
it again and then I'll also try to figure out the metaclasses problem
-- as Alex suggested, it can be a problem in sympy, but sympy works on
all cpython 2.4, 2.5 and 2.6, so definitely jython should not just
throw some java exception.

Getting sympy to import is just the first thing. The real fun will
begin when running the actuall tests, so I am sure more problems will
popup. But generally, jython seems to be in a pretty good shape. Btw,
pypy had similar problems too in the past and ironpython also cannot
import sympy.

Ondrej

P.S. I am going to the google mentors summit and I'll be in the bay
area from 27-29. If some of you are be around, let me know.

[Ondrej Certik]

On Fri, Oct 10, 2008 at 12:12 AM, Philip Jenvey <pje...@underboss.org> wrote:
>
> On Oct 8, 2008, at 7:29 PM, Philip Jenvey wrote:
>
>>
>> On Oct 8, 2008, at 5:58 PM, Ondrej Certik wrote:
>>

>>> Here is how to play with it (I tried jython latest alpha 3):
>>>
>>> $ git clone git://git.sympy.org/sympy.git
>>> $ cd sympy
>>> $ ~/ext/jython2.5a3/jython -c "import sympy"
>>> Traceback (most recent call last):
>>> File "<string>", line 1, in <module>
>>> File "sympy/__init__.py", line 16, in <module>
>>> from sympy.core import *
>>> File "sympy/core/__init__.py", line 4, in <module>
>>> from basic import Basic, S, C, sympify
>>> File "sympy/core/basic.py", line 236, in <module>
>>> class Basic(AssumeMeths):
>>> TypeError: Error when calling the metaclass bases
>>> mro() returned base with unsuitable layout ('Basic')
>>>
>>

>> Definitely log a ticket for this one.
>
> Actually, don't bother logging these. I've got this TypeError figured out.
>
> The NPE you posted is also still present in trunk, but I've about got a fix
> for that one. The compiler isn't handling lambda within a decorator's
> arglist correctly, the offending line was in numbers.py:
>
> @Memoizer((int, long), return_value_converter = lambda d: d.copy())
>
> With those two squashed I think I've unearthed some more odd TypeErrors.

I just found your blog post about this fix:

http://dunderboss.blogspot.com/2008/10/debugging-jython-compiler.html

nice job! When I get to it, I'll try it again and try to isolate the
remaining bugs if there are any.

Ondrej

[Ondrej Certik]

On Wed, Oct 15, 2008 at 2:59 AM, Philip Jenvey <pje...@underboss.org> wrote:

>
> On Oct 13, 2008, at 7:44 AM, Ondrej Certik wrote:
>>
>> nice job! When I get to it, I'll try it again and try to isolate the
>> remaining bugs if there are any.
>>
>

> On the latest Jython trunk I've got it importing and the tests run but die:
>
> Jython 2.5a3+ (trunk:5394, Oct 14 2008, 17:56:35)
> [Java HotSpot(TM) Client VM (Apple Inc.)] on java1.5.0_16
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import sympy
>>>> sympy.test("sympy/core")
> ============================= test process starts
> ==============================
> py.test like reporting.
>
> sympy/core/tests/test_arit.py[41] ...ff....................................
> sympy/core/tests/test_assumptions.py[27] ..f........................
> sympy/core/tests/test_basic.py[32] fE...................EEEEEEEEEEE
> sympy/core/tests/test_complex.py[10] EEEEEEEEEE

> Traceback (most recent call last):

> File "<stdin>", line 1, in <module>
> File "sympy/utilities/runtests.py", line 54, in test
> return t.test()
> File "sympy/utilities/runtests.py", line 89, in test
> self.test_file(f)
> File "sympy/utilities/runtests.py", line 103, in test_file
> module = imp.load_source(name, filename)
> File "/Users/pjenvey/src/python/sympy/sympy/core/tests/test_count_ops.py",
> line 3, in <module>
> x, y, z = symbols('xyz')
> File "sympy/core/symbol.py", line 197, in symbols
> result = [ Symbol(name, **kwargs) for name in names[0] ]
> RuntimeError: maximum __call__ recursion depth exceeded
>
> Any more narrowing down of this current blocker bug would be appreciated --
> it'd also be nice to know what the other failures and errors were about.

Excellent job! Now we are on a good track, I'll try to look at it soon
and try to figure out why it fails.

Ondrej