Problem with ValueError: <classname> has the wrong size, try recompiling

[santa]

Hi,
I've been running against a wall trying to solve the following
problem: in a flat package without subpackages I have a class defined
in cython with a .pyx and .pxd file, another .pyx file using that
class and a regular python file that imports the other pyx file. This
setup constantly throws a ValueError at runtime. I've created a
minimized testcase to show the issue:
http://www.senarclens.eu/~gerald/tc/c_test.tar.gz shows the issue
(needs to be extracted to a path that is included in PYTHONPATH)
http://www.senarclens.eu/~gerald/tc/c_test_w.tar.gz works properly
The only difference between the two is that the second does not use
package imports, thus a simple diff yields
$ diff c_test/ c_test_w/
Only in c_test/: __init__.py
diff c_test//problemreader.pyx c_test_w//problemreader.pyx
1,2c1,2
< from c_test.node cimport Node
< from c_test.node import Node
---
> from node cimport Node
> from node import Node
diff c_test//test.py c_test_w//test.py
2c2
< from c_test import problemreader
---
> import problemreader

Which options do I have to solve/ circumvent the problem? Why does it
even occur and is it just me who believes that the ValueError is
highly misleading? Any help is greatly appreciated,
Gerald

[Robert Bradshaw]

The short answer:

Recompile all your .pyx files again (you can force this by removing
all your .c files) or use a setup like
http://wiki.cython.org/enhancements/distutils_preprocessing that
understands interdependencies.

The long answer:

You have node.pyx, node.pxd, problemreader.pyx. When compiling
problemreader.pyx it looks at node.pxd and uses information about
object struct layouts, etc. to generate the .c file. If you then
modify node.pxd without re-compiling problemreader.pyx the C-level
layout of your Node class can change rendering it incompatible with
problemreader.c. Cython detects this at runtime (when it can) and
trows this value error.

- Robert

[santa]

Hi,

thanks for your answer! I just upgraded to the current version of cython to make sure I have a working cythonize (w/ sudo pip install cython --upgrade). 

cython --version yields Cython version 0.14.1

Recompile all your .pyx files again (you can force this by removing

all your .c files)

Doesn't work when removing all .c, .o and .so files and then directly calling setup.py afterwards; same issue. Manually cythonizing the pyx files first (cython *.pyx) and then calling the old setup works like a charm. :)

or use a setup like
http://wiki.cython.org/enhancements/distutils_preprocessing that
understands interdependencies.

Using the exact setup.py script proposed in your link (with cython 0.14.1), I get the following:

<terminal>

~/bitbucket/c_test >python setup.py build_ext --inplace

Traceback (most recent call last):

  File "setup.py", line 6, in <module>

    ext_modules = cythonize(["*.pyx"]),

  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 456, in cythonize

    aliases=aliases)

  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 426, in create_extension_list

    pkg = deps.package(file)

  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 21, in wrapper

    res = cache[args] = f(self, *args)

  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 287, in package

    return self.package(dir) + (os.path.basename(dir),)

  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 21, in wrapper

    res = cache[args] = f(self, *args)

  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 287, in package

    return self.package(dir) + (os.path.basename(dir),)

... # removed a few hundred repeating lines

  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 287, in package

    return self.package(dir) + (os.path.basename(dir),)

  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 21, in wrapper

    res = cache[args] = f(self, *args)

  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 286, in package

    if os.path.exists(os.path.join(dir, '__init__.py')):

  File "/usr/lib/python2.7/posixpath.py", line 68, in join

    elif path == '' or path.endswith('/'):

RuntimeError: maximum recursion depth exceeded in cmp

</terminal>

 

The long answer:

You have node.pyx, node.pxd, problemreader.pyx. When compiling
problemreader.pyx it looks at node.pxd and uses information about
object struct layouts, etc. to generate the .c file.

ok

If you then
modify node.pxd without re-compiling problemreader.pyx the C-level
layout of your Node class can change rendering it incompatible with
problemreader.c. Cython detects this at runtime (when it can) and
trows this value error.

 I think this would be much easier to understand if I actually modified anything... but the problem occurs on executing the setup script on a version without any c etc. files. Everything is generated, then nothing is changed and the error still occurs. Would be nice to be able to understand this, however I'm happy that I have now one way of creating everything that doesn't fail (manually cythonizing the .pyx files and then calling the old setup). I guess whatever goes wrong with the new setup script deserves a look as well - do you want me to file a bug? [1]

/gerald

[1] I couldn't find any related bug on http://trac.cython.org.

[Robert Bradshaw]

On Thu, Aug 4, 2011 at 1:31 AM, santa <gerald.s...@gmail.com> wrote:
> Hi,
> thanks for your answer! I just upgraded to the current version of cython to
> make sure I have a working cythonize (w/ sudo pip install cython
> --upgrade).
> cython --version yields Cython version 0.14.1
>>
>> Recompile all your .pyx files again (you can force this by removing
>>
>> all your .c files)
>
> Doesn't work when removing all .c, .o and .so files and then directly
> calling setup.py afterwards; same issue. Manually cythonizing the pyx files
> first (cython *.pyx) and then calling the old setup works like a charm. :)

Removing the .c files should cause it to create new ones as if you did
it manually. Are you sure you didn't have some old object files cached
somewhere? Are you invoking the same version of Cython from setup.py
as you are from the command line?

Hmm... I'll take a look at that. Could you re-post your code so I can
try to reproduce it?

>>
>> The long answer:
>>
>> You have node.pyx, node.pxd, problemreader.pyx. When compiling
>> problemreader.pyx it looks at node.pxd and uses information about
>> object struct layouts, etc. to generate the .c file.
>
> ok
>>
>> If you then
>> modify node.pxd without re-compiling problemreader.pyx the C-level
>> layout of your Node class can change rendering it incompatible with
>> problemreader.c. Cython detects this at runtime (when it can) and
>> trows this value error.
>
>  I think this would be much easier to understand if I actually modified
> anything... but the problem occurs on executing the setup script on a
> version without any c etc. files. Everything is generated, then nothing is
> changed and the error still occurs. Would be nice to be able to understand
> this, however I'm happy that I have now one way of creating everything that
> doesn't fail (manually cythonizing the .pyx files and then calling the old
> setup). I guess whatever goes wrong with the new setup script deserves a
> look as well - do you want me to file a bug? [1]
> /gerald
> [1] I couldn't find any related bug on http://trac.cython.org.

Yes, please do.

- Robert

[santa]

On Thursday, August 4, 2011 10:37:56 AM UTC+2, robertwb wrote:

 

> Doesn't work when removing all .c, .o and .so files and then directly
> calling setup.py afterwards; same issue. Manually cythonizing the pyx files
> first (cython *.pyx) and then calling the old setup works like a charm. :)

Removing the .c files should cause it to create new ones as if you did
it manually. Are you sure you didn't have some old object files cached
somewhere? Are you invoking the same version of Cython from setup.py
as you are from the command line?

I'm fairly sure that there's nothing but the .p* files left when running the old style setup. The curiosity is that the issue only occurs w/ package imports.

Concerning the version: is there any easy way to find out (there's no [cC]ython.VERSION or the like)? I had removed the outdated cython packages before pip installing the current one, but I'll gladly verify.

Hmm... I'll take a look at that. Could you re-post your code so I can

try to reproduce it?

Of course; the complete testcase is available at:http://www.senarclens.eu/~gerald/tc/c_test.tar.gz and needs to be extracted to a directory on the class path to "work".

 

> ... do you want me to file a bug? [1] 

Yes, please do.

 As soon as I get access to trac.cython.org (pm is on the way). If you prefer, I could post it to cython-devel instead.

/gerald