Trying to use generated module from Python 3, get error 'undefined symbol: PyString_Type'

[JDonner]

Hi, I have a text-handling module that wants to handle both unicode
and str in Python 2, and does so via:

cdef get_as_ucs4(object text):
if isinstance(text, unicode) or (PY_MAJOR_VERSION < 3 and
isinstance(text, str)):
ucs4_data = text.encode('utf-32')[4:]
else:
raise ValueError("Requires unicode or str text input, got %s"
% type(text))

but I get the subject's error when I use it in Python 3

$ python3
Python 3.2 (r32:88445, Dec 8 2011, 15:26:58)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more
information.
>>> import aho
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: ./aho.so: undefined symbol: PyString_Type


But even when I get rid of any mention of 'str', and even change
the .pyx string literals to unicode:

cdef get_as_ucs4(object text):
if isinstance(text, unicode):
ucs4_data = text.encode(u'utf-32')[4:]
else:
raise ValueError(u"Requires unicode or str text input, got
%s" % type(text))

I get the same error message.

I also have:

cdef extern from "Python.h":
cdef void Py_INCREF(object)
cdef void Py_DECREF(object)

Could PyString_Types be entering via "Python.h"?

Thanks for any help. The host Python is 2.7.x, and Cython is 0.15.1

[Stefan Behnel]

Hi,

just a stab in the dark, but you might be interested in this:

http://pypi.python.org/pypi/acora/1.7

JDonner, 09.03.2012 22:07:

> Hi, I have a text-handling module that wants to handle both unicode
> and str in Python 2, and does so via:
>
> cdef get_as_ucs4(object text):
> if isinstance(text, unicode) or (PY_MAJOR_VERSION < 3 and
> isinstance(text, str)):
> ucs4_data = text.encode('utf-32')[4:]
> else:
> raise ValueError("Requires unicode or str text input, got %s"
> % type(text))
>
> but I get the subject's error when I use it in Python 3
>
> $ python3
> Python 3.2 (r32:88445, Dec 8 2011, 15:26:58)
> [GCC 4.5.2] on linux2
> Type "help", "copyright", "credits" or "license" for more
> information.
> >>> import aho
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> ImportError: ./aho.so: undefined symbol: PyString_Type

Stupid question: you did rebuild your module in Py3 before importing it
there, didn't you? (not the generated C sources, just the shared library)

> But even when I get rid of any mention of 'str', and even change
> the .pyx string literals to unicode:
>
> cdef get_as_ucs4(object text):
> if isinstance(text, unicode):
> ucs4_data = text.encode(u'utf-32')[4:]
> else:
> raise ValueError(u"Requires unicode or str text input, got
> %s" % type(text))
>
> I get the same error message.
>
> I also have:
>
> cdef extern from "Python.h":
> cdef void Py_INCREF(object)
> cdef void Py_DECREF(object)

You can cimport those from the cpython.ref module, no need to define them
yourself. See Cython/Includes/

> Could PyString_Types be entering via "Python.h"?

No, not in Python 3.

> Thanks for any help. The host Python is 2.7.x, and Cython is 0.15.1

That should work.

Stefan

[JDonner]

On Friday, March 9, 2012 1:35:27 PM UTC-8, Stefan Behnel wrote:

Hi,

just a stab in the dark, but you might be interested in this:

http://pypi.python.org/pypi/acora/1.7

Hah, thanks - it didn't + doesn't build for me but I had not realized it was written in Cython. (Also, its use is a little awkward for my use-case, and I added a feature.)

>   Stupid question: you did rebuild your module in Py3 before importing it

>   there, didn't you? (not the generated C sources, just the shared library)

I did, but got this:

$ rm -f aho.cpp && rm -rf build && python3 aho_setup.py build_ext --inplace
running build_ext

Traceback (most recent call last):

  File "aho_setup.py", line 9, in <module>
    cmdclass={'build_ext': build_ext})
  File "/usr/lib/python3.2/distutils/core.py", line 149, in setup
    dist.run_commands()
  File "/usr/lib/python3.2/distutils/dist.py", line 919, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.2/distutils/dist.py", line 938, in run_command
    cmd_obj.run()
  File "/home/jd/software/Cython/Cython/Distutils/build_ext.py", line 135, in run
    _build_ext.build_ext.run(self)
  File "/usr/lib/python3.2/distutils/command/build_ext.py", line 347, in run
    self.build_extensions()
  File "/home/jd/software/Cython/Cython/Distutils/build_ext.py", line 142, in build_extensions
    ext.sources = self.cython_sources(ext.sources, ext)
  File "/home/jd/software/Cython/Cython/Distutils/build_ext.py", line 153, in cython_sources
    from Cython.Compiler.Main \
  File "/home/jd/software/Cython/Cython/Compiler/Main.py", line 369
    except UnicodeDecodeError, msg:
                             ^
SyntaxError: invalid syntax

Then tried in addition, running cython.py itself as python3 (#!/usr/bin/env python3). Then thought I'd ask :)

I've got python and python3 installed on Ubuntu (one version back) - I've got a custom Cython-0.15.1 at the head of my PATH (for cython.py) and PYTHONPATH. Any other configuration I should check / change?

[Stefan Behnel]

JDonner, 09.03.2012 23:41:

> On Friday, March 9, 2012 1:35:27 PM UTC-8, Stefan Behnel wrote:
>> just a stab in the dark, but you might be interested in this:
>>
>> http://pypi.python.org/pypi/acora/1.7
>>
> Hah, thanks - it didn't + doesn't build for me but I had not realized it
> was written in Cython.

And you don't even need Cython to use it because the source distribution
comes with the generated C sources.

> (Also, its use is a little awkward for my use-case,
> and I added a feature.)

... it's often better to add a feature to existing software than to rewrite it.

>> Stupid question: you did rebuild your module in Py3 before importing it
>> there, didn't you? (not the generated C sources, just the shared
>> library)
>
> I did, but got this:
> $ rm -f aho.cpp && rm -rf build && python3 aho_setup.py build_ext --inplace
> running build_ext
> Traceback (most recent call last):
> File "aho_setup.py", line 9, in <module>
> cmdclass={'build_ext': build_ext})
> File "/usr/lib/python3.2/distutils/core.py", line 149, in setup
> dist.run_commands()

> [...]

> File "/home/jd/software/Cython/Cython/Distutils/build_ext.py", line 153,
> in cython_sources
> from Cython.Compiler.Main \
> File "/home/jd/software/Cython/Cython/Compiler/Main.py", line 369
> except UnicodeDecodeError, msg:
> ^
> SyntaxError: invalid syntax

Ah, ok, you didn't report *that*. That's because you are using a Python 2
installation (or copy, it seems) of Cython instead of a Python 3
installation. Just run "setup.py install" or "pip install Cython" (or
whatever you do to install Python packages) in Python 3, that'll do the
right thing.

> Then tried in addition, running cython.py itself as python3 (#!/usr/bin/env
> python3). Then thought I'd ask :)

You should have. Python 3 is not syntax compatible with Python 2, therefore
we use different sources for both. The Python 3 version gets generated at
install time using the 2to3 tool.

> I've got python and python3 installed on Ubuntu (one version back) - I've
> got a custom Cython-0.15.1 at the head of my PATH (for cython.py) and
> PYTHONPATH.

Yes, that's exactly the problem.

Stefan

[Stefan Behnel]

Stefan Behnel, 10.03.2012 08:08:

> Stupid question: you did rebuild your module in Py3 before importing it
> there, didn't you? (not the generated C sources, just the shared
> library)

Oh, and as I hinted with the last sentences here: you don't need to run
Cython in Python 3 to build the C sources. It's enough to run it in Python
2 (as in "run it once"), and then reuse the generated C code to build in
any CPython environment you like (2.4-3.3, currently). That's why Cython
implemented libraries normally ship with their pre-built C sources, so that
their users don't need Cython at all. A serious feature.

Stefan