.so file names using Python 3.4

[Kevin Thornton]

Hi,

I'm working on a package primarily using Python 2.7.6 on an Ubuntu 14.04 system.  Everything "just works" here.

However, when I attempt to use Python 3.4 to compile my module, the resulting .so files have unusual names that prevent proper importing from the package.

My basic steps are 

alias python=python3

Then re-compile my module, and the .so names come out like this:

find . -name "*.so"

./fwdpy/fwdpy.cpython-34m.so

./fwdpy/fwdpyio/fwdpyio.cpython-34m.so

./fwdpy/libseq/libseq.cpython-34m.so

./fwdpy/internal/internal.cpython-34m.so

With python 2, the cpython-XXm bit is gone.

Here's a minimal example:

In foo.pyx:

def test():

    return 1

In setup.py

from distutils.core import setup, Extension

from Cython.Build import cythonize

import platform, glob, sys

extensions=[Extension("foo",sources=["foo.pyx"])]

setup(name="foo",

      ext_modules=cythonize(extensions))

And the output on my system:

kevin@:~/tmp/cython$ ls -lhrt

total 80K

-rw-rw-r-- 1 kevin kevin   25 Oct 12 14:39 foo.pyx

-rw-rw-r-- 1 kevin kevin  66K Oct 12 14:39 foo.c

-rw-rw-r-- 1 kevin kevin  217 Oct 12 14:41 setup.py

drwxrwxr-x 4 kevin kevin 4.0K Oct 12 14:42 build

kevin@:~/tmp/cython$ python setup.py build_ext -i

running build_ext

building 'foo' extension

[STUFF DELETED]

$ ls -lhrt

total 120K

-rw-rw-r-- 1 kevin kevin   25 Oct 12 14:39 foo.pyx

-rw-rw-r-- 1 kevin kevin  66K Oct 12 14:39 foo.c

-rw-rw-r-- 1 kevin kevin  217 Oct 12 14:41 setup.py

drwxrwxr-x 4 kevin kevin 4.0K Oct 12 14:42 build

-rwxrwxr-x 1 kevin kevin  37K Oct 12 14:43 foo.so

kevin@:~/tmp/cython$ alias python=python3

kevin@:~/tmp/cython$ python setup.py build_ext -i

running build_ext

building 'foo' extension

[STUFF DELETED]

$ !ls

ls -lhrt

total 160K

-rw-rw-r-- 1 kevin kevin   25 Oct 12 14:39 foo.pyx

-rw-rw-r-- 1 kevin kevin  66K Oct 12 14:39 foo.c

-rw-rw-r-- 1 kevin kevin  217 Oct 12 14:41 setup.py

drwxrwxr-x 4 kevin kevin 4.0K Oct 12 14:42 build

-rwxrwxr-x 1 kevin kevin  37K Oct 12 14:43 foo.so

-rwxrwxr-x 1 kevin kevin  40K Oct 12 14:43 foo.cpython-34m.so

Is this expected, or something to do with my system?

Thanks,

Kevin

[Stefan Behnel]

Kevin Thornton schrieb am 12.10.2015 um 23:38:
> I'm working on a package primarily using Python 2.7.6 on an Ubuntu 14.04
> system. Everything "just works" here.
> However, when I attempt to use Python 3.4 to compile my module, the
> resulting .so files have unusual names that prevent proper importing from
> the package.

No, they don't. Python 3.4 will find them by looking for the longer
extension first.

What makes you think they cannot be imported?

Note that the modules are specific to CPython 3.4, so you cannot import
them in earlier Python versions.

Stefan

[Kevin Thornton]

I'm still looking into this, but I see what you mean now with the example that I gave.  I can import that just fine.  However, the package I'm working on is failing, and I am trying to work out why.

The problem that I'm having may stem from my __init__.py being incorrect or too simplistic.

My package is set up with subdirectories, each of which contains a Cython module.  Each __init.py__ says "from X import *".

When using Python 2.7, this scheme is working fine.  With Python 3.4, I get "ImportError: No module named X" when attempting to run any example scripts, doctests, etc.

I'll try to work out a minimal example of my issue, but the code that I'm working with is here: https://github.com/molpopgen/fwdpy/tree/master/fwdpy

An example of a subdirectory's __init__.py is here: https://github.com/molpopgen/fwdpy/blob/master/fwdpy/internal/__init__.py

At this point, I'm assuming that I'm missing something about those init files.

Any advice would be appreciated.

Thanks,

Kevin

[Kevin Thornton]

I've worked out a minimal example of the issue I'm having with the more complex package.

It is here: https://github.com/molpopgen/cython_question

sh py2.sh to build and run the test script with python 2

sh py2.sh for python 3.

The former is working, the latter doesn't, which is the same problem I'm seeing in my more complex project.

On Tuesday, October 13, 2015 at 12:24:51 AM UTC-7, Stefan Behnel wrote:

[Stefan Behnel]

Kevin Thornton schrieb am 13.10.2015 um 19:03:
> I've worked out a minimal example of the issue I'm having with the more
> complex package.
>
> It is here: https://github.com/molpopgen/cython_question

Your imports are wrong. You need to use either relative or fully qualified
imports but you are using absolute imports of relative package names, and a
module "mod" in a package "pkg" cannot be found as "mod", only as
"pgk.mod", or ".mod" if imported from another module in the package "pkg".

This has nothing to do with Cython, BTW, it's happening in plain Python
code. You can get the same in Python 2.x if you add

from __future__ import absolute_import

That's generally a good idea anyway.

Stefan

[Kevin Thornton]

Thanks--I figured that this must be something basic that I'm missing.  I'm still new to putting packages together.

--Kevin