Cython extension wrapping a DLL can't be imported in Windows wheel

[Stephan Hügel]

Hi everyone,

I'm building a Cython extension ("cutil") which wraps an external library I've written (I'm making it available as a DLL in Windows). I'm including the DLL in the same directory as the module:

- setup.py

- module

-- __init__.py

-- cutil.pyx

-- rlib.h

-- module.dll

-- module.lib

When I run setup.py build_ext --inplace, Cython generates cutil.c, produces a cutil.pyd in the working directory, and my tests pass. However, when I run pip wheel. to produce a wheel, the compiler places the Cython extension alongside the source files (in the module directory), as you would expect, but this extension can't be imported:

======================================================================

ERROR: Failure: ImportError (No module named cutil)

----------------------------------------------------------------------

Traceback (most recent call last):

File "c:\python27.10-x64\lib\site-packages\nose\loader.py", line 418, in loadTestsFromName

addr.filename, addr.module)

File "c:\python27.10-x64\lib\site-packages\nose\importer.py", line 47, in importFromPath

return self.importFromDir(dir_path, fqname)

File "c:\python27.10-x64\lib\site-packages\nose\importer.py", line 94, in importFromDir

mod = load_module(part_fqname, fh, filename, desc)

File "C:\projects\convertbng\convertbng\test\test_convertbng.py", line 15, in <module>

from convertbng.cutil import convert_bng as cconvert_bng

ImportError: No module named cutil

 

I'm not experienced with Windows, but as I understand it, there's no equivalent of a relative path ($ORIGIN on Linux, @loader_path on OS X ) which I can pass to extra_link_args? So, is there a standard approach for relative import paths on Windows, or bundling DLLs in general?

(I should add that I can build working wheels for OS X and Linux, using the rpath arguments mentioned above)

Complete build output is here: https://ci.appveyor.com/project/urschrei/convertbng/build/job/fd1fql0ykd0c78pe

Thanks,

Steph

[David Vierra]

The current directory is always the first entry in `sys.path`. Thus, the
`convertbng` module in the working directory is the one that is found
when you do `import convertbng`, even though you have also installed
`convertbng` from a wheel previously.

What has changed is that when you stopped running `build_ext --inplace`,
the `convertbng` module in the working directory no longer contains
`cutil.pyd`. Your wheel seems to be intact. Perhaps try something like
`cd convertbng/tests; nosetests` to use a different working directory
and allow it to find the wheel-installed `convertbng` module.

On 6/21/2016 12:56 AM, Stephan Hügel wrote:
> Hi everyone,
> I'm building a Cython extension ("cutil") which wraps an external
> library I've written (I'm making it available as a DLL in Windows). I'm
> including the DLL in the same directory as the module:
>
> - setup.py
> - module
> -- __init__.py
> -- cutil.pyx
> -- rlib.h
> -- module.dll
> -- module.lib
>
> When I run setup.py build_ext --inplace, Cython generates cutil.c,
> produces a cutil.pyd in the working directory, and my tests pass.
> However, when I run pip wheel. to produce a wheel, the compiler places
> the Cython extension alongside the source files (in the

> moduledirectory), as you would expect, but this extension can't be imported:
>
> ======================================================================
> <https://ci.appveyor.com/project/urschrei/convertbng/build/1.0.49/job/fd1fql0ykd0c78pe#L207>ERROR:

> Failure: ImportError (No module named cutil)

> <https://ci.appveyor.com/project/urschrei/convertbng/build/1.0.49/job/fd1fql0ykd0c78pe#L208>----------------------------------------------------------------------
> <https://ci.appveyor.com/project/urschrei/convertbng/build/1.0.49/job/fd1fql0ykd0c78pe#L209>Traceback
> (most recent call last):
> <https://ci.appveyor.com/project/urschrei/convertbng/build/1.0.49/job/fd1fql0ykd0c78pe#L210>File

> "c:\python27.10-x64\lib\site-packages\nose\loader.py", line 418, in
> loadTestsFromName

> <https://ci.appveyor.com/project/urschrei/convertbng/build/1.0.49/job/fd1fql0ykd0c78pe#L211>addr.filename,
> addr.module)
> <https://ci.appveyor.com/project/urschrei/convertbng/build/1.0.49/job/fd1fql0ykd0c78pe#L212>File

> "c:\python27.10-x64\lib\site-packages\nose\importer.py", line 47, in
> importFromPath

> <https://ci.appveyor.com/project/urschrei/convertbng/build/1.0.49/job/fd1fql0ykd0c78pe#L213>return
> self.importFromDir(dir_path, fqname)
> <https://ci.appveyor.com/project/urschrei/convertbng/build/1.0.49/job/fd1fql0ykd0c78pe#L214>File

> "c:\python27.10-x64\lib\site-packages\nose\importer.py", line 94, in
> importFromDir

> <https://ci.appveyor.com/project/urschrei/convertbng/build/1.0.49/job/fd1fql0ykd0c78pe#L215>mod

> = load_module(part_fqname, fh, filename, desc)

> <https://ci.appveyor.com/project/urschrei/convertbng/build/1.0.49/job/fd1fql0ykd0c78pe#L216>File

> "C:\projects\convertbng\convertbng\test\test_convertbng.py", line 15, in
> <module>

> <https://ci.appveyor.com/project/urschrei/convertbng/build/1.0.49/job/fd1fql0ykd0c78pe#L217>from

> convertbng.cutil import convert_bng as cconvert_bng

> <https://ci.appveyor.com/project/urschrei/convertbng/build/1.0.49/job/fd1fql0ykd0c78pe#L218>ImportError:
> No module named cutil
> <https://ci.appveyor.com/project/urschrei/convertbng/build/1.0.49/job/fd1fql0ykd0c78pe#L219>
> <https://ci.appveyor.com/project/urschrei/convertbng/build/1.0.49/job/fd1fql0ykd0c78pe#L220>

>
> I'm not experienced with Windows, but as I understand it, there's no

> equivalent of a relative path ($ORIGINon Linux, @loader_pathon OS X )

> which I can pass to extra_link_args? So, is there a standard approach
> for relative import paths on Windows, or bundling DLLs in general?
>
> (I should add that I can build working wheels for OS X and Linux, using
> the rpath arguments mentioned above)
>
> Complete build output is
> here: https://ci.appveyor.com/project/urschrei/convertbng/build/job/fd1fql0ykd0c78pe
>
> Thanks,
> Steph
>

> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cython-users...@googlegroups.com
> <mailto:cython-users...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.


--
David Vierra
MCEdit, a Minecraft World Editor
http://www.mcedit.net
http://twitter.com/codewarrior0

[Stephan Hügel]

Argh that never occurred to me.

I've switched to running `setup.py nosetests` instead, as per the Python Appveyor demo.

> <mailto:cython-users+unsub...@googlegroups.com>.

[Chris Barker - NOAA Federal]

A wheel is all

Of the package bundled up. You'll need that do in the package -- did you tell wheel that it's needed? 

Something like data_files or ?? (Sorry, not sure how to spell it)

Windows should find a dull sitting alongside the pyd

-CHB

Sent from my iPhone

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.

[Stephan Hügel]

Chris,

I was completely overthinking the problem – my manifest.in is correct, and the wheel build is bundling both my DLL and the Cython module DLL correctly, side-by-side. I had forgotten about the working dir issue that David pointed out, and was puzzled by the Windows DLL search path convention (I'm not a Windows user).

My Appveyor test harness now runs setup.py nosetests, which I understand builds and tests the package without including the current working directory in PYTHONPATH, and, if the tests pass and following the Wheel build, creates and switches to a new directory, installs the wheel using the --no-index -f path/to/wheel switches, then runs nosetests packagename. I'm fairly confident that since both sets of tests pass that the Wheel's been built correctly, and the Cython module has been correctly linked against my DLL. 

Anyway, my full Appveyor build output is here, though of course if anyone is a Windows user and feels like testing the package, pip install convertbng and then nosetests convertbng is all it takes, at least 'til I'm physically close to Windows machines again.

Thanks for the additional clarification.

Steph