pyximport breaks on namespace packages

[Antony Lee]

Consider a namespace package `pkg` containing `cy_module.pyx` (but no `__init__`.py).

pyximporting the cython module fails, which I guess is acceptable if we just say "namespace modules are not supported"

$ python -c 'from pyximport import install; install(); from pkg import cy_module'
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 890, in _find_spec
AttributeError: 'PyxImporter' object has no attribute 'find_spec'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 954, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 892, in _find_spec
  File "<frozen importlib._bootstrap>", line 873, in _find_spec_legacy
  File "/usr/lib/python3.5/site-packages/pyximport/pyximport.py", line 247, in find_module
    fp, pathname, (ext,mode,ty) = imp.find_module(fullname,package_path)
  File "/usr/lib/python3.5/imp.py", line 270, in find_module
    "not {}".format(type(name)))
RuntimeError: 'list' must be None or a list, not <class 'str'>

But at least trying to import a non-existing module should still fail with ImportError, not RuntimeError.

$ python -c 'from pyximport import install; install(); from pkg import foo'
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 890, in _find_spec
AttributeError: 'PyxImporter' object has no attribute 'find_spec'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 954, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 892, in _find_spec
  File "<frozen importlib._bootstrap>", line 873, in _find_spec_legacy
  File "/usr/lib/python3.5/site-packages/pyximport/pyximport.py", line 247, in find_module
    fp, pathname, (ext,mode,ty) = imp.find_module(fullname,package_path)
  File "/usr/lib/python3.5/imp.py", line 270, in find_module
    "not {}".format(type(name)))
RuntimeError: 'list' must be None or a list, not <class 'str'>

Cython 0.23.2 on Python 3.5.

Antony

[Lisa Guo]

I'm hitting the same issue right now. Have you found a solution for this?

[Robert Bradshaw]

I don't know of a solution to this particular problem, other than that
once you reach a sufficient level of complexity it's better to write a
proper setup.py file rather than use pyximport.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cython-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Walter Woods]

I'm running into this problem too.  It would be nice to have a definite fix, as pyximport is way more handy than a setup.py when you're working with local development code.

One thing to note - I was able to work around this by fixing pyximporter locally:

_, pyxImporter = pyximport.install()

_pyx_oFind = pyxImporter.find_module

def _pyx_nFind(fullname, package_path=None):

    # Here, blacklist modules that are causing the issue.  For me, it was cuda

    if fullname in ['cuda_ndarray.cuda_ndarray']:

        return None

    return _pyx_oFind(fullname, package_path)

pyxImporter.find_module = _pyx_nFind

Hope that helps someone,

Walt