Cython module layout and visibility

[Tristan Partin]

Hello all,

I have been working on an open source key value storage engine at https://github.com/hse-project. As of right now the project only supports C. My current task is to create Python bindings for the project which will also be open sourced and published to PyPI when they are ready which should hopefully come with the 1.9/1.10 releases assuming I can fix the following issue.

This is my first time using Cython and I have been fairly impressed with it for the most part. Most of the process has been fairly straightforward. I have written most of the bindings correctly and tested them appropriately, so things are looking good. Right now the HSE public API is located here https://github.com/hse-project/hse/tree/master/include/hse. There you will find a couple header files, most notably hse.h and hse_limits.h. I have modeled both of these into pyx and pxd files such that a mapping looks like:

hse.h -> hse/__init__.{pxd,pyx}

hse_limits.h -> hse/limits.{pxd,pyx}

In __init__.pyx, I have `cimport limits` so that I can use some of the constants in limits.pxd. Those references work great!

The goal of the bindings is essentially to have the top level hse module with limits as a submodule. When I go to install or even do an inplace build_ext, I can import the hse module successfully! Where I am running into issues however is when I try to import the hse.limits submodule. I have tried `import hse.limits` and `from hse import limits`. Both raise import errors for me. dir(hse) returns exactly what I would expect minus the submodule.

I am running out of ideas on how to fix this issue. When I install the module to --user, the tree looks like:

hse-1.9-py3.6-linux-x86_64.egg/

hse-1.9-py3.6-linux-x86_64.egg/hse.cpython-36m-x86_64-linux-gnu.so

hse-1.9-py3.6-linux-x86_64.egg/hse.py

hse-1.9-py3.6-linux-x86_64.egg/EGG-INFO

hse-1.9-py3.6-linux-x86_64.egg/EGG-INFO/PKG-INFO

hse-1.9-py3.6-linux-x86_64.egg/EGG-INFO/SOURCES.txt

hse-1.9-py3.6-linux-x86_64.egg/EGG-INFO/dependency_links.txt

hse-1.9-py3.6-linux-x86_64.egg/EGG-INFO/native_libs.txt

hse-1.9-py3.6-linux-x86_64.egg/EGG-INFO/not-zip-safe

hse-1.9-py3.6-linux-x86_64.egg/EGG-INFO/top_level.txt

hse-1.9-py3.6-linux-x86_64.egg/__pycache__

hse-1.9-py3.6-linux-x86_64.egg/__pycache__/hse.cpython-36.pyc

hse-1.9-py3.6-linux-x86_64.egg/hse

hse-1.9-py3.6-linux-x86_64.egg/hse/limits.cpython-36m-x86_64-linux-gnu.so

hse-1.9-py3.6-linux-x86_64.egg/hse/limits.py

hse-1.9-py3.6-linux-x86_64.egg/hse/__pycache__

hse-1.9-py3.6-linux-x86_64.egg/hse/__pycache__/limits.cpython-36.pyc

which seems like a pretty good layout to me. Now when I start a Python 3.6 REPL, I can import hse appropriately. 

>>> from hse import limits

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

ImportError: cannot import name 'limits

That is the issue I am running into. My setup.py looks like this:

import os

from setuptools import setup, Extension

# Distribute the generated C source files so that consumers don't necessarily

# need Cython on their system to build the extensions.

USE_CYTHON = os.environ.get("USE_CYTHON")

PROJECT_DIR = os.path.dirname(os.path.realpath(__file__))

extensions = [

    Extension(

        "hse",

        [f"{PROJECT_DIR}/hse/__init__.{'pyx' if USE_CYTHON else 'c'}"],

        libraries=["hse_kvdb"]

    ),

    Extension(

        "hse.limits",

        [f"{PROJECT_DIR}/hse/limits.{'pyx' if USE_CYTHON else 'c'}"],

        libraries=["hse_kvdb"]

    )

]

cmdclass = {}

if USE_CYTHON:

    from Cython.Build import cythonize

    from Cython.Distutils import build_ext

    extensions = cythonize(extensions, include_path=["hse"])

    cmdclass["build_ext"] = build_ext

setup(

    name="hse",

    version="1.9",

    maintainer="Micron Technology",

    description="Python bindings to HSE's C API. " \

        "HSE is an embeddable key-value store designed for SSDs based on NAND " \

        "flash or persistent memory. HSE optimizes performance and endurance by " \

        "orchestrating data placement across DRAM and multiple classes of SSDs " \

        "or other solid-state storage. HSE is ideal for powering NoSQL, " \

        "Software-Defined Storage (SDS), High-Performance Computing (HPC), " \

        "Big Data, Internet of Things (IoT), and Artificial Intelligence (AI) " \

        "solutions.",

    license="Apache-2.0",

    url="https://github.com/hse-project",

    ext_modules=extensions,

    packages=["hse", "hse.limits"],

    cmdclass=cmdclass,

    package_dir={"hse": f"{PROJECT_DIR}/hse", "hse.limits": f"{PROJECT_DIR}/hse"},

    keywords="micron hse key value object store"

)

I don't really know how to fix my issue at all. If anyone here knows how to fix this issue, I would be eternally grateful.

Thanks,

Tristan Partin

[da-woods]

I'm a bit puzzled because your setup.py and your description don't seem to match your directory contents when installed to --user.

Is it possible that hse/__init__.pyx is getting cythonized to hse.<stuff>.so instead of hse/__init__.<stuff>.so?

I think what's happening is "hse.<stuff>.so" is being found, and that means that it never searches for the hse directory (and thus finds limits) This isn't a specifically Cython problem - I think the same would happen with regular Python modules.

You'd probably be advised not to Cythonize __init__ since it doesn't work well on Windows (at least not without some tweaking). Instead have a regular __init__.py that just imports from a Cython submodule (e.g. hse/_hse_implementation.pyx)

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/f94c742a-fbf2-44f7-9693-bf6de30e8d43n%40googlegroups.com.

[Tristan Partin]

da-woods,

Your response was spot on. I essentially did exactly what you said. I moved the currest hse module to hse.hse, created an __init__.py in the hse folder, and add a `from hse.hse import *` as to give a little bit better import path, but now the hse.limits module is able to be found. Thanks for your help.

Tristan

[vybhav ramachandran]

Hi,

I'm running into similar issues where I'm getting errors when doing cimports.

Can you please share your code? Perhaps that'll help me debug my issue.

Best,
Vybhav