setuptools+cythonize includes .c but not .pyx?

303 views
Skip to first unread message

Dan Stromberg

unread,
Apr 23, 2021, 3:23:09 PM4/23/21
to cython-users

Hi folks.

I'm attempting to package up a variable-length, content-based blocking algorithm for pypi.  There's a pure python version which is working fine, but there's also a cython version which I'm having trouble with.

What I have so far is at:

It's based on the following, which works (or worked?) well:
https://stromberg.dnsalias.org/svn/treap/trunk

So you don't have to svn checkout those URL's to see what I'm up against, I've got the following right now for the cython version of rolling_checksum_mod:

"""
Package up the Cython version of rolling_checksum_mod.

This code wants the pure python also installed.
"""

import os
import sys
import subprocess

from setuptools import setup, Extension
from Cython.Build import cythonize
# from distutils.extension import Extension

version = '1.0.9'


def is_newer(filename1, filename2):
    """Return True if filename1 is newer than filename2."""
    time1 = os.stat(filename1).st_mtime
    time2 = os.stat(filename2).st_mtime

    if time1 > time2:
        return True
    else:
        return False


def m4_it(infilename, outfilename, define):
    """
    Create outfilename from infilename in a make-like manner.

    If outfilename doesn't exit, create it using m4.
    If outfilename exists but is older than infilename, recreate it using m4.
    """
    build_it = False

    if os.path.exists(outfilename):
        if is_newer(infilename, outfilename):
            # outfilename exists, but is older than infilename, build it
            build_it = True
    else:
        # outfilename does not exist, build it
        build_it = True

    if build_it:
        try:
            subprocess.check_call('m4 -D"%s"=1 < "%s" > "%s"' % (define, infilename, outfilename), shell=True)
        except subprocess.CalledProcessError:
            sys.stderr.write('You need m4 on your path to build this code\n')
            sys.exit(1)


if os.path.exists('../rolling_checksum_mod.m4'):
    m4_it('../rolling_checksum_mod.m4', 'rolling_checksum_pyx_mod.pyx', 'pyx')

extensions = [
    Extension("rolling_checksum_pyx_mod", ["rolling_checksum_pyx_mod.pyx"]),
    ]

setup(
    name='rolling_checksum_pyx_mod',
    ext_modules=cythonize(extensions),
    version=version,
    description='Cython-augmented Python module providing a variable-length, content-based blocking algorithm',
    long_description="""
Chop a file into variable-length, content-based chunks.

Example use:
.. code-block:: python

    >>> import rolling_checksum_mod
    >>> # If you have both rolling_checksum_pyx_mod and rolling_checksum_py_mod installed, the software will
    >>> # automatically prefer the pyx version.  Both py and pyx versions require rolling_checksum_py_mod, but
    >>> # only the pyx version requires rolling_checksum_pyx_mod.
    >>> with open('/tmp/big-file.bin', 'rb') as file_:
    >>>     for chunk in rolling_checksum_mod.min_max_chunker(file_):
    >>>         # chunk is now a piece of the data from file_, and it will not always have the same length.
    >>>         # Instead, it has the property that if you insert a byte at the beginning of /tmp/big-file.bin,
    >>>         # most of the chunks of the file will remain the same.  This can be nice for a deduplicating
    >>>         # backup program.
    >>>         print(len(chunk))
""",
    author='Daniel Richard Stromberg',
    author_email='stro...@gmail.com',
    platforms='Cross platform',
    license='GPLv3',
    classifiers=[
        "Development Status :: 5 - Production/Stable",
        "Intended Audience :: Developers",
        "Programming Language :: Cython",
        "Programming Language :: Python :: 3",
        ],
    install_requires=[
        "rolling_checksum_py_mod",
        "cython",
        ],
    )

It seems to upload to test pypi OK, but when I try to install it on a Linux system, I get:
$ python3 -m pip install --no-clean -i https://test.pypi.org/simple/ rolling-checksum-pyx-mod==1.0.9
below cmd output started 2021 Fri Apr 23 09:07:49 AM PDT
Looking in indexes: https://test.pypi.org/simple/
Collecting rolling-checksum-pyx-mod==1.0.9
     |████████████████████████████████| 57 kB 507 kB/s 
    ERROR: Command errored out with exit status 1:
     command: /home/dstromberg/virtualenvs/rcm/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-mxonscoa/rolling-checksum-pyx-mod_d00fd93e79684552bb692f0ac8f25d5c/setup.py'"'"'; __file__='"'"'/tmp/pip-install-mxonscoa/rolling-checksum-pyx-mod_d00fd93e79684552bb692f0ac8f25d5c/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-2rfq0s_x
         cwd: /tmp/pip-install-mxonscoa/rolling-checksum-pyx-mod_d00fd93e79684552bb692f0ac8f25d5c/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-mxonscoa/rolling-checksum-pyx-mod_d00fd93e79684552bb692f0ac8f25d5c/setup.py", line 60, in <module>
        ext_modules=cythonize(extensions),
      File "/home/dstromberg/virtualenvs/rcm/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 972, in cythonize
        aliases=aliases)
      File "/home/dstromberg/virtualenvs/rcm/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 815, in create_extension_list
        for file in nonempty(sorted(extended_iglob(filepattern)), "'%s' doesn't match any files" % filepattern):
      File "/home/dstromberg/virtualenvs/rcm/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 114, in nonempty
        raise ValueError(error_msg)
    ValueError: 'rolling_checksum_pyx_mod.pyx' doesn't match any files
    ----------------------------------------
ERROR: Could not find a version that satisfies the requirement rolling-checksum-pyx-mod==1.0.9
ERROR: No matching distribution found for rolling-checksum-pyx-mod==1.0.9

I'm pretty sure I have cython on my $PATH, but the downloaded tarball appears to not include rolling_checksum_pyx_mod.pyx - I think this is probably where the error is stemming from.

BTW, I do not want to build a manylinux package for it. I just want to compile the .c, and possibly cython the .pyx.

What do I need to do to get this to install?

Thanks.

Stefan Behnel

unread,
Apr 23, 2021, 3:31:42 PM4/23/21
to cython...@googlegroups.com
Dan Stromberg schrieb am 23.04.21 um 18:10:
> I'm pretty sure I have cython on my $PATH, but the downloaded tarball
> appears to not include rolling_checksum_pyx_mod.pyx - I think this is
> probably where the error is stemming from.

In order to include non-Python-(source-)files in your source distribution,
you need to tell distutils/setuptools what it should include.

This can be done via a MANIFEST.in file or by declaring `.pyx` files as
package data in your `setup()` call. The latter would then also include
them in your binary distributions, which would allow users to see source
lines in tracebacks.

Stefan
Reply all
Reply to author
Forward
0 new messages