Cython and PEP 518

[Thomas Robitaille]

Hi everyone,

PEP 518 [1] specifies a way to list build-time dependencies for packages in a pyproject.toml file, e.g.

    [build-system]

    requires = ["setuptools", "wheel", "Cython==0.29.3"]

In addition, pip will automatically use build isolation if such a file is found in a package, meaning that e.g. Cython (in the above case) would be installed into an isolated environment regardless of whether it is installed or not for users.

The Cython docs [2] recommend that releases of packages include the generated C code so that users don't need to have Cython for things to work properly. However, if one has a pyproject.toml file as above, then Cython will always be installed (temporarily) during the build process. If Cython is left out of the pyproject.toml file, then people installing the code from the repo will not be able to since:

    pip install -e .

will fail as Cython wouldn't be in the build environment.

If Cython is kept in the pyproject.toml file, then it's not clear to me whether there is any value in still distributing generated C files with released packages since Cython will always be installed during the build. I'm therefore curious whether people here have any thoughts on what the 'best practices' look like going forward once pyproject.toml files become commonplace? Is it likely that the recommendation of shipping the generated C files would change, in favor of including Cython in the build-time dependencies in pyproject.toml?

Thanks!

Tom

[1] https://www.python.org/dev/peps/pep-0518/

[2] https://cython.readthedocs.io/en/latest/src/userguide/source_files_and_compilation.html#distributing-cython-modules

[Tomas Zulberti]

Hi. Was there any response in this subject? I am having the same issue on a project of mine so I wanted to know the best option

[Jérôme Kieffer]

Hi,

We use a lot Cython in our projects and stopped distributing C/C++
generated files as part of the _sources_ since it makes little sense:
* Processed files (C/C++ files) are not source files anymore (for
instance, Debian asked us to remove them)
* Those C/C++ files are likely dependent on Numpy ABI or on OpenMP
option which are likely to change on the build platform

For ~5 years, having Cython as build dependency did not show any major
drawback since the Python packaging matured a lot over this period of
time. I would advocate to change the documentation of Cython if
everybody agrees.

Cheers,

Jerome

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.


--

[da-woods]

> * Those C/C++ files are likely dependent on Numpy ABI or on OpenMP
> option which are likely to change on the build platform

The C/C++ files should be almost platform independent. The aim is to be able to generate the same files on any platform and have them compile on any other platform (where that doesn't happen it's mostly because of the vagaries of Python dict ordering and similar things, rather than anything more fundamental.

They aren't perfectly forward compatible (it's unlikely that files generated now will work on the final Py3.11 release for example) so it only covers you for current version of Python.

I think the value of distributing the C/C++ source files is the ability to say "we compiled our library with a known, supported, tested version of Cython, and thus we know exactly what code all our users are using". It's one less thing to debug.

Obviously if using Cython as a build dependency works for you, then you should keep doing it!

David

[Stefan Behnel]

Jérôme Kieffer schrieb am 09.01.22 um 14:42:

> We use a lot Cython in our projects and stopped distributing C/C++
> generated files as part of the _sources_ since it makes little sense:
> * Processed files (C/C++ files) are not source files anymore (for
> instance, Debian asked us to remove them)
> * Those C/C++ files are likely dependent on Numpy ABI or on OpenMP
> option which are likely to change on the build platform
>
> For ~5 years, having Cython as build dependency did not show any major
> drawback since the Python packaging matured a lot over this period of
> time. I would advocate to change the documentation of Cython if
> everybody agrees.

Yeah, given the improvements in the Python build system(s), I think it's
time to change that recommendation.

These days, with CPython's C-API becoming visible more fluid (and Cython
being part of the reason why that is feasible), it seems more helpful to
have Cython run at build time, so that it's easier to replace it with a
newer version when a new (possibly incompatible) CPython (or other) release
comes out.

Stefan

[Robert Bradshaw]

I completely agree.

- Robert