Should source distribution depend on Cython?

[Evgeny Yakimov]

Hello,

I was trying to understand the rationale behind the distribution instructions here: https://cython.readthedocs.io/en/latest/src/reference/compilation.html#distributing-cython-modules , which seem to suggest that your source distribution should be shipped with both your cython artifacts and your generated source code.

The way I see it, there are 3 options for creating a source distribution package for a cython project:
1. distribute just the pyx/pxd and included (not generated) source files, source files are then generated by the cython build process
2. distribute only the generated and included source files, cython does not need to be invoked on the generated package
3. distribute everything, and give the user the choice of whether to invoke cython or not

What is the justification between the 3rd method being the recommended practice and what are the merits over the others?

My view (based on rather limited experience with cython) would be that 2) would be more appropriate. Given that the primary purpose (imo) of a source distribution is to allow it to be build on any platform, 2) does satisfy this requirement, and introduces the least dependencies on the build system to be able to build the package, as at this point it is just the source files that actually need to be built. Would it not make sense to ship these over the pyx/pxd files as the source distribution?

Thanks in advance,
Evgeny

[Stefan Behnel]

Option 3) gives

- end users an easy build path by including all necessary files to build
without further dependencies (at least not Cython)

- end users a way to regenerate the .c files directly if anything goes
wrong, requires adaptation to newer Python releases, etc.

- the developers the safety that what end users install by default has been
tested on their side before shipping and does not depend on whatever Cython
version the end users may or may not have installed.

Shipping only the .c files makes last resort fixes on user side harder. Not
shipping the .c files makes the general installation harder and more error
prone.

Since I can't see a reason to make my end users' lives more difficult, I
recommend 3).

Stefan

[Evgeny Yakimov]

Thanks for the follow up. If we consider pip as the main source distribution mechanism where consumers may further alter the code from this source (rather than from the github repo) then I agree with with the rationale here about shipping both.

Given option 3 is desired, what should be the default build option? (i.e. what should pip's invocation of setup.py do by default?). I'd again make the case that compiling from .c/.cpp would be the easier (and faster) case for more users and should probably be the default.

Having played around with cython's build_ext and cythonize, I believe the current mechanisms aren't well suited for either the 2nd or 3rd options, but primarily for the first. i.e. you have to add a bunch of code into your setup.py to support these workflows as outlined in https://cython.readthedocs.io/en/latest/src/reference/compilation.html#distributing-cython-modules . I'm thinking of raising an issue on github to see what buy-in there is for improving support for these other workflows with lower overhead for users.

Evgeny

[Dima Pasechnik]

Is the advice in https://cython.readthedocs.io/en/stable/src/userguide/source_files_and_compilation.html#distributing-cython-modules

not outdated?

Today I cannot see a reason not to depend on Cython, but instead boat your source package with files which are very quick (compared to compiling them with C/C++ compiler etc) to generate, and installing Cython has become trivial with pip an PyPI ?

Last by not least, Cython is much more tied up to Python versions than Python, and now with Python 3.11 we see bug reports coming from

Cython generated C code witch does not support 3.11, bundled in source packages: e.g.

https://github.com/3-manifolds/SnapPy/issues/88

Dima

[Stefan Behnel]

Dima Pasechnik schrieb am 25.01.23 um 17:28:

> Is the advice in
> https://cython.readthedocs.io/en/stable/src/userguide/source_files_and_compilation.html#distributing-cython-modules
> not outdated?

See https://github.com/cython/cython/issues/5089

Stefan