Transition from setup.py to pyproject.toml only, with dynamic args to cythonize

[Didier VEZINET]

Hi,

First of all: thanks for the hard work !

I've been using cython for several years now and my library has had a very sketchy (setup.py + requirements + some custom scripts) to make it work.

Did all this years ago, and I feel like it's time to clean all that mess up and modernize with a `pyproject.toml` and get rid of the (setup.py + requirement + setup.cfg).

It work in progress on a dedicated branch:

I have set the pyproject.toml file (best I could).

Since my extensions needed some dynamic arguments (e.g.: testing whether openmp is installed or not), I've set up a custom build_py for setuptools following the advice on stackoverflow.

Seems to work.

Now I'm stuck at the point where I need to provide a similar dynamic argument to cythonize.

I've tried the same trick (overloading cythonize in a dedicated custom module and specifying it in the pyproject.toml), but it doesn't seem to work.

Looks like the [tool.cython] section of the pyproject.toml is being ignored, I think.

I've spent hours online trying to find a clean way to do this without success, any help would be very appreciated !

Thanks

Didier

 

[Jérôme Kieffer]

Hi,

I faced the same issue as you and decided to follow scipy & numpy which
switched to pyproject+meson-python on some important projects, at
least all the ones using cython because it makes this part so much easier.
The management of compiler, their option is so much simple. OpenMP is
just an optional dependency !
The cleanest project is probably silx, but pyFAI and fabio have paved
the way before ... https://github.com/silx-kit/silx
It is a big change, but today, a lot of documentation exists and there
are other examples. Two years after the migration, I am very happy with
the results. Thanks for the meson-python community who helped in the
transition.

Cheers,

Jerome

On Mon, 3 Nov 2025 10:07:38 -0800 (PST)
Didier VEZINET <didier....@gmail.com> wrote:

> Hi,
>
> First of all: thanks for the hard work !
>
> I've been using cython for several years now and my library

> <https://github.com/ToFuProject/tofu>has had a very sketchy (setup.py +

> requirements + some custom scripts) to make it work.
> Did all this years ago, and I feel like it's time to clean all that mess up
> and modernize with a `pyproject.toml` and get rid of the (setup.py +
> requirement + setup.cfg).
>
> It work in progress on a dedicated branch

> <https://github.com/ToFuProject/tofu/tree/Issue1032_uvPyprojectt>:

> I have set the pyproject.toml

> <https://github.com/ToFuProject/tofu/blob/Issue1032_uvPyprojectt/pyproject.toml>

> file (best I could).
> Since my extensions needed some dynamic arguments (e.g.: testing whether
> openmp is installed or not), I've set up a custom build_py

> <https://github.com/ToFuProject/tofu/blob/Issue1032_uvPyprojectt/_custom_build.py>

> for setuptools following the advice on stackoverflow

> <https://stackoverflow.com/a/74196255>.
>
> Seems to work.
>
> Now I'm stuck at the point where I need to provide a similar *dynamic
> argument to cythonize*.

>
> I've tried the same trick (overloading cythonize

> <https://github.com/ToFuProject/tofu/blob/Issue1032_uvPyprojectt/_custom_cythonize.py>

> in a dedicated custom module and specifying it in the pyproject.toml), but
> it doesn't seem to work.

> Looks like the* [tool.cython]* section of the pyproject.toml is being

> ignored, I think.
>
> I've spent hours online trying to find a clean way to do this without
> success, any help would be very appreciated !
>
> Thanks
> Didier
>
>
>
>
>
>

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/cython-users/4e24bbb2-be7d-4ef6-aa92-a46a5e7f56a2n%40googlegroups.com.


--

[Didier VEZINET]

Hi Jerome,

Thanks a lot for your answer, that looks like good advice, I'll give a it a try !

(and may come back with questions),

Thanks !

Didier

[Didier VEZINET]

A quick question: does anyone have an example of a pyproject.toml where openmp is declared as an optional dependency using meson as build backend ?

On Monday, 3 November 2025 at 16:13:59 UTC-5 Jerome Kieffer wrote:

[Bernie Roesler]

I am having a similar issue, where I have moved all of my configuration to pyproject.toml, and my setup.py just gets the list of include/library directories and creates the list of Extensions from the pyx files, then calls setup(ext_modules=extensions)..

I'd like to use "annotate=true" as if I had called setup(ext_modules=cythonize(extensions, annotate=True)) in setup.py, but using "[tool.cython] annotate = true" doesn't seem to do anything. Is there documentation anywhere on how the pyproject.toml section gets processed?

Thanks,

Bernie

[Didier VEZINET]

Also, I've recently move to using uv for package managing and virtual env,

It's really great, fast and all !

But looking only, it seems like it does not fare too well with meson as a build backend, does anyone have feedback on that ?

Thanks again,

Didier

[Jérôme Kieffer]

Full optional dependency in meson.build:
```
py_mod = import('python')
py = py_mod.find_installation(pure: false)
cy = meson.get_compiler('cython')
py_dep = py.dependency()
omp = dependency('openmp', required: false)

py.extension_module('interpolate', 'interpolate.pyx',
subdir: 'silx/math',
dependencies : [py_dep, omp],
install: true,
)
```

In this case, OpenMP will be used if found, unless, it won't be used.

There are also ways to propagate options from the PEP517 build-backend to meson to cython.
For this you need to define options in meson.build and pass the to the proper option to the PEP517-build-backend.
In the example of the silx project it is done like this at the meson level:
`meson setup build2 -Duse_openmp=disabled`
For the integration in the pyproject.toml:
https://mesonbuild.com/meson-python/how-to-guides/meson-args.html#how-to-guides-meson-args

Cheers,

Jerome

On Tue, 4 Nov 2025 05:37:56 -0800 (PST)

> To view this discussion visit https://groups.google.com/d/msgid/cython-users/0f9fcbb2-f2c3-48e1-b279-ba49505aee63n%40googlegroups.com.



--

[Didier VEZINET]

Thanks !

[Didier VEZINET]

One more question:

how do you specify dynamic cython arguments in meson:

With setuptools, since the `Extension()` class could be called in python I could do:

Extension(
        name="tofu.geom._openmp_tools",
        sources=["tofu/geom/_openmp_tools.pyx"],
        extra_compile_args=extra_compile_args,
        extra_link_args=extra_link_args,
        cython_compile_time_env=dict(TOFU_OPENMP_ENABLED=openmp_installed),
 ),

I haven't found how to provide the same dynamic compile-time argument with Meson:

py.extension_module(
     'tofu.geom._openmp_tools',
     'tofu/geom/_openmp_tools.pyx',
     subdir: 'tofu/geom/',
     dependencies : [py_dep, omp],
     cython_args : cython_args + ['cython_compile_time_env':dict(TOFU_OPENMP_ENABLED=openmp_installed)],
     install: true,  # true
)

The above doesn't work, 

* the syntax is probably wrong because meson.build is not a python module

* If I had the correct syntax, I guess I would need to replace openmp_installed by omp is not None is something alike ? (from omp = dependency('openmp', required: false))

Any help would be greatly appreciated !

 Cheers

Didier

[Oscar Benjamin]

On Wed, 12 Nov 2025 at 18:09, Didier VEZINET <didier....@gmail.com> wrote:
>
> With setuptools, since the `Extension()` class could be called in python I could do:
>
> Extension(
> name="tofu.geom._openmp_tools",
> sources=["tofu/geom/_openmp_tools.pyx"],
> extra_compile_args=extra_compile_args,
> extra_link_args=extra_link_args,
> cython_compile_time_env=dict(TOFU_OPENMP_ENABLED=openmp_installed),
> ),
>
>
> I haven't found how to provide the same dynamic compile-time argument with Meson:
>
> py.extension_module(
> 'tofu.geom._openmp_tools',
> 'tofu/geom/_openmp_tools.pyx',
> subdir: 'tofu/geom/',
> dependencies : [py_dep, omp],
> cython_args : cython_args + ['cython_compile_time_env':dict(TOFU_OPENMP_ENABLED=openmp_installed)],
> install: true, # true
> )

What exactly is openmp_installed here? Is it a boolean in meson's terms?

With meson you should assume that cython is going to be run via its
CLI and so you need to look for the CLI option for this:

$ cython --help
...
-E, --compile-time-env NAME=VALUE,...
Provides compile time env like DEF would do.

Now you need to tell meson to pass that argument to cython's CLI. This
sets the option globally:

if openmp_installed
add_project_arguments(
'--compile-time-env', 'TOFU_OPENMP_ENABLED=True',
language : 'cython'
)
endif

Instead of add_project_arguments you could use

cython_args = cython_args + ['--compile-time-env', 'TOFU_OPENMP_ENABLED=True']

which would also allow setting different options for different
extension modules if there was a reason not to just set this globally.

--
Oscar

[Didier VEZINET]

Thanks muchly !

That really helped too, it's working now !

I still bump into an error at compile time which I think shows I haven't fully grasped how meson builds / manages dependencies.

I have the fatal error: numpy/arrayobject.h: No such file or directory

I know it means meson is not finding numpy, but I'm not sure why because:

1/ I'm running from a uv env where numpy is installed (and I'm calling meson from within that env, in which it was uv-pip-installed, as well as ninja, although I'm not sure how to be certain ninja is called from within the env and not from the system install ?)

2/ Also, my pyproject.toml clearly declares numpy as a build dependency

[build-system]
duild-backend = 'mesonpy'
 requires = [ 'meson-python',  'cython',  'numpy' ]

3/ Only thing I see is that numpy is not explicitly declared as a dependency in the meson.build

    = > but isn't that the job of the pyproject.toml and to propagate whatever build dependency is in the pyproject.toml into the meson.build ?

          - I would have thought that's what the preliminary meson setup --reconfigure builddir command does ?

    => Do I have to declare explicitly all build dependencies both in pyproject.toml and meson.build ?

          - if so how / where exactly shall I declare it in the meson.build ?

Thanks again for all the valuable help !

Cheers

Didier

[Oscar Benjamin]

On Thu, 13 Nov 2025 at 16:21, Didier VEZINET <didier....@gmail.com> wrote:
>
> I still bump into an error at compile time which I think shows I haven't fully grasped how meson builds / manages dependencies.
>
> I have the fatal error: numpy/arrayobject.h: No such file or directory
>
> I know it means meson is not finding numpy, but I'm not sure why

I think it rather means that your C compiler is not finding the numpy
header. Presumably your setup.py uses the numpy.get_include() function
and gives the path to setuptools to add this header directory to the C
compiler include path. I guess that you need to do something similar
with meson.

There are some notes here about getting the header path when not using
setuptools:
https://numpy.org/devdocs/reference/generated/numpy.get_include.html#numpy.get_include

--
Oscar

[Didier VEZINET]

OK,

I found the appropriate keyword for extension_module:

py.extension_module(                                                                              
   'tofu.geom._basic_geom_tools',                                                                  
   'tofu/geom/_basic_geom_tools.pyx',                                                              
   subdir: 'tofu/geom/',                                                                            
   include_directories: include_dirs,                                                              
   dependencies : [py_dep, omp],                                                                    
   cython_args : cython_args,                                                                      

   install: true,  # true                                                                          
)       

Now to provide the proper dir, I need to get it from numpy.

I can't use the  CLI tool numpy-config because my libary is not (yet) compatible with Numpy 2 (that's the next big transition I have to do).

Question: How do I get that dir from inside the meson.build ?

Mimicking the syntax seen for python itself I've tried this but it doesn't work:

np = import('numpy')

np_header_dir = np.get_include()

include_dirs = [np_header_dir]

I get a numpy not found error upon calling meson setup --reconfigure builddir

meson.build:129:5: ERROR: Module "numpy" does not exist

Thanks again, making progress at every step, getting there !

And sorry, I'm a bit illiterate in meson (what language is the meson.build file btw ? it looks a bit like python and bash at the same time)

Cheers

[Oscar Benjamin]

On Thu, 13 Nov 2025 at 19:50, Didier VEZINET <didier....@gmail.com> wrote:
>
> Mimicking the syntax seen for python itself I've tried this but it doesn't work:
>
> np = import('numpy')
> np_header_dir = np.get_include()
> include_dirs = [np_header_dir]
>
> I get a numpy not found error upon calling meson setup --reconfigure builddir
>
> meson.build:129:5: ERROR: Module "numpy" does not exist
>
> Thanks again, making progress at every step, getting there !
> And sorry, I'm a bit illiterate in meson (what language is the meson.build file btw ? it looks a bit like python and bash at the same time)

The meson.build file is written in the meson language. It looks a bit
like Python but it is not Python (even though meson itself is
implemented in Python):
https://mesonbuild.com/Syntax.html

Because the meson language is not Python you cannot import a Python
module like numpy. Meson's import function is for importing meson
modules:
https://mesonbuild.com/Modules.html

The meson language is not Python and meson as a build system is not
only intended to be used for building Python packages. These are
significant differences from setuptools. It is always possible to
shell out from meson to Python code but I would not assume to be able
to access numpy that way since meson is itself just a CLI program that
happens to be on PATH and may not be running from the particular
Python environment where you installed numpy.

Assume that meson only uses CLI interfaces much like make. The link I
sent before shows how to get this path for numpy from the command line
rather than from Python running within the environment where numpy is
installed.

It may be better to tell meson that your extension modules depend on
numpy and use numpy's pkgconfig to locate the headers.

--
Oscar

[Jérôme Kieffer]

There are several possibilities exposed in this thread:
https://github.com/mesonbuild/meson/issues/9598

This issue is still open: it indicates it is still WIP but there are at
least 2 work-arround this problem.

BTW, I try to avoid numpy cimport dependency, it allows me to ship
wheels which work with any version of numpy (no ABI compatibility
issue). It is not that difficult ... but represents a bit of work.

Cheers,

Jerome

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.

> To view this discussion visit https://groups.google.com/d/msgid/cython-users/CAHVvXxSE2Qp28eqq-T1HemYX%3Drn%3DBJtVSNe%2Bzt1yar%3DRdXwtJg%40mail.gmail.com.
>


--

[Didier VEZINET]

Thanks Oscar and Jerome, 

Really helps !

So I've implemented the scipy solution, which presumably should work for all versions of numpy (I don't have numpy 2 compatibility yet).

It seems to work, looks like meson compile is running !

Next step: I now need to test the whole build workflow and automatize building sdist and wheel in github Actions and distributing, but at least I'm not stuck at compilation anymore. 

Thanks a lot for your help,

Best

Didier