cython3 cythonize issues due to import build_ext failure

[Laban Mwangi]

Hi all,

 I came across an issue when cythonizing a project by following a blog post.

 The project works with cython 0.x but not with cython3.x. On drilling down, it looks like an import issue was introduce by  this commit. 

So do folks think we should revert this commit? Pros? Cons?

Setup:

- MacOS Monterey 12.4 (21F79) x86_64

 -  $ python -VV  

     Python 3.9.13 (main, May 24 2022, 21:28:31)
     [Clang 13.1.6 (clang-1316.0.21.2)]

-  $ poetry run Cython -V   

     Cython version 3.0.0a10

Thanks,

Laban

PS:

I have also created an issue with the tutorial project for reference purposes. It's also documented below:

```
$  /tmp/example-cython-poetry-pypi 
 ❯ poetry build -vvv                                                                                                                                                                                                              
Using virtualenv: /Users/lmwangi/Library/Caches/pypoetry/virtualenvs/poetry-pypi-example-wSzIWKA2-py3.9
Building poetry-pypi-example (0.0.1)
  - Building sdist
  - Adding: /private/tmp/example-cython-poetry-pypi/poetry_pypi_example/main.cpython-39-darwin.so
  - Adding: README.rst
  - Adding: pyproject.toml
  - Built poetry-pypi-example-0.0.1.tar.gz
  - Building wheel
  - Executing build script: build.py
[1/1] Cythonizing poetry_pypi_example/main.py
Traceback (most recent call last):
  File "/private/tmp/example-cython-poetry-pypi/build.py", line 80, in <module>
    build()
  File "/private/tmp/example-cython-poetry-pypi/build.py", line 30, in build
    build_ext_cmd.copy_extensions_to_source()
  File "/Users/lmwangi/Library/Caches/pypoetry/virtualenvs/poetry-pypi-example-wSzIWKA2-py3.9/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 103, in __getattr__
    raise AttributeError(attr)
AttributeError: copy_extensions_to_source
```


On drilling down, it looks like the import for build_ext in cython3 does not have copy_extensions_to_source. To be precise in [line 14](https://github.com/cython/cython/blob/master/Cython/Distutils/build_ext.py#L14), build_ext is None when it is imported from setuptools. Please see below:

```python
Python 3.9.13 (main, May 24 2022, 21:28:31)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.6.0 -- An enhanced Interactive Python. Type '?' for help.

# Injected an IPython shell just before the setuptools import in Cython/Distutils/build_ext.py
In [1]: _build_ext_module

# Should return the build_ext module, but is currently unset. Hmm, we can't get the build_ext module. Let's check the parents.
In [2]:  sys.modules.get('setuptools.command.build_ext')

# We can get the setuptools module
In [3]:  sys.modules.get('setuptools')
Out[3]: <module 'setuptools' from '/Users/lmwangi/Library/Caches/pypoetry/virtualenvs/wallet-api-ZqjMZXwH-py3.9/lib/python3.9/site-packages/setuptools/__init__.py'>


# We can get the setuptools.command module
In [5]:  sys.modules.get('setuptools.command')
Out[5]: <module 'setuptools.command' from '/Users/lmwangi/Library/Caches/pypoetry/virtualenvs/wallet-api-ZqjMZXwH-py3.9/lib/python3.9/site-packages/setuptools/command/__init__.py'>

# On the filesystem, the module does exist.
In [6]: !ls /Users/lmwangi/Library/Caches/pypoetry/virtualenvs/wallet-api-ZqjMZXwH-py3.9/lib/python3.9/site-packages/setuptools/command
 __init__.py   alias.py       bdist_rpm.py    build_ext.py   develop.py     easy_install.py   install.py        install_lib.py    'launcher manifest.xml'   register.py   saveopts.py   setopt.py   upload.py
 __pycache__   bdist_egg.py   build_clib.py   build_py.py    dist_info.py   egg_info.py       install_egg_info.py   install_scripts.py     py36compat.py          rotate.py     sdist.py      test.py      upload_docs.py


# Hmm, import does work
In [8]: import setuptools.command.build_ext

# Now a get works....
In [9]:  sys.modules.get('setuptools.command.build_ext')
Out[9]: <module 'setuptools.command.build_ext' from '/Users/lmwangi/Library/Caches/pypoetry/virtualenvs/wallet-api-ZqjMZXwH-py3.9/lib/python3.9/site-packages/setuptools/command/build_ext.py'>

```

Digging further, it looks like this is the `[breaking commit](https://github.com/cython/cython/commit/c6f5c5ddcc021099febae7cb1194fedea01fd056)`.

```diff
< # Always inherit from the "build_ext" in distutils since setuptools already imports
< # it from Cython if available, and does the proper distutils fallback otherwise.
< # https://github.com/pypa/setuptools/blob/9f1822ee910df3df930a98ab99f66d18bb70659b/setuptools/command/build_ext.py#L16
<
< # setuptools imports Cython's "build_ext", so make sure we go first.
< _build_ext_module = sys.modules.get('setuptools.command.build_ext')
< if _build_ext_module is None:
<     import distutils.command.build_ext as _build_ext_module
<
< # setuptools remembers the original distutils "build_ext" as "_du_build_ext"
< _build_ext = getattr(_build_ext_module, '_du_build_ext', None)
< if _build_ext is None:
<     _build_ext = getattr(_build_ext_module, 'build_ext', None)
< if _build_ext is None:
---
> if 'setuptools' in sys.modules:
>     try:
>         from setuptools.command.build_ext import build_ext as _build_ext
>     except ImportError:
>         # We may be in the process of importing setuptools, which tries
>         # to import this.
>         from distutils.command.build_ext import build_ext as _build_ext
```

If we revert the import in Cython/Distutils/build_ext.py with the one in cython 0.x, the build now works

```sh

 ❯  head -17 /Users/lmwangi/Library/Caches/pypoetry/virtualenvs/poetry-pypi-example-wSzIWKA2-py3.9/lib/python3.9/site-packages/Cython/Distutils/build_ext.py
import sys
import os


import sys

if 'setuptools' in sys.modules:
    try:
        from setuptools.command.build_ext import build_ext as _build_ext
    except ImportError:
        # We may be in the process of importing setuptools, which tries
        # to import this.
        from distutils.command.build_ext import build_ext as _build_ext
else:
    from distutils.command.build_ext import build_ext as _build_ext


 ❯ poetry run Cython -V
Cython version 3.0.0a10

  ❯ poetry build  
Building poetry-pypi-example (0.0.1)
  - Building sdist
  - Built poetry-pypi-example-0.0.1.tar.gz
  - Building wheel
[1/1] Cythonizing poetry_pypi_example/main.py
  - Built poetry_pypi_example-0.0.1-cp39-cp39-macosx_12_0_x86_64.whl
```

[da-woods]

On 23/07/2022 09:55, Laban Mwangi wrote:
>
> So do folks think we should revert this commit? Pros? Cons?
>

I'm not convinced this justifies reverting it. The blog post admits that
it's using unsupported features, and Cython 3 is the place where we're
breaking some things to be able to make improvements.

[Stefan Behnel]

Laban Mwangi schrieb am 23.07.22 um 10:55:

> I came across an issue when cythonizing a project

> <https://github.com/aotuai/example-cython-poetry-pypi> by following a blog
> post
> <https://aotu.ai/en/blog/2021/01/19/publishing-a-proprietary-python-package-on-pypi-using-poetry/>
> .

> The project works with cython 0.x but not with cython3.x. On drilling
> down, it looks like an import issue was introduce by this commit

> <https://github.com/cython/cython/commit/c6f5c5ddcc021099febae7cb1194fedea01fd056>

Thanks. Given that distutils is deprecated and will be removed from the
stdlib soon, the way forward is to always use setuptools, and only fall
back to distutils if that's not available (and I think that's also what we
already do in most places). See

https://github.com/cython/cython/issues/4405
and
https://github.com/cython/cython/pull/4498

The change you referenced above came from a tricky edge case where both
setuptools and Cython try to replace the distutils "build_ext" command.
Can't say right now what room for improvement there is, but we obviously
need this to continue working when there is only setuptools and no longer
distutils.

Stefan