PyInit function name

[Ray Chen]

I see in the generated .c file from Cython that on Windows, it seems that the init function for __init__.py is PyInit___init__ not PyInit_<package name>. There is a clear #if directive for Windows:

#if !defined(CYTHON_NO_PYINIT_EXPORT) && (defined(_WIN32) || defined(WIN32) || defined(MS_WINDOWS))

__Pyx_PyMODINIT_FUNC PyInit___init__(void) { return PyInit_submodule(); }

#endif

This also means that if i turn off CYTHON_NO_PYINIT_EXPORT, then this wont happen. As I am new to cython, what exactly happens when i unset CYTHON_NO_PYINIT_EXPORT?

The reason why I am trying to do this is because I am replicating a Python library for merak which combines a Python package into a single shared library. So far, this works on Linux but on Windows what happens is all subpackages (which use a __init__.py file) have their init function in C called PyInit___init__ unlike on Linux, where it is called  PyInit_<package name>. In the C code PyInit_submodule is also defined, and all other code that was generated from my python code also uses the name __pyx_mdef_9_submodule_8___init__ (which i take to mean submodule.__init__).

[Ray Chen]

Ok edit: i meant if I turn on  CYTHON_NO_PYINIT_EXPORT. However that just breaks everything and I cannot import my built library. What makes the Linux and Windows code different so that on linux PyInit_submodule is used as the module name?

[Stefan Behnel]

Ray Chen schrieb am 02.09.21 um 12:51:

> On Thursday, September 2, 2021 at 11:20:32 AM UTC+1 Ray Chen wrote:
>
>> I see in the generated .c file from Cython that on Windows, it seems that
>> the init function for __init__.py is PyInit___init__ not PyInit_<package
>> name>. There is a clear #if directive for Windows:
>>
>> #if !defined(CYTHON_NO_PYINIT_EXPORT) && (defined(_WIN32) ||
>> defined(WIN32) || defined(MS_WINDOWS))
>> __Pyx_PyMODINIT_FUNC PyInit___init__(void) { return PyInit_submodule(); }
>> #endif
>>
>> This also means that if i turn off CYTHON_NO_PYINIT_EXPORT, then this wont
>> happen. As I am new to cython, what exactly happens when i unset
>> CYTHON_NO_PYINIT_EXPORT?

It hides the exported symbol of the PyInit module entry point, so that it's
only accessible from inside the C code file. That's intended specifically
for the use case of including Cython modules in external libraries or
applications, which use the inittab mechanism for initialising the
statically included extension modules.

>> The reason why I am trying to do this is because I am replicating a Python
>> library for merak which combines a Python package into a single shared
>> library. So far, this works on Linux but on Windows what happens is all
>> subpackages (which use a __init__.py file) have their init function in C
>> called PyInit___init__ unlike on Linux, where it is called PyInit_<package
>> name>. In the C code PyInit_submodule is also defined, and all other code
>> that was generated from my python code also uses the name
>> __pyx_mdef_9_submodule_8___init__ (which i take to mean submodule.__init__).
>

> Ok edit: i meant if I turn on CYTHON_NO_PYINIT_EXPORT. However that just
> breaks everything

Does it? I assume you're not using the symbol directly then, but trying to
access it externally, right?

> and I cannot import my built library. What makes the
> Linux and Windows code different so that on linux PyInit_submodule is used
> as the module name?

The code that you copied above is a work-around for a CPython bug on
Windows, which searches for the wrong symbol name of the PyInit function of
package modules (__init__.py). Thus, Cython exports two symbols on Windows,
the correct one and the one that CPython searches for, both leading to the
same PyInit function.

This obviously conflicts with the goal of linking together multiple packages.

Stefan

[Ray Chen]

>  Does it? I assume you're not using the symbol directly then, but trying to
access it externally, right?

No, I am using importlib.machinery.ExtensionFileLoader to load a specific module from the shared library. I do pretty much everything in pure Python, and I do not wish to use Cython syntax.

> Thus, Cython exports two symbols on Windows,

> the correct one and the one that CPython searches for, both leading to the
> same PyInit function.

So how do I make it work on Windows like how it works on Linux, with different PyInit function names? And what exactly will happen if i remove the declaration of the PyInit___init__ function and everywhere it is used?

[Ray Chen]

Also what is the CPython bug that you were referring to? Is there a link to it?

[Stefan Behnel]

Ray Chen schrieb am 02.09.21 um 15:28:

> Also what is the CPython bug that you were referring to? Is there a link to
> it?

Yes, there is, as a quick web search reveals:

https://bugs.python.org/issue35893

Stefan

[Stefan Behnel]

Ray Chen schrieb am 02.09.21 um 15:24:

>> I assume you're not using the symbol directly then, but trying
>> to access it externally, right?
>
> No, I am using importlib.machinery.ExtensionFileLoader to load a specific
> module from the shared library.

Not sure I understand – do you have a single shared library with multiple
extension modules linked together? Could you show the code that you use for
(loading and) initialising the modules?

>> Thus, Cython exports two symbols on Windows,
>> the correct one and the one that CPython searches for, both leading to the
>> same PyInit function.
>
> So how do I make it work on Windows like how it works on Linux, with
> different PyInit function names?

They are not different. There is only an *additional* symbol with name
"…__init__" being exported on Windows.

> And what exactly will happen if i remove
> the declaration of the PyInit___init__ function and everywhere it is used?

I looked up the CPython ticket. It won't build.

https://bugs.python.org/issue35893

It's not a problem at module load time, only at build time.

Stefan

[Ray Chen]

bootstrap.py: https://pastebin.com/1jWkFzUs

prepare.py: https://pastebin.com/BmE8myC3

setup.py: https://pastebin.com/C6mQymXA

prepare.py just generates the Cython files and patches them (specifically changes traceback file info to match the original). What it does is it replaces file separators with _SEP_. This makes something like pyunity.window.glfwWindow become pyunity_SEP_window_SEP_glfwWindow. All packages are renamed into folders but still keep their __init__.py file because that will distinguish them as a package, as opposed to just a normal module. It also patches bootstrap.py so that the custom loader knows what packages are bundled. By building with `python setup.py build_ext --build-lib build/lib` I get a single bootstrap.so file under build/lib, which I can import. Once i do that, my custom loader is injected to sys.meta_paths so I can import pyunity (from the bootstrap.so file).

[Ray Chen]

Side note: in prepare.py at the very end I replace the #if directive that I mentioned before. This was just a test and I forgot to take it out before pasting into the pastebin.

In prepare.py I do a lot of things that are dependent on things that will quite possibly change. I have acknowledged that which is why I will be findng a better way to do most things, but right now I am just experimenting.

[Ray Chen]

> Yes, there is, as a quick web search reveals:

> https://bugs.python.org/issue35893

> Stefan

That bug report describes a monkey patch to build_ext.get_export_symbols - is that safe? If I were to put that in my setup.py, what else do I need to do? I'm assuming I can remove the additional symbol?