cython access to C implementation of stdlib module

[jbrock...@gmail.com]

Looking at the standard library decimal module, its (default) implementation is in C.  AFAICt there is no corresponding .h file.  Still, it would be nice to be able to cimport from there for e.g. fast isinstance checks.

Is this a pipe-dream?

[Michael Wayne Goodman]

I'd like to echo this question. In my case, I want to use the C implementation of the re module. It has header files but my attempts at importing from them have failed. I'd like to be able to provide static typing for compiled Pattern objects and call matching function without hopping back to Python.

[Stefan Behnel]

jbrock...@gmail.com schrieb am 24.01.21 um 21:45:

> Looking at the standard library decimal module

> <https://github.com/python/cpython/blob/3.9/Lib/decimal.py>, its (default)
> implementation
> <https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/Modules/_decimal/_decimal.c>

> is in C. AFAICt there is no corresponding .h file. Still, it would be
> nice to be able to cimport from there for e.g. fast isinstance checks.

The decimal module's C-API that you were referring to was taken back out
before the release of CPython 3.10 in

https://bugs.python.org/issue43422

Thus, there is no "official" way to access the internals any more.

But if you can live with potential unanticipated breakage in the future,
you can choose to exploit the internals.

Write a, say, "cdecimal.pxd" file like this:

cdef extern from *:
"""
typedef struct {
PyObject_HEAD
/* remaining fields intentionally left out */
} PyDecObject;

/* copy more declarations from _decimal.c here */
"""

ctypedef class _decimal.Decimal [object PyDecObject,
check_size ignore]


and so on. Note that I left out the mpd_t etc. typed fields to avoid having
to compile against libmpdec. If you want to include them, you'll have to
add those declarations, too, or compile against the header files of the
library, which are not included in a CPython installation. That might
become a bit fiddly.

Anyway, the above .pxd file should give Cython and the C compiler
everything they need to make use of the _decimal module's internals that
you declared to them.

Note that the above is untested, but something like this should work.

Good luck! And if you can get this .pxd file to work, it's probably worth
its own package on PyPI.

Stefan

[Stefan Behnel]

Michael Wayne Goodman schrieb am 22.05.21 um 04:33:

> I want to use the C
> implementation of the re module. It has header files but my attempts at
> importing from them have failed. I'd like to be able to provide static
> typing for compiled Pattern objects and call matching function without
> hopping back to Python.

As far as I can see, the header file of the re module is not part of the
C-API and is not part of the Python installation, only of the source checkout.

Then, basically the same applies as I just wrote about the "_decimal"
module. You can copy the relevant parts out of the sources and write your
own mixed-C-Cython .pxd file with them. You must copy enough to make
everything compilable by the C compiler, but you're otherwise on your own
with how far you go. The more you copy, the higher the chances of future
breakage due to internal changes. If that happens, you can use preprocessor
version checks in your declarations to adapt at C compile time.

As for the "_decimal" module, this sounds worth a PyPI package if you can
get it to work.

Stefan

[Michael Wayne Goodman]

On Saturday, May 22, 2021 at 12:21:25 PM UTC+8 Stefan Behnel wrote:

Michael Wayne Goodman schrieb am 22.05.21 um 04:33:
> I want to use the C
> implementation of the re module. It has header files but my attempts at
> importing from them have failed. I'd like to be able to provide static
> typing for compiled Pattern objects and call matching function without
> hopping back to Python.

As far as I can see, the header file of the re module is not part of the
C-API and is not part of the Python installation, only of the source checkout.

Sorry, yes I just meant that the source has, e.g., sre.h and similar header files, not that it's part of the C-API. It now makes sense that this fact doesn't help us much.

 

Then, basically the same applies as I just wrote about the "_decimal"
module. You can copy the relevant parts out of the sources and write your
own mixed-C-Cython .pxd file with them. You must copy enough to make
everything compilable by the C compiler, but you're otherwise on your own
with how far you go. The more you copy, the higher the chances of future
breakage due to internal changes. If that happens, you can use preprocessor
version checks in your declarations to adapt at C compile time.

Great, thanks for explaining!

[Daniele Nicolodi]

On 22/05/2021 06:08, Stefan Behnel wrote:
> jbrock...@gmail.com schrieb am 24.01.21 um 21:45:
>> Looking at the standard library decimal module
>> <https://github.com/python/cpython/blob/3.9/Lib/decimal.py>, its (default)
>> implementation
>> <https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/Modules/_decimal/_decimal.c>
>> is in C. AFAICt there is no corresponding .h file. Still, it would be
>> nice to be able to cimport from there for e.g. fast isinstance checks.
>
> The decimal module's C-API that you were referring to was taken back out
> before the release of CPython 3.10 in
>
> https://bugs.python.org/issue43422
>
> Thus, there is no "official" way to access the internals any more.

I don't know if it is helpful for your use case, but I have some (C not
Cython code) to use the decimal module C implementation here:

https://github.com/beancount/beancount/blob/master/beancount/parser/decimal.h
https://github.com/beancount/beancount/blob/master/beancount/parser/decimal.c

This mimics what Python itself does for the exposed datetime module C
API. If you need more than instantiating Decimal objects, you would need
some additions to this.

Cheers,
Dan