[Python-Dev] Clarification regarding Stable ABI and _Py_*

[Guido van Rossum]

Hi Petr,

In PEP 384 it is written that no functions starting with an underscore are part of the stable ABI:

PEP 384 -- Defining a Stable ABI | Python.org

> All functions starting with _Py are not available to applications

OTOH there's a data file in the repo, Misc/stabe_abi.txt, which lists many functions starting with _Py_, for example _PyObject_GC_Malloc. Then again, that function is not listed in Doc/data/stable_abi.dat. (I didn't check other functions, but maybe there are others.)

So is Misc/stable_abi.txt just out of date? Or how can the discrepancy be explained?

--

--Guido van Rossum (python.org/~guido)

Pronouns: he/him (why is my pronoun here?)

[Petr Viktorin]

On 06. 12. 21 20:29, Guido van Rossum wrote:
> Hi Petr,
>
> In PEP 384 it is written that no functions starting with an underscore
> are part of the stable ABI:
>
> PEP 384 -- Defining a Stable ABI | Python.org

> <https://www.python.org/dev/peps/pep-0384/#excluded-functions>

> > All functions starting with _Py are not available to applications
>
> OTOH there's a data file in the repo, Misc/stabe_abi.txt, which lists
> many functions starting with _Py_, for example _PyObject_GC_Malloc. Then
> again, that function is not listed in Doc/data/stable_abi.dat. (I didn't
> check other functions, but maybe there are others.)
>
> So is Misc/stable_abi.txt just out of date? Or how can the discrepancy
> be explained?

These are not part of the limited API, so extension authors can't use
them in the C source. But they typically are (or have been) called by
macros from the limited API. So, they are part of the stable ABI; they
need to be exported.

Misc/stable_abi.txt says "abi_only" for all of these. They don't show up
in the user-facing docs.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/WB3W2QY4PLM5XGLQMEEC4HE3HSQ4M3KO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Guido van Rossum]

On Mon, Dec 6, 2021 at 12:12 PM Petr Viktorin <enc...@gmail.com> wrote:

On 06. 12. 21 20:29, Guido van Rossum wrote:
> Hi Petr,
>
> In PEP 384 it is written that no functions starting with an underscore
> are part of the stable ABI:
>
> PEP 384 -- Defining a Stable ABI | Python.org
> <https://www.python.org/dev/peps/pep-0384/#excluded-functions>
>  > All functions starting with _Py are not available to applications
>
> OTOH there's a data file in the repo, Misc/stabe_abi.txt, which lists
> many functions starting with _Py_, for example _PyObject_GC_Malloc. Then
> again, that function is not listed in Doc/data/stable_abi.dat. (I didn't
> check other functions, but maybe there are others.)
>
> So is Misc/stable_abi.txt just out of date? Or how can the discrepancy
> be explained?

These are not part of the limited API, so extension authors can't use
them in the C source. But they typically are (or have been) called by
macros from the limited API. So, they are part of the stable ABI; they
need to be exported.

Misc/stable_abi.txt says "abi_only" for all of these. They don't show up
in the user-facing docs.

Thanks, that helps. It's too bad that there's no comment at the top explaining the format (in fact it appears to discourage reading the file?).

Also, it looks like Mark is proposing to *remove* _PyObject_GC_Malloc from stable_abi.txt in https://github.com/python/cpython/pull/29879 Is that allowed? If it's being used by a macro it means code using that macro will fail unless recompiled for 3.11.

[Victor Stinner]

On Mon, Dec 6, 2021 at 9:54 PM Guido van Rossum <gu...@python.org> wrote:
> Also, it looks like Mark is proposing to *remove* _PyObject_GC_Malloc from stable_abi.txt

In Python 3.2, _PyObject_GC_Malloc() is implemented as a function.
PyObject_GC_New() macro calls _PyObject_GC_New() function. Internally,
PyType_GenericAlloc() and _PyObject_GC_New() call
_PyObject_GC_Malloc().

=> IMO _PyObject_GC_Malloc() doesn't need to be part of Python 3.2 stable ABI

In Python 3.11, _PyObject_GC_Malloc() is still implemented as a
function. It's not used in the C API header files, it's only used in
the implementation of other functions.

=> IMO again, IMO _PyObject_GC_Malloc() doesn't need to be part of
Python 3.11 stable ABI

It would be nice if someone could double check my analysis.


In Python 3.7, there was still "#undef _PyObject_GC_Malloc" in
Modules/gcmodule.c with the comment:

/* extension modules might be compiled with GC support so these
functions must always be available */

This code comes from an old commit made for ABI compatibility with...
Python 2.2!

https://github.com/python/cpython/commit/fec4eb1be1bb193ab3db0645a2c084c34868401c
---
commit fec4eb1be1bb193ab3db0645a2c084c34868401c
Author: Neil Schemenauer <nasc...@enme.ucalgary.ca>
Date: Fri Apr 12 02:41:03 2002 +0000

Allow PyObject_Del to be used as a function designator. Provide binary
compatibility function.

Make PyObject_GC_Track and PyObject_GC_UnTrack functions instead of
trivial macros wrapping functions. Provide binary compatibility
functions.
---

I don't think that Python 3.2 needs to provide ABI compatibility with
Python 2.2.

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/4SUSFYA2P3CLCP3ULYSXETM2QEBNUWQG/

[Petr Viktorin]

You can read it, but I want to discourage people from relying on the
format: Tools/scripts/stable_abi.py should be the only consumer.
I will add a comment though.

> Also, it looks like Mark is proposing to *remove* _PyObject_GC_Malloc
> from stable_abi.txt in https://github.com/python/cpython/pull/29879
> <https://github.com/python/cpython/pull/29879> Is that allowed? If it's
> being used by a macro it means code using that macro will fail unless
> recompiled for 3.11.

Generally, that's not allowed. In this particular case, Victor's
analysis is right: if you trawl through the history from 3.2 on, you can
see that you can't call _PyObject_GC_Malloc via macros in the limited
API. So yes, this one can be removed.

I'll also note that removing things that are "allowed" to go is not nice
to people who relied on PEP 384, which says that defining Py_LIMITED_API
"will hide all definitions that are not part of the ABI" -- even though
that's incompatible with the part where it says "All functions starting
with _Py are not available to applications".
PEP 384 is a historical document, but before 3.10 it was the best
available documentation. PEP 652 sort of changed the rules mid-course
(ref. https://www.python.org/dev/peps/pep-0652/#backwards-compatibility).


But for _PyObject_GC_Malloc specifically, IMO the speedup is worth it.
Go ahead and remove it.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/S7PL37E4Z4RHICOHTK32NCQTYN5C6FAQ/

[Guido van Rossum]

I don't mind that the format might change. But I feel that I should be able to understand the current format so I can know how to maintain the document without using cargo-culting (and trying to understand what the tool does is not an option). So thanks for adding the comment.

 

> Also, it looks like Mark is proposing to *remove* _PyObject_GC_Malloc
> from stable_abi.txt in https://github.com/python/cpython/pull/29879
> <https://github.com/python/cpython/pull/29879> Is that allowed? If it's
> being used by a macro it means code using that macro will fail unless
> recompiled for 3.11.

Generally, that's not allowed. In this particular case, Victor's
analysis is right: if you trawl through the history from 3.2 on, you can
see that you can't call _PyObject_GC_Malloc via macros in the limited
API. So yes, this one can be removed.

Okay, that's very subtle, so thanks for confirming.

 

I'll also note that removing things that are "allowed" to go is not nice
to people who relied on PEP 384, which says that defining Py_LIMITED_API
"will hide all definitions that are not part of the ABI" -- even though
that's incompatible with the part where it says "All functions starting
with _Py are not available to applications".

I don't actually really follow what you are trying to say here. Probably because I've never paid much attention to PEP 384. I guess the API is confusing because the "right" way to do it (having to define some symbol to *expose* extra stuff rather than to *hide* stuff) was not possible for backwards compatibility reasons. But the extra negative will forever make this confusing. Also, "All functions starting with _Py are not available" sounds like a clumsy way to say "No functions starting with _Py are available" (and you left out whether Py_LIMITED_API affects that availability, whether it was intended to affect it, whether it did in practice affect it in all cases, etc.

I assume it would be insensitive to ask whether we could just get rid of the stable ABI altogether and focus on the limited API? Just tell everyone they have to rebuild binary wheels for every Python feature release. Presumably the deprecation of the stable ABI itself would require a two-release waiting period. But maybe it would be worth it, given how subtle it is to do the historical research about even a single function.

 

PEP 384 is a historical document, but before 3.10 it was the best
available documentation. PEP 652 sort of changed the rules mid-course
(ref. https://www.python.org/dev/peps/pep-0652/#backwards-compatibility).


But for _PyObject_GC_Malloc specifically, IMO the speedup is worth it.
Go ahead and remove it.

It's gone. Mark landed his change earlier today.

[Christian Heimes]

On 07/12/2021 19.28, Guido van Rossum wrote:
> I assume it would be insensitive to ask whether we could just get rid of
> the stable ABI altogether and focus on the limited API? Just tell
> everyone they have to rebuild binary wheels for every Python feature
> release. Presumably the deprecation of the stable ABI itself would
> require a two-release waiting period. But maybe it would be worth it,
> given how subtle it is to do the historical research about even a single
> function.

The stable ABI is useful for Python packages that ship binary wheels.

Take PyCA cryptography [1] as an example. Alex and Paul already build,
upload, and ship 12 abi3 wheels for each release and combinations of CPU
arch, platform, and libc ABI. Without a stable ABI they would have to
create a total of 60 binary abi3 wheels for Python 3.6 to 3.10. The
number will only increase over time. Python 3.6 is very common on
LTS/Enterprise Linux distros.

If the current stable ABI makes performance improvements too complex
then we should consider to define a new stable ABI with less symbols.

Christian

[1] https://pypi.org/project/cryptography/#files

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/PQ7GNBTFAHIC6HWNZ62MQJJOV75N7UAT/

[Guido van Rossum]

On Tue, Dec 7, 2021 at 11:02 AM Christian Heimes <chri...@python.org> wrote:

On 07/12/2021 19.28, Guido van Rossum wrote:
> I assume it would be insensitive to ask whether we could just get rid of
> the stable ABI altogether and focus on the limited API? Just tell
> everyone they have to rebuild binary wheels for every Python feature
> release. Presumably the deprecation of the stable ABI itself would
> require a two-release waiting period. But maybe it would be worth it,
> given how subtle it is to do the historical research about even a single
> function.

The stable ABI is useful for Python packages that ship binary wheels.

Take PyCA cryptography [1] as an example. Alex and Paul already build,
upload, and ship 12 abi3 wheels for each release and combinations of CPU
arch, platform, and libc ABI. Without a stable ABI they would have to
create a total of 60 binary abi3 wheels for Python 3.6 to 3.10. The
number will only increase over time. Python 3.6 is very common on
LTS/Enterprise Linux distros.

Thanks, that's a very useful example.

 

If the current stable ABI makes performance improvements too complex
then we should consider to define a new stable ABI with less symbols.

But then we will run into backwards compatibility concerns. Suppose we want to delete *one* functions from the stable ABI. How many releases do we have to wait before we can actually delete (as opposed to just deprecate) it? It sounds like you're saying it would take 5 releases, i.e. if we deprecate it in 3.11, we can delete it in 3.16. It would probably be easier to just not bother with the deprecation.

[Petr Viktorin]

That really depends on what function we'd want to remove. There are
usually alternatives to deleting things, but the options depend on the
function. If we run out of other options we can make the function always
fail or make it leak memory.
And the regular backwards compatibility policy gives us 2 years to
figure something out :)


It is possible that we'll need a new stable ABI for nogil, though, since
refcounting is one of the few areas where even the stable ABI uses
direct struct access rather than functions.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/WN7TBSV6LM24X6CIEQQDG63OYPIR5ZLU/

[Petr Viktorin]

On 07. 12. 21 19:28, Guido van Rossum wrote:
> On Tue, Dec 7, 2021 at 12:58 AM Petr Viktorin <enc...@gmail.com

> <mailto:enc...@gmail.com>> wrote:
>
> On 06. 12. 21 21:50, Guido van Rossum wrote:

[...]

It's hard to say what PEP 384 was meant to say. My interpretation, PEP
652, is hopefully more consistent. But someone who had a different
interpretation of PEP 384 might feel that it broke some promise.

> I assume it would be insensitive to ask whether we could just get rid of
> the stable ABI altogether and focus on the limited API? Just tell
> everyone they have to rebuild binary wheels for every Python feature
> release. Presumably the deprecation of the stable ABI itself would
> require a two-release waiting period. But maybe it would be worth it,
> given how subtle it is to do the historical research about even a single
> function.

A honest question wouldn't be insensitive. Thanks for asking!

The part where you don't need to rebuild extensions (not just wheels) is
the main reason for both Stable ABI and the Limited API.
Without it, there might be some reduced API to focus on, but it wouldn't
be this feature.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/SBQHQ6DWSXMDVQCETVDMRLZ3CNG4OJUL/

[Paul Moore]

On Wed, 8 Dec 2021 at 12:42, Petr Viktorin <enc...@gmail.com> wrote:
> > I assume it would be insensitive to ask whether we could just get rid of
> > the stable ABI altogether and focus on the limited API? Just tell
> > everyone they have to rebuild binary wheels for every Python feature
> > release. Presumably the deprecation of the stable ABI itself would
> > require a two-release waiting period. But maybe it would be worth it,
> > given how subtle it is to do the historical research about even a single
> > function.
>
> A honest question wouldn't be insensitive. Thanks for asking!
>
> The part where you don't need to rebuild extensions (not just wheels) is
> the main reason for both Stable ABI and the Limited API.
> Without it, there might be some reduced API to focus on, but it wouldn't
> be this feature.

This I think is the critical point here. I don't follow the API/ABI
debate closely, but I do know that the number varieties of C extension
builds some projects have to make is a serious overhead. Being able to
target the stable ABI and say "this will work for all (essentially)
versions of CPython" is a huge benefit, as it reduces the number (and
hence the burden) significantly.

Not having to amend your source for each CPython release feels to me
like a subset of this, and loses a major advantage, so my personal
view is that the stable ABI is far more beneficial than the limited
API (the limited API is what you fall back to if you're unable to use
the stable ABI).

Paul

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/KCAIYXMKY7RZULJNG7BGM5OQ7RF5Z266/

[Eric Snow]

On Wed, Dec 8, 2021 at 2:23 AM Petr Viktorin <enc...@gmail.com> wrote:
> That really depends on what function we'd want to remove. There are
> usually alternatives to deleting things, but the options depend on the
> function. If we run out of other options we can make the function always
> fail or make it leak memory.
> And the regular backwards compatibility policy gives us 2 years to
> figure something out :)

What about the various symbols listed in Misc/stable_abi.txt that were
accidentally added to the limited API? Can we move toward dropping
them from the stable ABI?

Most notably, there are quite a few functions listed there that are in
the stable ABI but no longer in the limited API. This implies that
either they were already deprecated in the limited API (and removed)
or they were just removed. At least in some cases they were moved to
header files in Include/cpython or Include/internal. So I would not
expect extensions to be using them. This subset of those symbols
seems entirely appropriate to remove from the stable ABI. Is that
okay? Do we even need to bother deprecating them? What about just
the "private" ones?

For example, I went to change/remove _PyThreadState_Init() (internal
API declared in Include/internal/pycore_pystate.h) and found that it
is in the stable ABI but not the limited API. It's highly unlikely
anyone is using it and plan on double-checking. As far as I can tell,
the function was accidentally exposed in the limited API and stable
ABI and later removed from the limited API.

-eric

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/OJ65FPCJ2NVUFNZDXVNK5DU3R3JGLL3J/

[Petr Viktorin]

It's possible to remove them just like _PyObject_GC_Malloc was removed,
but check that it was unusable (e.g. not called from public macros) in
all versions of Python from 3.2 up to now.
Could you check if this PR makes things clear?
https://github.com/python/devguide/pull/778

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/T3M5AE6DFB2C6JAMCAMAIC2XQDNOKND5/

[Eric Snow]

On Thu, Dec 9, 2021 at 1:56 AM Petr Viktorin <enc...@gmail.com> wrote:
> It's possible to remove them just like _PyObject_GC_Malloc was removed,
> but check that it was unusable (e.g. not called from public macros) in
> all versions of Python from 3.2 up to now.

That's what I expected. Thanks.

> Could you check if this PR makes things clear?
> https://github.com/python/devguide/pull/778

Yeah, that text is super helpful.

-eric

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/TB7JEBQXUJJKK4SZVLCMUNOTRTD5KQ5C/