PEP 384: Defining a Stable ABI
"Martin v. Löwis"
found the time to write it down.
Please comment!
Regards,
Martin
PEP: 384
Title: Defining a Stable ABI
Version: $Revision: 72754 $
Last-Modified: $Date: 2009-05-17 21:14:52 +0200 (So, 17. Mai 2009) $
Author: Martin v. L�wis <mar...@v.loewis.de>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 17-May-2009
Python-Version: 3.2
Post-History:
Abstract
========
Currently, each feature release introduces a new name for the
Python DLL on Windows, and may cause incompatibilities for extension
modules on Unix. This PEP proposes to define a stable set of API
functions which are guaranteed to be available for the lifetime
of Python 3, and which will also remain binary-compatible across
versions. Extension modules and applications embedding Python
can work with different feature releases as long as they restrict
themselves to this stable ABI.
Rationale
=========
The primary source of ABI incompatibility are changes to the lay-out
of in-memory structures. For example, the way in which string interning
works, or the data type used to represent the size of an object, have
changed during the life of Python 2.x. As a consequence, extension
modules making direct access to fields of strings, lists, or tuples,
would break if their code is loaded into a newer version of the
interpreter without recompilation: offsets of other fields may have
changed, making the extension modules access the wrong data.
In some cases, the incompatibilities only affect internal objects of
the interpreter, such as frame or code objects. For example, the way
line numbers are represented has changed in the 2.x lifetime, as has
the way in which local variables are stored (due to the introduction
of closures). Even though most applications probably never used these
objects, changing them had required to change the PYTHON_API_VERSION.
On Linux, changes to the ABI are often not much of a problem: the
system will provide a default Python installation, and many extension
modules are already provided pre-compiled for that version. If additional
modules are needed, or additional Python versions, users can typically
compile them themselves on the system, resulting in modules that use
the right ABI.
On Windows, multiple simultaneous installations of different Python
versions are common, and extension modules are compiled by their
authors, not by end users. To reduce the risk of ABI incompatibilities,
Python currently introduces a new DLL name pythonXY.dll for each
feature release, whether or not ABI incompatibilities actually exist.
With this PEP, it will be possible to reduce the dependency of binary
extension modules on a specific Python feature release, and applications
embedding Python can be made work with different releases.
Specification
=============
The ABI specification falls into two parts: an API specification,
specifying what function (groups) are available for use with the
ABI, and a linkage specification specifying what libraries to link
with. The actual ABI (layout of structures in memory, function
calling conventions) is not specified, but implied by the
compiler. As a recommendation, a specific ABI is recommended for
selected platforms.
During evolution of Python, new ABI functions will be added.
Applications using them will then have a requirement on a minimum
version of Python; this PEP provides no mechanism for such
applications to fall back when the Python library is too old.
Terminology
-----------
Applications and extension modules that want to use this ABI
are collectively referred to as "applications" from here on.
Header Files and Preprocessor Definitions
-----------------------------------------
Applications shall only include the header file Python.h (before
including any system headers), or, optionally, include pyconfig.h, and
then Python.h.
During the compilation of applications, the preprocessor macro
Py_LIMITED_API must be defined. Doing so will hide all definitions
that are not part of the ABI.
Structures
----------
Only the following structures and structure fields are accessible to
applications:
- PyObject (ob_refcnt, ob_type)
- PyVarObject (ob_base, ob_size)
- Py_buffer (buf, obj, len, itemsize, readonly, ndim, shape,
strides, suboffsets, smalltable, internal)
- PyMethodDef (ml_name, ml_meth, ml_flags, ml_doc)
- PyMemberDef (name, type, offset, flags, doc)
- PyGetSetDef (name, get, set, doc, closure)
The accessor macros to these fields (Py_REFCNT, Py_TYPE, Py_SIZE)
are also available to applications.
The following types are available, but opaque (i.e. incomplete):
- PyThreadState
- PyInterpreterState
Type Objects
------------
The structure of type objects is not available to applications;
declaration of "static" type objects is not possible anymore
(for applications using this ABI).
Instead, type objects get created dynamically. To allow an
easy creation of types (in particular, to be able to fill out
function pointers easily), the following structures and functions
are available::
typedef struct{
int slot; /* slot id, see below */
void *pfunc; /* function pointer */
} PyType_Slot;
struct{
const char* name;
const char* doc;
int basicsize;
int itemsize;
int flags;
PyType_Slot *slots; /* terminated by slot==0. */
} PyType_Spec;
PyObject* PyType_FromSpec(PyType_Spec*);
To specify a slot, a unique slot id must be provided. New Python
versions may introduce new slot ids, but slot ids will never be
recycled. Slots may get deprecated, but continue to be supported
throughout Python 3.x.
The slot ids are named like the field names of the structures that
hold the pointers in Python 3.1, with an added ``Py_`` prefix (i.e.
Py_tp_dealloc instead of just tp_dealloc):
- tp_dealloc, tp_print, tp_getattr, tp_setattr, tp_repr,
tp_hash, tp_call, tp_str, tp_getattro, tp_setattro,
tp_doc, tp_traverse, tp_clear, tp_richcompare, tp_iter,
tp_iternext, tp_methods, tp_base, tp_descr_set, tp_descr_set,
tp_init, tp_alloc, tp_new, tp_is_gc, tp_bases, tp_del
- nb_add nb_subtract nb_multiply nb_remainder nb_divmod nb_power
nb_negative nb_positive nb_absolute nb_bool nb_invert nb_lshift
nb_rshift nb_and nb_xor nb_or nb_int nb_float nb_inplace_add
nb_inplace_subtract nb_inplace_multiply nb_inplace_remainder
nb_inplace_power nb_inplace_lshift nb_inplace_rshift nb_inplace_and
nb_inplace_xor nb_inplace_or nb_floor_divide nb_true_divide
nb_inplace_floor_divide nb_inplace_true_divide nb_index
- sq_length sq_concat sq_repeat sq_item sq_ass_item was_sq_ass_slice
sq_contains sq_inplace_concat sq_inplace_repeat
- mp_length mp_subscript mp_ass_subscript
- bf_getbuffer bf_releasebuffer
XXX Not supported yet: tp_weaklistoffset, tp_dictoffset
The following fields cannot be set during type definition:
- tp_dict tp_mro tp_cache tp_subclasses tp_weaklist
Functions and function-like Macros
----------------------------------
All functions starting with _Py are not available to applications.
Also, all functions that expect parameter types that are unavailable
to applications are excluded from the ABI, such as PyAST_FromNode
(which expects a ``node*``).
All other functions are available, unless excluded below.
Function-like macros (in particular, field access macros) remain
available to applications, but get replaced by function calls
(unless their definition only refers to features of the ABI, such
as the various _Check macros)
ABI function declarations will not change their parameters or return
types. If a change to the signature becomes necessary, a new function
will be introduced. If the new function is source-compatible (e.g. if
just the return type changes), an alias macro may get added to
redirect calls to the new function when the applications is
recompiled.
If continued provision of the old function is not possible, it may get
deprecated, then removed, in accordance with PEP 7, causing
applications that use that function to break.
Excluded Functions
------------------
Functions declared in the following header files are not part
of the ABI:
- cellobject.h
- classobject.h
- code.h
- frameobject.h
- funcobject.h
- genobject.h
- pyarena.h
- pydebug.h
- symtable.h
- token.h
- traceback.h
Global Variables
----------------
Global variables representing types and exceptions are available
to applications.
XXX provide a complete list.
XXX should restrict list of globals to truly "builtin" stuff,
excluding everything that can also be looked up through imports.
XXX may specify access to predefined types and exceptions through
the interpreter state, with appropriate Get macros.
Other Macros
------------
All macros defining symbolic constants are available to applications;
the numeric values will not change.
In addition, the following macros are available:
- Py_BEGIN_ALLOW_THREADS, Py_BLOCK_THREADS, Py_UNBLOCK_THREADS,
Py_END_ALLOW_THREADS
Linkage
-------
On Windows, applications shall link with python3.dll; an import
library python3.lib will be available. This DLL will redirect all of
its API functions through /export linker options to the full
interpreter DLL, i.e. python3y.dll.
XXX is it possible to redirect global variables in the same way?
If not, python3.dll would have to copy them, and we should verify
that all available global variables are read-only.
On Unix systems, the ABI is typically provided by the python
executable itself. PyModule_Create is changed to pass ``3`` as the API
version if the extension module was compiled with Py_LIMITED_API; the
version check for the API version will accept either 3 or the current
PYTHON_API_VERSION as conforming. If Python is compiled as a shared
library, it is installed as both libpython3.so, and libpython3.y.so;
applications conforming to this PEP should then link to the former.
XXX is it possible to make the soname libpython.so.3, and still
have some applications link to libpython3.y.so?
Implementation Strategy
=======================
This PEP will be implemented in a branch, allowing users to check
whether their modules conform to the ABI. To simplify this testing, an
additional macro Py_LIMITED_API_WITH_TYPES will expose the existing
type object layout, to let users postpone rewriting all types. When
the this branch is merged into the 3.2 code base, this macro will
be removed.
Copyright
=========
This document has been placed in the public domain.
Dirkjan Ochtman
> Excluded Functions
> ------------------
>
> Functions declared in the following header files are not part
> of the ABI:
> - cellobject.h
> - classobject.h
> - code.h
> - frameobject.h
> - funcobject.h
> - genobject.h
> - pyarena.h
> - pydebug.h
> - symtable.h
> - token.h
> - traceback.h
What kind of effect does this have on optimization efforts, for
example all the stuff done by Antoine Pitrou over the last few months,
and the first few results from unladen? Will it mean we won't get to
the good optimizations until 4.0? Or does it just mean unladen swallow
takes longer to come back to trunk (until 4.0) and every extension
author who wants to be compatible with it will basically have the same
burden as now?
Cheers,
Dirkjan
"Martin v. Löwis"
>> of the ABI:
>> - cellobject.h
>> - classobject.h
>> - code.h
>> - frameobject.h
>> - funcobject.h
>> - genobject.h
>> - pyarena.h
>> - pydebug.h
>> - symtable.h
>> - token.h
>> - traceback.h
>
> What kind of effect does this have on optimization efforts, for
> example all the stuff done by Antoine Pitrou over the last few months,
> and the first few results from unladen?
I fail to see the relationship, so: no effect that I can see.
Why do you think that optimization efforts could be related to
the PEP 384 proposal?
Regards,
Martin
Dirkjan Ochtman
> I fail to see the relationship, so: no effect that I can see.
>
> Why do you think that optimization efforts could be related to
> the PEP 384 proposal?
It would seem to me that optimizations are likely to require data
structure changes, for exactly the kind of core data structures that
you're talking about locking down. But that's just a high-level view,
I might be wrong.
Cheers,
Dirkjan
"Martin v. Löwis"
Ah. It's exactly the opposite: The purpose of the PEP is not to lock
the data structures down, but to allow more flexible evolution of
them - by completely hiding them from extension modules.
Currently, any data structure change must be weighed for its impact
on binary compatibility. With the PEP, changing structures can
be done fairly freely - with the exception of the very few structures
that do get locked down. In particular, the list of header files
that you quoted precisely contains the structures that can be
modified with no impact on the ABI.
I'm not aware that any of the structures that I propose to lock
would be relevant for optimization - but I might be wrong. If so,
I'd like to know, and it would be possible to add accessor functions
in cases where extension modules might still legitimately want to
access certain fields.
Certain changes to the VM would definitely be binary-incompatible,
such as removal of reference counting. However, such a change would
probably have a much wider effect, breaking not just binary
compatibility, but also source compatibility. It would be justified
to call a Python release that makes such a change 4.0.
Regards,
Martin
"Martin v. Löwis"
> Dirkjan Ochtman wrote:
>> It would seem to me that optimizations are likely to require data
>> structure changes, for exactly the kind of core data structures that
>> you're talking about locking down. But that's just a high-level view,
>> I might be wrong.
>>
>
>
> I would think not directly exposing the field and having inc/dec ref
> Functions (real methods, not macros) for it would give a lot more
> ability to change the API in the future.
In the context of optimization, I'm skeptical that introducing functions
for the reference counting would be useful. Making the INCREF/DECREF
macros functions just in case the reference counting goes away is IMO
an unacceptable performance cost.
Instead, such a change should go through the regular deprecation
procedure and/or cause the release of Python 4.0.
> It also might make it easier for alternate implementations to support
> the same API so some modules could work cross implementation - but I
> suspect that's a non-goal of this PEP :).
Indeed :-) I'm also skeptical that this would actually allow
cross-implementation modules to happen. The list of functions that
an alternate implementation would have to provide is fairly long.
The memory management APIs in particular also assume a certain layout
of Python objects in general, namely that they start with a header
whose size is a compile-time constant. Again, making this more flexible
"just in case" would also impact performance, and probably fairly badly
so.
> Other fields directly accessed (via macros or otherwise) might have similar
> problems but they don't seem as core as ref counting.
Access to the type object reference is probably similar. All the other
structs are used "directly" in C code, with no accessor macros.
Regards,
Martin
Dino Viehland
>
> It would seem to me that optimizations are likely to require data
> structure changes, for exactly the kind of core data structures that
> you're talking about locking down. But that's just a high-level view,
> I might be wrong.
>
In particular I would guess that ref counting is the biggest issue here.
I would think not directly exposing the field and having inc/dec ref
Functions (real methods, not macros) for it would give a lot more
ability to change the API in the future.
the same API so some modules could work cross implementation - but I
suspect that's a non-goal of this PEP :).
Michael Foord
> Dino Viehland wrote:
>
>> Dirkjan Ochtman wrote:
>>
>>> It would seem to me that optimizations are likely to require data
>>> structure changes, for exactly the kind of core data structures that
>>> you're talking about locking down. But that's just a high-level view,
>>> I might be wrong.
>>>
>>>
>> In particular I would guess that ref counting is the biggest issue here.
>> I would think not directly exposing the field and having inc/dec ref
>> Functions (real methods, not macros) for it would give a lot more
>> ability to change the API in the future.
>>
>
> for the reference counting would be useful. Making the INCREF/DECREF
> macros functions just in case the reference counting goes away is IMO
> an unacceptable performance cost.
>
> Instead, such a change should go through the regular deprecation
> procedure and/or cause the release of Python 4.0.
>
>
>> the same API so some modules could work cross implementation - but I
>> suspect that's a non-goal of this PEP :).
>>
>
> cross-implementation modules to happen. The list of functions that
> an alternate implementation would have to provide is fairly long.
>
>
Just in case you're unaware of it; the company I work for has an open
source project called Ironclad. This *is* a reimplementation of the
Python C API and gives us binary compatibility with [some subset of]
Python C extensions for use from IronPython.
http://www.resolversystems.com/documentation/index.php/Ironclad.html
It's an ambitious project but it is now at the stage where 1000s of the
Numpy and Scipy tests pass when run from IronPython. I don't think this
PEP impacts the project, but it is not completely unfeasible for the
alternative implementations to do this.
In particular we have had to address the issue of the GIL and extensions
(IronPython has no GIL) and reference counting (which IronPython also
doesn't) use.
Michael Foord
--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog
James Y Knight
On May 17, 2009, at 4:54 PM, Martin v. Löwis wrote:
> Currently, each feature release introduces a new name for the
> Python DLL on Windows, and may cause incompatibilities for extension
> modules on Unix. This PEP proposes to define a stable set of API
> functions which are guaranteed to be available for the lifetime
> of Python 3, and which will also remain binary-compatible across
> versions. Extension modules and applications embedding Python
> can work with different feature releases as long as they restrict
> themselves to this stable ABI.
It seems like a good ideal to strive for.
But I think this is too strong a promise. IMO it would be better to
say that ABI compatibility across releases is a goal. If someone does
make a change that breaks the ABI, I'd expect whomever is proposing it
to put forth a fairly strong argument towards why it's a worthwhile
change. But it should be possible and allowed, given the right
circumstances. Because I think it's pretty much inevitable that it
*will* need to happen, sometime.
(of course there will need to be ABI tests, so that any potential ABI
breakages are known about when they occur)
Python is much more defined by its source language than its C
extension API, so tying the python major version number to the C ABI
might not be the best idea from a "marketing" standpoint. (I can see
it now..."Python 4.0 major new features: we changed the C method
definition struct layout incompatibly" :)
James
"Martin v. Löwis"
>>> the same API so some modules could work cross implementation - but I
>>> suspect that's a non-goal of this PEP :).
>>>
>>
>> Indeed :-) I'm also skeptical that this would actually allow
>> cross-implementation modules to happen. The list of functions that
>> an alternate implementation would have to provide is fairly long.
>>
>>
>
> Just in case you're unaware of it; the company I work for has an open
> source project called Ironclad.
I was unaware indeed; thanks for pointing this out.
IIUC, it's not just an API emulation, but also an ABI emulation.
> In particular we have had to address the issue of the GIL and extensions
> (IronPython has no GIL) and reference counting (which IronPython also
> doesn't) use.
I think this somewhat strengthens the point I was trying to make: An
alternate implementation that tries to be API compatible has to consider
so many things that it is questionable whether making Py_INCREF/DECREF
functions would be any simplification.
So I just ask:
a) Would it help IronClad if it could restrict itself to PEP 384
compatible modules?
b) Would further restrictions in the PEP help that cause?
Regards,
Martin
Michael Foord
>>>> the same API so some modules could work cross implementation - but I
>>>> suspect that's a non-goal of this PEP :).
>>>>
>>>>
>>> Indeed :-) I'm also skeptical that this would actually allow
>>> cross-implementation modules to happen. The list of functions that
>>> an alternate implementation would have to provide is fairly long.
>>>
>>>
>>>
>> Just in case you're unaware of it; the company I work for has an open
>> source project called Ironclad.
>>
>
> I was unaware indeed; thanks for pointing this out.
>
> IIUC, it's not just an API emulation, but also an ABI emulation.
>
>
Correct.
>> In particular we have had to address the issue of the GIL and extensions
>> (IronPython has no GIL) and reference counting (which IronPython also
>> doesn't) use.
>>
>
> I think this somewhat strengthens the point I was trying to make: An
> alternate implementation that tries to be API compatible has to consider
> so many things that it is questionable whether making Py_INCREF/DECREF
> functions would be any simplification.
>
It would actually have been helpful for us, but I understand that it
would be a big performance hit. The Ironclad garbage collection
mechanism is described here:
http://www.voidspace.org.uk/python/weblog/arch_d7_2009_01_24.shtml#e1055
We artificially inflate the refcount of all objects that Ironclad
creates to 2 and hold a reference to them on the .NET side to make them
ineligible for garbage collection.
Because we can't always know when objects have been decreffed back down
to 1, there are some circumstances when we have to scan all the objects
we are holding onto. If their refcount is only 1 then we no longer need
to hold a reference them. When nothing is using them on the IronPython
side either normal .NET garbage collection kicks in and the IronPython
proxy object has a destructor that calls back into Ironclad and uses the
CPython dealloc method.
> So I just ask:
> a) Would it help IronClad if it could restrict itself to PEP 384
> compatible modules?
> b) Would further restrictions in the PEP help that cause?
>
I've forwarded these questions to the lead developer of Ironclad
(William Reade) along with a link to the PEP. He isn't on Python-dev so
I may have to be a proxy for him in discussion. His initial response was
"looks pretty sweet".
Michael
> Regards,
> Martin
Jeffrey Yasskin
I'm with the people who think the refcount should be accessed through
functions by apps that want ABI compatibility. In particular,
GIL-removal efforts are guaranteed to change how the refcount is
modified, but there's a good chance they wouldn't have to change the
API. (We have some ideas for how to maintain source compatibility in
the absence of a GIL:
http://code.google.com/p/unladen-swallow/wiki/ExtensionModules#Reference_Counting)
Over an 8-year lifetime for Python 3, Moore's law predicts that
desktop systems will have up to 64 cores, at which point even the
simplest GIL-removal strategy of making refcounts atomic will be a
win, despite the 2x performance loss for a single thread. I wouldn't
want an ABI to rule that out.
I do think the refcounting macros should remain present in the API
(not ABI) for apps that only need source compatibility and want the
extra speed.
I wonder if it makes sense to specify an API compatibility mode in
this PEP too.
"Py_LIMITED_API" may not be the right macro name—it didn't imply
anything about an ABI when I first saw it. Might it make sense to use
Py_ABI_COMPATIBILITY=### instead? (Where ### could be an ISO date like
20090520.) That would put "ABI" in the macro name and make it easier
to define new versions later if necessary. (New versions would help
people compile against a new version of Python and be confident they
had something that would run against old versions.) If we never define
a new version, defining it to a number instead of just anything
doesn't really hurt.
It's probably worth pointing out in the PEP that the fact that
PyVarObject.ob_size is part of the ABI means that PyObject cannot
change size, even by adding fields at the end.
Right now, the globals representing types are defined like
"PyAPI_DATA(PyTypeObject) PyList_Type;". To allow the core to use the
new type creation functions, it might be useful to make the ABI type
objects PyTypeObject* constants instead.
In general, this looks really good. Thanks!
Jeffrey
On Sun, May 17, 2009 at 1:54 PM, "Martin v. Löwis" <mar...@v.loewis.de> wrote:
>
> Thomas Wouters reminded me of a long-standing idea; I finally
> found the time to write it down.
>
> Please comment!
>
> Regards,
> Martin
>
> PEP: 384
> Title: Defining a Stable ABI
> Version: $Revision: 72754 $
> Last-Modified: $Date: 2009-05-17 21:14:52 +0200 (So, 17. Mai 2009) $
> Author: Martin v. Löwis <mar...@v.loewis.de>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 17-May-2009
> Python-Version: 3.2
> Post-History:
>
> Abstract
> ========
>
> Currently, each feature release introduces a new name for the
> Python DLL on Windows, and may cause incompatibilities for extension
> modules on Unix. This PEP proposes to define a stable set of API
> functions which are guaranteed to be available for the lifetime
> of Python 3, and which will also remain binary-compatible across
> versions. Extension modules and applications embedding Python
> can work with different feature releases as long as they restrict
> themselves to this stable ABI.
>
> Functions declared in the following header files are not part
> of the ABI:
> - cellobject.h
> - classobject.h
> - code.h
> - frameobject.h
> - funcobject.h
> - genobject.h
> - pyarena.h
> - pydebug.h
> - symtable.h
> - token.h
> - traceback.h
>
M.-A. Lemburg
> Thomas Wouters reminded me of a long-standing idea; I finally
> found the time to write it down.
>
> Please comment!
>
Up until this PEP proposal, we had a very simple scheme for
the Python C-API: all documented functions and variables with
a "Py" prefix were part of the C-API, everything else was not
and could change between releases (in particular the private
"_Py" prefix APIs).
Changing the published APIs was considered a bad thing in the
2.x development process and generally required a good reason
to get supported.
Changing private functions or ones that were not documented
was generally never a big problem.
Now, with the PEP, I have a feeling that the Python C-API
will in effect be limited to what's in the PEP's idea of
a usable ABI and open up the non-inluded public C-APIs
to the same rate of change as the private APIs.
If that's the case, the PEP should be discussed on the C-API
list first, in order to identify a complete list of APIs that
is supposed to define the Python C-API. Ideally, all other
APIs would then need to be made private. However, I doubt that
this is possible before switching to Python 4.0.
Then again, I'm not sure whether that's what you're aiming for...
An optional cross-version ABI would certainly be a good thing.
Limiting the Python C-API would be counterproductive.
> During the compilation of applications, the preprocessor macro
> Py_LIMITED_API must be defined. Doing so will hide all definitions
> that are not part of the ABI.
So extensions wanting to use the full Python C-API as documented
in the C-API docs will still be able to do this, right ?
> Type Objects
> ------------
>
> The structure of type objects is not available to applications;
> declaration of "static" type objects is not possible anymore
> (for applications using this ABI).
Hmm, that's going to create big problems for extensions that
want to expose a C-API for their types: Type checks are normally
done by pointer comparison using those static type objects.
> Functions and function-like Macros
> ----------------------------------
>
> Function-like macros (in particular, field access macros) remain
> available to applications, but get replaced by function calls
> (unless their definition only refers to features of the ABI, such
> as the various _Check macros)
Including Py_INCREF()/Py_DECREF() ?
> Excluded Functions
> ------------------
>
> Functions declared in the following header files are not part
> of the ABI:
> - cellobject.h
> - classobject.h
> - code.h
> - frameobject.h
> - funcobject.h
> - genobject.h
> - pyarena.h
> - pydebug.h
> - symtable.h
> - token.h
> - traceback.h
I don't think that's feasable: you basically remove all introspection
functions that way.
This will need a more fine-grained approach.
> Linkage
> -------
>
> On Windows, applications shall link with python3.dll;
You mean: extensions that were compiled with Py_LIMITED_API, right ?
> an import
> library python3.lib will be available. This DLL will redirect all of
> its API functions through /export linker options to the full
> interpreter DLL, i.e. python3y.dll.
What if you mix extensions that use the full C-API with ones
that restrict themselves to the limited version ?
Would creating a Python object in a full-API extension and
free'ing it in a limited-API extension cause problems ?
> Implementation Strategy
> =======================
>
> This PEP will be implemented in a branch, allowing users to check
> whether their modules conform to the ABI. To simplify this testing, an
> additional macro Py_LIMITED_API_WITH_TYPES will expose the existing
> type object layout, to let users postpone rewriting all types. When
> the this branch is merged into the 3.2 code base, this macro will
> be removed.
Now I'm confused again: this sounds a lot like you do want all extension
writers to only use the limited API.
[And in another post]
>> Something I haven't seen explicitly mentioned as yet (in the PEP or the
>> > python-dev list discussion) are the memory management APIs and the FILE*
>> > APIs which can cause the MSVCRT versioning issues on Windows.
>> >
>> > Those would either need to be excluded from the stable ABI or else
>> > changed to use opaque pointers.
>
> Good point. As a separate issue, I would actually like to deprecate,
> then remove these APIs. I had originally hoped that this would happen
> for 3.0 already, alas, nobody worked on it.
>
> In any case, I have removed them from the ABI now.
How do you expect Python extensions to allocate memory and objects
in a platform independent way without those APIs ?
And as an aside: Which API families are you referring to ? PyMem_Malloc,
PyObject_Malloc, or PyObject_New ?
Thanks,
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, May 25 2009)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
2009-06-29: EuroPython 2009, Birmingham, UK 34 days to go
::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
"Martin v. Löwis"
> will in effect be limited to what's in the PEP's idea of
> a usable ABI and open up the non-inluded public C-APIs
> to the same rate of change as the private APIs.
That's certainly not the plan. Instead, the plan is to have
a stable ABI. The policy on the API isn't affected, except
for restricting changes to the API that would break the ABI.
>> During the compilation of applications, the preprocessor macro
>> Py_LIMITED_API must be defined. Doing so will hide all definitions
>> that are not part of the ABI.
>
> So extensions wanting to use the full Python C-API as documented
> in the C-API docs will still be able to do this, right ?
Correct. They would link to the version-specific DLL on Windows.
>> The structure of type objects is not available to applications;
>> declaration of "static" type objects is not possible anymore
>> (for applications using this ABI).
>
> Hmm, that's going to create big problems for extensions that
> want to expose a C-API for their types: Type checks are normally
> done by pointer comparison using those static type objects.
I don't see the problem. During module initialization, you
create the type object and store it in a global variable, and
then both clients and the module compare against the stored
pointer.
>> Function-like macros (in particular, field access macros) remain
>> available to applications, but get replaced by function calls
>> (unless their definition only refers to features of the ABI, such
>> as the various _Check macros)
>
> Including Py_INCREF()/Py_DECREF() ?
Yes, although some people are requesting that these become functions.
>> Excluded Functions
>> ------------------
>>
>> Functions declared in the following header files are not part
>> of the ABI:
>> - cellobject.h
>> - classobject.h
>> - code.h
>> - frameobject.h
>> - funcobject.h
>> - genobject.h
>> - pyarena.h
>> - pydebug.h
>> - symtable.h
>> - token.h
>> - traceback.h
>
> I don't think that's feasable: you basically remove all introspection
> functions that way.
>
> This will need a more fine-grained approach.
What specifically is it that you want to do in a module that you
couldn't do anymore?
>> On Windows, applications shall link with python3.dll;
>
> You mean: extensions that were compiled with Py_LIMITED_API, right ?
Correct, see "Terminology" in the PEP.
>
>> an import
>> library python3.lib will be available. This DLL will redirect all of
>> its API functions through /export linker options to the full
>> interpreter DLL, i.e. python3y.dll.
>
> What if you mix extensions that use the full C-API with ones
> that restrict themselves to the limited version ?
Some link against python3.dll, others against python32.dll (say).
> Would creating a Python object in a full-API extension and
> free'ing it in a limited-API extension cause problems ?
No problem that I can see.
>> This PEP will be implemented in a branch, allowing users to check
>> whether their modules conform to the ABI. To simplify this testing, an
>> additional macro Py_LIMITED_API_WITH_TYPES will expose the existing
>> type object layout, to let users postpone rewriting all types. When
>> the this branch is merged into the 3.2 code base, this macro will
>> be removed.
>
> Now I'm confused again: this sounds a lot like you do want all extension
> writers to only use the limited API.
I certainly want to support as many modules as reasonable with the PEP.
Whether or not developers then chose to build version-independent
binaries is certainly outside the scope of the PEP - it only specifies
action items for Python, not for application authors.
>>> Something I haven't seen explicitly mentioned as yet (in the PEP or the
>>>> python-dev list discussion) are the memory management APIs and the FILE*
>>>> APIs which can cause the MSVCRT versioning issues on Windows.
>>>>
>>>> Those would either need to be excluded from the stable ABI or else
>>>> changed to use opaque pointers.
>> Good point. As a separate issue, I would actually like to deprecate,
>> then remove these APIs. I had originally hoped that this would happen
>> for 3.0 already, alas, nobody worked on it.
>>
>> In any case, I have removed them from the ABI now.
>
> How do you expect Python extensions to allocate memory and objects
> in a platform independent way without those APIs ?
I have only removed functions from the ABI that have FILE* in their
signatures.
> And as an aside: Which API families are you referring to ? PyMem_Malloc,
> PyObject_Malloc, or PyObject_New ?
Neither. PyRun_AnyFileFlags and friends.
Regards,
Martin
M.-A. Lemburg
>> will in effect be limited to what's in the PEP's idea of
>> a usable ABI and open up the non-inluded public C-APIs
>> to the same rate of change as the private APIs.
>
> That's certainly not the plan. Instead, the plan is to have
> a stable ABI. The policy on the API isn't affected, except
> for restricting changes to the API that would break the ABI.
Thanks for clarifying this.
>>> During the compilation of applications, the preprocessor macro
>>> Py_LIMITED_API must be defined. Doing so will hide all definitions
>>> that are not part of the ABI.
>> So extensions wanting to use the full Python C-API as documented
>> in the C-API docs will still be able to do this, right ?
>
> Correct. They would link to the version-specific DLL on Windows.
Good.
>>> The structure of type objects is not available to applications;
>>> declaration of "static" type objects is not possible anymore
>>> (for applications using this ABI).
>> Hmm, that's going to create big problems for extensions that
>> want to expose a C-API for their types: Type checks are normally
>> done by pointer comparison using those static type objects.
>
> I don't see the problem. During module initialization, you
> create the type object and store it in a global variable, and
> then both clients and the module compare against the stored
> pointer.
Ah, good point !
>>> Function-like macros (in particular, field access macros) remain
>>> available to applications, but get replaced by function calls
>>> (unless their definition only refers to features of the ABI, such
>>> as the various _Check macros)
>> Including Py_INCREF()/Py_DECREF() ?
>
> Yes, although some people are requesting that these become functions.
I'd opt against that, simply because it creates a lot of overhead
due to the function call and issues with cache locality.
>>> Excluded Functions
>>> ------------------
>>>
>>> Functions declared in the following header files are not part
>>> of the ABI:
>>> - cellobject.h
>>> - classobject.h
>>> - code.h
>>> - frameobject.h
>>> - funcobject.h
>>> - genobject.h
>>> - pyarena.h
>>> - pydebug.h
>>> - symtable.h
>>> - token.h
>>> - traceback.h
>> I don't think that's feasable: you basically remove all introspection
>> functions that way.
>>
>> This will need a more fine-grained approach.
>
> What specifically is it that you want to do in a module that you
> couldn't do anymore?
See my reply to Nick: some of the functions are needed even
if you don't want to do introspection, such as Py_FatalError()
or PyTraceBack_Print().
BTW: Given the headline, I take it that the various type checking
macros in these header will still be available, right ?
>>> On Windows, applications shall link with python3.dll;
>> You mean: extensions that were compiled with Py_LIMITED_API, right ?
>
> Correct, see "Terminology" in the PEP.
Good, thanks.
>>> an import
>>> library python3.lib will be available. This DLL will redirect all of
>>> its API functions through /export linker options to the full
>>> interpreter DLL, i.e. python3y.dll.
>> What if you mix extensions that use the full C-API with ones
>> that restrict themselves to the limited version ?
>
> Some link against python3.dll, others against python32.dll (say).
>
>> Would creating a Python object in a full-API extension and
>> free'ing it in a limited-API extension cause problems ?
>
> No problem that I can see.
Can we be sure that the MSVCRT used by python35.dll stays compatible
to the one used by say python32.dll ? What if the CRT memory
management changes between MSVCRT versions ?
Another aspect to consider:
How will this work in the light of having multiple copies of
Python installed on a Windows machine ?
They implementation section suggests that python3.dll would always
redirect to the python3x.dll for which it was installed, ie. if
I have Python 3.5 installed, but then need to run some app with
Python 3.2, the installed python3.dll would then point back to the
python32.dll.
Now, if I start a Python 3.5 application which uses a limited
API extension, this would try to load python32.dll into the
Python 3.5 process. AFAIK, that's not possible due to the
naming conflicts.
>>> This PEP will be implemented in a branch, allowing users to check
>>> whether their modules conform to the ABI. To simplify this testing, an
>>> additional macro Py_LIMITED_API_WITH_TYPES will expose the existing
>>> type object layout, to let users postpone rewriting all types. When
>>> the this branch is merged into the 3.2 code base, this macro will
>>> be removed.
>> Now I'm confused again: this sounds a lot like you do want all extension
>> writers to only use the limited API.
>
> I certainly want to support as many modules as reasonable with the PEP.
> Whether or not developers then chose to build version-independent
> binaries is certainly outside the scope of the PEP - it only specifies
> action items for Python, not for application authors.
Thanks for the clarification.
>>>> Something I haven't seen explicitly mentioned as yet (in the PEP or the
>>>>> python-dev list discussion) are the memory management APIs and the FILE*
>>>>> APIs which can cause the MSVCRT versioning issues on Windows.
>>>>>
>>>>> Those would either need to be excluded from the stable ABI or else
>>>>> changed to use opaque pointers.
>>> Good point. As a separate issue, I would actually like to deprecate,
>>> then remove these APIs. I had originally hoped that this would happen
>>> for 3.0 already, alas, nobody worked on it.
>>>
>>> In any case, I have removed them from the ABI now.
>> How do you expect Python extensions to allocate memory and objects
>> in a platform independent way without those APIs ?
>
> I have only removed functions from the ABI that have FILE* in their
> signatures.
>
>> And as an aside: Which API families are you referring to ? PyMem_Malloc,
>> PyObject_Malloc, or PyObject_New ?
>
> Neither. PyRun_AnyFileFlags and friends.
Good.
Thanks,
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, May 26 2009)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
2009-06-29: EuroPython 2009, Birmingham, UK 33 days to go
"Martin v. Löwis"
>>>> of the ABI:
>>>> - cellobject.h
>>>> - classobject.h
>>>> - code.h
>>>> - frameobject.h
>>>> - funcobject.h
>>>> - genobject.h
>>>> - pyarena.h
>>>> - pydebug.h
>>>> - symtable.h
>>>> - token.h
>>>> - traceback.h
>>> I don't think that's feasable: you basically remove all introspection
>>> functions that way.
>>>
>>> This will need a more fine-grained approach.
>> What specifically is it that you want to do in a module that you
>> couldn't do anymore?
>
> See my reply to Nick: some of the functions are needed even
> if you don't want to do introspection, such as Py_FatalError()
Ok. I don't know what Py_FatalError is doing in pydebug.h, so I
now propose to move it to pyerrors.h.
> or PyTraceBack_Print().
Ok; I have removed traceback.h from the list. By the other rules
of the PEP, the only function that becomes available then is
PyTraceBack_Print.
> BTW: Given the headline, I take it that the various type checking
> macros in these header will still be available, right ?
Which headers? The one on the list above? No; my idea would
be to completely hide them as-is.
All other type-checking macros will remain available, and
will remain being macros.
>>> Would creating a Python object in a full-API extension and
>>> free'ing it in a limited-API extension cause problems ?
>> No problem that I can see.
>
> Can we be sure that the MSVCRT used by python35.dll stays compatible
> to the one used by say python32.dll ? What if the CRT memory
> management changes between MSVCRT versions ?
It doesn't matter. For Python "things", the extension module will
use the pymem.h functions, which get routed through pythonxy.dll
to the CRT that Python was build with.
If the extension uses regular malloc(), it should also invoke
regular free() on the pointer. There is no API where Python
calls malloc directly and the extension calls free, or vice
versa.
> How will this work in the light of having multiple copies of
> Python installed on a Windows machine ?
Interesting question. One solution could be to use SxS, which
would allow multiple concurrent installations of python3.dll,
although we would need to make sure it always binds to the
"right" one in each context.
Another solution could be to keep the various copies of python3.dll
in their respective PYTHONHOMEs, and leave it to python.exe or the
app to load the right one; any subsequent extension modules should
then pick up the one that was already loaded.
> They implementation section suggests that python3.dll would always
> redirect to the python3x.dll for which it was installed, ie. if
> I have Python 3.5 installed, but then need to run some app with
> Python 3.2, the installed python3.dll would then point back to the
> python32.dll.
That depends on where they get installed. If they all go into system32,
only the most recent one would be available, which is probably not
desirable.
> Now, if I start a Python 3.5 application which uses a limited
> API extension, this would try to load python32.dll into the
> Python 3.5 process. AFAIK, that's not possible due to the
> naming conflicts.
I don't see this problem. As long as we manage to install multiple
versions of python3.dll on the system somehow, different processes
could certainly load different such DLLs, and the same extension
module would always use the right one.
Regards,
Martin