Something has been baffling me and is still present in py3.0 -rc2.
When initializing a (non-variable) PyTypeObject in Python 2,
PyObject_HEAD_INIT is used. The docs for Python 3 still show that:
http://docs.python.org/dev/3.0/extending/newtypes.html
However if you try to use it you get all sorts of severe warnings (it
looks like structure members don't line up). Looking through the Python
3.0 source, *all* initialization is done using
PyVarObject_HEAD_INIT(NULL, 0) instead - PyObject_HEAD_INIT is not used
at all. The Python 2.3 source shows the latter form being used almost
exclusively so at some point someone changed a lot of code.
It would seem that common practise, the examples and the documentation
don't all match each other!
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
iEYEARECAAYFAkkdC0MACgkQmOOfHg372QRErgCdGPyIr9eLNaoivePS5AhUFzJx
KEEAnR8oYK27C5ZueWnmtk+qecOh0bpP
=LDdB
-----END PGP SIGNATURE-----
_______________________________________________
Python-3000 mailing list
Pytho...@python.org
http://mail.python.org/mailman/listinfo/python-3000
Unsubscribe: http://mail.python.org/mailman/options/python-3000/python-3000-garchive-63646%40googlegroups.com
That's correct. I changed it for Python 3, for PEP 3123.
> It would seem that common practise, the examples and the documentation
> don't all match each other!
I cannot parse this sentence.
Regards,
Martin
Martin v. Löwis wrote:
>> It would seem that common practise, the examples and the documentation
>> don't all match each other!
>
> I cannot parse this sentence.
Py2 source: Uses PyObject_HEAD_INIT
Py2 code compiled under Py3: Gives serious warnings
Py3 examples: Say to use PyObject_HEAD_INIT
Py3 reference: Says to use PyObject_HEAD_INIT
Py3 source: Uses PyVarObject_HEAD_INIT
PEP 3123: Says PyVarObject_HEAD_INIT only
Obviously the Python 3 documentation and examples need to be updated.
Also why not remove PyObject_HEAD_INIT from Python 3 headers so that if
it is used then the compile fails?
The Python 3 examples show using PyObject_HEAD_INIT:
http://docs.python.org/dev/3.0/extending/newtypes.html
The Python 3 documentation says to use PyObject_HEAD_INIT:
http://docs.python.org/dev/3.0/search.html?q=PyObject_HEAD_INIT&check_keywords=yes&area=default
There are no matches for PyVarObject_HEAD_INIT:
http://docs.python.org/dev/3.0/search.html?q=PyVarObject_HEAD_INIT
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
iEYEARECAAYFAkkdUUUACgkQmOOfHg372QSAGwCgvx4N8GAe1ciUgedgu/3QB920
PqkAoMGQ0veEJ/FrkUiWxBJ/ZPw5ugjb
=+hZI
-----END PGP SIGNATURE-----
I see - please submit a bug report.
> Also why not remove PyObject_HEAD_INIT from Python 3 headers so that if
> it is used then the compile fails?
It's still needed for non-var objects.
Regards,
Martin
Martin v. Löwis wrote:
>> Also why not remove PyObject_HEAD_INIT from Python 3 headers so that if
>> it is used then the compile fails?
>
> It's still needed for non-var objects.
Wouldn't a var object have PyVarObject_HEAD and a non-var object have
PyObject_HEAD?
Coming from Python 2, I would expect the example as currently documented
to be perfectly correctly:
http://docs.python.org/dev/3.0/extending/newtypes.html
It isn't a var object and doesn't use the var form of the macros.
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
iEYEARECAAYFAkkdwcYACgkQmOOfHg372QSbPwCcC1V+v7x0clfJmRUaPiP+IvB0
msoAn1r5smhNdmXxENaYL5Y40h/QNHPB
=lqEM
-----END PGP SIGNATURE-----
Martin v. Löwis wrote:
>> Obviously the Python 3 documentation and examples need to be updated.
>
> I see - please submit a bug report.
Yet another site that wants another login to report bugs. So someone
else can report it.
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
iEYEARECAAYFAkkdwkMACgkQmOOfHg372QQY3ACg5Tb1nft1fheQLd1ORKghoutD
joUAn3Sf5daabNtvmtm5LJZqVNMammjd
=hpGA
-----END PGP SIGNATURE-----
That's in the types. In the objects, you need the *_INIT macros.
Regards,
Martni
My confusion was because I though that the HEAD for the data structure
had to use the same corresponding HEAD_INIT in the type. So for
whatever reason the PyTypeObject is declared as a var object which is
why the var HEAD_INIT is needed.
It still looks like PyObject_HEAD_INIT should be removed so that people
using earlier versions of Python, following the Py3 docs (before they
are fixed), using older tutorials etc don't get burnt.
Grepping through the py3 source shows only PyModuleDef_HEAD_INIT using
PyObject_HEAD_INIT. There are no other legitimate uses!
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
iEYEARECAAYFAkkd6osACgkQmOOfHg372QSXDgCbBg/zzqmAU4GaJL2qo4aNHocq
c+oAn3IqgPGCvQN8jVMjttA8h+5+MO4g
=f7os
-----END PGP SIGNATURE-----
> Obviously the Python 3 documentation and examples need to be updated.
> Also why not remove PyObject_HEAD_INIT from Python 3 headers so that if
> it is used then the compile fails?
Is there some reason not to define PyObject_HEAD_INIT so
that it expands into the appropriate PyVarObject_HEAD_INIT
call?
--
Greg
Except maybe a few thousand extensions already using it which are
waiting to be ported to Python 3.
Whether you write:
{PyObject_HEAD_INIT(0), 0, ...
or
{PyVarObject_HEAD_INIT(0, 0), ...
for your type definition doesn't really make much difference. They
both unwrap to the same code. Since PyTypeObjects are variable length
objects, you always need the ob_size entry.
However, the macros exist to be used for both variable size
and fixed size objects, so having both available is useful and
legitimate.
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Nov 20 2008)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
2008-11-12: Released mxODBC.Connect 0.9.3 http://python.egenix.com/
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
M.-A. Lemburg wrote:
> Whether you write:
>
> {PyObject_HEAD_INIT(0), 0, ...
>
> or
>
> {PyVarObject_HEAD_INIT(0, 0), ...
>
> for your type definition doesn't really make much difference.
Actually in Py 3 it does. If you use the former (which is how Py 2 does
it) then you get serious compiler warnings due to misaligned fields in
Py 3 and presumably even worse if run the code. See PEP 3123 as to why
things changed. That is why all the code in Python 3 was changed from
using the former to the latter.
> However, the macros exist to be used for both variable size
> and fixed size objects, so having both available is useful and
> legitimate.
...
> Except maybe a few thousand extensions already using it which are
> waiting to be ported to Python 3.
Can you point to any? All the ones I found (via Google) only use
PyObject_HEAD_INIT for PyTypeObjects and every single one of those will
have to change to using PyVarObject_HEAD_INIT.
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
iEYEARECAAYFAkklu7UACgkQmOOfHg372QTZBQCgt3kwtUYF3Us8hjPAS2PvDtpm
l+EAoJUata+K55mboNB0UpMsLlzoRpnA
=WbrQ
-----END PGP SIGNATURE-----
You might get warnings (esp. from GCC), but I have yet to see a compiler
that doesn't map the above to the same memory.
After all, Python 2 has been using this layout for years without any
compiler warnings or segfaults because of this.
> See PEP 3123 as to why
> things changed. That is why all the code in Python 3 was changed from
> using the former to the latter.
Right. Things are now more standard compliant and you get fewer
warnings.
>> However, the macros exist to be used for both variable size
>> and fixed size objects, so having both available is useful and
>> legitimate.
> ...
>> Except maybe a few thousand extensions already using it which are
>> waiting to be ported to Python 3.
>
> Can you point to any? All the ones I found (via Google) only use
> PyObject_HEAD_INIT for PyTypeObjects and every single one of those will
> have to change to using PyVarObject_HEAD_INIT.
True, because PyTypeObjects *are* in fact PyVarObjects and not
PyObjects, so they should have used PyVarObject_HEAD_INIT all
along.
It's only that compilers never really cared or always
did the right thing - depending on how you see it :-)
BTW: With the "few thousand extensions" I was referring to the current
use of the PyObject_HEAD_INIT() macro which you wanted to remove, not
to a few thousand extensions using it correctly. Note that it's rather
uncommon to create singletons like the type objects in C.
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Nov 21 2008)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
2008-11-12: Released mxODBC.Connect 0.9.3 http://python.egenix.com/
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
The definition of PyObject_HEAD_INIT and PyVarObject_HEAD_INIT
changed. We've gone from a series of common fields to a single struct
containing the fields. With the series of fields using
PyObject_HEAD_INIT followed by a size was perfectly correct, but with
a struct it's gibberish.
--
Adam Olsen, aka Rhamphoryncus
Sigh...
Python 2.5:
-----------
(gdb) print PyUnicode_Type
$2 = {ob_refcnt = 1, ob_type = 0x6404c0, ob_size = 0, tp_name = 0x505972 "unicode",
tp_basicsize = 48, tp_itemsize = 0, tp_dealloc = 0x470500 <unicode_dealloc>,
tp_print = 0,
tp_getattr = 0, tp_setattr = 0, tp_compare = 0, tp_repr = 0x4746c0 <unicode_repr>,
tp_as_number = 0x644f60, tp_as_sequence = 0x644f00, tp_as_mapping = 0x644ee0,
...
(gdb) print &PyUnicode_Type.tp_name
$3 = (const char **) 0x642758
(gdb) print &PyUnicode_Type.ob_refcnt
$4 = (Py_ssize_t *) 0x642740
Python 3.0:
-----------
(gdb) print PyUnicode_Type
$1 = {ob_base = {ob_base = {ob_refcnt = 1, ob_type = 0x733940}, ob_size = 0},
tp_name = 0x4f52e9 "str", tp_basicsize = 56, tp_itemsize = 0,
tp_dealloc = 0x42af80 <unicode_dealloc>, tp_print = 0, tp_getattr = 0,
tp_setattr = 0,
tp_compare = 0, tp_repr = 0x431ca0 <unicode_repr>, tp_as_number = 0x736420,
...
(gdb) print &PyUnicode_Type.tp_name
$3 = (const char **) 0x735bf8
(gdb) print &PyUnicode_Type.ob_base
$4 = (PyVarObject *) 0x735be0
In both cases, the fields are 24 bytes apart (on my 64-bit machine).
Yes, it's a different way of writing and accessing the resp. fields.
No, it's not a different memory layout.
Yes, this is binary compatible.
No, this is not going to help you, since the rest of Python 3 is not ;-)
Fortunately, you only rarely have to access the fields in question
directly in extensions. And Python 3.0 and 2.6 also add a few macros
for abstracting this:
#define Py_REFCNT(ob) ...
#define Py_TYPE(ob) ...
#define Py_SIZE(ob) ...
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Nov 21 2008)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
2008-11-12: Released mxODBC.Connect 0.9.3 http://python.egenix.com/
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
You're comparing already compiled code. The issue is with
recompiling, ie source compatibility.
In 2.5 the macros expanded to look like this:
PyTypeObject PyUnicode_Type = {
1, /* ob_refcnt */
&PyType_Type, /* ob_type */
0, /* ob_size */
"unicode", /* tp_name */
sizeof(PyUnicodeObject), /* tp_basicsize */
Try the same macro in 3.0 and it'll look like this:
PyTypeObject PyUnicode_Type = {
{
{
1, /* ob_refcnt */
&PyType_Type, /* ob_type */
},
/* Trailing ob_size gets implicitly initialized to 0 */
},
0, /* ob_size? Nope, tp_name! */
"unicode", /* tp_name? Nope, tp_basicsize! */
sizeof(PyUnicodeObject), /* tp_basicsize? Nope, tp_itemsize! */
The compiler knows what layout a PyTypeObject should have, but the
initializer doesn't match up.
--
Adam Olsen, aka Rhamphoryncus
M.-A. Lemburg wrote:
> You might get warnings (esp. from GCC), but I have yet to see a compiler
> that doesn't map the above to the same memory.
They don't map the same as Adam showed. Your fields end up off by one
which is why the compiler is giving warnings.
> After all, Python 2 has been using this layout for years without any
> compiler warnings or segfaults because of this.
Instead of speculating, please write some Python 3 code to initialize a
PyTypeObject using the Python 2 way and look at what the compiler is
showing.
> Right. Things are now more standard compliant and you get fewer
> warnings.
The underlying structures also changed.
> BTW: With the "few thousand extensions" I was referring to the current
> use of the PyObject_HEAD_INIT() macro which you wanted to remove, not
> to a few thousand extensions using it correctly. Note that it's rather
> uncommon to create singletons like the type objects in C.
That means you are agreeing with what I said in the first place! You
*cannot* use PyObject_HEAD_INIT to initialize a PyTypeObject in Python
3. The fields end up misaligned. You *have to* change to using
PyVarObject_HEAD_INIT. As you point out Py{Var,}Object_HEAD_INIT is for
singleton static objects and other than type objects is very uncommon in
C. I couldn't find any.
The reason I am suggesting removing PyObject_HEAD_INIT from Python 3 is
because there are no demonstrated uses of it. Python 2 code being
ported will be using it, but will be using it incorrectly to initialize
PyTypeObjects. The compiler will issue warnings, but not errors. The
code will then run and probably crash in various interesting ways.
Every single extension author porting to Python 3 is going to encounter
this. If PyObject_HEAD_INIT is removed then their code won't compile
and they will have to work out what is going on.
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
iEYEARECAAYFAkknBMMACgkQmOOfHg372QR8xgCghQs2f1+O0p7anwRPRlGyM+fV
7lAAmgIj2FYcqvQ9XTAdQ1C+38/CD0g1
=VFDH
-----END PGP SIGNATURE-----
Well, yes, of course. That's the whole purpose of PEP 3123, isn't it ?
Starting with Python 3, you have to use PyVarObject_HEAD_INIT()
on PyVarObjects and PyObject_HEAD_INIT() on PyObjects. I don't
see the problem. It's just another change to remember when porting
to Python 3.
The whole type slot interface has changed significantly between
Python 2 and 3, so this minor clarification is really harmless compared
to all the other changes:
* PyNumberMethods have changed due to removal of the division,
oct, hex and coercion slots
* PySequenceMethods have changed, but maintained binary compatibility
(why ?) by replacing the removed slice functions with dummy pointers
* PyBufferProcs is a completely new design
* A lot type flags were removed.
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Nov 21 2008)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
2008-11-12: Released mxODBC.Connect 0.9.3 http://python.egenix.com/
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
M.-A. Lemburg wrote:
> Starting with Python 3, you have to use PyVarObject_HEAD_INIT()
> on PyVarObjects and PyObject_HEAD_INIT() on PyObjects. I don't
> see the problem. It's just another change to remember when porting
> to Python 3.
The problem is that unless you are clairvoyant you have no way of
knowing about this change. Even in rc3 the documentation shows the old
(wrong) way:
http://docs.python.org/dev/3.0/extending/newtypes.html
PyObject_HEAD_INIT is documented:
http://docs.python.org/dev/3.0/search.html?q=PyObject_HEAD_INIT
PyVarObject_HEAD_INIT is not:
http://docs.python.org/dev/3.0/search.html?q=PyVarObject_HEAD_INIT
So anyone porting from Python 2 to Python 3 is just going to compile
their code. There will be some warnings but if they consult the docs
the code will still look correct. They will just assume it is a Python
quirk. Then the code will run and crash and they will have to examine
the Python 3 source to work out what the underlying issue is and how to
fix it.
If PyObject_HEAD_INIT were removed/renamed then the code wouldn't even
compile and so they would realize they have to fix it.
> * PyNumberMethods have changed due to removal of the division,
...
> * PySequenceMethods have changed, but maintained binary compatibility
> (why ?) by replacing the removed slice functions with dummy pointers
...
> * PyBufferProcs is a completely new design
PyTypeObjects are used in every extension. The above changes affect
fewer extensions and the compiler error/warnings will be more
meaningful. I would have expected minor changes like these.
> * A lot type flags were removed.
That is fine. The Python 2 code would fail to compile so you would at
least know what to look for and about.
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
iEUEARECAAYFAkknEGsACgkQmOOfHg372QR6RwCeOJ6Sj2hYWPVwpHnwc9yOvG2H
2YkAmOb6VXWaqHQL+Xd7ihq9gEWLHiA=
=EwXY
-----END PGP SIGNATURE-----
> The problem is that unless you are clairvoyant you have no way of
> knowing about this change. Even in rc3 the documentation shows the old
> (wrong) way:
>
> http://docs.python.org/dev/3.0/extending/newtypes.html
>
> PyObject_HEAD_INIT is documented:
>
> http://docs.python.org/dev/3.0/search.html?q=PyObject_HEAD_INIT
>
> PyVarObject_HEAD_INIT is not:
>
> http://docs.python.org/dev/3.0/search.html?q=PyVarObject_HEAD_INIT
Whatever happens to PyObject..., that should be fixed. Is there are
tracker item yet? A doc person will copy, paste, and format if given
raw text.
> So anyone porting from Python 2 to Python 3 is just going to compile
> their code.
A What's New in Py 3 API is needed that people could read first instead
of digging through the source after.
> There will be some warnings but if they consult the docs
> the code will still look correct. They will just assume it is a Python
> quirk. Then the code will run and crash and they will have to examine
> the Python 3 source to work out what the underlying issue is and how to
> fix it.
>
> If PyObject_HEAD_INIT were removed/renamed then the code wouldn't even
> compile and so they would realize they have to fix it.
>
>> * PyNumberMethods have changed due to removal of the division,
> ...
>> * PySequenceMethods have changed, but maintained binary compatibility
>> (why ?) by replacing the removed slice functions with dummy pointers
> ...
>> * PyBufferProcs is a completely new design
>
> PyTypeObjects are used in every extension. The above changes affect
> fewer extensions and the compiler error/warnings will be more
> meaningful. I would have expected minor changes like these.
>
>> * A lot type flags were removed.
This seems like a start.
I just created a release blocker pointing at this thread.
http://bugs.python.org/issue4385
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------