Porting Django to Python 3.0
Martin v. Löwis
process, I had to make a number of changes to Python, so this port
currently requires the Python 3.0 subversion head (soon to be released
as 3.0a4).
This port is just a start; it runs the tutorial with the sqlite3
backend, including the admin interface, but probably chokes on larger
applications that use more features.
The patch is designed to not break Django on any earlier Python
versions.
I put up the patch and my notes at
http://wiki.python.org/moin/PortingDjangoTo3k
I would like to know how I should proceed with that; options are
a) split the patch into separate chunks, and contribute them
separately.
b) hand the entire patch over to somebody interested in completing it
c) complete it, then submit it
d) abandon it
I would favor option a), even though this means that the django source
repository would only get a part of a working 3.0 port; I might
continue to work on it over the next months, although I would again
hope that regular Django users take over (which I am not).
Malcolm Tredinnick
On Sun, 2008-03-23 at 16:49 -0700, Martin v. Löwis wrote:
> At the PyCon sprint, I started porting Django to Python 3.0. In the
> process, I had to make a number of changes to Python, so this port
> currently requires the Python 3.0 subversion head (soon to be released
> as 3.0a4).
>
> This port is just a start; it runs the tutorial with the sqlite3
> backend, including the admin interface, but probably chokes on larger
> applications that use more features.
>
> The patch is designed to not break Django on any earlier Python
> versions.
>
> I put up the patch and my notes at
>
> http://wiki.python.org/moin/PortingDjangoTo3k
This is interesting. I read over this during lunch today and it's quite
nice. Thanks.
Some of the things you've identified as "Django should..." are a little
problematic since one of the current requirements (at least for 1.0) is
"Django should work out of the box with Python 2.3". Some of your
proposed changes don't even work with Python 2.5, so we need to add a
few more conditionals.
The only potential showstoppers I see are the need to change "except
Exception, e:" to "except Exception as e", since that isn't supported in
Python 2.5 or earlier and it's going to be hard to fake as it's a major
syntactic construction. Also, relative imports aren't supported in 2.4
or earlier whereas they seem to be required to be explicitly specified
in 2.6.
The latter problem can be fixed a bit by reshuffling some of the import
paths, as we've been intending to do in some cases. We use relative
imports in a few places to avoid importing a higher-level __init__ file
because it does so much (too much) work. That's a change that can be
made so everything can do "from django.foo.bar.baz..." and avoid
unnecessary "from ." ugliness. Painful and possibly quite disruptive in
places, but possible and probably worth doing.
We already have a lot of conditional code based on sys.version_info.
We'll need to introduce some more. That explains why, e.g., we're using
md5 in some places and are testing based on behaviour (try to import
hashlib and if that fails, use the fallback). Changing to be a
sys.version_info test is not impossible eventually, although it relies
on IronPython and Jython being comparable in their version numbering and
functionality.
Similarly, we can't unconditionally switch to "from io import ...",
since that doesn't exist in Python 2.5, let alone earlier versions.
Wrapping it in import guards will work, but the code is starting to get
a bit ugly to read. Ditto, the reason we use upper-case names when
import from email is because that's required by 2.3. They are backwards
compatible in later versions, so we intentionally stuck with them.
None of this is your fault, Martin, I realise that. I'm just commenting
from the notes I took when I read through the patch earlier, more as a
record of things we need to think about.
Almost all the things you've listed under the new-style classes section
are fixed in the queryset-refactor branch. I just checked and I've
forgotten to commit one local branch that removes the last ClassType
check. We need to retain one use of ClassType to generate the right
class type for our own exception subclasses -- they're old-style classes
there -- in Python 2.3 and 2.4, but that's guarded by a version_info
conditional, so hopefully 2to3.py won't be concerned about that. The one
in ModelBase.contribute_to_class can go away, though. That's the one I
haven't committed to the branch yet. Anyway, most of those issues will
become non-issues once the branch is in trunk.
The AdminLogNode sending strings as slices bit me in queryset-refactor,
too. I was a bit annoyed that Python allowed it because then I had to
work around it. I'm strongly tempted to fix that (i.e. fix AdminLogNode
to not be so slack) at a some point, but it's low-priority at the
moment, since it breaks indeterminate amounts of code, so I haven't had
the energy to work out the impact and/or mitigation.
It's kind of cool, though: in Python 2.x, you can write a class that
accepts fruit['apple':'banana'] and have it do something. Useful for
symbolic enumerated types, I guess.
> I would like to know how I should proceed with that; options are
>
> a) split the patch into separate chunks, and contribute them
> separately.
> b) hand the entire patch over to somebody interested in completing it
> c) complete it, then submit it
> d) abandon it
>
> I would favor option a), even though this means that the django source
> repository would only get a part of a working 3.0 port; I might
> continue to work on it over the next months, although I would again
> hope that regular Django users take over (which I am not).
I'm not going to make a call here; I'll leave that up to Jacob and
Adrian, but I would hope our plan isn't to leave Python 2.5 (and maybe
even 2.4) behind too quickly. It's a bit too easy to be blinkered in
Open Source development into thinking others should/must upgrade entire
system components (their entire Python install) as fast as we do just
because they want to use one package. Our strategy on that front guides
our porting strategy for 2.6/3.0 to a large extent. There's also the
question of how mainline to make the 3.0 support given it's only in
alpha at the moment. So maybe this is work for a branch. *shrug*
Anyway, that's just one man's thoughts from reading through the wiki
page and the patch. It's interesting to see the level of changes
required. Nothing too surprising, although a few of the necessary
shuffles strike me as annoying since they imply Python 2.6 is breaking
compatibility in potentially unnecessary ways.
Nice work, though.
Malcolm
--
What if there were no hypothetical questions?
http://www.pointy-stick.com/blog/
Martin v. Löwis
> problematic since one of the current requirements (at least for 1.0) is
> "Django should work out of the box with Python 2.3". Some of your
> proposed changes don't even work with Python 2.5, so we need to add a
> few more conditionals.
I think you misunderstood. In these cases, I already changed Django to
do what it should do in the patch. E.g. with the patch, it already
uses io.BytesIO
in 3k, but continues to use cStringIO.StringIO in earlier versions.
> The only potential showstoppers I see are the need to change "except
> Exception, e:" to "except Exception as e", since that isn't supported in
> Python 2.5 or earlier and it's going to be hard to fake as it's a major
> syntactic construction.
Yes; these are the remaining two cases left. There are *many* more cases
that need to be changed to use "except ... as ..."; these are all fixed by 2to3.
There is a bug in 2to3 which fails to update the try-except block if there is
a bare except. Once that's fixed in 2to3, these chunks can be removed from
the patch; this is bug http://bugs.python.org/issue2453
> Also, relative imports aren't supported in 2.4
> or earlier whereas they seem to be required to be explicitly specified
> in 2.6.
In 2.6, you can still use the implicit relative imports. In 3.0, you need
to make explicit relative imports. Again, this is fixed by 2to3. The
remaining imports were left in because the 2to3 fixer again had a
bug (http://bugs.python.org/issue2446); this should be fixed now.
> The latter problem can be fixed a bit by reshuffling some of the import
> paths, as we've been intending to do in some cases. We use relative
> imports in a few places to avoid importing a higher-level __init__ file
> because it does so much (too much) work.
Relative imports are fine. David Wolever wrote a 2to3 fixer in the PyCon
sprint just because I requested one to simplify porting Django.
> That's a change that can be
> made so everything can do "from django.foo.bar.baz..." and avoid
> unnecessary "from ." ugliness. Painful and possibly quite disruptive in
> places, but possible and probably worth doing.
However, unnecessary to support 3.0; see above.
> We already have a lot of conditional code based on sys.version_info.
> We'll need to introduce some more. That explains why, e.g., we're using
> md5 in some places and are testing based on behaviour (try to import
> hashlib and if that fails, use the fallback). Changing to be a
> sys.version_info test is not impossible eventually, although it relies
> on IronPython and Jython being comparable in their version numbering and
> functionality.
Right. I think both Jython and IronPython try to follow the CPython feature
sets by version number. In particular wrt. bytes-vs-string, it is actually
simpler for them to implement 3.x instead of 2.x, as their native string
type is already a Unicode type. Still, they would have to change APIs
when dividing strings and bytes, so they'll likely use the major version 3
to indicate presence of the separate bytes type.
> Similarly, we can't unconditionally switch to "from io import ...",
> since that doesn't exist in Python 2.5, let alone earlier versions.
But my patch doesn't do that!
> Wrapping it in import guards will work, but the code is starting to get
> a bit ugly to read.
If this happens in too many places, it can be factored out into a single
module, like the django.utils.py3 module that I propose.
> Ditto, the reason we use upper-case names when
> import from email is because that's required by 2.3. They are backwards
> compatible in later versions, so we intentionally stuck with them.
Which is fine, no need to change that. Again, if it gets too ugly, you
can factor it out into a single place (although I think each email class
is imported only once).
> We need to retain one use of ClassType to generate the right
> class type for our own exception subclasses -- they're old-style classes
> there -- in Python 2.3 and 2.4, but that's guarded by a version_info
> conditional, so hopefully 2to3.py won't be concerned about that. The one
> in ModelBase.contribute_to_class can go away, though. That's the one I
> haven't committed to the branch yet. Anyway, most of those issues will
> become non-issues once the branch is in trunk.
Ok. Out of curiosity: What was the rationale for clearing Meta.__dict__ when
iterating over it?
> The AdminLogNode sending strings as slices bit me in queryset-refactor,
> too. I was a bit annoyed that Python allowed it because then I had to
> work around it. I'm strongly tempted to fix that (i.e. fix AdminLogNode
> to not be so slack) at a some point, but it's low-priority at the
> moment, since it breaks indeterminate amounts of code, so I haven't had
> the energy to work out the impact and/or mitigation.
>
> It's kind of cool, though: in Python 2.x, you can write a class that
> accepts fruit['apple':'banana'] and have it do something. Useful for
> symbolic enumerated types, I guess.
Sure - the slice object is designed to hold arbitrary objects (and continues
to in 3.x); however, isn't this broken already in 2.x, and just not causing
an exception? IIUC, when limit is, say, '10', render will do
(k.start is None or k.start >= 0) and (k.stop is None or k.stop >= 0)
which happens to give True, but is actually incorrect. The culprit seems
to be connection.ops.limit_offset_sql, which accepts '10' as a limit;
_QuerySet seems to pass limit through as-is (except for the bogus
check above).
> I'm not going to make a call here; I'll leave that up to Jacob and
> Adrian, but I would hope our plan isn't to leave Python 2.5 (and maybe
> even 2.4) behind too quickly.
Again, with the patch, that would not at all be necessary. Django could
continue to support any 2.x versions it pleases to.
> Our strategy on that front guides
> our porting strategy for 2.6/3.0 to a large extent.
I would hope it only guides the tactics (i.e. *how* this is supported),
not the strategy (i.e. *whether* it is supported).
> There's also the
> question of how mainline to make the 3.0 support given it's only in
> alpha at the moment. So maybe this is work for a branch. *shrug*
Whatever is appropriate - if you normally do such stuff in branches
(which I don't know), that's fine. I agree that nobody would actually
use Django on 3.0 just yet - also, as the patch is incomplete, you
probably *couldn't* really use it without fixing more problems.
> Anyway, that's just one man's thoughts from reading through the wiki
> page and the patch. It's interesting to see the level of changes
> required. Nothing too surprising, although a few of the necessary
> shuffles strike me as annoying since they imply Python 2.6 is breaking
> compatibility in potentially unnecessary ways.
2.6 isn't breaking anything (in theory, at least). 3.0 is.
> Nice work, though.
Thanks!
Martin
Malcolm Tredinnick
Just responding to the pertinent bits. I stand corrected on all the
rest.
On Mon, 2008-03-24 at 11:10 +0100, Martin v. Löwis wrote:
> > Some of the things you've identified as "Django should..." are a little
> > problematic since one of the current requirements (at least for 1.0) is
> > "Django should work out of the box with Python 2.3". Some of your
> > proposed changes don't even work with Python 2.5, so we need to add a
> > few more conditionals.
>
> I think you misunderstood. In these cases, I already changed Django to
> do what it should do in the patch. E.g. with the patch, it already
> uses io.BytesIO
> in 3k, but continues to use cStringIO.StringIO in earlier versions.
My apologies. I misread the patch. You're correct.
[...]
> Ok. Out of curiosity: What was the rationale for clearing Meta.__dict__ when
> iterating over it?
That's code from the pre-open-source days (so before my involvement),
but I presume because it presented an easy way to check that all the
parameters were used by the end of processing. It happened to work for
whomever wrote that code. I've had to change it in the queryset-refactor
branch because Meta can be subclassed, so I rapidly hit all sorts of
problems there.
This is one of many things in Django that are there because "it was fit
for purpose when originally written years ago and has worked up until
now." Incremental improvements and all that. :-)
> > The AdminLogNode sending strings as slices bit me in queryset-refactor,
> > too. I was a bit annoyed that Python allowed it because then I had to
> > work around it. I'm strongly tempted to fix that (i.e. fix AdminLogNode
> > to not be so slack) at a some point, but it's low-priority at the
> > moment, since it breaks indeterminate amounts of code, so I haven't had
> > the energy to work out the impact and/or mitigation.
> >
> > It's kind of cool, though: in Python 2.x, you can write a class that
> > accepts fruit['apple':'banana'] and have it do something. Useful for
> > symbolic enumerated types, I guess.
>
> Sure - the slice object is designed to hold arbitrary objects (and continues
> to in 3.x); however, isn't this broken already in 2.x, and just not causing
> an exception? IIUC, when limit is, say, '10', render will do
>
> (k.start is None or k.start >= 0) and (k.stop is None or k.stop >= 0)
>
> which happens to give True, but is actually incorrect. The culprit seems
> to be connection.ops.limit_offset_sql, which accepts '10' as a limit;
> _QuerySet seems to pass limit through as-is (except for the bogus
> check above).
Indeed. It's fixed in the queryset-refactor branch (which is a big
rewrite of query.py's functionality), where we convert the arguments to
integers before trying to use them as things that should have integer
properties.
> > I'm not going to make a call here; I'll leave that up to Jacob and
> > Adrian, but I would hope our plan isn't to leave Python 2.5 (and maybe
> > even 2.4) behind too quickly.
>
> Again, with the patch, that would not at all be necessary. Django could
> continue to support any 2.x versions it pleases to.
I think I misunderstood some of the reasons you were patching things
(e.g. working around the bare except issue and things like that). Thanks
for straightening me out.
Cheers,
Malcolm
--
Experience is something you don't get until just after you need it.
http://www.pointy-stick.com/blog/
Graham Dumpleton
> At the PyCon sprint, I started porting Django to Python 3.0. In the
> process, I had to make a number of changes to Python, so this port
> currently requires the Python 3.0 subversion head (soon to be released
> as 3.0a4).
3.0 as well. When I tried Python 3.0a2, it would crash deep inside of
Python. When I tried 3.0a3 recently however it got further and
absolute minimum hello world program actually works.
Thus now seems to be just a case of working out changes to mod_wsgi in
line with what was worked out on Python WEB-SIG as necessary changes
to WSGI specification to make WSGI compatible with Python 3.0.
If I can get mod_wsgi working on Python 3.0, will be interesting to
see if you can tweak the Django WSGI interface, if not done already,
to make it WSGI for Python 3.0 compatible and thus try your changes on
top of mod_wsgi.
Graham
pbx
> At the PyCon sprint, I started porting Django to Python 3.0. In the
> process, I had to make a number of changes to Python, so this port
> currently requires the Python 3.0 subversion head (soon to be released
> as 3.0a4).
> separately.
> b) hand the entire patch over to somebody interested in completing it
> c) complete it, then submit it
> d) abandon it
Maybe it would be best not to apply it as separate patches on the
current trunk, but to make it a branch, at least until it can be
clearly established that the single-version-plus-2to3 approach will
actually work for all of Django.
I wasn't at PyCon, and haven't done any 3.0 porting work myself, so I
could be behind the times, but my understanding of current porting
advice (based on PEP 3000) was that it's not going to be possible to
support 2.x and 3.x from a single codebase in many cases (even with
2to3) if Python < 2.6 needs to be supported. Right?
I also wonder, is it worth adding this stuff to Django *now* for a
transition that is likely to be quite far off, with at least two big
merges (qs-rf, nfa, possibly gis) coming up soonish?
In any case, it's cool to see the work, and I look forward to the day
I'm running Django on Python 3.
pb
pb
Martin v. Löwis
> could be behind the times, but my understanding of current porting
> advice (based on PEP 3000) was that it's not going to be possible to
> support 2.x and 3.x from a single codebase in many cases (even with
> 2to3) if Python < 2.6 needs to be supported. Right?
It depends on whom you ask. People often give this advise *assuming*
that it couldn't possibly work. I happen to think differently.
I see no technical reason why you couldn't support all versions from,
say, 2.1 up to 3.0, from a single code base, if you rely on 2to3.
(there are severe technical reasons that make that difficult if you
don't want to rely on 2to3, such as not being able to get the exception
object in an except clause)
> I also wonder, is it worth adding this stuff to Django *now* for a
> transition that is likely to be quite far off, with at least two big
> merges (qs-rf, nfa, possibly gis) coming up soonish?
That's completely up to you; I don't want to interfere with the project
management. I just wanted to make it absolutely clear that there is
no *technical* reason why Django couldn't support 3.x. In addition
to whether that is wise from a project management point of view,
it's also a lot of work still to make it work fully, people might want
to focus on other aspects, this is free software, and all that.
Regards,
Martin