Python 3 and you
Jannis Leidel
After last week's sprint I wanted to get you up-to-speed about the
current state of porting Django to Python 3.
As some may be aware Martin von Löwis has been working on a port for
a while [1] but only recently I've had the chance to meet with him and
talk through the porting process.
I'm not going to hide the fact that it'll be a long process, but I'm
also convinced it's an important step for Django to make. I'm writing
this in the hope to find volunteers to join the porting efforts.
Goals
-----
To allow Django to run on Python 3 there are several goals to achieve,
some of which are our respsonsibility, some depend on 3rd party libraries
we use internally and some left to the users that use Django to build
their websites. It's my understanding that we can't solve everything
at once, so take this with a grain of salt:
- get Django to run on Python 3
- provide helpers and docs for porting Django-based projects
- help out 3rd party projects we rely only to make the jump (if needed)
Porting strategies
------------------
As you can imagine there are still quite a few open questions at
the moment about specific porting problems but taking from the
experience in the Python community I think we have a good general
strategy.
There are a few assumptions we're applying either because it's
unrealistic or impossible to maintain as long as Python 2.X is in
use for the forseeable future; so these strategy *don't* work:
- Create a Python 3 only port ("burning the bridges")
This is outright a no-go since it would leave all the Python 2.X
projects in dead water. Instead we need to provide a migration
path for them.
- Maintaing a separate Python 3 branch ("dual releases")
While this would allow for new projects to use Python 3, I'm
convinced this has the potential to split the community. It'd
also be a major burden for the core team to maintain both
branches. Instead we need a combined effort.
So as a result of that the only viable option is to support both major
versions of Python at the same time, with the same code base.
Fortunately the Python community gained lots of experience in the past
years to make this happen (e.g. Lennart Regebro's book [4]). There are
also tools to ease the transition of Django and the Django-based
projects. Some of which are:
- six [3] -- a compatibility library that includes many (if not all)
needed import proxies and utilities to prepare Django and Django-based
projects to be ported to Python 3.X. This only applies to API that
isn't syntactically changed, but only moved or enhanced in 3.X.
- 2to3 [2] -- an extensible library which is able to translate the rest
of the Python 2 code to the Python 3 equivalent. For every Django
specific feature that isn't covered by the default 2to3 "fixers" we can
write our own if needed. It integrates with distutils (in Python 3.X)
and is able to convert Django at installation time. Installing Django
with Python 2 wouldn't trigger the translation process, of course.
Code status
-----------
During the sprint we've moved Martin's code from a Bitbucket clone to
an own SVN branch:
https://code.djangoproject.com/browser/django/branches/features/py3k/
Some notable changes:
- a modified ``setup.py`` which automatically calls 2to3 during installation
- a ``py3ktest`` helper bash script which -- for now -- installs Django in
a directory called "3k" in the same directory to trigger the translation
from Python 2 to Python 3 code and then run the tests from the build
directory directly because they are not part of the installation in "3k"
because we don't include it. This script should be seen a temporary
workaround till we've found a better way to run the tests (Could we use
tox instead? [5]).
- a new django.utils.py3 module which contains some helpers that are used
throughout the code as a common API to ease the pain of maintaining a
project that runs on both Python 2 and 3. I expect it to grow in size
while we port Django, but even then it may not be complete enough to
be useful for Django-based user projects. Which is why I think Django
should ship the "six" library [3] instead, on the long run ("six" has
the advantage of being maintained by a Python core developer).
A good overview of the current changes can be seen on Bitbucket:
https://bitbucket.org/django/django/compare/features/py3k..default
Right now it's mostly changes to how byte and unicode strings are handled
by using a b() and u() function instead of the 'u' prefix. That said,
this is far from complete.
How to help
-----------
We have multiple big pieces of the puzzle to solve:
- Try out the branch by running the tests with the ``py3ktest`` script
and fix the failed tests (needs an installed ``python3`` binary), one
by one. This may be repetitive work, but could also be the chance for
you to dive into the internals of Django.
- Write a tutorial to prepare a Django app to for Python 3 by using one
of the tools we provide. Have a look at the official porting guides [6]
for inspiration.
- Help port the 3rd party libraries we rely on in Django (e.g. MySQLdb [7])
by getting in touch with their community.
There are probably lots of other small steps to make, but I'm confident that
we'll figure them out on the way.
Let's start the porting, Python 3 is waiting for us,
Jannis
1: https://bitbucket.org/loewis/django-3k/
2: http://docs.python.org/library/2to3.html
3: http://pypi.python.org/pypi/six
4: http://python3porting.com/
5: http://tox.readthedocs.org/
6: http://docs.python.org/py3k/howto/pyporting.html
7: http://sourceforge.net/projects/mysql-python/forums/forum/70460/topic/3831691
Daniel Lindsley
"You have my sword." I want to see this happen & would love to be a
part of it. A couple questions:
* How should patches be provided? Trac? BitBucket?
* Where should feedback go? This mailing list? Somewhere else?
* This is further off, but once we have a ported Django, how do get
the community (specifically pluggable apps) onboard? I'm assuming the
docs are meant to do this but wondering if there's anything else we
can be doing (like perhaps a Django-specific 2to3 (extension?) to
cover common Django conventions).
* Do we have a target date? I know this is hard with a volunteer-only
effort, but if we setup some sort of timeline, we'd at least have a
metric & something to shoot/push for.
Finally, a philosophical question on approach: Should we really be
doing 2to3 (leaving the Django codebase in Python 2.X for a long time)
or would it be better to port Django over to Python 3 & use 3to2 for
existing Python 2.X installs? I confess I don't know much about the
current state of 3to2 (nor how most other Python libraries are
handling the transition). But I do know Django will continue to grow
over time & I worry that, at some point in the future we'll be making
more even more work for someone else to do the 3-only work.
Regardless, I'm excited & will see what I can do to help.
Daniel
Jannis Leidel
> "You have my sword." I want to see this happen & would love to be a
> part of it.
Huzzah!
> A couple questions:
>
> * How should patches be provided? Trac? BitBucket?
For now via Trac, that's why we've moved the changes into a SVN branch.
Unless anyone has a better idea I could create a Trac component "Python 3"
so we can track the tickets easily.
> * Where should feedback go? This mailing list? Somewhere else?
Feedback should go here, on the developers mailing list, to get as many
eyes on it as possible.
> * This is further off, but once we have a ported Django, how do get
> the community (specifically pluggable apps) onboard? I'm assuming the
> docs are meant to do this but wondering if there's anything else we
> can be doing (like perhaps a Django-specific 2to3 (extension?) to
> cover common Django conventions).
Very good question, I'm uncertain as to how the "helpers" I mentioned
will look like in the end. Whether they will be part of Django (e.g.
a management command to run 2to3 on an app) or if we "only" provide the
necessary compatibility library (e.g. "six") so that 3rd party app
authors would still keep writing apps with Python 2 but would allow
their apps to be translated to Python 3 automatically. Documenting ways
of how to write a setup.py to do the conversion during install time
is *in* the scope of what we need to provide, IMO. Whether we need
Django-specific 2to3 fixers isn't clear at this time as the porting
has only just begun.
> * Do we have a target date? I know this is hard with a volunteer-only
> effort, but if we setup some sort of timeline, we'd at least have a
> metric & something to shoot/push for.
One assumption of the strategy I outlined was the fact that Django is
as close to 3.X as possible. Django 1.4 will require Python 2.5 or
higher, but I'm not sure how quick we can do the jump to 2.6, which
is recommended by the Python porting docs [1].
> Finally, a philosophical question on approach: Should we really be
> doing 2to3 (leaving the Django codebase in Python 2.X for a long time)
> or would it be better to port Django over to Python 3 & use 3to2 for
> existing Python 2.X installs? I confess I don't know much about the
> current state of 3to2 (nor how most other Python libraries are
> handling the transition). But I do know Django will continue to grow
> over time & I worry that, at some point in the future we'll be making
> more even more work for someone else to do the 3-only work.
I personally haven't ported a 2.X library completely to 3.X yet, so I
can also only guess. But from what I've seen in the community I'm afraid
of a "clean cut" port because it has a high risk of leaving many projects
and apps behind. In that sense it seems more sensible to me to see the
port to Python 3 just as another step of our Python version deprecation
policy, which we at some point take with a complete conversion. Basically
a "burn bridges as soon as everyone is safe" approach :)
I don't dare to guess when that moment could be though, but it would probably
happen after a potential Python 2.7 only release of Django -- whenever that is.
Jannis
1: http://docs.python.org/py3k/howto/pyporting.html#try-to-support-python-2-6-and-newer-only
Daniel Lindsley
I wasn't trying to suggest we leave anyone behind, far from it. I
was suggesting move the code to Python 3 now, while there's less code
there (than some future date) but using 3to2[1] to help others on
Python 2.X. Since Django still supports 2.5, it's possible that this
isn't even an option, as I don't know if 3to2 can translate back that
far reliably. Simply getting the question out there for others to mull
over.
Daniel
Daniel Lindsley
>
> read more »
Jannis Leidel
> I wasn't trying to suggest we leave anyone behind, far from it. I
> was suggesting move the code to Python 3 now, while there's less code
> there (than some future date) but using 3to2[1] to help others on
> Python 2.X. Since Django still supports 2.5, it's possible that this
> isn't even an option, as I don't know if 3to2 can translate back that
> far reliably. Simply getting the question out there for others to mull
> over.
Apologies for the misunderstanding. This is definitely an option which
should be tried out instead of forward porting with 2to3. That said,
it seems like there are a few technical limitations:
"However, there is no Distribute support for 3to2 and also Python 2.5
or earlier also do not include the required lib2to3 package. Therefore
3to2 currently remains only an interesting experiment, although this may
change in the future." -- http://python3porting.com/strategies.html#using-3to2
Besides that, I could also think that missing Python 3 expertise in the
community would prevent a smooth direct transition to Python 3. There
is no doubt that it needs to happen eventually but we just need to figure
out if we're ready for it.
Jannis
> On Sep 14, 10:36 am, Jannis Leidel <lei...@gmail.com> wrote:
>> Daniel,
>>
>>
>> Huzzah!
>>
>>
>>
>> For now via Trac, that's why we've moved the changes into a SVN branch.
>> Unless anyone has a better idea I could create a Trac component "Python 3"
>> so we can track the tickets easily.
>>
>>
>> Feedback should go here, on the developers mailing list, to get as many
>> eyes on it as possible.
>>
>>
>> Very good question, I'm uncertain as to how the "helpers" I mentioned
>> will look like in the end. Whether they will be part of Django (e.g.
>> a management command to run 2to3 on an app) or if we "only" provide the
>> necessary compatibility library (e.g. "six") so that 3rd party app
>> authors would still keep writing apps with Python 2 but would allow
>> their apps to be translated to Python 3 automatically. Documenting ways
>> of how to write a setup.py to do the conversion during install time
>> is *in* the scope of what we need to provide, IMO. Whether we need
>> Django-specific 2to3 fixers isn't clear at this time as the porting
>> has only just begun.
>>
>>
>> One assumption of the strategy I outlined was the fact that Django is
>> as close to 3.X as possible. Django 1.4 will require Python 2.5 or
>> higher, but I'm not sure how quick we can do the jump to 2.6, which
>> is recommended by the Python porting docs [1].
>>
>>
Andrews Medina
On Wed, Sep 14, 2011 at 10:03 AM, Jannis Leidel <lei...@gmail.com> wrote:
> Hi all,
>
> After last week's sprint I wanted to get you up-to-speed about the
> current state of porting Django to Python 3.
>
I'm very happy with this news.
> As some may be aware Martin von Löwis has been working on a port for
> a while [1] but only recently I've had the chance to meet with him and
> talk through the porting process.
>
I already started a django port to python3 in following MvL way, using
2to3 but I had steel problems with tests. I think that you already fix
this using python3k test.
I want help with this. I will get the branch code and help fixing some tests ok?
--
Andrews Medina
www.andrewsmedina.com
Ákos Péter Horváth
Really, I started to port that with a recursive 2to3. It is not too far from good working. There are no big magic things, altough I think a py2 and py3 support isn't possible from a common source tree. Some deep core improvement is needed too, mostly on the unicode line, but nothing really hard.
Anybody knows somebody who started a django/py3 port already? We should unify our efforts.
If core team doesn't like the py3 idea, we should start with a patch which follows always the latest devel version. I strongly anticipate a project fork (see what happened with the plone & bluebream).
2to3 (3to2), etc are good start for an initial port, but I think they aren't enough to do this on day-to-day base. Their conversion algo isn't enough, some handwork is needed.
thank you
MaXX
--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-d...@googlegroups.com.
To unsubscribe from this group, send email to django-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
![Cal Leeming [Simplicity Media Ltd]'s profile photo](http://lh3.googleusercontent.com/a-/ALV-UjWxmy9B2ZuM4H6f4BRSGX6Ym9xIa3xyDT0-2RyeSTW0CrcEcaXI=s40-c)
Cal Leeming [Simplicity Media Ltd]
Jannis Leidel
On 14.09.2011, at 18:57, Ákos Péter Horváth wrote:
> Really, I started to port that with a recursive 2to3. It is not too far from good working. There are no big magic things, altough I think a py2 and py3 support isn't possible from a common source tree. Some deep core improvement is needed too, mostly on the unicode line, but nothing really hard.
>
> Anybody knows somebody who started a django/py3 port already? We should unify our efforts.
As I said earlier in this thread, there is now a Python 3 branch in the Django SVN.
> If core team doesn't like the py3 idea, we should start with a patch which follows always the latest devel version. I strongly anticipate a project fork (see what happened with the plone & blue bream).
The core team likes the idea, there is no fork required.
> 2to3 (3to2), etc are good start for an initial port, but I think they aren't enough to do this on day-to-day base. Their conversion algo isn't enough, some handwork is needed.
Could you be a bit more specific about what the algorithm can't do? From what I saw in Martin von Löwis' patch (which is now in the Python 3 branch) we haven't hit any problem with 2to3. That said, it's of course a long way to go.
Jannis
Jannis Leidel
On 14.09.2011, at 19:19, Cal Leeming [Simplicity Media Ltd] wrote:
> Can I ask, have the django core team already accepted that Django will eventually be a 3.x framework, or will it be un-officially forked?
Yes, the core team has identified the port to Python 3 as a needed step which is why I've posted the first mail in this thread earlier today.
So this is not about forking Django, but a specific branch in the SVN which at some point in the distant future might be merged with trunk [1].
Jannis
1: https://code.djangoproject.com/browser/django/branches/features/py3k/
Ákos Péter Horváth
As I said earlier in this thread, there is now a Python 3 branch in the Django SVN.
> Anybody knows somebody who started a django/py3 port already? We should unify our efforts.
Thank you! Really good to see that.
The core team likes the idea, there is no fork required.
> If core team doesn't like the py3 idea, we should start with a patch which follows always the latest devel version. I strongly anticipate a project fork (see what happened with the plone & blue bream).
Thank you! Really good to see that.
Could you be a bit more specific about what the algorithm can't do? From what I saw in Martin von Löwis' patch (which is now in the Python 3 branch) we haven't hit any problem with 2to3. That said, it's of course a long way to go.
> 2to3 (3to2), etc are good start for an initial port, but I think they aren't enough to do this on day-to-day base. Their conversion algo isn't enough, some handwork is needed.
As I remember, it maked errors in the unicode handling of some *lazy* decorators in the django core. It isn't a wonder, because the functionality of these decorators contain a sophisticated functionality for the generalized unicode / ascii handling which in py3 is not needed. I think, the good fix is trivial for the core team and possibly it is done already in the official py3 branch (if not, I feel some urge to do this :-) )
bye,
MaXX
VernonCole
> One assumption of the strategy I outlined was the fact that Django is
> as close to 3.X as possible. Django 1.4 will require Python 2.5 or
> higher, but I'm not sure how quick we can do the jump to 2.6, which
> is recommended by the Python porting docs [1].
>
which installs and runs very nicely on any version of Python >= 2.3.
The automatic operation of 2to3 during the install is like magic. It
really works very well.
> > Finally, a philosophical question on approach: Should we really be
> > doing 2to3 (leaving the Django codebase in Python 2.X for a long time)
Python 2.4 until everyone is ready (or able) to jump to 2.5. You
simply avoid using new language features until that point. If
fortunate people have a later version, then they can use new language
features in their code, just us module authors cannot use them. But
we plan for them. For example, we add __enter__() and __exit__()
methods to objects where that makes sense. If the programmer is using
an older compiler, then they will never get called, but they will be
present when someone with a new compiler uses a "with" statement. We
cannot use "with" in module code, though, until all python 2.4 users
have disappeared.
Same thing when moving to Python 3.x -- you cannot use any new
features, but you have to refactor your old code in such a way that
2to3 can do the right thing when it installs. Mostly this means
cleaning up old code which was probably due for a maintenance pass
anyway. Get rid of string exceptions. Make all class definitions
into new style classes. Things like that.
> > or would it be better to port Django over to Python 3 & use 3to2 for
> > existing Python 2.X installs? I confess I don't know much about the
> > current state of 3to2 (nor how most other Python libraries are
> > handling the transition). But I do know Django will continue to grow
> > over time & I worry that, at some point in the future we'll be making
> > more even more work for someone else to do the 3-only work.
When that glorious day happens, we will make an archive copy of the
old version, and then run 2to3 for the last time. After that, code
would be maintained in the oldest supported 3.x version.
> I personally haven't ported a 2.X library completely to 3.X yet, so I
> can also only guess. But from what I've seen in the community I'm afraid
> of a "clean cut" port because it has a high risk of leaving many projects
> and apps behind. In that sense it seems more sensible to me to see the
> port to Python 3 just as another step of our Python version deprecation
> policy, which we at some point take with a complete conversion. Basically
> a "burn bridges as soon as everyone is safe" approach :)
guessing that the jump will be from 2.7 to about 3.3. No sense
supporting obsolete 3.x versions..
> I don't dare to guess when that moment could be though, but it would probably
> happen after a potential Python 2.7 only release of Django -- whenever that is.
>
others will have been running 3.x versions for years, and will have
long since worked out the kinks.
--
Vernon
Waylan Limberg
that as someone who maintains a module which installs and runs on
Python 2.4 - 3.2, Vernon's comments are spot on.
We first started supporting Python 3.0 almost immediately after its
release and we had a few misconceptions going in - some of the same
misconceptions I'm seeing here by people who have not done this yet.
But after working with it for a while now, those misconceptions are
gone (for me). You really do maintain one codebase to run in the
lowest version of python you support (with forward thinking code) and
have setup.py automagically run 2to3 when installing on Python 3+.
That is the only workable solution.
I realize there are various articles/tutorials out there which talk
about how to write code which runs on both 2.x and 3.x without
conversion. While it is possible, it is a bear to maintain and IMO
should only apply to the setup.py file itself (this means you need to
test setup.py in 2.x & 3.x every time you make a change to it - not
something we're used to testing regularly). The exception may be a
small one file module, which Django certainly is not.
> --
> You received this message because you are subscribed to the Google Groups "Django developers" group.
> To post to this group, send email to django-d...@googlegroups.com.
> To unsubscribe from this group, send email to django-develop...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
>
>
--
----
\X/ /-\ `/ |_ /-\ |\|
Waylan Limberg
richard.prosser
experience (mostly 2.4) and I have been out of work for a long time
due to illness, so a project like this may look good on my CV!
I have never been involved in an OS project before, though I am aware
of the guidelines.
At present the only machine I have available is a Windows Vista
laptop :-( Are there development/test servers available?
Cheers,
Richard
On Sep 14, 1:03 pm, Jannis Leidel <lei...@gmail.com> wrote:
> Hi all,
>
> After last week's sprint I wanted to get you up-to-speed about the
> current state of porting Django toPython3.
>
> As some may be aware Martin von Löwis has been working on a port for
> a while [1] but only recently I've had the chance to meet with him and
> talk through the porting process.
>
> I'm not going to hide the fact that it'll be a long process, but I'm
> also convinced it's an important step for Django to make. I'm writing
> this in the hope to find volunteers to join the porting efforts.
>
> Goals
> -----
>
> we use internally and some left to the users that use Django to build
> their websites. It's my understanding that we can't solve everything
> at once, so take this with a grain of salt:
>
> - get Django to run onPython3
> - provide helpers and docs for porting Django-based projects
> - help out 3rd party projects we rely only to make the jump (if needed)
>
> Porting strategies
> ------------------
>
> As you can imagine there are still quite a few open questions at
> the moment about specific porting problems but taking from the
>
> There are a few assumptions we're applying either because it's
>
> This is outright a no-go since it would leave all thePython2.X
> projects in dead water. Instead we need to provide a migration
> path for them.
>
>
> While this would allow for new projects to usePython3, I'm
> also be a major burden for the core team to maintain both
> branches. Instead we need a combined effort.
>
> So as a result of that the only viable option is to support both major
>
> Fortunately thePythoncommunity gained lots of experience in the past
> also tools to ease the transition of Django and the Django-based
> projects. Some of which are:
>
> - six [3] -- a compatibility library that includes many (if not all)
> needed import proxies and utilities to prepare Django and Django-based
>
> - 2to3 [2] -- an extensible library which is able to translate the rest
> write our own if needed. It integrates with distutils (inPython3.X)
> and is able to convert Django at installation time. Installing Django
> Code status
> -----------
>
> During the sprint we've moved Martin's code from a Bitbucket clone to
> an own SVN branch:
>
> https://code.djangoproject.com/browser/django/branches/features/py3k/
>
> Some notable changes:
>
> - a modified ``setup.py`` which automatically calls 2to3 during installation
>
> - a ``py3ktest`` helper bash script which -- for now -- installs Django in
> a directory called "3k" in the same directory to trigger the translation
> because we don't include it. This script should be seen a temporary
> workaround till we've found a better way to run the tests (Could we use
> tox instead? [5]).
>
> - a new django.utils.py3 module which contains some helpers that are used
> throughout the code as a common API to ease the pain of maintaining a
> be useful for Django-based user projects. Which is why I think Django
> should ship the "six" library [3] instead, on the long run ("six" has
> A good overview of the current changes can be seen on Bitbucket:
>
> https://bitbucket.org/django/django/compare/features/py3k..default
>
> Right now it's mostly changes to how byte and unicode strings are handled
> by using a b() and u() function instead of the 'u' prefix. That said,
> this is far from complete.
>
> How to help
> -----------
>
> We have multiple big pieces of the puzzle to solve:
>
> - Try out the branch by running the tests with the ``py3ktest`` script
> and fix the failed tests (needs an installed ``python3`` binary), one
> by one. This may be repetitive work, but could also be the chance for
> you to dive into the internals of Django.
>
> for inspiration.
>
> - Help port the 3rd party libraries we rely on in Django (e.g. MySQLdb [7])
> by getting in touch with their community.
>
> There are probably lots of other small steps to make, but I'm confident that
> we'll figure them out on the way.
>
> Jannis
>
> 1:https://bitbucket.org/loewis/django-3k/
> 2:http://docs.python.org/library/2to3.html3:http://pypi.python.org/pypi/six
> 5:http://tox.readthedocs.org/
> 6:http://docs.python.org/py3k/howto/pyporting.html
Jonas H.
> I may be willing to support this. I have 18 month's of Python
> experience (mostly 2.4) and I have been out of work for a long time
> due to illness, so a project like this may look good on my CV!
>
> I have never been involved in an OS project before, though I am aware
> of the guidelines.
>
> At present the only machine I have available is a Windows Vista
> laptop :-( Are there development/test servers available?
You don't need a Linux/Unix since Python/Django works on Windows.
richard.prosser
hot!
Plus I have very good Unix/Linux experience, which may help.
Richard
Brian Schott
https://www.virtualbox.org/
Brian Schott
bfsc...@gmail.com
