I've made good progress in porting Django to Python 3, and could use some help!
Vinay Sajip
terms of
test coverage pretty much almost there, but a few remaining test
failures are
eluding me, and I could probably use some help to speed things up.
I started with the features/py3k branch on the BitBucket Django mirror
(the one
at https://bitbucket.org/django/django), but whereas that was
approaching from
a point of view of running 2to3 over the Django codebase, I've
followed my
preferred strategy of aiming for a single codebase for 2.x and 3.x.
(This
strategy worked well for the virtualenv and pip ports which I did a
while ago.)
I've updated the django.utils.py3 module to add whatever I needed,
gratefully
borrowing ideas from Benjamin Peterson's six project as needed. I run
the
standard test suite on the codebase using Python 2.7.2 and 3.2.2 (I'm
not
supporting 3.0 or 3.1 - 3.2 is stable, 'callable' came back and is
liberally
used in Django, and I'm not sure it's worth bothering with support for
3.0/3.1.
The branch is available at
https://bitbucket.org/vinay.sajip/django (features/py3k branch)
Latest test result summaries are as follows:
2.7.2: ran 4229 tests in 301.690s
OK (skipped=71, expected failures=3)
3.2.2: Ran 4174 tests in 303.619s
FAILED (failures=6, errors=2, skipped=78, expected failures=2,
unexpected successes=1)
I think these results are encouraging, and I hope you agree! I added 7
skips,
mostly these are due to different representations on 2.x and 3.x e.g.
u'foo'
vs. 'foo'.
I'm posting the summary results to
https://gist.github.com/1373553
The verbose listings are too big to post together to Github, giving me
an
"Entity Too Large" error. So these are posted separately to
https://gist.github.com/1393994 (2.7.2)
and
https://gist.github.com/1394000 (3.2.2).
The failures are:
ERROR: test_bad_404_template
(regressiontests.test_client_regress.models.TemplateExceptionTests)
I think this is due to Django trying to handle an unaught exception
and failing
to find the 4500.html template, because the test has set the templates
path to a
bad value (deliberately, for the purpose of the test).
ERROR: test_cull
(regressiontests.cache.tests.DBCacheTests)
This raises a "sqlite3.IntegrityError: datatype mismatch" but I
haven't yet
found the underlying cause.
FAIL: test_modelform_factory_metaclass
(regressiontests.model_forms_regress.tests.CustomMetaclassTestCase)
This is down to the way metaclasses are implemented slightly
differently.
FAIL: test_lazy_objects
(regressiontests.i18n.tests.TranslationTests)
Proxy object comparison code differs between 2.x and 3.x, needs
looking at.
FAIL: test_RegexValidator_raises_error_23
(modeltests.validators.tests.TestSimpleValidators)
I haven't tracked this down yet.
FAIL: testParsing
(regressiontests.conditional_processing.models.ETagProcessing)
This is down to different representations on 2.x and 3.x:
e\"t\"ag vs. e"t"ag
FAIL: test_null_display_for_field
(regressiontests.admin_util.tests.UtilTests)
This is also down to proxy comparison logic.
FAIL: test_templates
(regressiontests.templates.tests.Templates)
This is down to representational differences between 2.x and 3.x.
I did the work and ran the tests on a VM running Ubuntu Oneiric Ocelot
(x64).
You can hopefully reproduce these by cloning my repository, updating
to the
features/py3k branch, and then running one of
PYTHONPATH=.. python2.7 -u runtests.py --settings test_sqlite
or
PYTHONPATH=.. python3.2 -u runtests.py --settings test_sqlite
in the tests directory. The work's taken about a week of elapsed time
(on and
off), and I haven't tracked any changes in the default branch since
then.
Of course, I realise there's still a lot left to do, but I hope I have
broken
the back of it. I've approached it just from the point of trying to
get the
tests to pass, and not (yet) focused on actually running realistic
scenarios.
Perhaps some of my fixes need changing - having other devs' eyeballs
on the
code can definitely help.
Please let me have your comments!
Regards,
Vinay Sajip
Vinay Sajip
there!
Regards,
Vinay Sajip
hiveNzin0
I'm looking forward to seeing the result of your work.
Cheers. :)
Kiril Vladimirov
Vinay Sajip
On Nov 27, 2:10 pm, Kiril Vladimirov <v.ki...@gmail.com> wrote:
> Have you made some sort of TODO list I could use?
> Or selecting some failing test and make it run fine would be fine as well?
I would say: just clone the repo, try to reproduce the results, report
any differences you find from my results, and select any failing test
to fix.
Another area to look at is to exercise other database backends, e.g.
PostgreSQL. I've only looked at sqlite as the obvious backend to use
first.
> Also, how about moving this project to GitHub?
> Django is there, too(https://github.com/django/) and I thing we could find
> more participants.
It just so happened that I found the BitBucket mirror first and used
that, but I wouldn't expect a serious developer, who wants to pitch
in, to object to using Mercurial and BitBucket instead. There's very
little to choose between them (BitBucket/GitHub), compared to the real
work still to be done in getting Django production-ready on 3.x.
Regards,
Vinay Sajip
Sindre Sorhus
Kiril Vladimirov
Russell Keith-Magee
> I'm working on a port of Django to Python 3. I'm getting close, and in
> terms of
> test coverage pretty much almost there, but a few remaining test
> failures are
> eluding me, and I could probably use some help to speed things up.
Hi Vinay,
This is awesome work! Getting down to less than a dozen test failures
is amazing progress.
I don't have anything specific to add beyond that; I just wanted to
let you know the core team (or, at least, this member of the core
team) has seen your work, and is really encouraged that people are
making serious inroads into making Django on Py3 a reality.
Yours,
Russ Magee %-)
Jannis Leidel
Many thanks for your efforts so far, as you can imagine I have a few
questions, both procedural and technical. I should note though that
I haven't reviewed all your changes in detail yet, since they are..
massive :)
I'm a bit concerned that you didn't get in touch with us before you
started with the work, since tracking the changes would have been
easier. FWIW, Martin von Löwis, Alex and me would be those you can
ask if you need any further help, usually also on IRC in #django-dev.
On 25.11.2011, at 18:33, Vinay Sajip wrote:
> I'm working on a port of Django to Python 3. I'm getting close, and in
> terms of
> test coverage pretty much almost there, but a few remaining test
> failures are
> eluding me, and I could probably use some help to speed things up.
>
> I started with the features/py3k branch on the BitBucket Django mirror
> (the one
> at https://bitbucket.org/django/django), but whereas that was
> approaching from
> a point of view of running 2to3 over the Django codebase, I've
> followed my
> preferred strategy of aiming for a single codebase for 2.x and 3.x.
> (This
> strategy worked well for the virtualenv and pip ports which I did a
> while ago.)
Have you worked on top of the current py3k branch in SVN (which should
be mirrored in the Mercurial repo)? Also, did you get in touch with
Martin von Löwis who has previously spearheaded the porting efforts
for a while and has been granted commit privileges to the py3k branch?
I'm asking simply because I'm wondering how we should get your changes
reviewed and committed to the official repo, it's a bit like facing a
done deal.
> I've updated the django.utils.py3 module to add whatever I needed,
> gratefully
> borrowing ideas from Benjamin Peterson's six project as needed.
Aha, meaning that you've copied over parts of the six module? Would you
(also as a way for helping out Django users later to port their code)
recommend putting six completely in Django? We've previously done that
with other libraries, e.g. unittest2.
> I run the standard test suite on the codebase using Python 2.7.2 and 3.2.2 (I'm
> not supporting 3.0 or 3.1 - 3.2 is stable, 'callable' came back and is liberally
> used in Django, and I'm not sure it's worth bothering with support for 3.0/3.1.
Yeah, we've talked about that and don't consider 3.0 or 3.1 to be sensible to
support in the future (given obvious bugs and/or lack of support from Python core).
> The branch is available at
>
> https://bitbucket.org/vinay.sajip/django (features/py3k branch)
>
> Latest test result summaries are as follows:
>
> 2.7.2: ran 4229 tests in 301.690s
> OK (skipped=71, expected failures=3)
>
> 3.2.2: Ran 4174 tests in 303.619s
> FAILED (failures=6, errors=2, skipped=78, expected failures=2,
> unexpected successes=1)
>
> I think these results are encouraging, and I hope you agree! I added 7
> skips, mostly these are due to different representations on 2.x and 3.x e.g.
> u'foo' vs. 'foo'.
Indeed they are, so the next step is to review them bit by bit and align
it with the work done before your work. Ideally committing it to SVN.
Jannis
Vinay Sajip
On Nov 28, 1:04 pm, Jannis Leidel <lei...@gmail.com> wrote:
>
> I'm a bit concerned that you didn't get in touch with us before you
> started with the work, since tracking the changes would have been
> easier. FWIW, Martin von Löwis, Alex and me would be those you can
> ask if you need any further help, usually also on IRC in #django-dev.
>
Well, it's only been about a week of elapsed time. I always start
these sort of ports on an experimental-and-potentially-throwaway
basis, and I didn't know until I started how things would progress -
and well, once I was into it, I was just dipping in and out amidst
doing other things, so I suppose I just focused on the task at hand
rather than the big picture.
> Have you worked on top of the current py3k branch in SVN (which should
> be mirrored in the Mercurial repo)? Also, did you get in touch with
> Martin von Löwis who has previously spearheaded the porting efforts
> for a while and has been granted commit privileges to the py3k branch?
Yes, I used the BitBucket repo and the features/py3k branch within
that. I didn't contact Martin directly about this, as my approach is a
little different to what he had started, in that Martin's approach is
based on running 2to3 on the code to get new 3-friendly code, whereas
my approach is to have a single codebase with runs on 2.x and 3.x.
It's more than a small philosophical difference - I value the
information that 2to3 gives, but it just acts as the driver for
porting "by hand" with my standard development tools.
> I'm asking simply because I'm wondering how we should get your changes
> reviewed and committed to the official repo, it's a bit like facing a
> done deal.
Well, I undertook the approach you mentioned in the "Python 3 and you"
thread back in September, which was to look at the features/py3k
branch and focus on test failures. Of course I didn't use the py3ktest
script (which involves the 2to3 pass), running just plain runtests.py
instead. And it's only been a little over a week, so it can't be that
hard to merge the changes assuming you agree with the approach. Even
if you start over, it's still less than a person-week of effort,
right?
> Aha, meaning that you've copied over parts of the six module? Would you
> (also as a way for helping out Django users later to port their code)
> recommend putting six completely in Django? We've previously done that
> with other libraries, e.g. unittest2.
I have no strong views on using six directly as a dependency, and I've
only used small parts of it (with_metaclass, reraise) directly. It's
certainly not needed as a dependency - you can see that the
django.utils.py3 module (which provides the functionality needed by
Django) is pretty small.
> Yeah, we've talked about that and don't consider 3.0 or 3.1 to be sensible to
> support in the future (given obvious bugs and/or lack of support from Python core).
Good, that rhymes with my thinking on it :-)
> Indeed they are, so the next step is to review them bit by bit and align
> it with the work done before your work. Ideally committing it to SVN.
Right, and I can provide some help with that (other work permitting).
Possibly some work could be done to look at merging the default branch
with the features/py3k branch, as a first step - presumably default
tracks SVN trunk closely. The main thing is to see whether you are
comfortable with the single codebase approach (requires some small
additional discipline from developers to consider str/bytes issues
carefully, do imports from utils.py3 where appropriate, etc. - nothing
too onerous). Carl Meyer may be able to chime in with his experience
re. pip/virtualenv, which were ported using the same approach a while
ago, and have seen maintenance work and new releases since then. Also
it's worth looking at the way metaclasses have been done in the port
and to see if you can identify any issues. A lot of the changes are
pretty mechanical, wrapping literals with u() and b() - nothing too
contentious there. Escape character handling esp. in regexes is
another area where closer inspection would be worthwhile. The test
codebase contained a lot of Unicode literals (i.e. Unicode literals in
the source code, which would have been encoded in UTF-8 in the files
actually stored on disk) and I have converted these to use Unicode
escapes, as this is more portable.
Regards,
Vinay Sajip
Vinay Sajip
On Nov 28, 9:29 am, Kiril Vladimirov <v.ki...@gmail.com> wrote:
> My point was the social factor. I mean the GitHub community is quite
> bigger. It's not big issue if we're using mercurial instead of git. Sure, I
> prefer git, it's faster and stuff, but it's not a big deal, right now.
Well, it's not as if there's a big team required, and presumably
interested parties who can contribute will be subscribed to this list
anyway. There probably isn't the kind of need for a lot of back-and-
forth, given that the overall approach and some specific
implementation decisions are acceptable to the core devs.
> Anyway, I'm trying to get into making the tests run and if I see some
> result from my work, will try to figure out some sync between github and
> bitbucket.
Great, we'd like to see what you find.
Regards,
Vinay Sajip
Jannis Leidel
> On Nov 28, 1:04 pm, Jannis Leidel <lei...@gmail.com> wrote:
>>
>> I'm a bit concerned that you didn't get in touch with us before you
>> started with the work, since tracking the changes would have been
>> easier. FWIW, Martin von Löwis, Alex and me would be those you can
>> ask if you need any further help, usually also on IRC in #django-dev.
>
> Well, it's only been about a week of elapsed time. I always start
> these sort of ports on an experimental-and-potentially-throwaway
> basis, and I didn't know until I started how things would progress -
> and well, once I was into it, I was just dipping in and out amidst
> doing other things, so I suppose I just focused on the task at hand
> rather than the big picture.
I see, that's absolutely fine, and to be honest, getting so much
progress done is definitely a good problem to have :)
>> Have you worked on top of the current py3k branch in SVN (which should
>> be mirrored in the Mercurial repo)? Also, did you get in touch with
>> Martin von Löwis who has previously spearheaded the porting efforts
>> for a while and has been granted commit privileges to the py3k branch?
>
> Yes, I used the BitBucket repo and the features/py3k branch within
> that. I didn't contact Martin directly about this, as my approach is a
> little different to what he had started, in that Martin's approach is
> based on running 2to3 on the code to get new 3-friendly code, whereas
> my approach is to have a single codebase with runs on 2.x and 3.x.
> It's more than a small philosophical difference - I value the
> information that 2to3 gives, but it just acts as the driver for
> porting "by hand" with my standard development tools.
Ah, that makes sense, in fact your approach is much closer to what I
remember doing when pip and virtualenv was ported. In that sense, I'm
definitely fine with that. How much of that approach needs to be
documented for end users is probably something we can deal with later.
>> I'm asking simply because I'm wondering how we should get your changes
>> reviewed and committed to the official repo, it's a bit like facing a
>> done deal.
>
> Well, I undertook the approach you mentioned in the "Python 3 and you"
> thread back in September, which was to look at the features/py3k
> branch and focus on test failures. Of course I didn't use the py3ktest
> script (which involves the 2to3 pass), running just plain runtests.py
> instead. And it's only been a little over a week, so it can't be that
> hard to merge the changes assuming you agree with the approach. Even
> if you start over, it's still less than a person-week of effort,
> right?
Honestly, I'm not sure how hard the merge is, as I'm not sure how much
changed. Martin could probably shed some light on it how he wants to deal
with it (e.g. svnmerge.py or not).
>> Aha, meaning that you've copied over parts of the six module? Would you
>> (also as a way for helping out Django users later to port their code)
>> recommend putting six completely in Django? We've previously done that
>> with other libraries, e.g. unittest2.
>
> I have no strong views on using six directly as a dependency, and I've
> only used small parts of it (with_metaclass, reraise) directly. It's
> certainly not needed as a dependency - you can see that the
> django.utils.py3 module (which provides the functionality needed by
> Django) is pretty small.
Fair enough, I just realized that's a discussion we need to have in a
separate thread (~"What's the best approach for migrating Django projects
from 2.X to 3.X?") that can be handled later in the porting process. When
in doubt I would rather use a module like six that has community traction
than writing our own though.
>> Indeed they are, so the next step is to review them bit by bit and align
>> it with the work done before your work. Ideally committing it to SVN.
>
> Right, and I can provide some help with that (other work permitting).
> Possibly some work could be done to look at merging the default branch
> with the features/py3k branch, as a first step - presumably default
> tracks SVN trunk closely. The main thing is to see whether you are
> comfortable with the single codebase approach (requires some small
> additional discipline from developers to consider str/bytes issues
> carefully, do imports from utils.py3 where appropriate, etc. - nothing
> too onerous).
Personally I'm fine with it, but as you say, it requires discipline
(I broke pip more than once). But it's definitely something that needs
some input from the other core devs, and probably a very good
documentation of the dos and don'ts.
> Carl Meyer may be able to chime in with his experience
> re. pip/virtualenv, which were ported using the same approach a while
> ago, and have seen maintenance work and new releases since then. Also
> it's worth looking at the way metaclasses have been done in the port
> and to see if you can identify any issues. A lot of the changes are
> pretty mechanical, wrapping literals with u() and b() - nothing too
> contentious there. Escape character handling esp. in regexes is
> another area where closer inspection would be worthwhile. The test
> codebase contained a lot of Unicode literals (i.e. Unicode literals in
> the source code, which would have been encoded in UTF-8 in the files
> actually stored on disk) and I have converted these to use Unicode
> escapes, as this is more portable.
Jannis
Vinay Sajip
On Nov 28, 5:36 pm, Jannis Leidel <lei...@gmail.com> wrote:
>
> Ah, that makes sense, in fact your approach is much closer to what I
> remember doing when pip and virtualenv was ported.
Right, since I did those ports originally :-)
> Honestly, I'm not sure how hard the merge is, as I'm not sure how much
> changed. Martin could probably shed some light on it how he wants to deal
> with it (e.g. svnmerge.py or not).
Sure.
> Fair enough, I just realized that's a discussion we need to have in a
> separate thread (~"What's the best approach for migrating Django projects
> from 2.X to3.X?") that can be handled later in the porting process. When
> in doubt I would rather use a module like six that has community traction
> than writing our own though.
There are areas where the current code needs to do metaclass-based
checks, and that involves delving into the specifics of the
implementation of with_metaclass. This being the case, I made a
modified version of Benjamin's which uses "_DjangoBase" as the
intermediate parent class. IMO we need this to distinguish from other
classes implemented using with_metaclass from the official six
package.
> Personally I'm fine with it, but as you say, it requires discipline
> (I broke pip more than once). But it's definitely something that needs
> some input from the other core devs, and probably a very good
> documentation of the dos and don'ts.
Having good code coverage helps to spot these potential breakages well
before a release (or even a checkin), and Django's extensive test
suite is a boon in this regard. Though having worked through the
tests, it doesn't seem like the DRY principle is followed as much as
it could be ... for example, the same literals being used over and
over again in copy/paste fashion, requiring patches in multiple
locations to add u() and b() wrappers, for example. I didn't have time
to rationalise this, as I was focused more on identifying and fixing
failures and errors.
Regards,
Vinay Sajip
Vinay Sajip
> suite is a boon in this regard. Though having worked through the
> tests, it doesn't seem like the DRY principle is followed as much as
> it could be ... for example, the same literals being used over and
> over again in copy/paste fashion, requiring patches in multiple
> locations to add u() and b() wrappers, for example. I didn't have time
> to rationalise this, as I was focused more on identifying and fixing
> failures and errors.
Ironically, I notice that I repeated myself in using "for example"
twice in the above paragraph ;-)
Regards,
Vinay Sajip