It looks like the sprint was fairly successful and we have a patch that updates Genshi to be Python 3 compatible (without losing Python 2 compatibility). The patch is available at either of:
Bitbucket has the full commit history. The sprint occurred in two stages. First we worked on simply porting to Python 3 without maintaining compatibility with Python 2. This work was done in the 2to3 bitbucket branch and took about seven hours. In the second stage we created a single code-base that supports Python 2 and Python 3 (via 2to3 and some helpers). This was done in the default branch and took about eight hours.
We've tested the result under Python 2.4, 2.5, 2.6, 3.1 and 3.2 and on Linux (Ubuntu Lucid and Maverick) and Mac OS X.
API Change ^^^^^^^^^^^
We made one important API change. Previously the default encoding used by Genshi in a number of functions was UTF-8. This is a little awkward in Python 3 since strings are unicode (and it's generally bytes that have to be explicitly asked for). We opted to change the default encoding to None (i.e. use a unicode string). Although this breaks compatibility, we think it eases the migration towards Python 3 (and it made porting easier).
Usage ^^^^^^
The patch should apply cleanly against trunk. After applying one can build, test and install under Python 2 using:
Running setup.py under Python 3 requires Distribute (which runs under Python 3 and provides hooks for running 2to3).
Patch Tricks ^^^^^^^^^^^
- "if not isinstance(s, unicode):" is converted by 2to3 while "if isinstance(s, str)" is not. - "u'foo'.encode('utf-8')" creates a byte string in UTF-8 both before and after 2to3 while "'foo'" does not.
Patch Details ^^^^^^^^^^^^
.hgignore README - Hg ignore file and note about the patch. No need to commit.
setup.py MANIFEST.in - changes to make setup.py importable by both Python 2 and Python 3 (without 2to3). - include tests in build so that "python3 setup.py test" works. - Distribute runs the tests on the build rather than on the source since the source is not modified by 2to3. - add Distribute 2to3 options when run with Python 3.
doc/common/doctools.py - change to an svn:external to make prints python3 compatible. - doctools is import by setup.py so this needs to be importable by python3 without running 2to3 on it.
examples_to_py3k.sh - script to run 2to3 on the python files in the examples folder.
fixes/fix_unicode_in_strings.py - We wrote a custom 2to3 fixer to fix unicode output strings inside doctests and other string constants.
genshi/_speedups.c - ported using #ifdefs for Python 3.
genshi/compat.py - new home for cross-python-version compatibility functions
genshi/core.py genshi/tests/core.py - change default encoding from UTF-8 to None (i.e. unicode strings) - stringrepr no longer used inside __repr__ in Python 3 repr() returns a string (i.e. unicode)
genshi/input.py genshi/tests/input.py - change default encoding for HTML() to None. - track changes to expat parser in Python 3 (mostly it accepts bytes instead of strings).
genshi/output.py genshi/tests/output.py - change default encoding for encode() to None.
genshi/util.py - move cross-Python-version compatibility fixes into genshi.compat
genshi/filters/html.py - minor changes to track encoding=None API change
genshi/filters/tests/html.py - renamed to test_html.py. In Python 3 there is a top-level module named html and having this overridden when running "python3 genshi/filters/tests/__init__.py" or similar was incredibly frustrating. - left genshi/filters/html.py with its own name since genshi/filters/ doesn't typically end up in sys.path. :)
genshi/filters/i18n.py genshi/filters/tests/i18n.py - ugettext and friends are gone in Python 3 (and only gettext and friends exist and they now handle unicode). - Some \ line continuations inside doctests confused 2to3 so we removed them. - Testing picked up a problem (already present in trunk) where Translator.__call__ could end up defining gettext as an endlessly recursive function. Noted with a TODO.
genshi/filters/transform.py genshi/filters/tests/transform.py - minor changes to track encoding=None API change
genshi/template/astutil.py - AST for raise has changed in Python 3. - Python 3 adds AST nodes for individual arguments and Bytes.
genshi/template/base.py - distinguish between bytes and unicode in Python 3 compatible way.
genshi/template/directives.py genshi/template/tests/directives.py - slightly odd syntax changes to make the 2to3 .next() fixer pick up *stream.next(). - minor test fix for change in behaviour of division (/) in Python 3.
genshi/template/eval.py genshi/template/tests/eval.py - use genshi.compat functions for dealing with code objects. - replace doctests that reply on exception names with uglier but more compatible try:.. except:.. doctest - handle filename preferences of Python 2 and 3 (2 prefers bytes, 3 prefers unicode). - ifilter is gone from itertools in Python 3 so use repeat for tests instead.
A more minor change is that the ASTTransformer in Genshi coerces byte strings to unicode (in genshi/templates/eval.py -- visit_Str). I'm not sure I fully understand the rationale for this in Python 2 but in Python 3 it's definitely wrong. We've retained the old behaviour in Python 2 and left bytes as they are in Python 3.
genshi/template/loader.py genshi/template/tests/loader.py - add 'b' to file modes to ensure it's loaded as bytes in Python 3.
genshi/template/tests/markup.py - use BytesIO compatibility function in test
-On [20100905 15:58], Simon Cross (hodges...@gmail.com) wrote:
>Hi Jeroen (and others)
>The patch should apply cleanly against trunk. After applying one can >build, test and install under Python 2 using:
In the meantime I have found out that I don't have commit privileges to Genshi. I thought I had them (at some point), but it turns out I do not (anymore).
You will need to ask Pedro to apply any of the patches.
-- Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B In every colour there's the Light...
> It looks like the sprint was fairly successful and we have a patch > that updates Genshi to be Python 3 compatible (without losing Python 2 > compatibility). The patch is available at either of:
> Bitbucket has the full commit history. The sprint occurred in two > stages. First we worked on simply porting to Python 3 without > maintaining compatibility with Python 2. This work was done in the > 2to3 bitbucket branch and took about seven hours. In the second stage > we created a single code-base that supports Python 2 and Python 3 (via > 2to3 and some helpers). This was done in the default branch and took > about eight hours.
> We've tested the result under Python 2.4, 2.5, 2.6, 3.1 and 3.2 and on > Linux (Ubuntu Lucid and Maverick) and Mac OS X.
> API Change > ^^^^^^^^^^^
> We made one important API change. Previously the default encoding used > by Genshi in a number of functions was UTF-8. This is a little awkward > in Python 3 since strings are unicode (and it's generally bytes that > have to be explicitly asked for). We opted to change the default > encoding to None (i.e. use a unicode string). Although this breaks > compatibility, we think it eases the migration towards Python 3 (and > it made porting easier).
> Usage > ^^^^^^
> The patch should apply cleanly against trunk. After applying one can > build, test and install under Python 2 using:
> Running setup.py under Python 3 requires Distribute (which runs under > Python 3 and provides hooks for running 2to3).
> Patch Tricks > ^^^^^^^^^^^
> - "if not isinstance(s, unicode):" is converted by 2to3 while "if > isinstance(s, str)" is not. > - "u'foo'.encode('utf-8')" creates a byte string in UTF-8 both before > and after 2to3 while "'foo'" does not.
> Patch Details > ^^^^^^^^^^^^
> .hgignore > README > - Hg ignore file and note about the patch. No need to commit.
> setup.py > MANIFEST.in > - changes to make setup.py importable by both Python 2 and Python 3 > (without 2to3). > - include tests in build so that "python3 setup.py test" works. > - Distribute runs the tests on the build rather than on the source > since the source is not modified by 2to3. > - add Distribute 2to3 options when run with Python 3.
> doc/common/doctools.py > - change to an svn:external to make prints python3 compatible. > - doctools is import by setup.py so this needs to be importable by > python3 without running 2to3 on it.
> examples_to_py3k.sh > - script to run 2to3 on the python files in the examples folder.
> fixes/fix_unicode_in_strings.py > - We wrote a custom 2to3 fixer to fix unicode output strings inside > doctests and other string constants.
> genshi/_speedups.c > - ported using #ifdefs for Python 3.
> genshi/compat.py > - new home for cross-python-version compatibility functions
> genshi/core.py > genshi/tests/core.py > - change default encoding from UTF-8 to None (i.e. unicode strings) > - stringrepr no longer used inside __repr__ in Python 3 repr() > returns a string (i.e. unicode)
> genshi/input.py > genshi/tests/input.py > - change default encoding for HTML() to None. > - track changes to expat parser in Python 3 (mostly it accepts bytes > instead of strings).
> genshi/output.py > genshi/tests/output.py > - change default encoding for encode() to None.
> genshi/util.py > - move cross-Python-version compatibility fixes into genshi.compat
> genshi/filters/html.py > - minor changes to track encoding=None API change
> genshi/filters/tests/html.py > - renamed to test_html.py. In Python 3 there is a top-level module > named html and having this overridden when > running "python3 genshi/filters/tests/__init__.py" or similar was > incredibly frustrating. > - left genshi/filters/html.py with its own name since > genshi/filters/ doesn't typically end up in sys.path. :)
> genshi/filters/i18n.py > genshi/filters/tests/i18n.py > - ugettext and friends are gone in Python 3 (and only gettext and > friends exist and they now handle unicode). > - Some \ line continuations inside doctests confused 2to3 so we removed > them. > - Testing picked up a problem (already present in trunk) where > Translator.__call__ could end up defining gettext > as an endlessly recursive function. Noted with a TODO.
> genshi/filters/transform.py > genshi/filters/tests/transform.py > - minor changes to track encoding=None API change
> genshi/template/astutil.py > - AST for raise has changed in Python 3. > - Python 3 adds AST nodes for individual arguments and Bytes.
> genshi/template/base.py > - distinguish between bytes and unicode in Python 3 compatible way.
> genshi/template/directives.py > genshi/template/tests/directives.py > - slightly odd syntax changes to make the 2to3 .next() fixer pick > up *stream.next(). > - minor test fix for change in behaviour of division (/) in Python 3.
> genshi/template/eval.py > genshi/template/tests/eval.py > - use genshi.compat functions for dealing with code objects. > - replace doctests that reply on exception names with uglier but > more compatible try:.. except:.. doctest > - handle filename preferences of Python 2 and 3 (2 prefers bytes, 3 > prefers unicode). > - ifilter is gone from itertools in Python 3 so use repeat for tests > instead.
> A more minor change is that the ASTTransformer in Genshi coerces byte > strings to unicode (in genshi/templates/eval.py -- visit_Str). I'm not > sure I fully understand the rationale for this in Python 2 but in > Python 3 it's definitely wrong. We've retained the old behaviour in > Python 2 and left bytes as they are in Python 3.
> genshi/template/loader.py > genshi/template/tests/loader.py > - add 'b' to file modes to ensure it's loaded as bytes in Python 3.
> genshi/template/tests/markup.py > - use BytesIO compatibility function in test
> genshi/template/text.py > - use not isinstance(s, unicode) instead of isinstance(s, str)
> Schiavo > Simon
> -- > You received this message because you are subscribed to the Google Groups > "Genshi" group. > To post to this group, send email to genshi@googlegroups.com. > To unsubscribe from this group, send email to > genshi+unsubscribe@googlegroups.com<genshi%2Bunsubscribe@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/genshi?hl=en.
On Mon, Sep 6, 2010 at 11:13 AM, Joshua Rowley <jos...@vannelluna.com> wrote: > Brilliant - nice work guys. I'm going to give it a whirl asap.
Woot. Let us know how it goes. :)
> Will you be adding a release on the Genshi homepage and on pypi's py3k > module list?
I think the first step is to get the patch into trunk but I've added the "Programming Language :: Python :: 3" trove classifier to setup.py in the bitbucket branch.
On 9/6/2010 11:13 AM, Jeroen Ruigrok van der Werven wrote:
> -On [20100905 15:58], Simon Cross (hodges...@gmail.com) wrote: >> Hi Jeroen (and others)
>> The patch should apply cleanly against trunk. After applying one can >> build, test and install under Python 2 using: > In the meantime I have found out that I don't have commit privileges to > Genshi. I thought I had them (at some point), but it turns out I do not > (anymore).
You still have them, but they were only given on trunk. I've added you to sandboxers (hope Christopher doesn't mind), so that you could integrate that branch somewhere below /branches/experimental if you want, which I believe is the appropriate next step before Chris integrates it in trunk if he'd like to.
On Mon, Sep 6, 2010 at 11:41 AM, Christian Boos <cb...@neuf.fr> wrote: > You still have them, but they were only given on trunk. > I've added you to sandboxers (hope Christopher doesn't mind), so that > you could integrate that branch somewhere below /branches/experimental > if you want, which I believe is the appropriate next step before Chris > integrates it in trunk if he'd like to.
Cool. Thanks!
Would it be easier if one of us (i.e. the patch creators) got commit access to the experimental branch? Then we could apply any fixes or changes requested there ourselves without continually having to involve someone else? I already have commit access to the Bitten repository (username: hodgestar).
> On Mon, Sep 6, 2010 at 11:41 AM, Christian Boos<cb...@neuf.fr> wrote: >> You still have them, but they were only given on trunk. >> I've added you to sandboxers (hope Christopher doesn't mind), so that >> you could integrate that branch somewhere below /branches/experimental >> if you want, which I believe is the appropriate next step before Chris >> integrates it in trunk if he'd like to. > Cool. Thanks!
> Would it be easier if one of us (i.e. the patch creators) got commit > access to the experimental branch? Then we could apply any fixes or > changes requested there ourselves without continually having to > involve someone else? I already have commit access to the Bitten > repository (username: hodgestar).
Sure! You're also a sandboxer now.
Note also that as I co-administrator on edgewall.org, I'm only trying to facilitate collaboration on the projects there, but I don't want to step over the actual owners and maintainers of the projects.
So be sure to have a chat with Christopher at some point, to see if he's OK with the general idea of having such an experimental branch later to be included in trunk. I think that having the code "at hand" in the Genshi repository, in easily digestible chunks may only make this easier, so you could take this opportunity to fold some of the original changes in logical pieces, without the trials and errors, for easier review.
On Mon, Sep 6, 2010 at 12:17 PM, Christian Boos <cb...@neuf.fr> wrote: > Sure! You're also a sandboxer now.
Woot.
> Note also that as I co-administrator on edgewall.org, I'm only trying to > facilitate collaboration on the projects there, but I don't want to step > over the actual owners and maintainers of the projects.
Understood.
> So be sure to have a chat with Christopher at some point, to see if he's OK > with the general idea of having such an experimental branch later to be > included in trunk. I think that having the code "at hand" in the Genshi > repository, in easily digestible chunks may only make this easier, so you > could take this opportunity to fold some of the original changes in logical > pieces, without the trials and errors, for easier review.
I've been kind of hoping Christopher would see these emails and chime in. :)
Re-organizing the original work into themed commits sounds sensible but it would probably be best to get a basic acceptance of the approach (use of 2to3, API changes, etc) before putting the effort in (I guess it'd need a few hours from me at some point).
On Mon, Sep 6, 2010 at 12:26 PM, Simon Cross <hodges...@gmail.com> wrote: > I've been kind of hoping Christopher would see these emails and chime in. :)
Hmm. Nickserv on Freenode IRC says:
-NickServ- Information on cmlenz (account cmlenz): -NickServ- Registered : Aug 30 12:41:20 2004 (6 years, 1 week, 0 days, 21:48:14 ago) -NickServ- Last addr : ~cmlenz@2002:543a:3f0c:0:223:6cff:fe99:3440 -NickServ- Last seen : Apr 23 23:11:02 2010 (19 weeks, 2 days, 11:18:32 ago)
So Christopher hasn't been seen on IRC since the end of April.
> It looks like the sprint was fairly successful and we have a patch > that updates Genshi to be Python 3 compatible (without losing Python 2 > compatibility). The patch is available at either of:
> Bitbucket has the full commit history. The sprint occurred in two > stages. First we worked on simply porting to Python 3 without > maintaining compatibility with Python 2. This work was done in the > 2to3 bitbucket branch and took about seven hours. In the second stage > we created a single code-base that supports Python 2 and Python 3 (via > 2to3 and some helpers). This was done in the default branch and took > about eight hours.
Awesome work! I'm currently unhealthily swamped with work (and have been for way too long), so I've had no time for Genshi. This is just to let you know I'm still alive (kind of) and this work you've put in here is much appreciated. I'd be happy to give out more commit bits.
On Mon, Sep 6, 2010 at 3:08 PM, Christopher Lenz <cml...@gmail.com> wrote: > Awesome work! I'm currently unhealthily swamped with work (and have been for way > too long), so I've had no time for Genshi. This is just to let you know I'm still alive (kind > of) and this work you've put in here is much appreciated. I'd be happy to give out > more commit bits.
Thanks! And glad to hear you're still alive. :D
Since you're busy could you perhaps appoint / bless someone to look at the patch in your stead?
I'm happy to be given commit access and take on the responsibility of getting something committed to trunk if you'd like that (although I really think it would be better if someone with more Genshi development experience did it).
At the moment I envisage the process as:
0) Get general okay of approach taken in patch from eventual committer (and make changes as needed). 1) Try out patch with Trac under Python 2 (and fix any obvious problems). 2) Speak to Trac developers and get their buy-in on the patch (and make any changes needed). 3) Re-organize changesets and commit to experimental branch. 4) Poll user community for feedback for a few (say four) weeks. 5) Commit to trunk. 6) Fix any issues that arise from users who didn't participate in step 4. :)
I know from my own Genshi use that I pulled in Genshi trunk by default for awhile so we probably don't want to surprise people by landing straight to trunk in step 3 (although it would short circuit the process a bit).
On Mon, Sep 6, 2010 at 3:55 PM, Simon Cross <hodges...@gmail.com> wrote: > 0) Get general okay of approach taken in patch from eventual committer > (and make changes as needed).
I'm taking this as read for the moment and moving on. :)
> 1) Try out patch with Trac under Python 2 (and fix any obvious problems).
With the attached two-line patch to Trac (which I believe is backwards compatible with Genshi 0.6), Trac works with the genshi-py3k hg branch.
> 2) Speak to Trac developers and get their buy-in on the patch (and > make any changes needed).
> On Mon, Sep 6, 2010 at 3:55 PM, Simon Cross<hodges...@gmail.com> wrote: >> 0) Get general okay of approach taken in patch from eventual committer >> (and make changes as needed). > I'm taking this as read for the moment and moving on. :)
>> 1) Try out patch with Trac under Python 2 (and fix any obvious problems). > With the attached two-line patch to Trac (which I believe is backwards > compatible with Genshi 0.6), Trac works with the genshi-py3k hg > branch.
>> 2) Speak to Trac developers and get their buy-in on the patch (and >> make any changes needed). > This is next on the agenda. :)
No worries, the patch looks fine and will soon be in trunk ;-)
Now Babel is next on the list, for the people wanting to get Trac3k ;-)
On Sat, Oct 2, 2010 at 9:44 AM, Christian Boos <cb...@neuf.fr> wrote: > No worries, the patch looks fine and will soon be in trunk ;-)
Woot. Thanks!
> Now Babel is next on the list, for the people wanting to get Trac3k ;-)
My l10n foo is lacking but David Fraser on the other hand has had lots of experience with Pootle. Anyway, one step at a time for now (for me anyway). :)
> On Sat, Oct 2, 2010 at 9:44 AM, Christian Boos<cb...@neuf.fr> wrote: >> No worries, the patch looks fine and will soon be in trunk ;-) > Woot. Thanks!
I've committed it, and indeed this seems to work well. Congrats!
However, there are a couple issues related to the tests, e.g.
ERROR: Regression test related to #5795 ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Workspace\src\trac\repos\trunk\trac\mimeview\tests\patch.py", line 38, in setUp self.patch_html = Stream(list(HTMLParser(patch_html))) File "build\bdist.win32\egg\genshi\core.py", line 273, in _ensure event = stream.next() File "build\bdist.win32\egg\genshi\input.py", line 443, in _coalesce for kind, data, pos in chain(stream, [(None, None, None)]): File "build\bdist.win32\egg\genshi\input.py", line 335, in _generate raise UnicodeError("source returned bytes, but no encoding specified") UnicodeError: source returned bytes, but no encoding specified
And a couple more. Some seem to be specific to the tests themselves, but a couple of failures in the notification tests may actually correspond to real problems (in total, failures=1, errors=24 with 'make unit-test', and no problems at all with 'make functional-test').
I briefly had a look but didn't find obvious fixes, so you may want to help us there as well ;-)
> On Sat, Oct 2, 2010 at 1:08 PM, Christian Boos<cb...@neuf.fr> wrote: >> However, there are a couple issues related to the tests, e.g. > Patch for tests and issues revealed by tests attached.
Hm, just saw that I could have said in the commit message that this was a follow-up to r10168. Now that we have the hg and git mirrors I can't fix anymore the svn:log property after the fact... too bad.
I opted not to attempt to break the patch up into pieces that individually pass all their tests but instead broke the patch up thematically into seven commits based on the area of Genshi involved. I don't think it's particular useful to think of this patch as a series of steps -- there's really only one step (adding support for Python 3). I hope the thematic break down should allow interested people to follow what was happening to each part of Genshi without having to juggle lots of small changesets or searching through a single giant changeset.
On Tue, Oct 5, 2010 at 7:40 PM, Christian Boos <cb...@neuf.fr> wrote: > Great! Patch is in with a little bonus ;-)
Thanks! And thanks for the mention in the thanks file. :)
Could I ask one more commit favour? :) The attached patch upgrades doctools.py from https://svn.edgewall.org/repos/edgewall/tools/doc/doctools.py to work under both Python 2 and Python 3. Genshi imports it from its setup.py file so I can't rely on 2to3 to fix it. Could you commit the patch for me if you think it's okay?
On Mon, Oct 25, 2010 at 1:10 AM, Simon Cross <hodges...@gmail.com> wrote: > Could I ask one more commit favour? :) The attached patch upgrades > doctools.py from > https://svn.edgewall.org/repos/edgewall/tools/doc/doctools.py to work > under both Python 2 and Python 3. Genshi imports it from its setup.py > file so I can't rely on 2to3 to fix it. Could you commit the patch for > me if you think it's okay?
The other Simon (osimons) gave me commit access to edgewall/tools and I've committed the change. Genshi Py3k branch is now all set for hordes of users to descend on it. :)
