web.py and python 3

[Tomas Schertel]

I saw a few posts talking about web.py run with python 3 and noticed one that said there's no reason in porting web.py to python 3 if wsgi layers still runs with python 2.
Why don't make web.py able to run on both python, 2.X and 3.x?
Bottle and Cherrypy does that.
Flup already has a branch for python 3.

[Michael Diamond]

I'm not sure if the developers have any sort of Python 3 comparability roadmap (I didn't see anything on the issue tracker, and only this note https://bugs.launchpad.net/webpy/+bug/277266 from 2008) but I agree, I think it should be possible to enable web.py to support both.

I just discovered web.py in the last week, and I'm totally hooked!  It's exactly what I imagined wanting out of a Python web-framework.  I have a Python project I've been working on, that I'd love to have hook into web.py, but it's Python 3 :(  So I spent a few hours today trying to figure out what needed to be done to get it working, and it doesn't seem all that bad.

1. Pull in CherryPy's updated web server code, which is now 2/3 compatible.

2. Run python -3:

      I ran this on a small test setup and saw the following:

$ python -3 code.py

http://0.0.0.0:8080/

/usr/lib/python2.6/site-packages/web/template.py:925: DeprecationWarning: the compiler package has been removed in Python 3.0

  import compiler

/usr/lib/python2.6/site-packages/markdown/blockprocessors.py:163: DeprecationWarning: classic int division

  indent_level = len(m.group(1))/markdown.TAB_LENGTH

127.0.0.1:57576 - - [22/Jul/2012 01:03:37] "HTTP/1.1 GET /Welcome" - 200 OK

127.0.0.1:57576 - - [22/Jul/2012 01:03:38] "HTTP/1.1 GET /" - 200 OK

    This seems pretty promising to me.  I think the compiler error can be resolved by switching to http://docs.python.org/library/ast.html#ast.parse but even if it's more complicated than that, the import is in a try block, and fails reasonably gracefully.  It looks like Jython supports AST as well, which would make this an even better switch: http://www.jython.org/docs/library/ast.html .  The markdown error is of no concern, since Python Markdown has a 3.0 compatible version available: http://freewisdom.org/projects/python-markdown/News

3. Run 2to3:

    2to3 runs successfully, and by and large cleans things up well.  It runs into a fair bit of trouble with web.py's unicode handling, such as utils.safestr and utils.safeunicode, generating this diff of safestr:

@@ -364,12 +364,12 @@

         >>> safestr(2)

         '2'

     """

-    if isinstance(obj, unicode):

+    if isinstance(obj, str):

         return obj.encode(encoding)

     elif isinstance(obj, str):

         return obj

     elif hasattr(obj, 'next'): # iterator

-        return itertools.imap(safestr, obj)

+        return map(safestr, obj)

     else:

         return str(obj)

    You'll notice that it replaces unicode with str, breaking the method's expected behavior.  I'm not sure if it makes more sense to define a custom 2to3 fixer, or figure out a better way to define / compartmentalize this code, but this will need to be cleaned up.

4. Fix runtime exceptions:

    Running 2to3 isn't enough to kick of the server though, I got a handful of stack traces that I was able to correct with the following patch:

Series of fixes to resolve stack traces in 3.2.

Fixed bad reference to __builtin__: http://pydev.blogspot.com/2008/11/making-code-work-in-python-2-and-3.html #5

Changed UserDict.DictMixin to collections.MutableMapping per http://bugs.python.org/issue2876 but this may not be enough: http://bugs.python.org/issue7975

Removed 'object' parent from TemplateResult: http://stackoverflow.com/questions/3003053/metaclass-multiple-inheritance-inconsistency

Removed 'exceptions' import

---- Enough to start the server

Removed str decode('UTF-8') call

diff -r 1b5bb3b5399b -r c823902ee50f application.py

--- a/application.py Sat Jul 21 23:27:20 2012 -0400

+++ b/application.py Sat Jul 21 23:37:09 2012 -0400

@@ -15,7 +15,6 @@

 import itertools

 import os

 import types

-from exceptions import SystemExit

 

 try:

     import wsgiref.handlers

@@ -374,12 +373,6 @@

 

         ctx.fullpath = ctx.path + ctx.query

         

-        for k, v in ctx.items():

-            # convert all string values to unicode values and replace 

-            # malformed data with a suitable replacement marker.

-            if isinstance(v, str):

-                ctx[k] = v.decode('utf-8', 'replace') 

-

         # status must always be str

         ctx.status = '200 OK'

         

diff -r 1b5bb3b5399b -r c823902ee50f template.py

--- a/template.py Sat Jul 21 23:27:20 2012 -0400

+++ b/template.py Sat Jul 21 23:37:09 2012 -0400

@@ -40,7 +40,7 @@

 import sys

 import glob

 import re

-from UserDict import DictMixin

+from collections import MutableMapping

 import warnings

 

 from .utils import storage, safeunicode, safestr, re_compile

@@ -721,8 +721,8 @@

     "__import__", # some c-libraries like datetime requires __import__ to present in the namespace

 ]

 

-import builtins

-TEMPLATE_BUILTINS = dict([(name, getattr(__builtin__, name)) for name in TEMPLATE_BUILTIN_NAMES if name in builtins.__dict__])

+import builtins as __builtin__

+TEMPLATE_BUILTINS = dict([(name, getattr(__builtin__, name)) for name in TEMPLATE_BUILTIN_NAMES if name in __builtin__.__dict__])

 

 class ForLoop:

     """

@@ -1209,7 +1209,7 @@

         e = SecurityError("%s:%d - execution of '%s' statements is denied" % (self.filename, lineno, nodename))

         self.errors.append(e)

 

-class TemplateResult(object, DictMixin):

+class TemplateResult(MutableMapping):

     """Dictionary like object for storing template output.

     

     The result of a template execution is usally a string, but sometimes it

diff -r 1b5bb3b5399b -r c823902ee50f utils.py

--- a/utils.py Sat Jul 21 23:27:20 2012 -0400

+++ b/utils.py Sat Jul 21 23:37:09 2012 -0400

@@ -259,7 +259,7 @@

 counter = Counter

 

 iters = [list, tuple]

-import builtins

+import builtins as __builtin__

 if hasattr(__builtin__, 'set'):

     iters.append(set)

 if hasattr(__builtin__, 'frozenset'):

    This was enough to start a server, but there are likely more robust ways of solving these exceptions.

5. Update test suite

    I saw 4 failures running the test suite locally on Python 2.7, and 41 failures and 12 errors on 3.2.  Most of these were just encoding issues, like:

File ".\web\utils.py", line 337, in web.utils.safeunicode

Failed example:

    safeunicode('hello')

Expected:

    u'hello'

Got:

    'hello' 

    But the tests will need to be updated and expanded to ensure no regressions.

Needless to say, it looks to me like there's still work to be done, but it's not impossible.  To the maintainers, do you have any plans to move towards Python 3?  I would be happy to help make this possible.

Michael

[Tomas Schertel]

Very nice Michael.
I hope web.py changes to work with python 3.

[Michael Diamond]

Pulling this conversation back to the Python 3 thread Thomas started, didn't mean to hijack my own patch thread with a Python 3 discussion.

On Thu, Aug 2, 2012 at 1:56 PM, Anand Chitipothu <anand...@gmail.com> wrote:

>> I posted some details of my initial thoughts on the 22nd, but two immediate
>> action items were to update the wsgiserver code from CherryPy as that is now
>> Python 2/3 compatible, and to identify how to best upgrade to Python 3's
>> unicode handling (moving encoding/decoding to a python 2 only module,
>> perhaps) since that was the biggest issue I ran into with my first attempt.
>
> I think we should start depending on cherrypy-wsgiserver instead of
> shipping a copy with web.py. When we started using it, it was not
> available on pypi.
>
> cherrypy-wsgiserver is not available for Python3, at least on pypi. We
> should provide alternate implementation of dev server using wsgiref.

Personally, I disagree - I hate having to go out and grab (and build, install, etc.) additional projects when I want to try out some new tool, I loved that I was able to drop the web.py codebase onto my machine and get everything up and running in a matter of moments, without any hassle.  Especially given that the cherrypy code isn't intended to be used in production, just for development and testing, it seems doubly unhelpful to pull it out into a dependency.

The current wsgiserver code on CherryPy's source repository is Python 2/3 compatible, handled by a version check in the package's __init__.py: https://bitbucket.org/cherrypy/cherrypy/src/b0c48210b250/cherrypy/wsgiserver/__init__.py

It would be a fairly safe and very simple change to upgrade web.py's wsgiserver directory with this code.  Even if you'd prefer ultimately to move to a dependency model, that's a serious, and backwards-incompatible change, where upgrading is reasonably trivial.  I'd suggest solving the 2/3 problem first, then outsourcing the CherryPy dependencies later, if that's really valuable for the project.

One of the burdens web.py carries is python2.3 compatibility. When I
last heard, Python 2.3 was still supported on Redhat servers. It might
have changed now. Anybody knows what is the status now?

Thanks for the input Zhang.  Personally, I have no objection to dropping 2.3 support (or for that matter, all of 2!), but I don't really think that's the proper thing to do.  Like above, backwards-incompatible changes are a dangerous design decision, and if we can in any way implement Python 3 support without breaking existing support, that would be ideal.  CherryPy still supports 2.3, according to the documentation in this file: https://bitbucket.org/cherrypy/cherrypy/src/b0c48210b250/cherrypy/_cpcompat.py which also details how they handled the string encoding issues I described earlier, so I think it should be possible.

I hadn't actually spotted that file before googling to see if CherryPy supported 2.3, and coincidentally (or not) the page I found was their solution to the biggest sticking point I'd hit thus far.  I'll try to play with applying this methodology to some of web.py when I have a chance.

From their source code:

Python 2 uses str and '' for byte strings, while Python 3
uses str and '' for unicode strings. We will call each of these the 'native
string' type for each version. Because of this major difference, this module
provides new 'bytestr', 'unicodestr', and 'nativestr' attributes, as well as
two functions: 'ntob', which translates native strings (of type 'str') into
byte strings regardless of Python version, and 'ntou', which translates native
strings to unicode strings.

Michael 

[Tomas Schertel]

Hi guys...
Anyone thinking on porting web.py to python 3??
There are a plenty of python 3 compatible wsgi servers on street. Isn't time to make web.py support python 3?
Michael, how about you request a pull with your version?

[CEBILE ELLEN SKHOSANA]

Hey guys I'm a software engineer at Socialfeet just wana let you know that Socialfeet is developing web.py into a much artistic philosophical way. They releasing the new version of web.py next month with advances much greater than Django or any other framework. The new version will include the engineering documentation and the programming documentation. They are focusing their engineers into building web.py. While the ease of web.py remains same, the advances are much great. The new version will come with its own web.py IDE and database tools. Don't freak that webpy about the big changes they just easy adaptable. The IDE will having the ability to edit the templates and code at a webpy style.

[CEBILE ELLEN SKHOSANA]

[CEBILE ELLEN SKHOSANA]

[CEBILE ELLEN SKHOSANA]

[Tomas Schertel]

Sorry, but can you explain it better?

Is this Socialfeet version a fork of web.py?