byte vs string issues with pyramid_viewgroup on python3

[Andreas Reuleaux]

I have ported a little pyramid app of mine to python3, almost
successfully, i. e. after some tweaking (using waitress etc,
won't go in the details here) it is already running.

But when it comes to rendering some pages, I get error msgs
concering byte vs string, see below. Those pages are assembled with
pyramid_viewgroup, which did install fine also for python3.

my app running on python2 had no particular problems (but see below: "what makes
things worse") when it came to rendering something like

<div tal:replace="structure provider('content')" />

omitting my configuration details here (but can provide them of course,
if needed), as I think they are fairly in line with the pyramid_viewgroup
docs.

Now, when running this same app on python3 I get:

...
File "/tmp/tmpdt09n6/page_936675d2dcac12f54a19c37066990109e30ddb34.py", line 341, in render
__cache_58307216 = getitem('provider')('content')
File "/home/reuleaux/eggs/pyramid_viewgroup-0.5-py3.2.egg/pyramid_viewgroup/__init__.py", line 37, in __call__
return render_view(self.context, self.request, name, secure)
File "/home/reuleaux/work/website/py3/pyramid/pyramid/view.py", line 139, in render_view
return ''.join(iterable)
TypeError: sequence item 0: expected str instance, bytes found

- Expression: "provider('content')"
- Filename: ... /reuleaux/work/website/py3/src/website/templates/page.pt
- Location: (85:29)
- Source: ... tal:replace="structure provider('content')" />
^^^^^^^^^^^^^^^^^^^
- Arguments: repeat: {...} (0)
renderer_name: templates/page.pt
req: <Request - at 0x3774f50>
Markup: <function Markup at 0x371a2f8>
request: <Request - at 0x3774f50>
provider: <Provider - at 0x37870d0>
renderer_info: <RendererHelper - at 0x3787110>
context: <NotFound - at 0x3787150>
escape: False
view: <NoneType - at 0x84bf40>


Any idea? I guess/hope that pyramid_viewgroup is just not all that
popular, and noone has actually tried running it on python3.

What makes things worse, is that my simple case

<div tal:replace="structure provider('content')" />

worked even on python 2 only for pages that had no umlauts in them,
but I am german, living in France and we have lots of umlauts here
in Europe, so my workaround in python 2 was to
use

<div tal:replace="structure Markup(provider('content'))" />

instead, with Markup() defined as

def Markup(s):
return s.decode('utf-8')

and provided for rendering with

return render_to_response(
'templates/page.pt',
{ 'provider': Provider(cntxt, req),
'Markup': Markup,
},
request=req,
)

Now in python3 not even the simple pages without umlauts render any
more, no matter if I use this Markup() or not. But of course, any
page with or without umlauts (templates written in utf-8) should render.


Any idea?

-Andreas

[Andreas Reuleaux]

OK, not having gotten any feedback, I guess my problem description was
to vague.

I have therefore prepared a little sample program, that should be easy
to extract and run from this e-mail: when running it with python2 I have
no problems, when running it with python3 I get the aforementioned byte
vs. string problems (also shown below once more).

While I roughly know, what has changed between python2 and python3
concerning strings/bytes, I am kind of stuck in this situation: I don't
know what to do in this case and I would hope that someone will try out
this sample program and point me in the right direction.

To make the installation experience easier, I have avoided to use to
many different python/configuration files: the whole program lives in
one file sample.py, which make use of a couple of templates.

To try out the program, you should therefore extract from this e-mail

* the file sample.py

* as well as the templates page.pt, home.pt, other.pt, and notfound.pt
(note that only page.pt is a complete html page, the others are
just html snippets)

steps to install/run the program on python2

create the environment
$ virtualenv foo
$ cd foo
$ ./bin/easy_install pyramid
$ ./bin/easy_install pyramid_viewgroup

copy the python script and the page templates to your foo env
$ cp .../simple.py .
$ cp .../page.pt .
$ cp .../home.pt .
$ cp .../other.pt .
$ cp .../notfound.pt .

adjust the path to the templates in simple.py, like
path=os.path.dirname('/path/to/my/foo/')

start the server
$ ./bin/python simple.py -port 8080

and have a look at some pages
http://myserver:8080/home
http://myserver:8080/other
http://myserver:8080/whatever - not found!
http://myserver:8080/ - default view

(see the explanations below, what's going on under the hood.)

--
now do the same with python3

create the environment
$ virtualenv-3.2 bar3
$ cd bar3
$ ./bin/easy_install-3.2 pyramid
$ ./bin/easy_install-3.2 pyramid_viewgroup

copy the simple.py script and the page templates as above,
and adjust the path to the templates

start the server with
$ ./bin/python3 simple.py -port 8080

and have a look at some pages as above.

While http://myserver:8080/, the default view renders fine,
as soon as I look at some of the other pages: home, other, whatever (not found)
I get

File "/home/reuleaux/tmp/bar3/lib/python3.2/site-packages/pyramid-1.3.2-py3.2.egg/pyramid/view.py", line 139, in render_view

return ''.join(iterable)
TypeError: sequence item 0: expected str instance, bytes found

- Expression: "provider('content')"

- Filename: None
- Location: (24:33)
- Source: ... l:replace="structure provider('content')">page content</div>

^^^^^^^^^^^^^^^^^^^
- Arguments: repeat: {...} (0)

renderer_name: /home/reuleaux/tmp/bar3/page.pt
req: <NoneType - at 0x84bf40>
request: <NoneType - at 0x84bf40>
provider: <Provider - at 0x28059d0>
renderer_info: <RendererHelper - at 0x2805a10>
context: <Home - at 0x2805a50>

escape: False
view: <NoneType - at 0x84bf40>

192.168.1.7 - - [21/Jul/2012 01:58:49] "GET /home HTTP/1.1" 500 59

As mentioned before: I am lost in this case, any help is appreciated.

Some explanations of what's going on in the python2 case /
how it is supposed to work:

While viewlets/content providers are not everyone's preferred style,
I have used them here to assemble some pages from building blocks
e. g. my home page looks just like this:

@implementer(IPage)
class Home(object):
content=HomeContent()
title='Home'

I. e. it is a page (implements IPage), and its contents is just
the home content.

Another page, Other, is very similar, but has of course some other
content:

@implementer(IPage)
class Other(object):
content=OtherContent()
title='Other'

I have used traversal to find those objects, e. g.

http://myserver:port/home

will render Home, .../other will render Other and so on

The views operate just on the interfaces IPage, IContent,
i. e. view_page can render any page, and will make use of
view_content to fill in / show the content provided for
this particular page.


Anyway, thanks for reading, and thanks in advance for trying out.


-Andreas


simple.py
--------------------------------------------------
from wsgiref.simple_server import make_server
from pyramid.config import Configurator
from pyramid.response import Response

from pyramid_viewgroup import Provider

from pyramid.renderers import render_to_response

import sys
import os

import argparse


opts=None


# path = os.path.dirname('/home/reuleaux/tmp/foo/')
path = os.path.dirname('/home/reuleaux/work/website/py3/')



from zope.interface import Interface, implementer


class IContent(Interface):
pass


class IPage(Interface):
pass


@implementer(IContent)
class HomeContent(object):
msg="hi from home"
tmplt=os.path.join(path, 'home.pt')


@implementer(IPage)
class Home(object):
content=HomeContent()
title='Home'


@implementer(IContent)
class OtherContent(object):
msg="hi from other"
tmplt=os.path.join(path, 'other.pt')


@implementer(IPage)
class Other(object):
content=OtherContent()
title='Other'


@implementer(IContent)
class NotFoundContent(object):
tmplt=os.path.join(path, 'notfound.pt')
msg="not found!"


@implementer(IPage)
class NotFound(object):
content=NotFoundContent()
title='not found'



app={
'home': Home,
'other': Other,
}


@implementer(IPage)
class App(object):
def __getitem__(self, name):
return app.get(name, NotFound)()


class Root(object):
def __init__(self, request):
self.request = request

def __getitem__(self, name=None):
if name in ['static',
]:
raise KeyError
else:
return App()[name]




def default_view(request):
return Response('Hello world!')





def content_view(request):
content=getattr(request.context, 'content', None)

if hasattr(content, 'tmplt'):
return render_to_response(content.tmplt,
{ },
)
else:
return Response('<span class="mark">content</span>')


def page_view(request):
context=request.context
provider=Provider(context, request)
return render_to_response(
os.path.join(path, 'page.pt'),
{
'provider': provider,
'context': context,
},
)




def parser():

parser=argparse.ArgumentParser()

parser.add_argument("-port",
'--port',
dest="port",
action="store",
default=6543,
help="port")

return parser


def run(args):

p=parser()
global opts
opts = p.parse_args(args)
# print(opts)

config = Configurator(root_factory=Root)


config.add_view('.default_view',)

config.add_view('.page_view',
context='.IPage',)


config.add_view('.content_view',
name='content',
context='.IPage',)


app = config.make_wsgi_app()
server = make_server('0.0.0.0', int(opts.port), app)
server.serve_forever()


if __name__ == '__main__':
run(sys.argv[1:])
--------------------------------------------------





page.pt
--------------------------------------------------
<?xml version="1.0" encoding="utf-8"?>

<html xmlns="http://www.w3.org/1999/xhtml"
xmlns:tal="http://xml.zope.org/namespaces/tal"
>
<head>
<title>${context.title}</title>

<style type="text/css"
media="screen">
.mark {
font-weight: bold;
font-style: italic;
}
</style>


</head>
<body>
<p>any page will have this structure</p>

<p>and here comes the content:</p>

<div tal:replace="structure provider('content')">page content</div>

<p>rest of the page...</p>

</body>
</html>
--------------------------------------------------




home.pt
--------------------------------------------------
<div
xmlns:tal="http://xml.zope.org/namespaces/tal"
tal:omit-tag=""
>


<p>
this is my <span class="mark">home page</span>...
</p>


</div>
--------------------------------------------------





other.pt
--------------------------------------------------
<div
xmlns:tal="http://xml.zope.org/namespaces/tal"
tal:omit-tag=""
>


<p>
this is my <span class="mark">other page</span>
</p>


</div>
--------------------------------------------------




notfound.pt
--------------------------------------------------
<div
xmlns:tal="http://xml.zope.org/namespaces/tal"
tal:omit-tag=""
>

Sorry, page not found

</div>
--------------------------------------------------

[Andreas Reuleaux]

OK, I have a solution: if I patch the function

render_view(context, request, name='', secure=True)

in my .../pyramid-1.3.2-py3.2.egg/pyramid/view.py file
by replacing

return ''.join(iterable)

at the end with the following if-then-else code

if isinstance(iterable[0], str):
return ''.join(iterable)
else:
return b''.join(iterable)

then everything works as expected, but I'd certainly not
want to do this forever / would very much appreciate, if
someone could have a closer look at the problem.

-Andreas

[Simon Yarde]

You'd probably want to walk the iterable and cast all elements to string? Then do your join operation with the empty unicode string. Your code only checks the first item and assumes all other elements are also strings - perhaps your code is working because there is only one element in the iterable for your circumstances.

I'm not familiar enough with this part of the Pyramid library to say more than this.. but something must be casting to bytes somewhere otherwise all strings would be defaulting to Unicode under Python 3.

> --
> You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
> To post to this group, send email to pylons-...@googlegroups.com.
> To unsubscribe from this group, send email to pylons-discus...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en.
>

[Andreas Reuleaux]

Well, yes you are right, seems to work in my case, because all entries
in the iterable are bytes (but haven't even bothered to see how many
entries there are in my iterable).

But the real thing is: I don't want to mess at all with
patching the pyramid source, I just want to use its api properly,
and if it doesn't work,

* then this could either be a flaw in my
program / usage of the pyramid api
- and I'd hope that someone would point me
in the right direction of how to do it better - maybe there is some
magic to use when working with python3?

* or there could be a flaw in the pyramid / pyramid_viewlet
source - I guess, that can only be answered by someone
familiar with the source

Anyway, thanks for answering.

-Andreas

[Andreas Reuleaux]

OK, I found a workaround/solution:

Instead of patching render_view() in
.../pyramid-1.3.2-py3.2.egg/pyramid/view.py I am overriding Providers'
__call__() method with my own my_render_view() function now like so:

from pyramid.view import render_view_to_iterable

# cf. pyramid-1.3.2-py3.2.egg/pyramid/view.py
def my_render_view(context, request, name, secure):
iterable = render_view_to_iterable(context, request, name, secure)
if iterable is None:
return None

# rx
# return ''.join(iterable)

if isinstance(iterable[0], str):
return ''.join(iterable)
else:
return b''.join(iterable)

from pyramid_viewgroup import Provider

class MyProvider(Provider):

def __call__(self, name='', secure=True):
return my_render_view(self.context, self.request, name, secure)

then I use this MyProvider (instead of just Provider formerly)
in page_view():


def page_view(request):
context=request.context
# provider=Provider(context, request)
provider=MyProvider(context, request)

return render_to_response(
os.path.join(path, 'page.pt'),
{
'provider': provider,
'context': context,
},
)

While there still might be a better / easier solution
- and I would like to here about it - this workaround
at least allows my to use pyramid_viegroup in the way
I used to in python2.x

My guess is still, that noone else has bothered to
try pyramid_viewgroup on python3.