"Header names must be latin1 string" AssertionError from WebTest

[Sean Hammond]

Hi,

I'm wondering if anyone has ideas for how to deal with a "Header names must be latin1 string" AssertionError from WebTest that occurs when Pyramid apps are run in Python 2 with `from __future__ import unicode_literals`? Or what people think of my tween-based workaround below?


## Problem

We're in the process of migrating our Pyramid app from Python 2 to 3, and added `from __future__ import unicode_literals` to all our Python files, and this caused the following error from WebTest in Python 2:

AssertionError: Header names must be latin1 string (not Py2 unicode or Py3 bytes type).

Here's a full traceback in case anyone's interested, though I don't think it's very enlightening: https://gist.github.com/seanh/1e8bee4476eca4d7f29e4c8d62f01171

The AssertionError is raised whenever your app sets any response header. For example this line would trigger it:

response.headers["Access-Control-Allow-Origin"] = "*"

Since we have `unicode_literals` those string literals are unicode strings in Python 2, whereas before `unicode_literals` they would have been byte strings, that's why adding `unicode_literals` triggered the AssertionError from WebTest.

In Python 3 no error occurs.

The reason WebTest has this AssertionError is that https://www.python.org/dev/peps/pep-3333 requires HTTP response headers to be native strings - byte strings in Python 2 and unicode strings in Python 3. Here's the WebTest issue and pull request that added the assert:

https://github.com/Pylons/webtest/issues/119
https://github.com/Pylons/webtest/pull/180


## Workarounds

b-prefixing the strings like response.headers[b"Access-Control-Allow-Origin"] = b"*" would get rid of the AssertionError in Python 2 but cause the error to appear if the tests were run in Python 3.

Wrapping the strings in str() like response.headers[str("Access-Control-Allow-Origin")] = str("*") will fix it in both Python 2 and 3, but requires you to find and str()-wrap every response header string throughout your app.

Adding a tween that str()-wraps all response headers seems to be a good fix:

def encode_headers_tween_factory(handler, registry):
def encode_headers_tween(request):
resp = handler(request)
for key in resp.headers.keys():
values = resp.headers.getall(key)
del resp.headers[key]
for value in values:
resp.headers.add(str(key), str(value))
return resp
return encode_headers_tween

config.add_tween('h.tweens.encode_headers_tween_factory')