TypeError: WSGI response header value u'PUT, GET' is not of type str.

[Andrew Back]

Hello,

I've put together a Docker container with TiddlyWebWiki running via twanager server and it seems to work up until the point of attempting to save a tiddler, which results in "Error saving New Tiddler: connection could not be established" and a WSGI error in the server logs, details of which can be found at the bottom of this message.

The GitHub repo is at:

https://github.com/abopen/tiddlywebwiki-docker

Docker Hub image is abopen/tiddlywebwiki.

Any thoughts as to what the problem might be would be much appreciated.

Cheers,

Andrew

//

WARNING:root:Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/httpexceptor/__init__.py", line 62, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/site-packages/tiddlyweb/web/wsgi.py", line 190, in __call__
    output = self.application(environ, start_response)
  File "/usr/local/lib/python2.7/site-packages/tiddlyweb/web/serve.py", line 161, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/site-packages/tiddlyweb/web/serve.py", line 117, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/site-packages/tiddlyweb/web/query.py", line 44, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/site-packages/tiddlyweb/web/wsgi.py", line 128, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/site-packages/tiddlyweb/web/extractor.py", line 34, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/site-packages/tiddlyweb/web/wsgi.py", line 38, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/site-packages/tiddlyweb/web/negotiate.py", line 30, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/site-packages/selector.py", line 137, in __call__
    return app(environ, start_response)
  File "/usr/local/lib/python2.7/site-packages/selector.py", line 23, in method_not_allowed
    ('Content-Type', 'text/plain')])
  File "/usr/local/lib/python2.7/site-packages/tiddlyweb/web/wsgi.py", line 147, in replacement_start_response
    return start_response(status, headers, exc_info)
  File "/usr/local/lib/python2.7/site-packages/tiddlyweb/web/wsgi.py", line 78, in replacement_start_response
    return start_response(status, headers, exc_info)
  File "/usr/local/lib/python2.7/site-packages/cheroot/wsgi.py", line 180, in start_response
    'WSGI response header value %r is not of type str.' % v)
TypeError: WSGI response header value u'PUT, GET' is not of type str.

[chris...@gmail.com]

On Sun, 30 Dec 2018, Andrew Back wrote:

> Hello,

Hi Andrew!

> I've put together a Docker container with TiddlyWebWiki running via
> twanager server and it seems to work up until the point of attempting to
> save a tiddler, which results in "Error saving New Tiddler: connection
> could not be established" and a WSGI error in the server logs, details of
> which can be found at the bottom of this message.

The root of the problem here is a combination of cheroot (the newer
cherrypy webserver used by the tiddlywebplugins.cherrypy package)
being strictly aligned with pep 3333 when it comes to headers (and
enforcing it) and selector (the URL routing library used by
tiddlyweb) decoding each line (to unicode) when it reads the
urls.map file.

The result of this is that when you make a request to tiddlyweb that
causes a 405 response, the value in the 'Allowed' header is being
sent, while in python 2.7, as unicode, not a str. According to PEP
3333, header values should always be whatever the native 'str' type
is. This was likely a horrible decision as it means the values need
to be different between Python 2 and 3.

If you have the option of using the python:3-alpine base for your
docker image, that will make the problem go away without needing to
change selector or cheroot. However you don't really have that
option until some changes are made:

* tiddlywebwiki hasn't been updated for Python 3, yet. I suppose I
should do that at some point but I wasn't aware of anyone using
it. Since you seem to be using it, that might be the push I need,
so that people don't need to use Python 2.

* tiddlywebplugins.urls also has some issues with not supporting
Python 3. For that you'll have to contact bengillies, probably via
an issue on https://github.com/bengillies/tiddlywebplugins.urls
If he not interested but wants to transfer the repo and pypi
ownership to the tiddlyweb org and me that would be fine.

Another option would be to fix selector so that its behavior is
better when sending headers. I've looked at this but I'm not sure it
is worth doing given it works in python3 and the limited lifetime
remaining on python2.

So until those things are fixed that leaves us with hacks. I've
tried this modification to the Docker file and it seems to work:

-=-=-
FROM python:2-alpine
MAINTAINER Andrew Back <and...@abopen.com>

VOLUME /data

RUN mkdir /usr/src/tww

COPY . /usr/src/tww
WORKDIR /usr/src/tww

RUN pip install -r requirements.txt

# hack up selector to remove line encoding
RUN sed -i'' -e '/line\.decode/d' $(python -c 'import sys; import selector; print(sys.modules["selector"].__file__)' | sed -e 's/c$//')

ENTRYPOINT ["/usr/src/tww/entrypoint.sh"]

EXPOSE 80
-=-=-

Note, however, that as done, this tiddlywiki this container presents
can only be accessed as http://0.0.0.0/ . If you try to use
localhost or a different port, the tiddlers won't save (because the
server.host field will be http://0.0.0.0/).

To use something different you'll need to create a
tiddlywebconfig.py with the values you want to create that url,
see: https://tank.peermore.com/tanks/tiddlyweb/server_host

Another potential fix is to use an older version of cherrypy, by
changing tiddlywebplugins.cherrypy and CherryPy in the
requiements.txt as follows:

-=-=-
setuptools
tiddlywebwiki
tiddlywebplugins.form
tiddlywebplugins.urls
tiddlywebplugins.cherrypy==0.1.1
CherryPy==8.9.1
-=-=-

(Usual caveats about using old stuff potentially having security
issues.)

But long term you should start thinking in terms of Python 3, so
I'll see about making tiddlywebwiki python3 compatible.

I suppose it might also be time to start considering TiddlyWiki 5,
which can talk to tiddlyweb without the need for tiddlywebwiki.

In any case, I hope some of the above is useful. And Happy New Year.

--
Chris Dent https://anticdent.org/
[...]

[chris...@gmail.com]

On Sun, 30 Dec 2018, chris...@gmail.com wrote:

> * tiddlywebwiki hasn't been updated for Python 3, yet. I suppose I
> should do that at some point but I wasn't aware of anyone using
> it. Since you seem to be using it, that might be the push I need,
> so that people don't need to use Python 2.

To follow up with a bit more info on this.

Getting this to go will be a bit of an undertaking. The testing
infrastructure with tiddlywebwiki is considerably more convoluted
and customized when compared to tiddlyweb, so getting my head back
around how to make it go and the many dependencies involved is
probably going to need some help from FND. I'm certain that if we
were to spend a bit of time on it, it would go quickly, but looking
at it by myself, right now, it's more than I want to think about.

In part this is because the testing depends on the twp.ibuilder and
twp.imaker modules, neither of which are python3-ready and the
dependency management for testing is not aligned with modern
expectations. Which is not really all that surprising given that the
first commit on the tiddlywebwiki repo was 'Thu Jul 2 19:03:23
2009'. Tiddlyweb itself started on 'Sat Feb 16 14:52:53 2008'. In
SVN.

So a few things need to happen here:

* People who would like tiddlywebwiki to support python3 should
speak up so that we can know there is demand.

* FND and I need to decide if we want to do some quick work to make
this go. I think this is a matter of hours, not days, if we do
things to a Chris standard. If we do them to a FND standard I
can't count high enough in any units.

[Andrew Back]

Hi Chris,

Firstly, Happy New Year!

Thanks for taking the time to look into this and for such a speedy and detailed response.

I am kind of surprised I was — seemingly at least — the first to discover the breakage, but maybe it is in fact pretty recent or others simply fixed by using older dependencies.

While it would be cool to be able to use TiddlyWebWiki on an ongoing basis, I don't want to create a lot of work for you and potentially FND also.

Cheers,

Andrew

[Ben Gillies]

Hi Chris

* tiddlywebplugins.urls also has some issues with not supporting

   Python 3. For that you'll have to contact bengillies, probably via
   an issue on https://github.com/bengillies/tiddlywebplugins.urls
   If he not interested but wants to transfer the repo and pypi
   ownership to the tiddlyweb org and me that would be fine.

I'm happy to update things if required, I'll keep an eye out here for any decisions.

Ben