Encoding problem with a web2py application that implements an XMLRPC webservice

[Lisandro]

I need to understand if this is a bug or if it is the expected behaviour and I'm doing something wrong.

I have two web2py applications. One of them implements an XMLRPC webservice, and the other one consumes it.

All the controllers (in both applications) have this first line:

# -*- coding: utf-8 -*-

One of the applications connects to the webservice to call a specific function, and it needs to pass some string arguments:

def make_the_call():
    from xmlrpclib import ServerProxy
    service = ServerProxy(webservice_url)
    data = {
        'title': 'áéíóú',
        'detail': 'Detail with special characters like ñ or Ç'
    }
    service.add_content(data)

The application that implements the webservice:

from gluon.tools import Service

service = Service()


def call():
    return service()


def add_content(data):
    db.content.insert(
        title=data.get('title'),
        detail=data.get('detail')
    )
    return {'success': True}

But the sentence service.add_content(data) fails with this error and traceback:

Traceback (most recent call last):
  File "/home/gonguinguen/medios/gluon/restricted.py", line 219, in restricted
    exec(ccode, environment)
  File "/home/gonguinguen/medios/applications/webmedios/controllers/admin.py", line 707, in <module>
  File "/home/gonguinguen/medios/gluon/globals.py", line 421, in <lambda>
    self._caller = lambda f: f()
  File "/home/gonguinguen/medios/applications/webmedios/controllers/admin.py", line 704, in test
    admin_password_sitio='93c824d1-91c4-428f-8542-db5db9d4594b')
  File "/home/gonguinguen/medios/applications/webmedios/controllers/admin.py", line 695, in instalar_demo_contenido
    r = server.add_content(data)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1306, in single_request
    return self.parse_response(response)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1482, in parse_response
    return u.close()
  File "/usr/lib64/python2.7/xmlrpclib.py", line 794, in close
    raise Fault(**self._stack[0])
Fault: <Fault 1: "<type 'exceptions.UnicodeEncodeError'>:'ascii' codec can't encode character u'\\xe1' in position 1: ordinal not in range(128)">

To make it work, I need to reload sys and set default encoding in the add_content() method of the webservice:

def add_content(data):
    import sys
    reload(sys)
    sys.setdefaultencoding('utf8')
    db.content.insert(
        title=data.get('title'),
        detail=data.get('detail')
    )
    return {'success': True}

After that, it works ok.

But here is the weird part: after having made one successfull call to the webservice method, I can remove the lines where I reload the sys, and it keeps working ok. But if I reload uwsgi, it starts throwing error again.

Anyway, I've read that reloading sys is not a good practise at all. So I'm pretty lost. I presumed that I could add "# -*- coding: utf-8 -*-" and forget about the encoding problems. 

What should I do?

[Dave S]

On Thursday, July 4, 2019 at 2:04:49 PM UTC-7, Lisandro wrote:

 I presumed that I could add "# -*- coding: utf-8 -*-" and forget about the encoding problems. 

What should I do?

I believe that line only specifies that the source code is in utf-8.

 

You probably need to use a codec, which is covered in

<URL:https://docs.python.org/2/library/codecs.html#encodings-and-unicode>

My own examples aren't on this computer, but there are some in gluon/...

/dps

[Alfonso Serra]

sys.setdefaultencoding will work thorough the server execution process. So as long it is alive you wont need to call it several times.

It will probably work if you set that up on routes.py or web2py.py so its not called per request.

As long as it fixes your problem i wouldnt care whether it is a bad practice. Python 2.7 by default encodes in ascii so there is not a workaround other than changing the default encoding.

I believe it fixes encoding issues from strings coming from your app, but you will have to encode/decode at some point f you are uploading text files from the browser.

[Lisandro]

Thank you for your suggestions.

As I'm not uploading text files from the browser, I don't need to encode/decode, I just need to set default encoding to utf8. 

Considering that, I've gone ahead with Alfonso's suggestion and I've moved sys.setdefaultencoding('utf8') inside my routes.py. 

It's working smoothly :)

Thanks again!

Best regards,

Lisandro.