escaped bytes in apache log messages

[ru...@yahoo.com]

Unable to write unicode log messages

I have a Flask app running with Apache and modwsgi with uses the Python
logging module to write log messages to stderr.  These messages sometimes
contain non-ascii text (Japanese specifically).  When run with the Flask
development server (no Apache, no modwsgi) the log messages are fine, eg:
  ERROR:root:test message: 日本語

When run under Apache/modwsgi, the message appearing in the Apache log
file is escaped bytes string:
  ERROR:root:test message: \xe6\x97\xa5\xe6\x9c\xac\xe8\xaa\x9e

I am using Ubuntu-18.04, apache2-2.4.29, libapache2-mod-wsgi-py3-4.5.17,
python3-3.6.7, flask-0.12.2, all installed from the usual Ubuntu
repositories.

What I have done:
- Lots of googling.  Similar problems seem to be old (eg python-2),
  involve unicode encode or decode exceptions (not a problem in my
  case) and the suggested solutions I tried don't fix my problem.
- Added following settings to the WSGIDaemonProcess line in the
  mod_wsgi config file:
     locale=en_US.UTF-8 lang=en_US.UTF-8
 - In /etc/apache2/envvars, uncommented line as directed:
     ## Uncomment the following line to use the system default locale instead:
     . /etc/default/locale
 - In /etc/apache2/envvars, added below the two lines above the lines:
     export LANG='en_US.UTF-8'
     export LC_ALL='en_US.UTF-8'
     export PYTHONIOENCODING=UTF-8
 - Printed out following values in the python app code just before
   the problem logging line:
     sys.filesystemencoding = utf-8
     sys.defaultencoding = utf-8
     type(sys.stderr) = <class '_io.TextIOWrapper'>, encoding = utf-8
     locale = ('en_US', 'UTF-8')
 - In the app, wrote the log message line directly to sys.stderr
   without going through the logging module.  The results were
   still escaped bytes rather then the correct encoded unicode text.

None of the above made any difference.  What now???

Thanks for any advice!

Addendum: here is a minimal test program.

======== test.py ========
import sys, logging
from flask import Flask
App = Flask(__name__)
L = logging.getLogger
logging.basicConfig()

@App.route('/')
def hello_world():
        L().error("test message: 日本語")
        return 'Hello, World!'

if __name__ == '__main__':
        App.run (debug=True)

======== test.wsgi ========
import sys, os
appdir, _ = os.path.split (os.path.abspath(__file__))
sys.path.insert (0, appdir)
from test import App as application

======== test.conf ========
WSGIDaemonProcess jmtest processes=2 threads=4 \
                        display-name=apache2-jmtest \
                        locale=en_US.UTF-8 lang=en_US.UTF-8
WSGIProcessGroup jmtest

WSGIScriptAlias /jmtest /home/.../jmtest/test.wsgi
<Directory /home/.../jmtest>
    Require all granted
    </Directory>

[Graham Dumpleton]

The string will be getting output correctly from Python. The problem is that the Apache logging system when passed any strings which contain characters outside of the ASCII character set, will convert them to encoded byte values. So that string is the escaped UTF-8 encoded value of the byte string.

If you need the original original strings, you should use the Python logging module and set it up to save log output to a separate file.

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/538fb660-7145-49d0-89da-8eaf7e1f694a%40googlegroups.com.

[ru...@yahoo.com]

OK, thanks.  I was kind of afraid of that but found it hard to accept that going into the third decade of the 21st century, a major software project like Apache would not be able to handle unicode in its log files.

To unsubscribe from this group and stop receiving emails from it, send an email to mod...@googlegroups.com.

[Graham Dumpleton]

But Apache is from last century. Once a popular system has been implemented in a certain way, it is hard to change that behaviour as you will break expectations of how it works. In this case likely would break a lot of log parsing software out there, so would have been left the way it always has been.

To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/1d2a375a-d191-4e73-b59a-6f6a106080be%40googlegroups.com.