LookupError: no codec search functions registered - expected behavior when loading WSGI script?

3,793 views
Skip to first unread message

Andreas Sommer

unread,
Mar 15, 2011, 6:45:28 AM3/15/11
to mod...@googlegroups.com
Hi,

I'm getting the exception

  File "/var/www/django-sites/351b488d-80bb-42f0-b1bb-927aa89a1d5c/wsgi_autogen.py", line 9, in <module>
    if os.path.exists(virtualenvDirectory + 'bin')
  File "/usr/lib/python2.6/genericpath.py", line 18, in exists
    st = os.stat(path)
LookupError: no codec search functions registered: can't find encoding


in my WSGI script (autogenerated by my program Site Deploy):

import os
import sys

sys.path.insert(0, u'/var/www/django-sites/351b488d-80bb-42f0-b1bb-927aa89a1d5c')

virtualenvDirectory = u'/var/www/django-sites/351b488d-80bb-42f0-b1bb-927aa89a1d5c/django_simple_todo_list/env/'
activateScriptFilename = ((virtualenvDirectory + 'bin/activate_this.py')
                          if os.path.exists(virtualenvDirectory + 'bin')
                          else (virtualenvDirectory + 'Scripts/activate_this.py'))
execfile(activateScriptFilename, dict(__file__ = activateScriptFilename))

os.environ['DJANGO_SETTINGS_MODULE'] = 'django_simple_todo_list.settings_site_deploy_absolute_url_compatibility'

import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()


Putting "import codecs" at the top makes the WSGI app work again. Now my question: Is it expected behavior of mod_wsgi that it does not register any codec search function? If not, consider this as bug report ;-)

Related old thread: http://www.mailinglistarchive.com/html/mod...@googlegroups.com/2010-01/msg00178.html

Andreas

Graham Dumpleton

unread,
Mar 15, 2011, 1:54:05 PM3/15/11
to mod...@googlegroups.com
On 15 March 2011 06:45, Andreas Sommer <andreas....@googlemail.com> wrote:
> Hi,
>
> I'm getting the exception
>
>   File
> "/var/www/django-sites/351b488d-80bb-42f0-b1bb-927aa89a1d5c/wsgi_autogen.py",
> line 9, in <module>
>     if os.path.exists(virtualenvDirectory + 'bin')
>   File "/usr/lib/python2.6/genericpath.py", line 18, in exists
>     st = os.stat(path)
> LookupError: no codec search functions registered: can't find encoding
>
> in my WSGI script (autogenerated by my program Site Deploy):
>
> import os
> import sys
>
> sys.path.insert(0,
> u'/var/www/django-sites/351b488d-80bb-42f0-b1bb-927aa89a1d5c')
>
> virtualenvDirectory =
> u'/var/www/django-sites/351b488d-80bb-42f0-b1bb-927aa89a1d5c/django_simple_todo_list/env/'

And if you don't use unicode strings in sys.path?

I have never heard of anyone doing that and didn't know you even could
do it with Python 2.X.

Graham

> activateScriptFilename = ((virtualenvDirectory + 'bin/activate_this.py')
>                           if os.path.exists(virtualenvDirectory + 'bin')
>                           else (virtualenvDirectory +
> 'Scripts/activate_this.py'))
> execfile(activateScriptFilename, dict(__file__ = activateScriptFilename))
>
> os.environ['DJANGO_SETTINGS_MODULE'] =
> 'django_simple_todo_list.settings_site_deploy_absolute_url_compatibility'
>
> import django.core.handlers.wsgi
> application = django.core.handlers.wsgi.WSGIHandler()
>
> Putting "import codecs" at the top makes the WSGI app work again. Now my
> question: Is it expected behavior of mod_wsgi that it does not register any
> codec search function? If not, consider this as bug report ;-)
>
> Related old thread:
> http://www.mailinglistarchive.com/html/mod...@googlegroups.com/2010-01/msg00178.html
>
> Andreas
>

> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To post to this group, send email to mod...@googlegroups.com.
> To unsubscribe from this group, send email to
> modwsgi+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/modwsgi?hl=en.
>

Andreas Sommer

unread,
Mar 15, 2011, 3:40:51 PM3/15/11
to mod...@googlegroups.com
On 15.03.2011 18:54, Graham Dumpleton wrote:
> On 15 March 2011 06:45, Andreas Sommer <andreas....@googlemail.com> wrote:
>> Hi,
>>
>> I'm getting the exception
>>
>> File
>> "/var/www/django-sites/351b488d-80bb-42f0-b1bb-927aa89a1d5c/wsgi_autogen.py",
>> line 9, in <module>
>> if os.path.exists(virtualenvDirectory + 'bin')
>> File "/usr/lib/python2.6/genericpath.py", line 18, in exists
>> st = os.stat(path)
>> LookupError: no codec search functions registered: can't find encoding
>>
>> in my WSGI script (autogenerated by my program Site Deploy):
>>
>> import os
>> import sys
>>
>> sys.path.insert(0,
>> u'/var/www/django-sites/351b488d-80bb-42f0-b1bb-927aa89a1d5c')
>>
>> virtualenvDirectory =
>> u'/var/www/django-sites/351b488d-80bb-42f0-b1bb-927aa89a1d5c/django_simple_todo_list/env/'
> And if you don't use unicode strings in sys.path?
>
> I have never heard of anyone doing that and didn't know you even could
> do it with Python 2.X.
>
> Graham
Sure, using byte strings instead works fine because then Python doesn't
have to decode Unicode to any filesystem encoding. But that is not a
solution for me... Unicode is the way to go. It's especially important
if you have special characters in some files and transfer them from one
filesystem to another. Try committing a file in GIT on Windows =
windows-1252, and then check it out on Linux = utf-8... GIT does not
support Unicode and thus f***s up filenames.

Just out of interest: It seems that mod_wsgi is Python 3 compatible -
wouldn't you run into the described codecs problem with 3.x if you use
Unicode strings?

Graham Dumpleton

unread,
Mar 15, 2011, 3:51:30 PM3/15/11
to mod...@googlegroups.com
The #python IRC channel says that yes, unicode in sys.path is okay and
should be converted based on file system encoding. That was qualified
by comment that 'it doesn't always work' as expected.

Given that you are using a virtualenv I would suspect that maybe
something is broken with that. That or mod_wsgi is running against
different Python installation than what it was built for.

Can you provide the result of doing the following tests:

http://code.google.com/p/modwsgi/wiki/CheckingYourInstallation#Python_Shared_Library
http://code.google.com/p/modwsgi/wiki/CheckingYourInstallation#Python_Installation_In_Use

Graham

Andreas Sommer

unread,
Mar 15, 2011, 4:30:33 PM3/15/11
to mod...@googlegroups.com
On 15.03.2011 20:51, Graham Dumpleton wrote:
> The #python IRC channel says that yes, unicode in sys.path is okay and
> should be converted based on file system encoding. That was qualified
> by comment that 'it doesn't always work' as expected.
>
> Given that you are using a virtualenv I would suspect that maybe
> something is broken with that. That or mod_wsgi is running against
> different Python installation than what it was built for.
No, both the Debian Python installation and the virtualenv are the same
interpreter version. You can actually comment out the virtualenv lines
from my example WSGI file and get the same codec problem. I assume that
mod_wsgi starts up the interpreter without importing the codecs module.
Checked this by putting the following at the very beginning of the WSGI
file:

import sys
with open("/tmp/wsgimodules", "wb") as f: print >>f, sys.modules

The outcome was a modules dict without the "codecs" module!

And for completeness the output of the tests you wanted me to run:

Without virtualenv activation:

sys.version = '2.6.6 (r266:84292, Dec 26 2010, 22:48:11) \n[GCC 4.4.5]'
sys.prefix = '/usr'

With virtualenv (notice Unicode prefix because I inserted that into the
sys.path):

sys.version = '2.6.6 (r266:84292, Dec 26 2010, 22:48:11) \n[GCC 4.4.5]'
sys.prefix =
u'/var/www/django-sites/351b488d-80bb-42f0-b1bb-927aa89a1d5c/django_simple_todo_list/env'

And linkage of mod_wsgi.so-2.6 which is the .so file used by my Apache
process:
linux-vdso.so.1 => (0x00007fff74806000)
libpython2.6.so.1.0 => /usr/lib/libpython2.6.so.1.0 (0x00007fbd474d0000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007fbd472b4000)
libdl.so.2 => /lib/libdl.so.2 (0x00007fbd470af000)
libutil.so.1 => /lib/libutil.so.1 (0x00007fbd46eac000)
libm.so.6 => /lib/libm.so.6 (0x00007fbd46c2a000)
libc.so.6 => /lib/libc.so.6 (0x00007fbd468c8000)
libssl.so.0.9.8 => /usr/lib/libssl.so.0.9.8 (0x00007fbd46673000)
libcrypto.so.0.9.8 => /usr/lib/libcrypto.so.0.9.8 (0x00007fbd462d2000)
libz.so.1 => /usr/lib/libz.so.1 (0x00007fbd460ba000)
/lib64/ld-linux-x86-64.so.2 (0x00007fbd47bb2000)

As said above, I think not importing the "codecs" module by default is
the problem in my eyes. So the question is should mod_wsgi always
include this? -- you mentioned that most users use byte strings and thus
don't need the module. I have no problem with the manual "import codecs"
workaround, but that should get documented somewhere in the wiki.

Joonas Lehtolahti

unread,
Mar 15, 2011, 4:45:26 PM3/15/11
to mod...@googlegroups.com
> As said above, I think not importing the "codecs" module by default is
> the problem in my eyes. So the question is should mod_wsgi always
> include this?

The problem is that codecs doesn't get imported. I have also that problem
in another project where Python is included into another software. Well,
personally I don't have the problem but other people using the same
software do have problem that "no codec lookup functions have been
registered". Normally these modules are automatically loaded when the
Python interpreter is initialized. For some yet unknown reason for some
people embedded Python interpreters fail to do it. We have yet to find out
what exactly causes the issue, but I had a theory.

Well, Python initializes the encodings package, which imports codecs
module. Now the interesting thing is that if any exception happens inside
this encodings package importing during Python initialization, that error
is just silently ignored. So in case for some reason importing the codecs
module fails during initialization, there will be no signs of the error
anywhere except that when a codec is needed you are faced with the error
that there are no lookup functions available.

Now, you could try to find out what the error actually is by wrapping the
whole contents of encodings/__init__.py into a try: except: construct and
dump the error to a file or something in the exception handler there. That
should, in my theory, reveal where the error exactly happens. And if it
does, please share the findings.

Graham Dumpleton

unread,
Mar 15, 2011, 4:48:00 PM3/15/11
to mod...@googlegroups.com
Are you running this in daemon mode? Is the application the only thing
running, be it embedded mode or daemon mode?

If you are running one application, add:

WSGIApplicationGroup %{GLOBAL}

This will force use of main interpreter, which presumably may have
already had codecs imported as part of main Python initialisation.

In other words, not working in sub interpreters perhaps as indicated
maybe by other email if seen in other embedded systems.

Graham

Graham Dumpleton

unread,
Mar 15, 2011, 5:22:16 PM3/15/11
to mod...@googlegroups.com
BTW, on the platform you are using, just so can compare with my MacOS
X system, can you write a hello world WSGI program which returns
pretty printed version of sys.modules.keys() and WSGI environ. Do this
for WSGIApplicationGroup set to '%{GLOBAL}' and for it set to 'xxx'.
Interested to see if see a difference in what modules are preloaded.

Graham

Andreas Sommer

unread,
Mar 16, 2011, 8:33:30 AM3/16/11
to mod...@googlegroups.com
Tried that. Each of my four Apache processes imports
"encodings/__init__.py" as soon as I restart Apache ("service apache2
restart"), and the import runs without any exceptions!

Here is the output of sys.modules.keys() from the WSGI script (sorted
alphabetically):

('UserDict', '__builtin__', '__main__', '_abcoll',
'_mod_wsgi_a9b03b1837e0ae52d38b86c3232eac95', '_warnings', 'abc',
'apache', 'copy_reg', 'errno', 'genericpath', 'linecache', 'mod_wsgi',
'os', 'os.path', 'posix', 'posixpath', 'pwd', 'signal', 'site',
'sitecustomize', 'stat', 'sys', 'types', 'warnings', 'zipimport')

No "codecs" module here obviously. What I don't understand: Why is the
"encodings" module missing here? As said above, mod_wsgi seems to load
the "encodings" module somewhere when starting inside an Apache process.
Does it create a new interpreter for executing the WSGI script?

Andreas Sommer

unread,
Mar 16, 2011, 8:56:39 AM3/16/11
to mod...@googlegroups.com
On 15.03.2011 22:22, Graham Dumpleton wrote:
BTW, on the platform you are using, just so can compare with my MacOS
X system, can you write a hello world WSGI program which returns
pretty printed version of sys.modules.keys() and WSGI environ. Do this
for WSGIApplicationGroup set to '%{GLOBAL}' and for it set to 'xxx'.
Interested to see if see a difference in what modules are preloaded.
I wrote the following simple WSGI file:

def application(environ, start_response):
    status = '200 OK'
    import sys
    from pprint import pformat
    output = "WSGI environ:\n%s\n\nsys.modules.keys() (sorted):\n%s\n" % (pformat(environ), pformat(sorted(sys.modules.keys())))
    response_headers = [('Content-type', 'text/plain'),
                        ('Content-Length', str(len(output)))]
    start_response(status, response_headers)
    return [output]


Here's the output (my VM is a Debian squeeze BTW):

WITH "WSGIApplicationGroup %{GLOBAL}"

WSGI environ:
{'DOCUMENT_ROOT': '/var/www',
 'GATEWAY_INTERFACE': 'CGI/1.1',
 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7',
 'HTTP_ACCEPT_ENCODING': 'gzip,deflate',
 'HTTP_ACCEPT_LANGUAGE': 'en-us,en;q=0.5',
 'HTTP_CACHE_CONTROL': 'max-age=0',
 'HTTP_CONNECTION': 'keep-alive',
 'HTTP_COOKIE': 'csrftoken=695f1b1ccc8923adb1e2d8bd5e19869b; sessionid=344ba6127ec7540c71afb2c4b3d6c97c',
 'HTTP_HOST': '127.0.0.1',
 'HTTP_KEEP_ALIVE': '300',
 'HTTP_USER_AGENT': 'Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.16) Gecko/20110107 Iceweasel/3.5.16 (like Firefox/3.5.16)',
 'PATH_INFO': '/login/',
 'PATH_TRANSLATED': '/var/www/login/',
 'QUERY_STRING': 'next=/todosite/todo/',
 'REMOTE_ADDR': '127.0.0.1',
 'REMOTE_PORT': '32914',
 'REQUEST_METHOD': 'GET',
 'REQUEST_URI': '/todosite/login/?next=/todosite/todo/',
 'SCRIPT_FILENAME': '/var/www/django-sites/351b488d-80bb-42f0-b1bb-927aa89a1d5c/wsgi_autogen.py',
 'SCRIPT_NAME': '/todosite',
 'SERVER_ADDR': '127.0.0.1',
 'SERVER_ADMIN': 'webmaster@localhost',
 'SERVER_NAME': '127.0.0.1',
 'SERVER_PORT': '80',
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'SERVER_SIGNATURE': '<address>Apache/2.2.16 (Debian) Server at 127.0.0.1 Port 80</address>\n',
 'SERVER_SOFTWARE': 'Apache/2.2.16 (Debian)',
 'mod_wsgi.application_group': '',
 'mod_wsgi.callable_object': 'application',
 'mod_wsgi.handler_script': '',
 'mod_wsgi.input_chunked': '0',
 'mod_wsgi.listener_host': '',
 'mod_wsgi.listener_port': '80',
 'mod_wsgi.process_group': 'somesite',
 'mod_wsgi.request_handler': 'wsgi-script',
 'mod_wsgi.script_reloading': '1',
 'mod_wsgi.version': (3, 3),
 'wsgi.errors': <mod_wsgi.Log object at 0x7fef136442b0>,
 'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7fef13694210>,
 'wsgi.input': <mod_wsgi.Input object at 0x7fef13644230>,
 'wsgi.multiprocess': True,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 1)}

sys.modules.keys() (sorted):
['UserDict',

 '__builtin__',
 '__main__',
 '_abcoll',
 '_codecs',
 '_collections',
 '_functools',
 '_mod_wsgi_a9b03b1837e0ae52d38b86c3232eac95',
 '_sre',

 '_warnings',
 'abc',
 'apache',
 'cStringIO',
 'codecs',
 'collections',
 'copy_reg',
 'encodings',
 'encodings.__builtin__',
 'encodings.aliases',
 'encodings.ascii',
 'encodings.codecs',
 'encodings.encodings',
 'encodings.os',
 'encodings.sys',
 'encodings.threading',
 'errno',
 'exceptions',
 'functools',
 'genericpath',
 'keyword',
 'linecache',
 'mod_wsgi',
 'operator',

 'os',
 'os.path',
 'posix',
 'posixpath',
 'pprint',
 'pwd',
 're',

 'signal',
 'site',
 'sitecustomize',
 'sre_compile',
 'sre_constants',
 'sre_parse',
 'stat',
 'sys',
 'thread',
 'threading',
 'time',
 'traceback',
 'types',
 'warnings',
 'zipimport']

WITH "WSGIApplicationGroup something"

WSGI environ:
{'DOCUMENT_ROOT': '/var/www',
 'GATEWAY_INTERFACE': 'CGI/1.1',
 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7',
 'HTTP_ACCEPT_ENCODING': 'gzip,deflate',
 'HTTP_ACCEPT_LANGUAGE': 'en-us,en;q=0.5',
 'HTTP_CACHE_CONTROL': 'max-age=0',
 'HTTP_CONNECTION': 'keep-alive',
 'HTTP_COOKIE': 'csrftoken=695f1b1ccc8923adb1e2d8bd5e19869b; sessionid=344ba6127ec7540c71afb2c4b3d6c97c',
 'HTTP_HOST': '127.0.0.1',
 'HTTP_KEEP_ALIVE': '300',
 'HTTP_USER_AGENT': 'Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.16) Gecko/20110107 Iceweasel/3.5.16 (like Firefox/3.5.16)',
 'PATH_INFO': '/login/',
 'PATH_TRANSLATED': '/var/www/login/',
 'QUERY_STRING': 'next=/todosite/todo/',
 'REMOTE_ADDR': '127.0.0.1',
 'REMOTE_PORT': '32915',
 'REQUEST_METHOD': 'GET',
 'REQUEST_URI': '/todosite/login/?next=/todosite/todo/',
 'SCRIPT_FILENAME': '/var/www/django-sites/351b488d-80bb-42f0-b1bb-927aa89a1d5c/wsgi_autogen.py',
 'SCRIPT_NAME': '/todosite',
 'SERVER_ADDR': '127.0.0.1',
 'SERVER_ADMIN': 'webmaster@localhost',
 'SERVER_NAME': '127.0.0.1',
 'SERVER_PORT': '80',
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'SERVER_SIGNATURE': '<address>Apache/2.2.16 (Debian) Server at 127.0.0.1 Port 80</address>\n',
 'SERVER_SOFTWARE': 'Apache/2.2.16 (Debian)',
 'mod_wsgi.application_group': 'something',
 'mod_wsgi.callable_object': 'application',
 'mod_wsgi.handler_script': '',
 'mod_wsgi.input_chunked': '0',
 'mod_wsgi.listener_host': '',
 'mod_wsgi.listener_port': '80',
 'mod_wsgi.process_group': 'somesite',
 'mod_wsgi.request_handler': 'wsgi-script',
 'mod_wsgi.script_reloading': '1',
 'mod_wsgi.version': (3, 3),
 'wsgi.errors': <mod_wsgi.Log object at 0x7fbb50d158b0>,
 'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7fbb50d0ddc8>,
 'wsgi.input': <mod_wsgi.Input object at 0x7fbb50d157f0>,
 'wsgi.multiprocess': True,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 1)}

sys.modules.keys() (sorted):
['UserDict',

 '__builtin__',
 '__main__',
 '_abcoll',
 '_mod_wsgi_a9b03b1837e0ae52d38b86c3232eac95',
 '_warnings',
 'abc',
 'apache',
 'cStringIO',

 'copy_reg',
 'errno',
 'genericpath',
 'linecache',
 'mod_wsgi',
 'os',
 'os.path',
 'posix',
 'posixpath',
 'pprint',

 'signal',
 'site',
 'sitecustomize',
 'stat',
 'sys',
 'types',
 'warnings',
 'zipimport']


On 15 March 2011 16:48, Graham Dumpleton <graham.d...@gmail.com> wrote:
Are you running this in daemon mode? Is the application the only thing
running, be it embedded mode or daemon mode?

If you are running one application, add:

 WSGIApplicationGroup %{GLOBAL}

This will force use of main interpreter, which presumably may have
already had codecs imported as part of main Python initialisation.

In other words, not working in sub interpreters perhaps as indicated
maybe by other email if seen in other embedded systems.

Tried that setting and you were right, it does indeed work. My application is the only thing running (testing VM), and embedded/daemon mode does not make any difference - actually I was using daemon mode.

So that means the problem is the interpreter initialization in mod_wsgi (if the global interpreter is not used)?!

Joonas Lehtolahti

unread,
Mar 16, 2011, 11:37:04 AM3/16/11
to mod...@googlegroups.com
[cut out the long post]

Interesting, it seems like there really happens a difference between
initialization of sub-interpreters to the main one. But was there no
errors importing encodings in the sub-interpreters either, or did it not
even try to import the encodings package at all?

For the record, I have all the encodings and stuff in all my Python
sub-interpreters and none of my pages are using global application group,
so it is not a generic problem that happens for everyone; just like with
EventScripts (the other project embedding Python I mentioned earlier) it
works fine for most people but fails to include the codecs for some people.

It's too bad the error doesn't happen to me. If it did, I would surely dig
as deep as would be needed (even to edit Python's source code to add
additional debug messages) to find out what exactly causes it. I don't
know the internal structure of Python enough to be able to provide such
edits for remote debugging, though. But this is certainly an interesting
problem and not just mod_wsgi specific. If we can find out the cause and
solution for it, that would help not only your case but also plenty of
people using other embedded Python interpreters. (You might see that my
motivation for actively trying to participate in this particular problem
is mostly because of the similar problem in ES and there is a major pain
that has been reported to happen to more than just one user)

Graham Dumpleton

unread,
Mar 16, 2011, 12:32:16 PM3/16/11
to mod...@googlegroups.com
Am about to get on a couple of planes and will see if I can duplicate
it on the Mac and get a small test case together that can log Python
bug report for if needed. I wander if it is something more related to
virtualenv only. Will be important to see if can duplicate it without
virtualenv involved. Can't remember if you sent a standalone test
case. Need to look through discussion properly.

Graham

Andreas Sommer

unread,
Mar 16, 2011, 1:17:06 PM3/16/11
to mod...@googlegroups.com
You can take the WSGI script from my first post as test case. I said before that the problem has nothing to do with the virtualenv, you can just remove those few lines and get the same problem just because as soon as an import happens, Python encodes the Unicode entry from sys.path into the filesystem encoding - and since no codec search function is registered, the LookupError is raised. Here is a minimal WSGI test case (using Django import):

import sys
sys.path.insert(0, u'/tmp') # any Unicode will do
import django.core.handlers.wsgi

def application(environ, start_response):
    response_headers = [("Content-Type", "text/plain"),
                        ("Content-Length", "7")]
    start_response("200 OK", response_headers)
    return ["Worked."]


This will raise LookupError when trying to import the Django submodule. Strangely, replacing it with "import time" (which is not in sys.modules at that place!) works without problems, even though I put the Unicode entry at the very beginning of sys.path...

And again my configuration: Debian squeeze, Apache 2.2, Django 1.2.5 installed with aptitude. Let's see if you can reproduce the same on Mac.

Andreas Sommer

unread,
Mar 16, 2011, 1:21:48 PM3/16/11
to mod...@googlegroups.com
It did not even try to import them. You can see the sys.modules state of the WSGI script in my previous response (no "encodings" and "codecs" modules imported) and I also verified that with file-write debugging in the "encodings" module - it really does not get loaded in the subinterpreters.

Graham Dumpleton

unread,
Mar 17, 2011, 6:39:46 PM3/17/11
to mod...@googlegroups.com
My first attempts to reproduce this in MacOS X didn't work.

What I did do though is modify mod_wsgi C code to do stuff that would
force loading of encoding/codes modules for sub interpreters. Ie.,
added:

Index: mod_wsgi.c
===================================================================
--- mod_wsgi.c (revision 1755)
+++ mod_wsgi.c (working copy)
@@ -4393,6 +4393,17 @@
}

/*
+ * Force loading of codecs into interpreter. This has to be
+ * done as not otherwise done in sub interpreters and if not
+ * done, code running in sub interpreters can fail on some
+ * platforms if a unicode string is added in sys.path and an
+ * import then done.
+ */
+
+ item = PyCodec_Encoder("ascii");
+ Py_XDECREF(item);
+
+ /*
* If running in daemon process, override as appropriate
* the USER, USERNAME or LOGNAME environment variables
* so that they match the user that the process is running

If you were eager to try this patch and see if it fixes issue then
patch will likely not apply properly as mod_wsgi code for 4.0 has
moved around a fair bit and line numbers will be way off. The two
lines though are added to newInterpreterObject() just before
USER/USERNAME/LOGNAME are setup for daemon mode processes.

Graham

Andreas Sommer

unread,
Mar 18, 2011, 5:01:34 AM3/18/11
to mod...@googlegroups.com
Tried against current mod_wsgi sources in Debian (v3.3). This actually
solves the issue - well done!

Best regards,
Andreas

Reply all
Reply to author
Forward
0 new messages