And if you don't use unicode strings in sys.path?
I have never heard of anyone doing that and didn't know you even could
do it with Python 2.X.
Graham
> activateScriptFilename = ((virtualenvDirectory + 'bin/activate_this.py')
> if os.path.exists(virtualenvDirectory + 'bin')
> else (virtualenvDirectory +
> 'Scripts/activate_this.py'))
> execfile(activateScriptFilename, dict(__file__ = activateScriptFilename))
>
> os.environ['DJANGO_SETTINGS_MODULE'] =
> 'django_simple_todo_list.settings_site_deploy_absolute_url_compatibility'
>
> import django.core.handlers.wsgi
> application = django.core.handlers.wsgi.WSGIHandler()
>
> Putting "import codecs" at the top makes the WSGI app work again. Now my
> question: Is it expected behavior of mod_wsgi that it does not register any
> codec search function? If not, consider this as bug report ;-)
>
> Related old thread:
> http://www.mailinglistarchive.com/html/mod...@googlegroups.com/2010-01/msg00178.html
>
> Andreas
>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To post to this group, send email to mod...@googlegroups.com.
> To unsubscribe from this group, send email to
> modwsgi+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/modwsgi?hl=en.
>
Just out of interest: It seems that mod_wsgi is Python 3 compatible -
wouldn't you run into the described codecs problem with 3.x if you use
Unicode strings?
Given that you are using a virtualenv I would suspect that maybe
something is broken with that. That or mod_wsgi is running against
different Python installation than what it was built for.
Can you provide the result of doing the following tests:
http://code.google.com/p/modwsgi/wiki/CheckingYourInstallation#Python_Shared_Library
http://code.google.com/p/modwsgi/wiki/CheckingYourInstallation#Python_Installation_In_Use
Graham
import sys
with open("/tmp/wsgimodules", "wb") as f: print >>f, sys.modules
The outcome was a modules dict without the "codecs" module!
And for completeness the output of the tests you wanted me to run:
Without virtualenv activation:
sys.version = '2.6.6 (r266:84292, Dec 26 2010, 22:48:11) \n[GCC 4.4.5]'
sys.prefix = '/usr'
With virtualenv (notice Unicode prefix because I inserted that into the
sys.path):
sys.version = '2.6.6 (r266:84292, Dec 26 2010, 22:48:11) \n[GCC 4.4.5]'
sys.prefix =
u'/var/www/django-sites/351b488d-80bb-42f0-b1bb-927aa89a1d5c/django_simple_todo_list/env'
And linkage of mod_wsgi.so-2.6 which is the .so file used by my Apache
process:
linux-vdso.so.1 => (0x00007fff74806000)
libpython2.6.so.1.0 => /usr/lib/libpython2.6.so.1.0 (0x00007fbd474d0000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007fbd472b4000)
libdl.so.2 => /lib/libdl.so.2 (0x00007fbd470af000)
libutil.so.1 => /lib/libutil.so.1 (0x00007fbd46eac000)
libm.so.6 => /lib/libm.so.6 (0x00007fbd46c2a000)
libc.so.6 => /lib/libc.so.6 (0x00007fbd468c8000)
libssl.so.0.9.8 => /usr/lib/libssl.so.0.9.8 (0x00007fbd46673000)
libcrypto.so.0.9.8 => /usr/lib/libcrypto.so.0.9.8 (0x00007fbd462d2000)
libz.so.1 => /usr/lib/libz.so.1 (0x00007fbd460ba000)
/lib64/ld-linux-x86-64.so.2 (0x00007fbd47bb2000)
As said above, I think not importing the "codecs" module by default is
the problem in my eyes. So the question is should mod_wsgi always
include this? -- you mentioned that most users use byte strings and thus
don't need the module. I have no problem with the manual "import codecs"
workaround, but that should get documented somewhere in the wiki.
The problem is that codecs doesn't get imported. I have also that problem
in another project where Python is included into another software. Well,
personally I don't have the problem but other people using the same
software do have problem that "no codec lookup functions have been
registered". Normally these modules are automatically loaded when the
Python interpreter is initialized. For some yet unknown reason for some
people embedded Python interpreters fail to do it. We have yet to find out
what exactly causes the issue, but I had a theory.
Well, Python initializes the encodings package, which imports codecs
module. Now the interesting thing is that if any exception happens inside
this encodings package importing during Python initialization, that error
is just silently ignored. So in case for some reason importing the codecs
module fails during initialization, there will be no signs of the error
anywhere except that when a codec is needed you are faced with the error
that there are no lookup functions available.
Now, you could try to find out what the error actually is by wrapping the
whole contents of encodings/__init__.py into a try: except: construct and
dump the error to a file or something in the exception handler there. That
should, in my theory, reveal where the error exactly happens. And if it
does, please share the findings.
If you are running one application, add:
WSGIApplicationGroup %{GLOBAL}
This will force use of main interpreter, which presumably may have
already had codecs imported as part of main Python initialisation.
In other words, not working in sub interpreters perhaps as indicated
maybe by other email if seen in other embedded systems.
Graham
Graham
Here is the output of sys.modules.keys() from the WSGI script (sorted
alphabetically):
('UserDict', '__builtin__', '__main__', '_abcoll',
'_mod_wsgi_a9b03b1837e0ae52d38b86c3232eac95', '_warnings', 'abc',
'apache', 'copy_reg', 'errno', 'genericpath', 'linecache', 'mod_wsgi',
'os', 'os.path', 'posix', 'posixpath', 'pwd', 'signal', 'site',
'sitecustomize', 'stat', 'sys', 'types', 'warnings', 'zipimport')
No "codecs" module here obviously. What I don't understand: Why is the
"encodings" module missing here? As said above, mod_wsgi seems to load
the "encodings" module somewhere when starting inside an Apache process.
Does it create a new interpreter for executing the WSGI script?
BTW, on the platform you are using, just so can compare with my MacOS
X system, can you write a hello world WSGI program which returns
pretty printed version of sys.modules.keys() and WSGI environ. Do this
for WSGIApplicationGroup set to '%{GLOBAL}' and for it set to 'xxx'.
Interested to see if see a difference in what modules are preloaded.
On 15 March 2011 16:48, Graham Dumpleton <graham.d...@gmail.com> wrote:Are you running this in daemon mode? Is the application the only thing running, be it embedded mode or daemon mode? If you are running one application, add: WSGIApplicationGroup %{GLOBAL} This will force use of main interpreter, which presumably may have already had codecs imported as part of main Python initialisation. In other words, not working in sub interpreters perhaps as indicated maybe by other email if seen in other embedded systems.
Interesting, it seems like there really happens a difference between
initialization of sub-interpreters to the main one. But was there no
errors importing encodings in the sub-interpreters either, or did it not
even try to import the encodings package at all?
For the record, I have all the encodings and stuff in all my Python
sub-interpreters and none of my pages are using global application group,
so it is not a generic problem that happens for everyone; just like with
EventScripts (the other project embedding Python I mentioned earlier) it
works fine for most people but fails to include the codecs for some people.
It's too bad the error doesn't happen to me. If it did, I would surely dig
as deep as would be needed (even to edit Python's source code to add
additional debug messages) to find out what exactly causes it. I don't
know the internal structure of Python enough to be able to provide such
edits for remote debugging, though. But this is certainly an interesting
problem and not just mod_wsgi specific. If we can find out the cause and
solution for it, that would help not only your case but also plenty of
people using other embedded Python interpreters. (You might see that my
motivation for actively trying to participate in this particular problem
is mostly because of the similar problem in ES and there is a major pain
that has been reported to happen to more than just one user)
Graham
What I did do though is modify mod_wsgi C code to do stuff that would
force loading of encoding/codes modules for sub interpreters. Ie.,
added:
Index: mod_wsgi.c
===================================================================
--- mod_wsgi.c (revision 1755)
+++ mod_wsgi.c (working copy)
@@ -4393,6 +4393,17 @@
}
/*
+ * Force loading of codecs into interpreter. This has to be
+ * done as not otherwise done in sub interpreters and if not
+ * done, code running in sub interpreters can fail on some
+ * platforms if a unicode string is added in sys.path and an
+ * import then done.
+ */
+
+ item = PyCodec_Encoder("ascii");
+ Py_XDECREF(item);
+
+ /*
* If running in daemon process, override as appropriate
* the USER, USERNAME or LOGNAME environment variables
* so that they match the user that the process is running
If you were eager to try this patch and see if it fixes issue then
patch will likely not apply properly as mod_wsgi code for 4.0 has
moved around a fair bit and line numbers will be way off. The two
lines though are added to newInterpreterObject() just before
USER/USERNAME/LOGNAME are setup for daemon mode processes.
Graham
Best regards,
Andreas