Exception KeyError in module threading

13,948 views
Skip to first unread message

tmdk

unread,
Apr 9, 2010, 12:46:11 AM4/9/10
to modwsgi
I have a Django app running that *appears* to work fine (from a user's
perspective). But the error_log is constantly filled with these
threading KeyErrors (see below).

My environment:
CentOS 5.4 64bit on Intel
Stock Apache 2.2.3
Python 2.6.5 built from source with --enable_shared
mod_wsgi 2.8

I removed Django from the equation and set up the basic hello world
WSGI app from http://code.google.com/p/modwsgi/wiki/QuickConfigurationGuide
.

I tried to eliminate multithreading issues by going to single threaded
processes in the WSGI daemon:
WSGIDaemonProcess myapp processes=10 threads=1 display-name=%{GROUP}

But still, all I have to do is start apache and stop apache. And I
get ~15 of these errors in the error_log, even without a single
request being processed. And when requests are being processed, they
come more often (but not predictably).

What is this? I can't find any reference to this error. My setup is
so basic, I don't understand what could be going wrong.


[Fri Apr 09 04:34:54 2010] [error] Exception KeyError:
KeyError(47125770724704,) in <module 'threading' from '/usr/local/lib/
python2.6/threading.pyc'> ignored
[Fri Apr 09 04:34:54 2010] [error] Exception KeyError:
KeyError(47125770724704,) in <module 'threading' from '/usr/local/lib/
python2.6/threading.pyc'> ignored
[Fri Apr 09 04:34:54 2010] [error] Exception KeyError:
KeyError(47125770724704,) in <module 'threading' from '/usr/local/lib/
python2.6/threading.pyc'> ignored
[Fri Apr 09 04:34:54 2010] [error] Exception KeyError:
KeyError(47125770724704,) in <module 'threading' from '/usr/local/lib/
python2.6/threading.pyc'> ignored
[Fri Apr 09 04:34:54 2010] [error] Exception KeyError:
KeyError(47125770724704,) in <module 'threading' from '/usr/local/lib/
python2.6/threading.pyc'> ignored
[Fri Apr 09 04:34:54 2010] [error] Exception KeyError:
KeyError(47125770724704,) in <module 'threading' from '/usr/local/lib/
python2.6/threading.pyc'> ignored
[Fri Apr 09 04:34:54 2010] [error] Exception KeyError:
KeyError(47125770724704,) in <module 'threading' from '/usr/local/lib/
python2.6/threading.pyc'> ignored
[Fri Apr 09 04:34:54 2010] [error] Exception KeyError:
KeyError(47125770724704,) in <module 'threading' from '/usr/local/lib/
python2.6/threading.pyc'> ignored
[Fri Apr 09 04:34:54 2010] [error] Exception KeyError:
KeyError(47125770724704,) in <module 'threading' from '/usr/local/lib/
python2.6/threading.pyc'> ignored
[Fri Apr 09 04:34:54 2010] [error] Exception KeyError:
KeyError(47125770724704,) in <module 'threading' from '/usr/local/lib/
python2.6/threading.pyc'> ignored
[Fri Apr 09 04:34:54 2010] [error] Exception KeyError:
KeyError(47125770724704,) in <module 'threading' from '/usr/local/lib/
python2.6/threading.pyc'> ignored
[Fri Apr 09 04:34:54 2010] [error] Exception KeyError:
KeyError(47125770724704,) in <module 'threading' from '/usr/local/lib/
python2.6/threading.pyc'> ignored
[Fri Apr 09 04:34:54 2010] [error] Exception KeyError:
KeyError(47125770724704,) in <module 'threading' from '/usr/local/lib/
python2.6/threading.pyc'> ignored
[Fri Apr 09 04:34:54 2010] [error] Exception KeyError:
KeyError(47125770724704,) in <module 'threading' from '/usr/local/lib/
python2.6/threading.pyc'> ignored
[Fri Apr 09 04:34:54 2010] [error] Exception KeyError:
KeyError(47125770724704,) in <module 'threading' from '/usr/local/lib/
python2.6/threading.pyc'> ignored
[Fri Apr 09 04:34:54 2010] [error] Exception KeyError:
KeyError(47125770724704,) in <module 'threading' from '/usr/local/lib/
python2.6/threading.pyc'> ignored
[Fri Apr 09 04:34:54 2010] [error] Exception KeyError:
KeyError(47125770724704,) in <module 'threading' from '/usr/local/lib/
python2.6/threading.pyc'> ignored
[Fri Apr 09 04:34:54 2010] [error] Exception KeyError:
KeyError(47125770724704,) in <module 'threading' from '/usr/local/lib/
python2.6/threading.pyc'> ignored

Any help is greatly appreciated.

Graham Dumpleton

unread,
Apr 9, 2010, 1:04:01 AM4/9/10
to mod...@googlegroups.com
Provide all the mod_wsgi related configuration from Apache configuration files.

Provide output of running:

ldd mod_wsgi.so

on mod_wsgi.so module file where ever it is installed.

Indicate whether Apache is being run as you or out of standard root
init startup. If as you, supply result of doing:

env | grep PYTHON

Graham

> --
> You received this message because you are subscribed to the Google Groups "modwsgi" group.
> To post to this group, send email to mod...@googlegroups.com.
> To unsubscribe from this group, send email to modwsgi+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
>
>

Dan Kamins

unread,
Apr 9, 2010, 3:23:39 AM4/9/10
to mod...@googlegroups.com

Hi Graham,

Thanks for the quick response.

Hopefully some of this information will help pinpoint the problem.

1.

When we built Python, we did not set a --prefix, but did only:
$ ./configure --enable-shared
$ make
$ sudo make install

The installation seemed to put files into various places that
weren't being seen by other programs until we ran:
$ sudo ln -s /usr/local/lib/libpython2.6.so.1.0 /usr/lib64/libpython2.6.so.1.0
$ sudo ln -s /usr/local/bin/python2.6 /usr/bin/python2.6

Python 2.4 is still also running fine on this system
(as seems to be needed for yum, and possibly other CentOS tools, etc.)

2.

The WSGI test app shows what it thinks sys.version is:

Hello World (Test WSGI)! from Python sys.version=
2.6.5 (r265:79063, Apr 7 2010, 21:10:55)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)]

... as expected.

3.

Here is more information you (Graham) asked for about apache + mod_wsgi:

Apache is being run by standard system init:
$ sudo /sbin/chkconfig httpd on
$ sudo /sbin/service httpd start


Here is the mod_wsgi ldd dump:
$ ldd /usr/lib64/httpd/modules/mod_wsgi.so
libpython2.6.so.1.0 => /usr/lib64/libpython2.6.so.1.0 (0x00002b61a541b000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b61a57ca000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002b61a59e5000)
libutil.so.1 => /lib64/libutil.so.1 (0x00002b61a5bea000)
libm.so.6 => /lib64/libm.so.6 (0x00002b61a5ded000)
libc.so.6 => /lib64/libc.so.6 (0x00002b61a6070000)
/lib64/ld-linux-x86-64.so.2 (0x00000037eaa00000)

Here is our mod_wsgi related httpd.conf configuration:

LoadModule wsgi_module modules/mod_wsgi.so
WSGISocketPrefix /var/run/wsgi

and from within our VirtualHost:

WSGIDaemonProcess myapp processes=10 threads=1 display-name=%{GROUP}

WSGIProcessGroup myapp
WSGIScriptAlias / /home/myapp/wsgi/test.wsgi

Note that we already disabled mod_python (and many other unused modules).


For completeness,
$ uname -a
Linux host.domain.com 2.6.18-164.15.1.el5 #1 SMP Wed Mar 17 11:30:06 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

4.

This comment appears in threading.py's Thread.__delete method concerning
KeyErrors. I have no idea what it means or if it's even relevant:

def __delete(self):
"Remove current thread from the dict of currently running threads."

# Notes about running with dummy_thread:
#
# Must take care to not raise an exception if dummy_thread is being
# used (and thus this module is being used as an instance of
# dummy_threading). dummy_thread.get_ident() always returns -1 since
# there is only one thread if dummy_thread is being used. Thus
# len(_active) is always <= 1 here, and any Thread instance created
# overwrites the (if any) thread currently registered in _active.
#
# An instance of _MainThread is always created by 'threading'. This
# gets overwritten the instant an instance of Thread is created; both
# threads return -1 from dummy_thread.get_ident() and thus have the
# same key in the dict. So when the _MainThread instance created by
# 'threading' tries to clean itself up when atexit calls this method
# it gets a KeyError if another Thread instance was created.
#
# This all means that KeyError from trying to delete something from
# _active if dummy_threading is being used is a red herring. But
# since it isn't if dummy_threading is *not* being used then don't
# hide the exception.

So anyway, to reiterate the situation, web apps seem to work...
They just keeps logging these threading KeyErrors in error_log,
even when we strip the installation down the to minimum wsgi app:

[Fri Apr 09 04:34:54 2010] [error] Exception KeyError:
KeyError(47125770724704,) in <module 'threading' from '/usr/local/lib/
python2.6/threading.pyc'> ignored
[Fri Apr 09 04:34:54 2010] [error] Exception KeyError:
KeyError(47125770724704,) in <module 'threading' from '/usr/local/lib/
python2.6/threading.pyc'> ignored

...

Thank you again.

--
dk

Dan Kamins

unread,
Apr 9, 2010, 3:44:20 AM4/9/10
to mod...@googlegroups.com
I have also set in httpd.conf:
LogLevel debug

And I have reduced to 4 processes for easier visibility of logs:
WSGIDaemonProcess myapp processes=4 threads=1 display-name=%{GROUP}


Here is the log from starting apache:

[Fri Apr 09 07:41:12 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri Apr 09 07:41:12 2010] [info] mod_wsgi (pid=1087): Attach interpreter ''.
[Fri Apr 09 07:41:12 2010] [debug] mod_wsgi.c(8702): mod_wsgi (pid=1087): Starting 1 threads in daemon process 'myapp'.
[Fri Apr 09 07:41:12 2010] [debug] mod_wsgi.c(8708): mod_wsgi (pid=1087): Starting thread 1 in daemon process 'myapp'.
[Fri Apr 09 07:41:12 2010] [info] mod_wsgi (pid=1091): Attach interpreter ''.
[Fri Apr 09 07:41:12 2010] [info] mod_wsgi (pid=1087): Enable deadlock thread in process 'myapp'.
[Fri Apr 09 07:41:12 2010] [info] mod_wsgi (pid=1087): Enable monitor thread in process 'myapp'.
[Fri Apr 09 07:41:12 2010] [debug] mod_wsgi.c(8542): mod_wsgi (pid=1087): Deadlock timeout is 300.
[Fri Apr 09 07:41:12 2010] [debug] mod_wsgi.c(8545): mod_wsgi (pid=1087): Inactivity timeout is 0.
[Fri Apr 09 07:41:12 2010] [debug] mod_wsgi.c(8702): mod_wsgi (pid=1091): Starting 1 threads in daemon process 'myapp'.
[Fri Apr 09 07:41:12 2010] [debug] mod_wsgi.c(8708): mod_wsgi (pid=1091): Starting thread 1 in daemon process 'myapp'.
[Fri Apr 09 07:41:12 2010] [info] mod_wsgi (pid=1091): Enable monitor thread in process 'myapp'.
[Fri Apr 09 07:41:12 2010] [debug] mod_wsgi.c(8542): mod_wsgi (pid=1091): Deadlock timeout is 300.
[Fri Apr 09 07:41:12 2010] [debug] mod_wsgi.c(8545): mod_wsgi (pid=1091): Inactivity timeout is 0.
[Fri Apr 09 07:41:12 2010] [info] mod_wsgi (pid=1091): Enable deadlock thread in process 'myapp'.
[Fri Apr 09 07:41:12 2010] [info] mod_wsgi (pid=1092): Attach interpreter ''.
[Fri Apr 09 07:41:12 2010] [debug] mod_wsgi.c(8702): mod_wsgi (pid=1092): Starting 1 threads in daemon process 'myapp'.
[Fri Apr 09 07:41:12 2010] [debug] mod_wsgi.c(8708): mod_wsgi (pid=1092): Starting thread 1 in daemon process 'myapp'.
[Fri Apr 09 07:41:12 2010] [info] mod_wsgi (pid=1092): Enable deadlock thread in process 'myapp'.
[Fri Apr 09 07:41:12 2010] [info] mod_wsgi (pid=1092): Enable monitor thread in process 'myapp'.
[Fri Apr 09 07:41:12 2010] [debug] mod_wsgi.c(8542): mod_wsgi (pid=1092): Deadlock timeout is 300.
[Fri Apr 09 07:41:12 2010] [debug] mod_wsgi.c(8545): mod_wsgi (pid=1092): Inactivity timeout is 0.
[Fri Apr 09 07:41:12 2010] [info] mod_wsgi (pid=1099): Attach interpreter ''.
[Fri Apr 09 07:41:12 2010] [info] mod_wsgi (pid=1099): Enable deadlock thread in process 'myapp'.
[Fri Apr 09 07:41:12 2010] [info] mod_wsgi (pid=1099): Enable monitor thread in process 'myapp'.
[Fri Apr 09 07:41:12 2010] [debug] mod_wsgi.c(8542): mod_wsgi (pid=1099): Deadlock timeout is 300.
[Fri Apr 09 07:41:12 2010] [debug] mod_wsgi.c(8545): mod_wsgi (pid=1099): Inactivity timeout is 0.
[Fri Apr 09 07:41:12 2010] [debug] mod_wsgi.c(8702): mod_wsgi (pid=1099): Starting 1 threads in daemon process 'myapp'.
[Fri Apr 09 07:41:12 2010] [debug] mod_wsgi.c(8708): mod_wsgi (pid=1099): Starting thread 1 in daemon process 'myapp'.
[Fri Apr 09 07:41:12 2010] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations

Here is the log from stopping apache (immediately after starting) (see 3 errors per process):

[Fri Apr 09 07:41:18 2010] [info] mod_wsgi (pid=1099): Shutdown requested 'myapp'.
[Fri Apr 09 07:41:18 2010] [info] mod_wsgi (pid=1099): Stopping process 'myapp'.
[Fri Apr 09 07:41:18 2010] [info] mod_wsgi (pid=1099): Cleanup interpreter ''.
[Fri Apr 09 07:41:18 2010] [info] mod_wsgi (pid=1099): Terminating Python.
[Fri Apr 09 07:41:18 2010] [error] Exception KeyError: KeyError(47274411468128,) in <module 'threading' from '/usr/local/lib/python2.6/threading.pyc'> ignored
[Fri Apr 09 07:41:18 2010] [error] Exception KeyError: KeyError(47274411468128,) in <module 'threading' from '/usr/local/lib/python2.6/threading.pyc'> ignored
[Fri Apr 09 07:41:18 2010] [error] Exception KeyError: KeyError(47274411468128,) in <module 'threading' from '/usr/local/lib/python2.6/threading.pyc'> ignored
[Fri Apr 09 07:41:18 2010] [error] Exception KeyError: KeyError(47274411468128,) in <module 'threading' from '/usr/local/lib/python2.6/threading.pyc'> ignored
[Fri Apr 09 07:41:18 2010] [info] mod_wsgi (pid=1091): Shutdown requested 'myapp'.
[Fri Apr 09 07:41:18 2010] [info] mod_wsgi (pid=1091): Stopping process 'myapp'.
[Fri Apr 09 07:41:18 2010] [info] mod_wsgi (pid=1091): Cleanup interpreter ''.
[Fri Apr 09 07:41:18 2010] [info] mod_wsgi (pid=1087): Shutdown requested 'myapp'.
[Fri Apr 09 07:41:18 2010] [info] mod_wsgi (pid=1087): Stopping process 'myapp'.
[Fri Apr 09 07:41:18 2010] [info] mod_wsgi (pid=1087): Cleanup interpreter ''.
[Fri Apr 09 07:41:18 2010] [info] mod_wsgi (pid=1092): Shutdown requested 'myapp'.
[Fri Apr 09 07:41:18 2010] [error] Exception KeyError: KeyError(47274411468128,) in <module 'threading' from '/usr/local/lib/python2.6/threading.pyc'> ignored
[Fri Apr 09 07:41:18 2010] [error] Exception KeyError: KeyError(47274411468128,) in <module 'threading' from '/usr/local/lib/python2.6/threading.pyc'> ignored
[Fri Apr 09 07:41:18 2010] [error] Exception KeyError: KeyError(47274411468128,) in <module 'threading' from '/usr/local/lib/python2.6/threading.pyc'> ignored
[Fri Apr 09 07:41:18 2010] [error] Exception KeyError: KeyError(47274411468128,) in <module 'threading' from '/usr/local/lib/python2.6/threading.pyc'> ignored
[Fri Apr 09 07:41:18 2010] [info] mod_wsgi (pid=1092): Stopping process 'myapp'.
[Fri Apr 09 07:41:18 2010] [info] mod_wsgi (pid=1092): Cleanup interpreter ''.
[Fri Apr 09 07:41:18 2010] [error] Exception KeyError: KeyError(47274411468128,) in <module 'threading' from '/usr/local/lib/python2.6/threading.pyc'> ignored
[Fri Apr 09 07:41:18 2010] [info] mod_wsgi (pid=1087): Terminating Python.
[Fri Apr 09 07:41:18 2010] [error] Exception KeyError: KeyError(47274411468128,) in <module 'threading' from '/usr/local/lib/python2.6/threading.pyc'> ignored
[Fri Apr 09 07:41:18 2010] [info] mod_wsgi (pid=1091): Terminating Python.
[Fri Apr 09 07:41:18 2010] [info] mod_wsgi (pid=1092): Terminating Python.
[Fri Apr 09 07:41:18 2010] [error] Exception KeyError: KeyError(47274411468128,) in <module 'threading' from '/usr/local/lib/python2.6/threading.pyc'> ignored
[Fri Apr 09 07:41:18 2010] [error] Exception KeyError: KeyError(47274411468128,) in <module 'threading' from '/usr/local/lib/python2.6/threading.pyc'> ignored
[Fri Apr 09 07:41:18 2010] [notice] caught SIGTERM, shutting down

Graham Dumpleton

unread,
Apr 9, 2010, 5:23:04 AM4/9/10
to mod...@googlegroups.com
In Python 2.6.4 in Py_Main(), which is used by command line Python but
not mod_wsgi, they had:

WaitForThreadShutdown();

Py_Finalize();

In Python 2.6.5 they have made a change and now just have:

Py_Finalize();

Then in Py_Finalize() they have no added.

wait_for_thread_shutdown();

In mod_wsgi have explicit code which is commented as:

/*
* In Python 2.5.1 an exit function is no longer used to
* shutdown and wait on non daemon threads which were created
* from Python code. Instead, in Py_Main() it explicitly
* calls 'threading._shutdown()'. Thus need to emulate this
* behaviour for those versions.
*/

So, I had a workaround for them screwing with this stuff once before.

Another comment before that is:

/*
* Because the thread state we are using was created outside
* of any Python code and is not the same as the Python main
* thread, there is no record of it within the 'threading'
* module. We thus need to access current thread function of
* the 'threading' module to force it to create a thread
* handle for the thread. If we do not do this, then the
* 'threading' modules exit function will always fail
* because it will not be able to find a handle for this
* thread.
*/

so, looks like I have code in there specifically to avoid what looks
like a similar issue. I cant quickly see any changes which would cause
that fixup to fail however.

What is frankly annoying is that they do this sort of stuff in minor
Python patch revisions. Thus if have to cope with this, need a change
which can dynamically work no matter what patch revision of a specific
major/minor version is used without having to force people to
recompile if was first compiled against 2.6.4, but Python was since
upgraded to 2.6.5.

Also not happy that they are in effect changing externally visible
behaviour, which although they compensate with it in command line
Python, will stuff up anyone doing embedded Python work as it is
changing what one expects Py_Finalize() to do.

If this is an issue, am surprised it has taken almost a month since
Python 2.6.5 was released for it to be brought up.

Is anyone else using Python 2.6.5 yet? Are you seeing the same issue?

I will have to do some more investigation and install 2.6.5 to play
with. Will take me a few days or so depending on time I have.

Graham

Graham Dumpleton

unread,
Apr 9, 2010, 7:25:59 AM4/9/10
to mod...@googlegroups.com
I cant duplicate the same error message on Python 2.6.5 on MacOS X.

I do note though that some processes just don't shutdown cleanly at all.

[Fri Apr 09 21:21:04 2010] [info] mod_wsgi (pid=26874): Destroying interpreters.
[Fri Apr 09 21:21:04 2010] [info] mod_wsgi (pid=26874): Cleanup interpreter ''.
[Fri Apr 09 21:21:08 2010] [warn] child process 26874 still did not
exit, sending a SIGTERM

Trying to run Django also fails miserably.

[Fri Apr 09 21:20:08 2010] [error] [client 127.0.0.1] mod_wsgi
(pid=26872): Target WSGI script '/usr/local/django/apache/django.wsgi'
cannot be loaded as Python module.
[Fri Apr 09 21:20:08 2010] [error] [client 127.0.0.1] mod_wsgi
(pid=26872): Exception occurred processing WSGI script
'/usr/local/django/apache/django.wsgi'.
[Fri Apr 09 21:20:08 2010] [error] [client 127.0.0.1] Traceback (most
recent call last):
[Fri Apr 09 21:20:08 2010] [error] [client 127.0.0.1] File
"/usr/local/django/apache/django.wsgi", line 7, in <module>
[Fri Apr 09 21:20:08 2010] [error] [client 127.0.0.1] import
django.core.management
[Fri Apr 09 21:20:08 2010] [error] [client 127.0.0.1] File
"/usr/local/pythonenv/DJANGO-2/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/core/management/__init__.py",
line 3, in <module>
[Fri Apr 09 21:20:08 2010] [error] [client 127.0.0.1] from
optparse import OptionParser, NO_DEFAULT
[Fri Apr 09 21:20:08 2010] [error] [client 127.0.0.1] File
"/usr/local/python-2.6.5/frameworks/Python.framework/Versions/2.6/lib/python2.6/optparse.py",
line 77, in <module>
[Fri Apr 09 21:20:08 2010] [error] [client 127.0.0.1] import textwrap
[Fri Apr 09 21:20:08 2010] [error] [client 127.0.0.1] File
"/usr/local/python-2.6.5/frameworks/Python.framework/Versions/2.6/lib/python2.6/textwrap.py",
line 10, in <module>
[Fri Apr 09 21:20:08 2010] [error] [client 127.0.0.1] import string, re
[Fri Apr 09 21:20:08 2010] [error] [client 127.0.0.1] File
"/usr/local/python-2.6.5/frameworks/Python.framework/Versions/2.6/lib/python2.6/string.py",
line 528, in <module>
[Fri Apr 09 21:20:08 2010] [error] [client 127.0.0.1] from strop
import maketrans, lowercase, uppercase, whitespace
[Fri Apr 09 21:20:08 2010] [error] [client 127.0.0.1] SystemError:
dynamic module not initialized properly

So, they have obviously broken something in Python 2.6.5.

This was using Python 2.6.5 compiled from source code as fully fat and
running under 64 bit Apache on MacOS X (10.5).

Anyone using this combination?

Graham

Evren Esat Özkan

unread,
Apr 9, 2010, 1:07:01 PM4/9/10
to modwsgi
I'm getting same errors on CentOS with mod_wsgi-3.2, Python-2.6.5,
Django trunk. But my site working anyway...


WSGIDaemonProcess .......... processes=10 threads=10 inactivity-
timeout=2000 maximum-requests=1000

[Fri Apr 09 18:10:09 2010] [error] Exception KeyError:
KeyError(-1209010416,) in <module 'threading' from '/opt/python2.6/lib/
python2.6/threading.pyc'> ignored
[Fri Apr 09 18:10:09 2010] [error] Exception KeyError:
KeyError(-1209010416,) in <module 'threading' from '/opt/python2.6/lib/
python2.6/threading.pyc'> ignored
[Fri Apr 09 18:10:09 2010] [error] Exception KeyError:
KeyError(-1209010416,) in <module 'threading' from '/opt/python2.6/lib/
python2.6/threading.pyc'> ignored


Evren Esat Özkan


On 9 Nisan, 12:23, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:

Graham Dumpleton

unread,
Apr 12, 2010, 9:29:41 PM4/12/10
to mod...@googlegroups.com
Hooray, I have an environment set up where have been able to duplicate
this now. :-)

2010/4/10 Evren Esat Özkan <sle...@gmail.com>:

Dan Kamins

unread,
Apr 12, 2010, 9:38:06 PM4/12/10
to mod...@googlegroups.com
Great to hear. We're still running with (and concerned about) this config, so as soon as you have more information, please do update the list.

Graham Dumpleton

unread,
Apr 12, 2010, 10:02:48 PM4/12/10
to mod...@googlegroups.com
FWIW, this is the problem they were trying to fix when making the changes.

http://bugs.python.org/issue1722344

Graham

Thijs Triemstra | Collab

unread,
Apr 13, 2010, 4:36:16 AM4/13/10
to mod...@googlegroups.com
I opened a ticket yesterday (http://code.google.com/p/modwsgi/issues/detail?id=197) and I'm also looking forward to any updates. My brand new cloud server with 3GB of RAM is brought to it's knees after about 12 hours and I think it's due to this mod_wsgi / Python 2.6.5 bug. That machine dies with the following error messages and can only be brought back to live after a hard reboot:

[62282.116349] Out of memory: kill process 3763 (apache2) score 176572 or a child                                       
[62282.116367] Killed process 8913 (apache2)                                                                            
[62340.321916] Out of memory: kill process 3763 (apache2) score 176297 or a child                                       
[62340.321935] Killed process 8943 (apache2)                                                                            
[62404.888786] Out of memory: kill process 3763 (apache2) score 176028 or a child                                       
[62404.888806] Killed process 8947 (apache2) 

I'm now going to see if things get better with Python 2.6.4 and report back.

Cheers,

Thijs

Graham Dumpleton

unread,
Apr 13, 2010, 7:07:58 AM4/13/10
to mod...@googlegroups.com
On 13 April 2010 18:36, Thijs Triemstra | Collab <li...@collab.nl> wrote:
> I opened a ticket yesterday
> (http://code.google.com/p/modwsgi/issues/detail?id=197) and I'm also looking
> forward to any updates. My brand new cloud server with 3GB of RAM is brought
> to it's knees after about 12 hours and I think it's due to this mod_wsgi /
> Python 2.6.5 bug. That machine dies with the following error messages and
> can only be brought back to live after a hard reboot:
>
> [62282.116349] Out of memory: kill process 3763 (apache2) score 176572 or a
> child
> [62282.116367] Killed process 8913 (apache2)
>
> [62340.321916] Out of memory: kill process 3763 (apache2) score 176297 or a
> child
> [62340.321935] Killed process 8943 (apache2)
>
> [62404.888786] Out of memory: kill process 3763 (apache2) score 176028 or a
> child
> [62404.888806] Killed process 8947 (apache2)
>
> I'm now going to see if things get better with Python 2.6.4 and report back.

I still don't see that as related. Having checked through the Python
2.6.5 code, the KeyError message is noise only and not indicative of a
serious problem.

It comes about because shutdown of threading is now being done twice
and on second time through, the very last line of code, which removes
the current thread details for the set of threads fails because it was
already done previously. It is being done twice because Python never
used to do it for embedded Python but now it does. It isn't a simple
matter of mod_wsgi not doing it as then you end up with ordering
issues.

Anyway, if you are having issues with processes not shutting down
properly under 2.6.5, then it is likely a different issue.

Graham

Graham Dumpleton

unread,
Apr 16, 2010, 8:35:26 AM4/16/10
to mod...@googlegroups.com
If keen to get rid of the KeyError message in logs, add the following
two lines just before the call to Py_Finalize() in mod_wsgi.c code.

if (!PyImport_AddModule("dummy_threading"))
PyErr_Clear();

Note that the error in the logs is harmless. These two lines just get
rid of it by devious means rather than adding a run time check as to
whether Python 2.6.5+ or Python 3.1.2+ is being used.

Graham

Thijs Triemstra | Collab

unread,
Apr 19, 2010, 2:45:39 PM4/19/10
to mod...@googlegroups.com
Hi Graham,

> I still don't see that as related. Having checked through the Python
> 2.6.5 code, the KeyError message is noise only and not indicative of a
> serious problem.

Just to let you know that you were right and the crashes were not related to this mod_wsgi issue.

Cheers,

Thijs

Dan Kamins

unread,
Jun 24, 2010, 4:16:56 PM6/24/10
to mod...@googlegroups.com
Resurrecting this old thread about these errors spamming in the error log all the time:

Exception KeyError: KeyError(47435225036128,) in <module 'threading' from '/usr/local/lib/python2.6/threading.pyc'> ignored
Exception KeyError: KeyError(47435225036128,) in <module 'threading' from '/usr/local/lib/python2.6/threading.pyc'> ignored
Exception KeyError: KeyError(47435225036128,) in <module 'threading' from '/usr/local/lib/python2.6/threading.pyc'> ignored

Is your suggested fix included in the official mod_wsgi release yet?
Or any other fix that would address this issue? Or any updates to Python itself that addresses this?

Since this was apparently a "harmless" (but very annoying) error, we have been living with it...

Thank you.

dk

Graham Dumpleton

unread,
Jun 24, 2010, 6:45:06 PM6/24/10
to mod...@googlegroups.com
On 25 June 2010 06:16, Dan Kamins <dan.k...@gmail.com> wrote:
> Resurrecting this old thread about these errors spamming in the error log all the time:
>
> Exception KeyError: KeyError(47435225036128,) in <module 'threading' from '/usr/local/lib/python2.6/threading.pyc'> ignored
> Exception KeyError: KeyError(47435225036128,) in <module 'threading' from '/usr/local/lib/python2.6/threading.pyc'> ignored
> Exception KeyError: KeyError(47435225036128,) in <module 'threading' from '/usr/local/lib/python2.6/threading.pyc'> ignored

If you are using daemon mode, and not using embedded mode at all, have you set:

WSGIRestrictEmbedded On

This will get rid of most of these messages where they relate to
Apache server child processes. See:

http://blog.dscpl.com.au/2009/11/save-on-memory-with-modwsgi-30.html

It is assumed you are using mod_wsgi 3.X.

> Is your suggested fix included in the official mod_wsgi release yet?

Only in subversion. There has been no reason to come out with a new
2.X/3.X version as no serious issues have come up.

> Or any other fix that would address this issue?

There is no other fix. The alternatives are a pain as have to modify
the run time behaviour of mod_wsgi based on patch revision of Python.

> Or any updates to Python itself that addresses this?

There is nothing they can do now. They already made made the arguable
mistake of changing observable C API behaviour in a patch revision
rather than new minor version where compile time checks can be made,
thus the genie is out of the bottle and they cant go back.

Graham

> Since this was apparently a "harmless" (but very annoying) error, we have been living with it...
>
> Thank you.
>
> dk
>
>
> On Apr 16, 2010, at 5:35 AM, Graham Dumpleton wrote:
>
>> If keen to get rid of the KeyError message in logs, add the following
>> two lines just before the call to Py_Finalize() in mod_wsgi.c code.
>>
>>    if (!PyImport_AddModule("dummy_threading"))
>>        PyErr_Clear();
>>
>> Note that the error in the logs is harmless. These two lines just get
>> rid of it by devious means rather than adding a run time check as to
>> whether Python 2.6.5+ or Python 3.1.2+ is being used.
>>
>> Graham
>

tmdk

unread,
Aug 13, 2010, 2:12:42 PM8/13/10
to modwsgi
Last night we installed mod_wsgi 3.3, and the KeyErrors have totally
disappeared. I highly recommend the 3.3 upgrade if you're having this
problem. Thanks, Graham.

dk


On Jun 24, 1:16 pm, Dan Kamins <dan.kam...@gmail.com> wrote:
> Resurrecting this old thread about these errors spamming in the error log all the time:
>
> ExceptionKeyError:KeyError(47435225036128,) in <module 'threading' from '/usr/local/lib/python2.6/threading.pyc'> ignored
> ExceptionKeyError:KeyError(47435225036128,) in <module 'threading' from '/usr/local/lib/python2.6/threading.pyc'> ignored
> ExceptionKeyError:KeyError(47435225036128,) in <module 'threading' from '/usr/local/lib/python2.6/threading.pyc'> ignored

Graham Dumpleton

unread,
Feb 13, 2013, 4:51:53 PM2/13/13
to modwsgi
You should be upgrading mod_wsgi from version 2.8. Simple as that.

So long as you aren't using mod_wsgi 3.X, you will see this message.

Graham


On 14 February 2013 05:06, Kellen Fox <kellen....@gmail.com> wrote:
I can confirm that this issue is still present in python 2.7.3 as well
--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.

To post to this group, send email to mod...@googlegroups.com.

Graham Dumpleton

unread,
Feb 25, 2013, 5:43:38 PM2/25/13
to modwsgi
Sorry, mod_wsgi 3.3 is minimum to get rid of that warning. Not 3.0.

Graham


On 26 February 2013 03:40, Joel Taylor <joela...@gmail.com> wrote:
Hmmm, that's weird, I'm running Python 2.6.6 & mod_wsgi 3.2

cat /var/log/httpd/error_log | grep mod_wsgi

[Sun Feb 24 03:56:43 2013] [warn] mod_wsgi: Compiled for Python/2.6.2.
[Sun Feb 24 03:56:43 2013] [warn] mod_wsgi: Runtime using Python/2.6.6.
[Sun Feb 24 03:56:43 2013] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.0-fips mod_wsgi/3.2 Python/2.6.6 mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations

But I'm still getting the error logs:
[Mon Feb 25 11:28:54 2013] [error] Exception KeyError: KeyError(139946213496800,) in <module 'threading' from '/usr/lib64/python2.6/threading.pyc'> ignored

Graham Dumpleton

unread,
May 30, 2013, 6:06:23 AM5/30/13
to mod...@googlegroups.com
It is because you are using a very old version of mod_wsgi. The problem started when they made a change in how Python does cleanup on shutdown.

If you cannot upgrade to a newer mod_wsgi version (latest is 3.4), then at least add the following directive to Apache:

WSGIRestrictedEmbedded On

This will stop Python being initialised in the main Apache worker processes, which is where the message is likely coming from because of process churn with your Apache configuration and MPM choice.

If that causes stuff to fail in other ways, then likely you aren't actually delegating applications to the daemon process properly. In that case make sure you always have WSGIProcessGroup set appropriately.

Graham

On 30/05/2013, at 3:58 AM, Leandro de Oliverira <leandro.o...@gmail.com> wrote:

Hi. do to solve this problem:
# yum update mod_wsgi
Reply all
Reply to author
Forward
0 new messages