What Apache version are you using? The Apache supplied one or have you
built Apache from very recent Apache source code?
Graham
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/modwsgi/-/FTE7A8WZ8msJ.
> To post to this group, send email to mod...@googlegroups.com.
> To unsubscribe from this group, send email to
> modwsgi+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/modwsgi?hl=en.
>
> TLS = Thread Local Storage
Ah. That makes a lot more sense :-)
> What Apache version are you using? The Apache supplied one or have you
> built Apache from very recent Apache source code?
The built-in one.
Apache/2.2.17 (Unix) PHP/5.3.4 mod_ssl/2.2.17 OpenSSL/0.9.8r DAV/2 mod_wsgi/3.3 Python/2.7.2
rg
How are you determining that there are two threads, a core stack dump?
If yes, I would like to see those stack dumps.
FWIW, I did see this issue recently, but was with a custom Apache
source build and Apple Python 2.6.5. Using Apache and Python supplied
by Apple, have no problems for same program that was causing the
issue. If you can provide a simple WSGI hello world that shows the
problem, then I can perhaps test it on my Apache/Python combo which I
know might be susceptible as still have it set up.
Graham
> --
> You received this message because you are subscribed to the Google Groups "modwsgi" group.
> How are you determining that there are two threads, a core stack dump?
No, it's the result of calling thread.get_ident() within the application itself. See below.
> FWIW, I did see this issue recently, but was with a custom Apache
> source build and Apple Python 2.6.5. Using Apache and Python supplied
> by Apple, have no problems for same program that was causing the
> issue. If you can provide a simple WSGI hello world that shows the
> problem, then I can perhaps test it on my Apache/Python combo which I
> know might be susceptible as still have it set up.
Here you go. It's about as simple as it can be.
import subprocess, thread
def hello_world(environ, start_response):
s = subprocess.Popen(['whoami'], stdout=subprocess.PIPE).stdout.read()
start_response( '200 OK', [('Content-type', 'text/plain')])
return [str(thread.get_ident()), ' - ', s]
application = hello_world
If all the process is then doing is subsequent exec then it is
generally not a problem. It is when it wants to do mored complicate
stuff that it becomes an issue. So to blanket say it is a really bad
idea is plain wrong.
Sorry that haven't responded in this issue properly. Been tied up
doing other changes in mod_wsgi related to work that have been trying
to get on top of first. I will try and investigate it soon.
If someone wants to see if they can come up with a test program that
doesn't involve subprocess module and uses just fork/exec directly
then that would help.
The subprocess module has been a source of problems in the past with
assumptions it makes, such as flushing stdout/stderr after the fork,
ignoring the fact that there may be buffered data that parent would
also flush. End result is duplicated content in logs.
Graham
> If anyone wants details let me know.
>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/modwsgi/-/ocxE7791-QgJ.
[Fri Sep 23 15:05:13 2011] [notice] child pid 97541 exit signal Abort trap (6)
Fatal Python error: XXX block stack underflow
So, something entirely different.
If you can distill that test case down to exclude subprocess it would
thus help because if it is anything would suspect subprocess doing
something.
If you aren't already, try adding:
WSGIApplicationGroup %{GLOBAL}
into Apache configuration just for that WSGI application.
This issue if there is one is likely going to relate to subprocess and
forking in sub interpreters.
Graham
> Hmmm, the local setup I thought I was getting this issue on, I am not,
> The error on my local setup was:
>
> [Fri Sep 23 15:05:13 2011] [notice] child pid 97541 exit signal Abort trap (6)
> Fatal Python error: XXX block stack underflow
>
> So, something entirely different.
>
> If you can distill that test case down to exclude subprocess it would
> thus help because if it is anything would suspect subprocess doing
> something.
I can reliably reproduce the problem thusly:
pid = os.fork()
if pid:
sys.stderr.write('Fork succeeded (PID=%s)\n' % pid)
return ['OK - ', pid]
else:
sys.stderr.write('Fork succeeded (child PID=%s)\n' % os.getpid())
os._exit(0)
Here's a small excerpt from my error log, the result of reloading the page four times:
[Fri Oct 07 08:15:15 2011] [error] Fork succeeded (PID=90595)
[Fri Oct 07 08:15:15 2011] [error] Fork succeeded (child PID=90595)
[Fri Oct 07 08:15:35 2011] [error] Fork succeeded (PID=90599)
Fatal Python error: Couldn't create autoTLSkey mapping
[Fri Oct 07 08:15:43 2011] [error] Fork succeeded (PID=90600)
[Fri Oct 07 08:15:43 2011] [error] Fork succeeded (child PID=90600)
[Fri Oct 07 08:15:45 2011] [error] Fork succeeded (PID=90601)
Fatal Python error: Couldn't create autoTLSkey mapping
> If you aren't already, try adding:
>
> WSGIApplicationGroup %{GLOBAL}
>
> into Apache configuration just for that WSGI application.
That made the problem go away.
> This issue if there is one is likely going to relate to subprocess and
> forking in sub interpreters.
I'm pretty sure the error is being generated by this code in pystate.py:
/* Reset the TLS key - called by PyOS_AfterFork.
* This should not be necessary, but some - buggy - pthread implementations
* don't flush TLS on fork, see issue #10517.
*/
void
_PyGILState_Reinit(void)
{
PyThreadState *tstate = PyGILState_GetThisThreadState();
PyThread_delete_key(autoTLSkey);
if ((autoTLSkey = PyThread_create_key()) == -1)
Py_FatalError("Could not allocate TLS entry");
/* re-associate the current thread state with the new key */
if (PyThread_set_key_value(autoTLSkey, (void *)tstate) < 0)
Py_FatalError("Couldn't create autoTLSkey mapping");
}
> If all the process is then doing is subsequent exec then it is
> generally not a problem. It is when it wants to do mored complicate
> stuff that it becomes an issue. So to blanket say it is a really bad
> idea is plain wrong.
Yes, but this is not a C program, this is a Python program. Unless the subprocess module is re-coded in C, it is not possible to call exec immediately after fork. After calling fork from Python, control is necessarily returned to the Python interpreter.
rg
This code doesn't exist in Python 2.7.1 code that I have on my box.
This must be a new change and whatever they are doing has broken
things for a fork done from a sub interpreter. So,could be a bug in
Python.
Querying the bug report for original issue and whether they checked
fix for sub interpreters. Often they don't bother.
Will do standalone test when get chance.
Graham
>> If all the process is then doing is subsequent exec then it is
>> generally not a problem. It is when it wants to do mored complicate
>> stuff that it becomes an issue. So to blanket say it is a really bad
>> idea is plain wrong.
>
>
> Yes, but this is not a C program, this is a Python program. Unless the subprocess module is re-coded in C, it is not possible to call exec immediately after fork. After calling fork from Python, control is necessarily returned to the Python interpreter.
>
> rg
>
> --
> You received this message because you are subscribed to the Google Groups "modwsgi" group.
It was apparently introduced in 2.7.2.
> This must be a new change and whatever they are doing has broken
> things for a fork done from a sub interpreter. So,could be a bug in
> Python.
I tried reproducing the problem standalone but was unable to. It seems like it must be something mod_wsgi is doing because of the intermittent reproducibility. Also the fact that WSGIApplicationGroup %{GLOBAL} makes the problem go away is a pretty big clue.
For the record, I had WSGIApplicationGroup set to the name of my WSGIDaemonProcess. I have half a dozen WSGI applications running on the same machine. Each one has its own named WSGIDaemonProcess and a corresponding WSGIApplicationGroup set to the same name. But I don't actually understand what WSGIApplicationGroup is supposed to do, so this whole thing could be a configuration error on my part.
> Querying the bug report for original issue and whether they checked
> fix for sub interpreters. Often they don't bother.
>
> Will do standalone test when get chance.
Thanks.
rg
Graham
Sub interpreters can not be created ....
> The only way to reproduce it is going to be wrote a C program which
> embeds Python and manually initialises Python via C API and then
> creates sub interpreter, manages sub interpreter thread states
> correctly and then executes problem code in the context of that. It
> isn't straight forward thing that most would be able to do. Sub
> interpreters can [not] be created from pure Python code so must use C code.
> Is that what you have done so far to replicate it standalone?
No, I did a pure-python test. I thought that my mod_wsgi configuration wasn't using sub-interpreters because I'm running (I thought) in daemon mode with one process per wsgi application.
rg
WSGIApplicationGroup %{GLOBAL}
you are running in a sub interpreter of the daemon process.
The %{GLOBAL} value to that directive says to use the main interpreter.
Graham
http://bugs.python.org/issue13156
Basically your screwed if you want to use Python 2.7.2 or 3.2.1+ and
fork sub processes from application running in mod_wsgi sub
interpreters.
You are forced to use the main interpreter.
To work around the problem in mod_wsgi would require a very naughty
hack to force initialisation of auto thread state for main interpreter
and then manipulate its reference count manually up by one so it isn't
auto deleted.
Graham
BTW, I get the exact same result on Linux. --
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To view this discussion on the web visit https://groups.google.com/d/msg/modwsgi/-/NoRmhODnADAJ.
To post to this group, send email to mod...@googlegroups.com.
To unsubscribe from this group, send email to modwsgi+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
-- Ademir Francisco da Silva Skype ...: Ademir_Francisco_da_Silva