confused about WSGIDaemonProcess and WSGIApplicationGroup
giohappy
on Pyramid framework.
The will run in apache mpm prefork 2.2, on a linux system.
I'm trying to figure out the various options given by daemon
processes, daemon group and application groups, but I'm lost between
the relations between each other.
I'f I'm not wrong, I can dedicate each app a daemon group of
processes. The flow for each application is:
1. Incoming request
2. apache evaulates the process group of the requested application
3. it starts / reuse processes (or threads) from the selected daemon
process.
First doubt:
if apache requires to spawn a new process, because of requests peaks,
how does this new process relate to the daemon processes? I suppose it
starts up a new group od damen processes. If it's right, it can
happens that two requests redirected to the same app can be processes
in two different daemns. Am I far from the truth?
Second doubt:
The application group, by default, will map an application to the same
sub-interpreter, on into the main interpreter if GLOBAL is used. What
does it mean exactly, in relation to the daemon group of processes? I
mean, how mod_wsgi guarantee that the same (sub) interpreter will be
used, if apache can spawn new processes?
Third, and last doubt:
If I set GLOBAL for the application group, but I have various daemon
groups, what is the "main interpreter"?
I'm sorry for my basic (and probably dummy) questions, but these are
my very first steps into the world of mod_wsgi...
Thanks for the help,
giovanni
Graham Dumpleton
http://code.google.com/p/modwsgi/wiki/ProcessesAndThreading
For first lot of questions. In daemon mode additional daemon mode
processes are not spawned, the number of daemon mode process is fixed.
This doesn't mean Apache may not create additional child processes
which do the proxying of requests to the daemon mode processes.
Read:
http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html
for a bit more information about how Apache spins up new child
processes and its bad effects on using embedded mode, especially with
prefork.
As to where requests for a specific application are handled, normally
that would all be in the same daemon process group as that is the sole
place you would have told it to run the requests. So, the only way
that requests for one application would be handled in separate daemon
process groups is if you had configured it that way by saying to send
certain URL sub requests to a different daemon process group.
For second lot of questions. Any process, be it an embedded mode
process if that is being used, or a daemon process can have multiple
separate contexts in which code can run, ie., separate sub
interpreters within the one process. Thus sub interpreters are not one
to one with processes as a process can have multiple sub interpreter
contexts within them.
This means that when a daemon process group has multiple processes,
there is a same named sub interpreter in each of them. So if an
applications requests end up being handle by a process, it goes to the
appropriate sub interpreter (application group) within that process.
At the same time requests for same application handled by another
process in same daemon process group would be handled by same named
sub interpreter in the other process.
For third lot of questions. The main interpreter is just the first
interpreter created in a process when Python is initialised and is the
equivalent of what you get when you run command line Python. The first
interpreter has special properties because the Python C API simplified
GIL API only works properly in the main interpreter. Thus in some
cases it is necessary to force applications to run in the main
interpreter if using C extension modules that use this API.
Sorry if I have explained that badly, am snowed under with work right
now and behind on emails.
Graham
> --
> You received this message because you are subscribed to the Google Groups "modwsgi" group.
> To post to this group, send email to mod...@googlegroups.com.
> To unsubscribe from this group, send email to modwsgi+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
>
>
G. Allegri
Let me recap:
mod_wsgi daemon mode with prefork (I can't use worker in this moment).
I assign a distinct daemon process group to each application.
The deamon processes and threads will be alive until apache isn't stopped. They do not depend on the apache child processes (or threads in worker). Tell if it's right.
Even if apache decides to spawn a new process, the requests for app1 will be processed by one of the processes / one of the threads in its daemon group.
The ApplicationGroup "sticks" an app to be run inside a named sub-interpreter, but it can be inside any process. You say that two requests could be managed by two different processes (in the same process group) but would directed to the "appropriate sub-interpreter". Could you help me to understand the benefit of that? If the two requests are going to be managed in different processes, what the gain of having assigned a specific sub-interpreter?
(I also have a doubt on how global data, e.g. sessions, are granted to work when different processes from the same daemon group can handle the same application... but I will ask this in another thread)
Sorry for flushing all my doubts in a few rows, but things are getting clearer now thanks to your patience! :)
thanks a lot,
giovanni
Graham Dumpleton
> Thanks Grahama for the useful informations. Your blog post was great.
> Let me recap:
>
> mod_wsgi daemon mode with prefork (I can't use worker in this moment).
>
> I assign a distinct daemon process group to each application.
> The deamon processes and threads will be alive until apache isn't stopped.
Or an Apache restart is done. For the restart the daemon processes
will be shutdown and restarted straight away.
> They do not depend on the apache child processes (or threads in worker).
Except for restarts or shutdown of Apache as a whole, the life of the
daemon processes is not linked to the Apache child processes.
> Tell if it's right.
>
> Even if apache decides to spawn a new process, the requests for app1 will be
> processed by one of the processes / one of the threads in its daemon group.
The child processes that Apache may spawn to initially handle requests
only acts as proxies for request through to the daemon process. Those
child processes obviously could also handle static file requests if
Apache is configured for that.
> The ApplicationGroup "sticks" an app to be run inside a named
> sub-interpreter, but it can be inside any process. You say that two requests
> could be managed by two different processes (in the same process group) but
> would directed to the "appropriate sub-interpreter". Could you help me to
> understand the benefit of that?
Concurrency. If your application wasn't multithreaded and so you had
multiple single threaded processes for the daemon process group, then
concurrent requests would be handled at the same time still, each one
in different process of the daemon process group, but still in the
same named interpreter in the respective processes.
> If the two requests are going to be managed
> in different processes, what the gain of having assigned a specific
> sub-interpreter?
The sub interpreter separation allows you to host different WSGI
applications and generally not interfere with each other.
For example, you can't run two Django instances in same sub
interpreter because Django uses global data for settings. You can run
those in different sub interpreters of the same process though.
> (I also have a doubt on how global data, e.g. sessions, are granted to work
> when different processes from the same daemon group can handle the same
> application... but I will ask this in another thread)
If you have a multi process daemon process group the session store
needs to be outside of the process, ie., database, and can't be in
process. If in process, because subsequent request for same session
may not go back to same process, then wouldn't as the session store
would be different.
> Sorry for flushing all my doubts in a few rows, but things are getting
> clearer now thanks to your patience! :)
Graham
G. Allegri
About sessions, now I suppose that a Beaker memory session (which is kept a dictionary) must not be used with a multiprocess daemon, while it fits in a multithreaded daemon group (given my app isn't not multithreaded, I don't have problems with session management).
Just a yes/no answer is ok ;)
Graham Dumpleton
> Very clear explanation, thanks a lot Graham.
>
> About sessions, now I suppose that a Beaker memory session (which is kept a
> dictionary) must not be used with a multiprocess daemon,
Correct.
> while it fits in a
> multithreaded daemon group (given my app isn't not multithreaded, I don't
> have problems with session management).
So long as single process in daemon process group.
Just avoid inactivity-timeout, maximum-requests options and be aware
that touching the WSGI script file will all cause the daemon process
to restart and so drop all the session data and log your users out. If
that is an issue still use a file system or database based session
store.
> Just a yes/no answer is ok ;)
I can never give just a yes/no. ;-)
Graham