multiple virtualenvs, daemon mode

[alecf]

I'm trying to set up a "metaserver" for a project at my company - the
basic idea is something like a hosting provide might want, but with
completely trusted code. The idea is that a single 'container' apache
could run one or more distinct virtualenv-based WSGI applications.

Each of these applications can/should live in their own virtualenv and
"declare" how many daemons (or embedded apache children, if that is
even possible) should serve the application. (like I said, very
trusted code! :))

I've been reading through the docs at http://code.google.com/p/modwsgi/wiki/VirtualEnvironments
and I can't QUITE figure out how to set up what I need to set up.

I've got a 'virgin' virtualenv for the 'master' server, and it looks
like each child needs to do some of the tricks outlined in the
'application environments' section... but I wasn't sure how this
played into the 'process environments' section. Do you have to do
both?

For my purposes I really want each application to have its own set of
children/deamon processes because the applications have very different
memory profiles. i.e. I've got one very heavyweight application that I
want to allocate just 15 processes to because the processes can
balloon to 300+ megs, and another very lightweight high-availability
process that usually sticks around 50 megs, so I'll probably allocated
a good 150 processes to it.

Anyone have some advice?

Alec

[Graham Dumpleton]

2009/4/28 alecf <al...@metaweb.com>:

>
> I'm trying to set up a "metaserver" for a project at my company - the
> basic idea is something like a hosting provide might want, but with
> completely trusted code. The idea is that a single 'container' apache
> could run one or more distinct virtualenv-based WSGI applications.
>
> Each of these applications can/should live in their own virtualenv and
> "declare" how many daemons (or embedded apache children, if that is
> even possible) should serve the application. (like I said, very
> trusted code! :))
>
> I've been reading through the docs at http://code.google.com/p/modwsgi/wiki/VirtualEnvironments
> and I can't QUITE figure out how to set up what I need to set up.
>
>  I've got a 'virgin' virtualenv for the 'master' server, and it looks
> like each child needs to do some of the tricks outlined in the
> 'application environments' section... but I wasn't sure how this
> played into the 'process environments' section. Do you have to do
> both?

No you don't need to do both. Preferably use just python-path option
to WSGIDaemonProcess. Ensure you are using mod_wsgi 2.4 as that
addresses the path ordering issue mentioned in documentation and in
part does away with the need to build on a virgin virtual environment
as additional virtual environments will always take precedence over
system wide site-packages in mod_wsgi 2.4.

> For my purposes I really want each application to have its own set of
> children/deamon processes because the applications have very different
> memory profiles. i.e. I've got one very heavyweight application that I
> want to allocate just 15 processes to because the processes can
> balloon to 300+ megs, and another very lightweight high-availability
> process that usually sticks around 50 megs, so I'll probably allocated
> a good 150 processes to it.

Are your applications not thread safe? What do your applications do?
What are average and maximum request times?

I ask that as that seems to be an excessive number of processes to be
creating. For typical sub second request times you shouldn't need such
a great number.

> Anyone have some advice?

Have you at least got a single daemon mode instance running against
any sort of virtual environment? What is your current configuration
for that?

Have you worked through examples in:

http://code.google.com/p/modwsgi/wiki/QuickConfigurationGuide

and also read:

http://code.google.com/p/modwsgi/wiki/ConfigurationGuidelines
http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGIDaemonProcess

Graham

[Alec Flett]

On Apr 27, 2009, at 9:39 PM, Graham Dumpleton wrote:

>
> No you don't need to do both. Preferably use just python-path option
> to WSGIDaemonProcess.

Great, that simplifies things..

EXCEPT that I'm just discovering that WSGIProcessGroup doesn't seem to
apply to <Location>, the docs say the Context is "server config,
virtual host". I guess I was hoping for this (these are just two of a
couple of projects - the 'me' project is the CPU-bound, low memory
footprint project)

#############################
# Project: me
##############################
WSGIDaemonProcess me-freebase.com processes=50 threads=2 display-name=%
{GROUP} python-path=/mw/alecf/me/pyroot/_install/lib/python2.6/site-
packages

WSGIScriptAlias /api /mw/alecf/me/pyroot/scripts/me.wsgi
WSGISocketPrefix /mw/alecf/metaserver/data/metaserver-me

# Server configuration for me
<Location /XXXX>
WSGIProcessGroup me-freebase.com
</Location>


#############################
# Project: client
##############################
WSGIDaemonProcess client-freebase.com processes=10 threads=2 display-
name=%{GROUP} python-path=/mw/alecf/client/_install/lib/python2.6/site-
packages

WSGIScriptAlias / /mw/alecf/client/scripts/client.wsgi
WSGISocketPrefix /mw/alecf/metaserver/data/metaserver-client

# Server configuration for client
<Location />
WSGIProcessGroup client-freebase.com
</Location>

But I'm finding that everything seems to get bound to "client-
freebase.com"

>
> Are your applications not thread safe? What do your applications do?
> What are average and maximum request times?
>

In general responses are sub-second, but some queries (such as some
RESTful API stuff) can take up to 30 seconds. I'm on application
servers that have lots of RAM so I'm not so worried about creating too
many processes..

> I ask that as that seems to be an excessive number of processes to be
> creating. For typical sub second request times you shouldn't need such
> a great number.


>
>> Anyone have some advice?
>
> Have you at least got a single daemon mode instance running against
> any sort of virtual environment? What is your current configuration
> for that?
>

So right now we have just one virtualenv with regular apache-pre-fork
and mod_wsgi, in traditional apache child mode (i.e. we migrated
seamlessly from mod_python) - what we're trying to resolve is that
SOME applications have a huge memory profiles are sort of "poisoning"
apache children by ballooning up their resident size. Once a child
runs this application, that apache child might stay at 200-300M of
resident size even though it might spend the rest of its life serving
up small-memory application requests, and the small-memory application
dominates traffic. (The 200-300M of resident size is expected and
acceptable, FYI, its just rare)

The hope is to have one WSGIProcessGroup per type of application...

Thanks for your help, I'm going to keep hacking away at this but I'm
still curious to hear about the WSGIDaemonProcess context issue.

(oh and yes I'm on mod_wsgi 2.4, happy to even experiment with trunk/
3.0 if that's what it takes :))

Alec

[Graham Dumpleton]

2009/4/28 Alec Flett <al...@metaweb.com>:

>
>
> On Apr 27, 2009, at 9:39 PM, Graham Dumpleton wrote:
>
>>
>> No you don't need to do both. Preferably use just python-path option
>> to WSGIDaemonProcess.
>
> Great, that simplifies things..
>
> EXCEPT that I'm just discovering that WSGIProcessGroup doesn't seem to
> apply to <Location>, the docs say the Context is "server config,
> virtual host".

The WSGIDaemonProcess directive is 'server config, virtual host', but
WSGIProcessGroup is 'server config, virtual host, directory'. In
Apache speak, 'directory' also covers 'Location' directive'.

> I guess I was hoping for this (these are just two of a
> couple of projects - the 'me' project is the CPU-bound, low memory
> footprint project)
>
> #############################
> # Project: me
> ##############################
> WSGIDaemonProcess me-freebase.com processes=50 threads=2 display-name=%
> {GROUP} python-path=/mw/alecf/me/pyroot/_install/lib/python2.6/site-
> packages
>
> WSGIScriptAlias /api /mw/alecf/me/pyroot/scripts/me.wsgi
> WSGISocketPrefix /mw/alecf/metaserver/data/metaserver-me

You shouldn't be setting WSGISocketPrefix multiple times like you are.
Normally you shouldn't be setting it all. This isn't like external
mode of FASTCGI where you need to define the location of the socket,
mod_wsgi will work it all out for you. Only time you need you usually
need to set WSGISocketPrefix is on RedHat based systems as Apache
'logs' directory is not readable to anyone but root. In that case you
would usually have to set it to 'run/wsgi'. See:

http://code.google.com/p/modwsgi/wiki/ConfigurationIssues

> # Server configuration for me
> <Location /XXXX>

You mean '/api' don't you. As long as the URL matches mount point of
application it should work. That said, that isn't the way I would
recommend it be done. You instead would normally put WSGIProcessGroup
in appropriate Directory context.

<Directory /mw/alecf/me/pyroot/scripts>
WSGIProcessGroup me-freebase.com
</Directory>

This is more precise as then doesn't matter where application is mounted.

> WSGIProcessGroup me-freebase.com
> </Location>
>
>
> #############################
> # Project: client
> ##############################
> WSGIDaemonProcess client-freebase.com processes=10 threads=2 display-
> name=%{GROUP} python-path=/mw/alecf/client/_install/lib/python2.6/site-
> packages
>
> WSGIScriptAlias / /mw/alecf/client/scripts/client.wsgi
> WSGISocketPrefix /mw/alecf/metaserver/data/metaserver-client
>
> # Server configuration for client
> <Location />
> WSGIProcessGroup client-freebase.com
> </Location>

The Location directive is redundant in this case, as '/' is the default context.

As before though, better off having it in Directory context bound to
where script is actually located.

<Directory /mw/alecf/client/scripts>
WSGIProcessGroup client-freebase.com
</Directory>

So, drop WSGISocketPrefix and use WSGIProcessGroup in Directory
context for where script is located instead of using Location context.

Graham

[Alec Flett]

This is really helpful, thanks...

On Apr 28, 2009, at 4:55 PM, Graham Dumpleton wrote:

>
> <Directory /mw/alecf/me/pyroot/scripts>
> WSGIProcessGroup me-freebase.com
> </Directory>
>

>

> So, drop WSGISocketPrefix and use WSGIProcessGroup in Directory
> context for where script is located instead of using Location context.

My only issue is that both urls "/" and "/api" are roots of WSGI
applications (paste.deploy config-based applications, really, but
that's a side issue) so there's no "scripts" directory per say... is
<Location> still going to be problematic?

Alec

[Graham Dumpleton]

2009/4/29 Alec Flett <al...@metaweb.com>:

Not sure what you mean. The script directory I refer to is where the
WSGI script files are located, ie., the directory containing the file
listed as last argument to WSGIScriptAlias.

Can you explain better what you mean?

Graham

[Alec Flett]

On Apr 28, 2009, at 7:37 PM, Graham Dumpleton wrote:

My only issue is that both urls "/" and "/api" are roots of WSGI

applications (paste.deploy config-based applications, really, but

that's a side issue) so there's no "scripts" directory per say... is

<Location> still going to be problematic?

Not sure what you mean. The script directory I refer to is where the
WSGI script files are located, ie., the directory containing the file
listed as last argument to WSGIScriptAlias.

Can you explain better what you mean?

Just to follow up, I figured out where I was confused, switched to <Directory> and everything is running very smoothly now. Thanks for the help.

Alec

Graham