High Availability TG2 via Paster, Apache+modWSGI and/or nginx

[Derick Eisenhardt]

I'm getting ready to setup a site that could potentially have quite a
bit of traffic (at least in bursts). I've already decided I want to
use nginx as my front-end/reverse-proxy server. However, I'm not sure
what my setup should be beyond that.

I've already done a successful test simply using nginx to proxy to a
standard paster/virtualenv setup. Would it be faster/more efficient to
use Apache + modwsgi, as from what I'm reading Apache is multi-
threaded and paster isn't? I'm also unsure as to how that works. It
appears when using Apache+modwsgi it replaces paster, or do you still
have to run paster too, thus effectively running 3 different services?

My current assumption is if there's alot of cacheable content Paster +
nginx would be just fine, but if I'm gonna have a bunch of unique,
user specific requests I'd probably want to use Apache/WSGI + nginx.
Am I right? And come someone with more experience please expound on
this?

thanks!
- Derick

PS: We really really really need some guides/faqs/how-tos in the new
TG2 docs about such topics (most of the info I've found came from
mailing lists or the Pylons site)

[Jorge Vargas]

The answer is "it depends" both setups you explain below are good and
valid and it really depends on your system/software/personal
experience. I have apps running with both setups.

in general paster > mod_wsgi out of the box for small sites. mod_wsgi
> paster ones you know what you are dealing with.

More details below

On Wed, Apr 29, 2009 at 3:55 PM, Derick Eisenhardt
<derick.e...@gmail.com> wrote:
>
> I'm getting ready to setup a site that could potentially have quite a
> bit of traffic (at least in bursts). I've already decided I want to
> use nginx as my front-end/reverse-proxy server. However, I'm not sure
> what my setup should be beyond that.

You should also configure nginx to serve all your static content, that
is one of the biggest advantages of it.

>
> I've already done a successful test simply using nginx to proxy to a
> standard paster/virtualenv setup. Would it be faster/more efficient to
> use Apache + modwsgi, as from what I'm reading Apache is multi-
> threaded and paster isn't?

actually this is incorrect. paster is multithreaded, but apache is
able to run in multiprocess mode which on linux is better than
threads.

> I'm also unsure as to how that works. It
> appears when using Apache+modwsgi it replaces paster, or do you still
> have to run paster too, thus effectively running 3 different services?
>

yes you get rid of paster and use modwsgi as your wsgi server, there
is an article/program in the docs that sets this up for you.

> My current assumption is if there's alot of cacheable content Paster +
> nginx would be just fine, but if I'm gonna have a bunch of unique,
> user specific requests I'd probably want to use Apache/WSGI + nginx.

the bottom line here is that a badly configured apache+mod_wsgi could
scrawl while a stock paster is good.

> Am I right? And come someone with more experience please expound on
> this?
>

hope that helps.

> thanks!
> - Derick
>
> PS: We really really really need some guides/faqs/how-tos in the new
> TG2 docs about such topics (most of the info I've found came from
> mailing lists or the Pylons site)

Totally agreed, deployment is probably the worst part of our docs.
It's probably because we have so many alternatives that it hurts, but
we should have documentation of at least those two options.

[Graham Dumpleton]

On May 1, 2:36 pm, Jorge Vargas <jorge.var...@gmail.com> wrote:
> The answer is "it depends" both setups you explain below are good and
> valid and it really depends on your system/software/personal
> experience. I have apps running with both setups.
>
> in general paster > mod_wsgi out of the box for small sites. mod_wsgi
>
> > paster ones you know what you are dealing with.
>
> More details below
>
> On Wed, Apr 29, 2009 at 3:55 PM, Derick Eisenhardt
>

> <derick.eisenha...@gmail.com> wrote:
>
> > I'm getting ready to setup a site that could potentially have quite a
> > bit of traffic (at least in bursts). I've already decided I want to
> > use nginx as my front-end/reverse-proxy server. However, I'm not sure
> > what my setup should be beyond that.
>
> You should also configure nginx to serve all your static content, that
> is one of the biggest advantages of it.
>
>
>
> > I've already done a successful test simply using nginx to proxy to a
> > standard paster/virtualenv setup. Would it be faster/more efficient to
> > use Apache + modwsgi, as from what I'm reading Apache is multi-
> > threaded and paster isn't?
>
> actually this is incorrect. paster is multithreaded, but apache is
> able to run in multiprocess mode which on linux is better than
> threads.

Except to the extent described in:

http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html

Yes it is mainly focusing on mod_python, but embedded mode of mod_wsgi
has the same issues.

So, Apache configuration is quite important if using prefork and
embedded mode. That or make sure you have lots of memory.

Graham

[cd34]

If you're going to use nginx, there's no reason to involve apache.
nginx has a 3rd party wsgi module that works quite well. You'll need
to make a minor change to your middleware.py to remove the 3 lines
that allow turbogears to serve the static files. With those lines
removed, only the dynamic pages would be served through wsgi and the
rest is handled through nginx. nginx will outserve apache on static
files 5:1. It's mod_wsgi performance is about 15% faster.

http://turbogears.org/2.0/docs/main/StaticFile.html

Apache does have threaded models - mpm-worker for one, which does
support mod_wsgi. It is incompatible with mod_php which requires mpm-
prefork. If you need php and wsgi and want to use mpm-worker, you can
run php with fastcgi.

nginx also has ncache.org which does caching. If you were going to
use apache, I'd suggest varnish in front of apache with mod_wsgi.
There are unique issues with varnish dealing with IP addresses handed
to apache, so, if you are doing anything with IP addresses, you'll
need to use X-Forwarded-For.

[Graham Dumpleton]

On May 5, 3:32 pm, cd34 <mcd...@gmail.com> wrote:
> If you're going to use nginx, there's no reason to involve apache.
> nginx has a 3rd party wsgi module that works quite well.  You'll need
> to make a minor change to your middleware.py to remove the 3 lines
> that allow turbogears to serve the static files.  With those lines
> removed, only the dynamic pages would be served through wsgi and the
> rest is handled through nginx.

Except that static files would need to be served by a separate nginx
instance in front of the nginx instance running nginx/mod_wsgi. This
is because WSGI is blocking for individual requests and so any
concurrent static file requests handled by same nginx worker process
will block if handled by same process as WSGI application.

> nginx will outserve apache on static
> files 5:1.  It's mod_wsgi performance is about 15% faster.

If there is a performance difference for dynamic Python web
application, it is only going to be for a simple hello world type
program. When you load up a large Python web framework such as
TurboGears the bottlenecks are not going to be in the web hosting
mechanism but the application and database. When looking at what the
resultant overall request time is, that part which relates to the
underlying web hosting mechanism is going to be a very small
percentage, much less than 15%. If you are seeing as much as a 15%
difference for a realistic application test, then you aren't comparing
comparable configurations and so could be configuring one or the other
in a sub optimal fashion.

> http://turbogears.org/2.0/docs/main/StaticFile.html
>
> Apache does have threaded models - mpm-worker for one, which does
> support mod_wsgi.  It is incompatible with mod_php which requires mpm-
> prefork.  If you need php and wsgi and want to use mpm-worker, you can
> run php with fastcgi.

If you need Apache prefork MPM because of PHP running inside of
Apache, then run your Python web application under mod_wsgi daemon
mode. This is the fastcgi equivalent in mod_wsgi which separates
Python application into its own process, with number of processes and
threads being able to be configured separately to the main Apache
server child processes.

> nginx also has ncache.org which does caching.  If you were going to
> use apache, I'd suggest varnish in front of apache with mod_wsgi.
> There are unique issues with varnish dealing with IP addresses handed
> to apache, so, if you are doing anything with IP addresses, you'll
> need to use X-Forwarded-For.

Given that for nginx/mod_wsgi you still have to use a separate nginx
instance for static files, and that for large applications there is no
appreciable difference in performance between nginx/mod_wsgi and
Apache/mod_wsgi, you may as well just use nginx in front of Apache/
mod_wsgi.

If you look around you will find this configuration is quite popular,
as you get fast static file serving with nginx, but stability of
Apache/mod_wsgi for dynamic web application. Use of nginx front also
helps to buffer Apache/mod_wsgi from slow clients as request will not
be sent to Apache/mod_wsgi until completely received. The nginx also
acts as a buffer of sorts for the response when dealing with slow
clients as well. This means that Apache/mod_wsgi is only tied up with
request for minimum time possible and with keep alive off in Apache,
you don't tie up processes/threads with keep alive connections and
thus get better utilisation and reduced memory usage.

With nginx as front end, you can also for static files that need to be
password protected by the web application, use X-Accel-Redirect
response header back to nginx front end to have nginx serve static
file from a private area of URL namespace. This yields much better
performance over the WSGI application returning its own iterable
wrapper for files or even than wsgi.file_wrapper extension.

Overall, for real world dynamic Python applications the performance
differences between different web hosting mechanisms are going to be
very little. So, since all can be made to work, you are generally
better off just choosing that solution which fits your mind set and
which you find to be easy to install, configure and maintain. Issues
such as stability and quality of documentation and ongoing support
should also be considered for critical production systems.

Graham

[Mark Ramm]

> Overall, for real world dynamic Python applications the performance
> differences between different web hosting mechanisms are going to be
> very little. So, since all can be made to work, you are generally
> better off just choosing that solution which fits your mind set and
> which you find to be easy to install, configure and maintain. Issues
> such as stability and quality of documentation and ongoing support
> should also be considered for critical production systems.

The only exception to this is when you make extensive use of features
of the webserver.

The example that comes to mind immediately is the nginx-memcached
plugin which serves rendered pages up out of memcache directly. If
your application can work with this world view, you can get really
significant performance benefits by effectively cutting your app code
entirely out of the request/response and just letting the web server
do it.

But at that point the web server is effectively a part of your
application stack, and not just a WSGI server ;)

--Mark Ramm

[Derick Eisenhardt]

Thanks for all the info so far guys. I really like the idea of having
nginx be the only server accessible by the web, and then a different
service running the python side of things that is only accessible to
the localhost to give an added layer of security...but maybe that's
negligible, maybe there's really no security benefit there? I just
know I really like that nginx can run as a very limited user account,
as while Apache requires you to run it in a chroot jail if you don't
want it having root access.

[Graham Dumpleton]

On May 6, 3:32 am, Derick Eisenhardt <derick.eisenha...@gmail.com>
wrote:

Huh? Unless you are using SELinux or similar, for nginx to be able to
accept connections on port 80, it must be started up as root. This is
the same situation as Apache. For Apache, any server child processes
which handle connections do not run as root, instead they have dropped
privileges and run as a special user, usually www, www-data, wwwserv
or similar. I would expect/hope that nginx would also do similar. In
that respect, there shouldn't really be any differences between the
two.

With Apache/mod_wsgi, you can also use its daemon mode and delegate
specific WSGI applications to run in their own process group (like
fastcgi), with mod_wsgi being able to be told to run them as a
different user again, with that user not being restricted to just
being the Apache user. This is done as an inbuilt feature of mod_wsgi
and it isn't necessary to have to fiddle with suexec or similar like
with fastcgi implementations which also perform process management.

I think perhaps your understanding of how Apache works isn't
complete. :-)

Graham

[hhsuper]

you may search for the topic "TG2 deployment issue"

a ref http://www.cnblogs.com/sharplife/archive/2009/03/28/1424019.html, hope give you some help.

--
Yours sincerely

Jack Su