Methods deploying a TurboGears 2.0 application
7 views
Skip to first unread message
cd34
May 19, 2009, 12:45:10 PM5/19/09
to TurboGears
I am working on a turbogears app and decided to test the different
methods of deploying an app and I'm a little concerned. Since this
app requires SSL, and paster doesn't handle SSL, what are the
suggested methods for deployment?
Paster gives reasonably good performance during development for a
single user, however, I cannot use SSL with Paster.
mod_wsgi appears to work ok until there are multiple requests coming
in and response times on reloaded pages falls off considerably. I
don't believe I would have too many issues with this application, but,
a reasonably successful site would appear to have issues even without
SSL.
How is turbogears.org set up? mod_wsgi behind mpm-prefork? or mpm-
worker?
apachebench on a very simple test, hitting a page that requires
authentication, 100 requests, no keepalives, no concurrent connections
comes up with:
paster: Requests per second: Requests per second: 76.32 [#/sec]
(mean)
apache2-mpm-prefork Requests per second: 57.38 [#/sec] (mean)
apache2-mpm-worker Requests per second: 89.98 [#/sec] (mean)
As a baseline that is relatively decent. The current php application
we're running is at: Requests per second: 72.51 [#/sec] (mean) on
lesser hardware.
If we bump up the concurrency, we start to run into issues:
At 10 concurrent requests, we get the following:
paster: Requests per second: 52.94 [#/sec] (mean)
apache2-mpm-prefork: Requests per second: 54.45 [#/sec] (mean)
apache2-mpm-worker: Requests per second: 53.61 [#/sec] (mean)
whereas our php application with 10 concurrent sessions is at:
Requests per second: 62.35 [#/sec] (mean).
If we test 100 concurrent requests, mpm-worker really shines through
by staying at roughly Requests per second: 53.46 [#/sec] (mean) and
paster comes through with Requests per second: 54.85 [#/sec] (mean)
These results don't come close to the Requests per second: 86.33 [#/
sec] (mean) at http://turbogears.org/2.0/ Since you have ServerTokens
set to prod, I couldn't really guess how you had deployed the site.
Since you use Discus, I'm assuming Identity is turned off.
So even with slight concurrency, mod_wsgi (or PasteWSGIServer/0.5) is
taking a rather huge hit. Unless I have missed something obvious with
the testing, writing a scaleable application with TurboGears is going
to require considerable server resources.
A quick test to check mod_wsgi performance on my setup resulted in:
Requests per second: 2370.11 [#/sec] (mean)
Concurrency set to 20, 1000 requests. Even at 50 or 100 concurrent
requests, the numbers don't change that dramatically.
This seems to suggest the load isn't from mod_wsgi, but from TG2.
What have I done differently than the Turbogears site? Even without
authentication, with 10 concurrent requests I cannot get my
installation anywhere near 86.33 requests per second even on fairly
powerful hardware (X3220 @ 2.40GHz, quadcore, 8gb ram) The only way
I could come close to the 86.33 was to run the app on a 2 server
cluster.
I have yet to try nginx/mod_wsgi, but, I believe my results might be
similar. Apache was not configured to serve static images. Since
apachebench only pulled a single page without any stylesheets/images,
the test should mimic a properly configured setup by testing only the
response time that turbogears took to check to see if the current
session was authenticated and presented a login page.
What deployment options are there?
paster
proxy -> paster
apache2-mpm-prefork -> mod_wsgi
apache2-mpm-worker -> mod_wsgi
nginx -> mod_wsgi
apache2 -> mod_python - run the entire virtualenvironment within
mod_python?
Based on the numbers above, apache2-mpm-worker appears to give me the
flexibility to allow TG2 to handle only the pages it needs while
apache serves the static content.
For a site that will see 100k unique visitors with 4-5 pageviews a
person a day, what bottlenecks should I expect?
The site shootq.com which is linked from the frontpage of
turbogears.org/2.0 as a site using TG2 in production appears to be
using CakePHP as their framework. Does anyone use TurboGears on a
forward-facing website? What issues have you run into? How did you
deploy and what type of traffic are you seeing?
methods of deploying an app and I'm a little concerned. Since this
app requires SSL, and paster doesn't handle SSL, what are the
suggested methods for deployment?
Paster gives reasonably good performance during development for a
single user, however, I cannot use SSL with Paster.
mod_wsgi appears to work ok until there are multiple requests coming
in and response times on reloaded pages falls off considerably. I
don't believe I would have too many issues with this application, but,
a reasonably successful site would appear to have issues even without
SSL.
How is turbogears.org set up? mod_wsgi behind mpm-prefork? or mpm-
worker?
apachebench on a very simple test, hitting a page that requires
authentication, 100 requests, no keepalives, no concurrent connections
comes up with:
paster: Requests per second: Requests per second: 76.32 [#/sec]
(mean)
apache2-mpm-prefork Requests per second: 57.38 [#/sec] (mean)
apache2-mpm-worker Requests per second: 89.98 [#/sec] (mean)
As a baseline that is relatively decent. The current php application
we're running is at: Requests per second: 72.51 [#/sec] (mean) on
lesser hardware.
If we bump up the concurrency, we start to run into issues:
At 10 concurrent requests, we get the following:
paster: Requests per second: 52.94 [#/sec] (mean)
apache2-mpm-prefork: Requests per second: 54.45 [#/sec] (mean)
apache2-mpm-worker: Requests per second: 53.61 [#/sec] (mean)
whereas our php application with 10 concurrent sessions is at:
Requests per second: 62.35 [#/sec] (mean).
If we test 100 concurrent requests, mpm-worker really shines through
by staying at roughly Requests per second: 53.46 [#/sec] (mean) and
paster comes through with Requests per second: 54.85 [#/sec] (mean)
These results don't come close to the Requests per second: 86.33 [#/
sec] (mean) at http://turbogears.org/2.0/ Since you have ServerTokens
set to prod, I couldn't really guess how you had deployed the site.
Since you use Discus, I'm assuming Identity is turned off.
So even with slight concurrency, mod_wsgi (or PasteWSGIServer/0.5) is
taking a rather huge hit. Unless I have missed something obvious with
the testing, writing a scaleable application with TurboGears is going
to require considerable server resources.
A quick test to check mod_wsgi performance on my setup resulted in:
Requests per second: 2370.11 [#/sec] (mean)
Concurrency set to 20, 1000 requests. Even at 50 or 100 concurrent
requests, the numbers don't change that dramatically.
This seems to suggest the load isn't from mod_wsgi, but from TG2.
What have I done differently than the Turbogears site? Even without
authentication, with 10 concurrent requests I cannot get my
installation anywhere near 86.33 requests per second even on fairly
powerful hardware (X3220 @ 2.40GHz, quadcore, 8gb ram) The only way
I could come close to the 86.33 was to run the app on a 2 server
cluster.
I have yet to try nginx/mod_wsgi, but, I believe my results might be
similar. Apache was not configured to serve static images. Since
apachebench only pulled a single page without any stylesheets/images,
the test should mimic a properly configured setup by testing only the
response time that turbogears took to check to see if the current
session was authenticated and presented a login page.
What deployment options are there?
paster
proxy -> paster
apache2-mpm-prefork -> mod_wsgi
apache2-mpm-worker -> mod_wsgi
nginx -> mod_wsgi
apache2 -> mod_python - run the entire virtualenvironment within
mod_python?
Based on the numbers above, apache2-mpm-worker appears to give me the
flexibility to allow TG2 to handle only the pages it needs while
apache serves the static content.
For a site that will see 100k unique visitors with 4-5 pageviews a
person a day, what bottlenecks should I expect?
The site shootq.com which is linked from the frontpage of
turbogears.org/2.0 as a site using TG2 in production appears to be
using CakePHP as their framework. Does anyone use TurboGears on a
forward-facing website? What issues have you run into? How did you
deploy and what type of traffic are you seeing?
Mark Ramm
May 19, 2009, 2:50:23 PM5/19/09
to turbo...@googlegroups.com
What mod_wsgi settings are you using?
I tend to use modwsgi daemon mode and at least 1 process per CPU, and
then keep adding threads to the point where we have one thread per
apache process, or one thread per allowed database connection. But
slow clients, database hit free pages, and other factors can impact
the ratios a lot, so I'd do a couple of tests and figure out what
settings work best for you.
On a work project of mine we are actually running modwsgi behing nginx
and serving up all the static content from nginx.
turbogears.org currently serves static files via apache, on a shared
host, so it's not at all a fair comparison ;)
--
Mark Ramm-Christensen
email: mark at compoundthinking dot com
blog: www.compoundthinking.com/blog
I tend to use modwsgi daemon mode and at least 1 process per CPU, and
then keep adding threads to the point where we have one thread per
apache process, or one thread per allowed database connection. But
slow clients, database hit free pages, and other factors can impact
the ratios a lot, so I'd do a couple of tests and figure out what
settings work best for you.
On a work project of mine we are actually running modwsgi behing nginx
and serving up all the static content from nginx.
turbogears.org currently serves static files via apache, on a shared
host, so it's not at all a fair comparison ;)
Mark Ramm-Christensen
email: mark at compoundthinking dot com
blog: www.compoundthinking.com/blog
cd34
May 19, 2009, 4:10:34 PM5/19/09
to TurboGears
I tried multiple options, ending with
WSGIDaemonProcess daemon processes=8 threads=8
where processes was 2x the number of cpus (mutex issues appear to hit
at times)
number of threads didn't seem to affect performance much after 8.
I thought the docs section of the turbogears site ran turbogears -- or
is that mod_python/sphinx doing the ajax search?
In looking at it further, it appears to be an application stack
issue. SQLAlchemy doesn't quite do things in the most efficient
manner when dealing with backreferences, so, I am probably going to
change things around a little to specify the joins myself.
I guess I need to do a bit of profiling to see where the bottleneck
really is, then, figure out how to engineer around it.
I'll eventually deploy with nginx, but, the webserver/wsgi end doesn't
appear to be the bottleneck.
WSGIDaemonProcess daemon processes=8 threads=8
where processes was 2x the number of cpus (mutex issues appear to hit
at times)
number of threads didn't seem to affect performance much after 8.
I thought the docs section of the turbogears site ran turbogears -- or
is that mod_python/sphinx doing the ajax search?
In looking at it further, it appears to be an application stack
issue. SQLAlchemy doesn't quite do things in the most efficient
manner when dealing with backreferences, so, I am probably going to
change things around a little to specify the joins myself.
I guess I need to do a bit of profiling to see where the bottleneck
really is, then, figure out how to engineer around it.
I'll eventually deploy with nginx, but, the webserver/wsgi end doesn't
appear to be the bottleneck.
Graham Dumpleton
May 19, 2009, 5:12:41 PM5/19/09
to TurboGears
Haven't the time right now to read your post properly, but a comment
and a question.
Paste server would likely preload TG. In Apache/mod_wsgi it would lazy
load it on demand when first request to a process arrives. How did you
ensure your tests against Apache/mod_wsgi weren't counting the cost of
loading TG?
Graham
and a question.
Paste server would likely preload TG. In Apache/mod_wsgi it would lazy
load it on demand when first request to a process arrives. How did you
ensure your tests against Apache/mod_wsgi weren't counting the cost of
loading TG?
Graham
cd34
May 19, 2009, 6:35:11 PM5/19/09
to TurboGears
On May 19, 5:12 pm, Graham Dumpleton <Graham.Dumple...@gmail.com>
wrote:
eliminated any apache child creation issues and the wsgi environment
should have been created and preloaded. During ab, it was obvious
when a preload was hit and that result was thrown out. mpm-worker
seemed more consistent from first through third run. Best case of the
'worst case' scenario seemed to be a reasonable assessment.
I am sure there are settings in TG2 that will affect this. Perhaps I
don't have all of the debugging messages turned off properly or have
missed some of the options, but, I don't think that is going to
materially affect the results.
I am somewhat sure that the 2nd and 3rd runs would have had things
preloaded since I was calling the same page over and over again. The
fact that a simple 'Hello World' wsgi app handled over 2000 requests
per second suggests that wsgi is working.
At this point, I need to profile TG2, SQLAlchemy and many of the other
modules. SQLAlchemy backreferences appear to induce a hit. Looking
at the queries that hit mysql, I'm sure there are some improvements
that can be made to my auth handler to remove half a dozen queries and
reference links, but, my initial goal was to figure out where the
bottlenecks were and to figure out what other people were doing for
deployment.
Right now, it seems like apache2|nginx/ssl/mod_wsgi is what I need for
this project. For my future project, I've got to do quite a bit more
investigation to see what sort of resources I'm going to need.
wrote:
> Paste server would likely preload TG. In Apache/mod_wsgi it would lazy
> load it on demand when first request to a process arrives. How did you
> ensure your tests against Apache/mod_wsgi weren't counting the cost of
> loading TG?
3 runs of apachebench, fastest time recorded. For mpm-prefork, this
> load it on demand when first request to a process arrives. How did you
> ensure your tests against Apache/mod_wsgi weren't counting the cost of
> loading TG?
eliminated any apache child creation issues and the wsgi environment
should have been created and preloaded. During ab, it was obvious
when a preload was hit and that result was thrown out. mpm-worker
seemed more consistent from first through third run. Best case of the
'worst case' scenario seemed to be a reasonable assessment.
I am sure there are settings in TG2 that will affect this. Perhaps I
don't have all of the debugging messages turned off properly or have
missed some of the options, but, I don't think that is going to
materially affect the results.
I am somewhat sure that the 2nd and 3rd runs would have had things
preloaded since I was calling the same page over and over again. The
fact that a simple 'Hello World' wsgi app handled over 2000 requests
per second suggests that wsgi is working.
At this point, I need to profile TG2, SQLAlchemy and many of the other
modules. SQLAlchemy backreferences appear to induce a hit. Looking
at the queries that hit mysql, I'm sure there are some improvements
that can be made to my auth handler to remove half a dozen queries and
reference links, but, my initial goal was to figure out where the
bottlenecks were and to figure out what other people were doing for
deployment.
Right now, it seems like apache2|nginx/ssl/mod_wsgi is what I need for
this project. For my future project, I've got to do quite a bit more
investigation to see what sort of resources I'm going to need.
Graham Dumpleton
May 19, 2009, 7:25:01 PM5/19/09
to TurboGears
Graham Dumpleton
May 19, 2009, 7:26:38 PM5/19/09
to TurboGears
blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-
usage.html
http://blog.dscpl.com.au/2009/05/blocking-requests-and-nginx-version-of.html
Sorry about the earlier empty followup. :-)
Graham
cd34
May 19, 2009, 9:20:53 PM5/19/09
to TurboGears
I had seen you mention the blocking issue in another post, but, it is
good to read a detailed description of the issue without having to dig
into the code. I would have found the effect, then had to dig into
the code to figure out why. I am quite webserver agnostic and have
the benefit of being able to build whatever hosting environment I need
for a project.
In looking at my numbers, I'm not sure 50 requests per second is
unreasonable for the other project.
I will contact you offline with an issue I've run into with mod_wsgi
on an unrelated issue which is related to another blog post of yours.
On May 19, 7:26 pm, Graham Dumpleton <Graham.Dumple...@gmail.com>
wrote:
good to read a detailed description of the issue without having to dig
into the code. I would have found the effect, then had to dig into
the code to figure out why. I am quite webserver agnostic and have
the benefit of being able to build whatever hosting environment I need
for a project.
In looking at my numbers, I'm not sure 50 requests per second is
unreasonable for the other project.
I will contact you offline with an issue I've run into with mod_wsgi
on an unrelated issue which is related to another blog post of yours.
On May 19, 7:26 pm, Graham Dumpleton <Graham.Dumple...@gmail.com>
wrote:
> Some related reading for your then:
>
> blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-
> usage.html
> http://blog.dscpl.com.au/2009/05/blocking-requests-and-nginx-version-...
>
> blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-
> usage.html
Reply all
Reply to author
Forward
0 new messages