Turbogears 1.1 performance issues (nginx, memcached)
14 views
Skip to first unread message
Nils - VODO
Dec 2, 2010, 7:24:15 AM12/2/10
to TurboGears
Hi,
I need some help with sorting out our performance issues for http://vodo.net,
an online distribution site for free-to-share films.
Our basic problem is that, under even slightly heavy load, turbogears
slows down a great deal. We don't see any of the typical signs of high
load. CPU and RAM usage are low, and mysql seems to have no trouble
dealing with the requests. Rather, it seems that something is blocking
all the turbogears threads.
We are running the site under Nginx which handles all requests for the
website and serves static files from disk and cached files from
memcached. It also acts as a load balancer dividing requests between 3
turbogears instances.
We also have a bittorrent tracker and a bittorrent seeder running on
the same machine. The tracker handles somewhere upwards of 10 million
hits per day, although the size of most queries and responses is quite
small.
We are putting out a new release on the 13th of December and need to
address this issue before then. For the last couple of releases we
have been forced to use static landing page and bypass turbogears but
we need to enable functionlity for this one.
Please get in touch if you have experience with setting up high-
traffic sites using nginx, turbogears and memcached and let me know
your availability, costs and proposed scope for having a look at our
setup. Please keep in mind that we're a very small team and can't pay
a lot. More documentation is available as a PDF and you would get
access to the server to sort out any problems.
Best Regards
Nils Hellberg
VODO
http://vodo.net
I need some help with sorting out our performance issues for http://vodo.net,
an online distribution site for free-to-share films.
Our basic problem is that, under even slightly heavy load, turbogears
slows down a great deal. We don't see any of the typical signs of high
load. CPU and RAM usage are low, and mysql seems to have no trouble
dealing with the requests. Rather, it seems that something is blocking
all the turbogears threads.
We are running the site under Nginx which handles all requests for the
website and serves static files from disk and cached files from
memcached. It also acts as a load balancer dividing requests between 3
turbogears instances.
We also have a bittorrent tracker and a bittorrent seeder running on
the same machine. The tracker handles somewhere upwards of 10 million
hits per day, although the size of most queries and responses is quite
small.
We are putting out a new release on the 13th of December and need to
address this issue before then. For the last couple of releases we
have been forced to use static landing page and bypass turbogears but
we need to enable functionlity for this one.
Please get in touch if you have experience with setting up high-
traffic sites using nginx, turbogears and memcached and let me know
your availability, costs and proposed scope for having a look at our
setup. Please keep in mind that we're a very small team and can't pay
a lot. More documentation is available as a PDF and you would get
access to the server to sort out any problems.
Best Regards
Nils Hellberg
VODO
http://vodo.net
Diez B. Roggisch
Dec 2, 2010, 7:45:24 AM12/2/10
to turbo...@googlegroups.com
Some free tips:
- I guess you are using nginx in proxy mode. Thus the TG-instances are multi-threaded. Python does not scale really great with multiple threading, so upping the number of TG-instances might help.
- the low CPU and RAM usage are suspicious. Whenever we encounter this within our Apache/TG2/Postgres stack, we take a look at the DB. Most of the time, that's the culprit, because of lock congestion. I guess MySQL offers similar insights.
Diez
Sam
Dec 4, 2010, 2:48:28 AM12/4/10
to TurboGears
Hi Nils...
I +1 Diez's suggestion of paying attention the the database. What
does your disk subsystem look like? What are the rest of the specs on
the hardware?
Do you have an environment where you can simulate load and reproduce
the problem? Without having to change things in a production
environment?
> I need some help with sorting out our performance issues forhttp://vodo.net,
I +1 Diez's suggestion of paying attention the the database. What
does your disk subsystem look like? What are the rest of the specs on
the hardware?
Do you have an environment where you can simulate load and reproduce
the problem? Without having to change things in a production
environment?
> I need some help with sorting out our performance issues forhttp://vodo.net,
Reply all
Reply to author
Forward
0 new messages