Web2py in large web scenarios

[Alfonso de la Guarda]

Hi,

I'm about to start a major project development and although I am a
regular user of web2py in my development, they have been mostly in
intranets, what worries me is if anyone has had previous experience of
putting an application based on web2py with several thousand users, in
which case I would like your job with this.

Previously, I have dealt with a project based on django super (500000
unique users month) and problems of performance, scalability and
others were so serious that they chose to pass it to php, I would not
do the same with this application in web2py.

Thanks....

Saludos,

--------------------------------
Alfonso de la Guarda
Centro Open Source(COS)
http://www.cos-la.net
http://alfonsodg.net
Twitter: @alfonsodg
Redes sociales: alfonsodg
   Telef. 991935157
1024D/B23B24A4
5469 ED92 75A3 BBDB FD6B  58A5 54A1 851D B23B 24A4

[Bruno Rocha]

My first tip is, avoid /models have your code in /modules import what you need only when and where you need!

--

Bruno Rocha

[http://rochacbruno.com.br]

[Ross Peoples]

This is very good advice. I have moved many of my plugins and apps from using models to modules because of the performance gain. There is nothing wrong with the models implementation, but it's really meant to define tables and that's it. Functionality that doesn't belong in a controller should go to modules.

Some other advice that I have been gathering for my own deployments is to use Nginx instead of Apache. You will see an incredible increase in the number of requests per second that Nginx can handle over Apache. Also setting up an Nginx front-facing server that is proxying requests to another server running Nginx + web2py might be a good idea for lots of traffic. Have the front-facing server handle the SSL traffic, and maybe even enable caching on the front-end as well. When you need to add another server to handle increased traffic, this front-facing Nginx server can be configured to load balance so that you can scale as much as you need.

[Ross Peoples]

One other thing I noticed after reading your question a second time: Python (Django and web2py) will run much faster than PHP in almost every case. If you needed to start passing things to PHP in order to speed things up, then you must have been using a highly-tuned PHP configuration and a very untuned Python configuration.

[Bruno Rocha]

I also noted  best performance directly rendering the templates and caching views where it can be cached.

just replacing

return dict()

with

return response.render(filename, context)

in conttrollers and also usins @cached controllers and cached queries.

the bottleneck is always server and database, so it is better to use pure Python to sort, find, filter Rows objects than using a lot of database requests.

DAL provides .sort .find .exclude and Python has a lot of good things like map, reduce, filter. With one db request you can fetch records put them in cache and use Python in some cases to avoid more sql queries. Even paginations can be done without the need to go to db again.

Redis is being a good cache solution.

http://zerp.ly/rochacbruno

[Magnitus]

>This is very good advice. I have moved many of my plugins and apps from using models to modules because of the performance gain. There is nothing wrong with the models implementation, but it's really meant to define tables and that's it. Functionality that doesn't belong in a controller should go to modules.<

Interesting, so to clarify: are you minimizing the amount of code into the model (limiting it to table definitions) or you avoid using a model at all?

[Vinicius Assef]

It really depends.

Querying and returning large datasets can become a huge bottleneck.
More than letting db server work on its own relations. If you use a
real RDBMS, it's designed to work on it.

Web2py doesn't create indexes. You have to make it manually. Making
so, RDBMS thanks you a lot and can be your close friend.

--
Vinicius Assef.

[pbreit]

Are there any good instructions for how to do this?

[pbreit]

1. It's much easier to scale than it is to gain enough users where scaling is a problem.

2. The framework is almost never the culprit. For example, Disqus has 500 million monthly uniques on Django.

3. There are lots of ways to optimize system performance from caching, to optimizing queries, to refactoring, to upgrading ram/cpu, etc.

4. In most cases, only a tiny percentage of users will be on at the same time so several thousand users should be pretty easy for most frameworks to handle.

[Michele Comitini]

About db bottlenecks:

I think that using executesql gives a performance gain of 2 orders of
magnitude over DAL on datasets with more than 1000 records. The DAL
is a huge bottleneck if used inappropriately.

I suggest using posgtresql with proper indexing. How?
Just take your query and run it inside an EXPLAIN statement, see where
indexes are needed and *create those indexes*. Speed improvements are
*dramatic*.
Pass to the DAL only small datasets: even if python has the best
sorting and filtering algorithms around (no kidding), postgresql
having those same algorithims is simply faster, because of the way
objects are handled in memory.

When Massimo says the db is the bottleneck, he does not mean that
relational DBs are slow in general, IMHO he means that _your_ DB is
slow, badly designed and/or badly queried.

I also suggest to pay attention to one more thing. When you write the
application and you do some test you do it in a quasi-singlethreaded
setup. Well everything works fine. Then you go in production and
after you reach 20 hits/sec your db becomes a bottleneck whatever
hardware you have. If you are using postgresql use a concurrency
model based on procesess, *avoid threads*. i.e. configure web2py to
work as uwsgi, scgi or fcgi service.

mic

2012/1/27 Vinicius Assef <vinic...@gmail.com>:

[Bruno Rocha]

I am using /models just to define small global functions and to set some response and request keys.

All my code including datamodels I  am putting into /modules

In my mind I changed the /models to /scripts or /batches so it is more easy to understand why avoiding it.

my current structure is:

/modules/datamodels/someentity.py - Database definitions

/modules/handlers/someentity.py - my code logic and template rendering

/modules/helpers/* - miscelanious

/modules/myappname.py - My.custom Auth, db, Mail, Service etc..

/controllers/someentytity.py - it will just be a point of entry, here I do selective imports, intantiate the entities and call the template rendering.

views files can be stored anywhere filesystem or database.

http://zerp.ly/rochacbruno

[pbreit]

What confuses me is what modules have access to and what needs to be imported (in modules and controllers).

[pbreit]

I wonder if DAL should have an ability to return JSON data sets that are then manipulated with JavaScript? Or something like that?

[Alfonso de la Guarda]

Hi,

Nice Bruno,
But those approachs how many traffic deals?
By the way, is the first time that i notice about use the /modules
approach to gain performance, are you measure the performance por
models inside /model and /modules?

To all:
Mores experiences? Problem resolutions?

Maybe put all of this inside a document will be great for help to many
users when scalability is their minds.

Saludos,

--------------------------------
Alfonso de la Guarda
Centro Open Source(COS)
http://www.cos-la.net
http://alfonsodg.net
Twitter: @alfonsodg
Redes sociales: alfonsodg
   Telef. 991935157
1024D/B23B24A4
5469 ED92 75A3 BBDB FD6B  58A5 54A1 851D B23B 24A4

[Vinicius Assef]

I'd use folders inside models.
I think I'd use web2py infrastructure that's already available to me.

And, AFAIK, /modules were not planned for that. Except for pluggable apps.

--
Vinicius Assef.

[Bruno Rocha]

submodels are good, but you cant import models. So you cant reuse code. you end with too much repetition.

in modules you can have a DRY structure.

https://github.com/rochacbruno/web2py_model_less_app

http://zerp.ly/rochacbruno

[Johann Spies]

On 27 January 2012 20:24, Bruno Rocha <rocha...@gmail.com> wrote:

I also noted  best performance directly rendering the templates and caching views where it can be cached.

just replacing

return dict()

with

return response.render(filename, context)

Is there documentation about this?  How do you code the view differently in such a case? 

the bottleneck is always server and database, so it is better to use pure Python to sort, find, filter Rows objects than using a lot of database requests.

DAL provides .sort .find .exclude and Python has a lot of good things like map, reduce, filter. With one db request you can fetch records put them in cache and use Python in some cases to avoid more sql queries. Even paginations can be done without the need to go to db again.

This might be true for smaller queries, but when working with thousands of records Postgresql is much more efficient than Python.

Regards
Johann

--
Because experiencing your loyal love is better than life itself,
my lips will praise you.  (Psalm 63:3)

[Johann Spies]

On 28 January 2012 01:06, Michele Comitini <michele....@gmail.com> wrote:

About db bottlenecks:

I think that using executesql gives a performance gain of 2 orders of
magnitude over DAL on datasets with more than 1000 records.  The DAL
is a huge bottleneck if used inappropriately.

I agree. I often find it easier to use executesql - many times just because formulating a complex query for me  is not that easy using DAL.  Also DAL cannot handle all the query types that one can do in Postgresql.

It is a pity that SQLFORM.grid seems to depend on DAL queries.   The drawback is that one cannot use it in direct combination with executesql - or can one?  I would like to know whether this is possible.

Regards
Johann

[Anthony]

I also noted  best performance directly rendering the templates and caching views where it can be cached.

just replacing

return dict()

with

return response.render(filename, context)

Is there documentation about this?  How do you code the view differently in such a case?

There's an example at the end of the section on caching: http://web2py.com/books/default/chapter/29/4#cache. response.render() is briefly mentioned here: http://web2py.com/books/default/chapter/29/4#response. Note, if you leave out the filename, it will assume the current response.view. No changes are necessary in the view code.

Anthony 

[Johann Spies]

On 30 January 2012 15:51, Anthony <abas...@gmail.com> wrote:

There's an example at the end of the section on caching: http://web2py.com/books/default/chapter/29/4#cache. response.render() is briefly mentioned here: http://web2py.com/books/default/chapter/29/4#response. Note, if you leave out the filename, it will assume the current response.view. No changes are necessary in the view code.

Thanks.

Johann