Scalability of web2py?

[Lorin Rivers]

The project I'm working on has hired a consultant who is now recommending .Net in place of web2py or even rails.

What's the 'largest' scale web2py is known to perform well on?

--
Lorin Rivers
Mosasaur: Killer Technical Marketing <http://www.mosasaur.com>
<mailto:lri...@mosasaur.com>
512/203.3198 (m)

[mdipierro]

You achieve scalability by replicating the web server behind a load
balancer. This is documented in the book, chapter 11, using HAProxy.
All frameworks work the same way in this respect. web2py has no
intrinsic limitations. The bottle neck is the database connection. All
frameworks have the same problem. You can replicate the database too
and web2py supports multiple database clients with Round-Robin.

On a small VPS, web2py in average, should execute one page in 20ms.
Depending on how many requests/second you need you can determine how
many servers you need.

web2py apps run on Google App Engine and that means arbitrary
scalability as long as you can live with the constraints imposed by
the Google datastore (these limitations will go away as soon as Google
releases MySQL in the cloud, which they announced some time ago).

Please ask the consultant: which .NET feature makes it scale any
better than web2py or Rails? If he explains we can address it more
specifically.

Massimo

On Nov 29, 11:56 am, Lorin Rivers <lriv...@mosasaur.com> wrote:
> The project I'm working on has hired a consultant who is now recommending .Net in place of web2py or even rails.
>
> What's the 'largest' scale web2py is known to perform well on?
>
> --
> Lorin Rivers
> Mosasaur: Killer Technical Marketing <http://www.mosasaur.com>

> <mailto:lriv...@mosasaur.com>
> 512/203.3198 (m)

[Lorin Rivers]

We're looking at utility scale deployments with thousands of nodes reporting data back to the server. That and the ability to compile .NET.

<mailto:lri...@mosasaur.com>
512/203.3198 (m)

[Julio Schwarzbeck]

And this without considering "vendor lock-in". web2py can run on a
variety of platforms such as windows, macs. Linux and others, same
goes for the selection of the back-end database. Much more flexibility
under web2py in my opinion and prototyping is much faster in python.

[Lorin Rivers]

Unfortunately, the killing argument is "we know .NET will scale to thousands of nodes, blah, blah, blah".

This from (a guy who's smart and I respect, honestly) who uses his brand-new top-of-the-line 17" MBP to run Windows VMs in Parallels.

<mailto:lri...@mosasaur.com>
512/203.3198 (m)

[ron_m]

Beware the consultant that only recommends what they know about. :-)

Sounds like early optimisation attempts by the consultant if you ask
me. You have to do a traffic analysis on the network - sounds like
these are scales at remote locations - how often do they report in,
how much information per transaction, how much database activity per
transaction? My belief is either the network or the database will be
the real limiting factor. Worrying about whether or not the code is
compiled at this point is blowing smoke IMHO with 2GHz+ 4 core Nehalem
Xeons running $300 or $400 at most. CPU hardware is ridiculously
cheap. On the busiest system I am looking after I have trouble pushing
one core to 20% and I am moving 8 frames/sec of 704x480 pixel JPEG
camera video from 24 cameras to disk, each camera generates about 1
GByte of data per hour to put it in perspective. This is all done in
Python layered over the C/C++ library gstreamer which is admittedly
nothing to do with web2py but worth mentioning because of the
languages used. The reason Python can be so fast is all the heavy
lifting is done in C with the excellent integration to C libraries
Python provides. Same thing goes for when you hit the database. I come
from a Java background prior to Python and know what a pain JNI can be
which is why Java types try to do it all in Java. With .Net you are in
a similar environment to Java and have to do all the programming
boilerplate other than what they can generate for you. Expect
somewhere around 10x lines of code comparing .Net or Java to Python.

A far more important consideration is what does the programming team
know and is efficient working in. Having worked in both Java and
Python I can say with a certainty Python is like the walls have been
knocked down dealing with collections for example. Not meaning to
start any religious wars about languages but at age 59 I think I am
officially an old fart, I have been in the industry since 1977 and
have kept up to date with the technology else I would still be looking
for Fortran jobs. :-)

Ron

[ron_m]

Sticking my nose in one more time :-)

Another consideration for system choice is uptime loss for patch
maintenance. From the tiny bit of information so far, the data
collection off these scales may have to be nearly real time - web
system down = nothing moves in the field. Is that true? I am having a
vision of highways dpartment truck scales or something like that. Then
you will have to design for fail-over so machines can be patched.

A very common architecture in the securities trading business is to do
as little as possible on the front end except ingest the data and get
it into a queue with persistence, for example RabbitMQ. Then process
the downstream data with multiple systems as needed to deal with the
throughput. RabbitMQ is written in Erlang which has high availability
built in because of the way it is architected. The Erlang language,
libraries and OTP the queue system came out of Ericsson and their
Swedish telco switch business, I tend to believe they know what they
are talking about when it comes to high availability. RabbitMQ is
built on top of that. This is readily available on Ubuntu as a
standard install package. There are Python interfaces and I believe
orbited discussed recently does STOMP protocol as an interface into
RabbitMQ or ActiveMQ. At this point I may sound like I am guilty of
trying to design your system and am jumping ahead of knowing the true
requirements. :-) I am just pointing out other considerations and
products all readily available in the open source domain that may be
worth looking at. Once you are forced onto the .Net platform the
alternatives are much more restrictive. I am also not looking for a
job, I have too much to do right now. :-/

Cheers, hope you end up with something workable that will solve the
problem.

Ron

[mdipierro]

Some political considerations (which may be wrong and off topic and
improper)...

Here is a problem with external consultants. They make more per hours
than the average employees. They get hired because of their specific
expertise to tell you what the boss wants to say but he prefers
somebody else to say (so he does not take the responsibility for
saying it).

You cannot win this argument on technical merits. I would dismiss this
argument and point to Google as a scalability example and it is not
written in .net. I would address the real concern... you push web2py
therefore you are a single point of failure. If you leave who takes
care of this software? Not a problem with .net, they can always hire a
consultant.

I would stress that using web2py is good for rapid prototyping and it
will allow the company to have a test product much sooner than
with .net and at much lower cost. Once the prototype is built you will
be in a better situation to assess whether web2py or .net is the best
tool for the job. If you start developing in .net you will have higher
startup costs and limited flexibility to change the specs. web2py code
is much more compact and readable than .net code and it will be easier
to train other people to work with it and learn how it works than
with .net. Tell them experts4solutions.com can sell them long term
support contracts and code review.

The scalability bottle neck is the database. Offer something to the
consultant. .net uses mssql. If he claims mssql scales well for your
case, web2py will use mssql.

If mssql does not scale well with web2py you have other options and do
not need to rewrite code.

You can always reuse most of the design (html, js, css, images).

Management costs. I am sure you can make the case it costs less to run
linux vps than windows ones (although I have no experience with the
latter).

Massimo

> <mailto:lriv...@mosasaur.com>
> 512/203.3198 (m)

[Michele Comitini]

@Lorin

Ask the consultant which architectures and languages he knows and let
him make a table of advantages and pitfalls of each.
You will be able to take your decisions then

A solution that fits for any problems does not exist, but theorical
scalability is a non issue on many frameworks.
Indeed programmers can make such errors that can make run slow even
the most powerful system :)

To scale .NET well you must run on OS' that scale well, so you better
run .NET on Mono and choose a proper OS.
Scalability is not only a matter of application framework, systems
like Linux allows you to put new nodes fast with no
licensing fights, you can make 1000s VMs in seconds, you cannot do
that on commercial OS'.
Also if database is involved that is another critical point for
scalability, commercial DBs scale at high cost.

mic

2010/11/29 ron_m <ron.m...@gmail.com>:

[Lorin Rivers]

The good news is that the guy who hired ME picked web2py in the first place…

--

Lorin Rivers
Mosasaur: Killer Technical Marketing <http://www.mosasaur.com>

<mailto:lri...@mosasaur.com>
512/203.3198 (m)

[Michele Comitini]

great!

2010/11/29 Lorin Rivers <lri...@mosasaur.com>:

[ron_m]

:-)

> <mailto:lriv...@mosasaur.com>
> 512/203.3198 (m)

[Christopher Steel]

+1

I agree, Web2py is excellent for low cost prototyping, much less
than .NET and you will actually have something that works...

I hear that these guys can kick one out in under a month ->
http://experts4solutions.com/

; )

Chris

[Branko Vukelic]

We know .NET will scale to thousands of nodes IF you write the .NET
code right. If you write crappy code (and that's inevitable if you
don't like .NET or you don't know .NET), it will not only NOT run on
thousands of nodes, but will probably crash all of them.

Having said that... if they can help you write better code on .NET
than you currently write in web2py, the above argument turns on you.

--
Branko Vukelić

bg.b...@gmail.com
stu...@brankovukelic.com

Check out my blog: http://www.brankovukelic.com/
Check out my portfolio: http://www.flickr.com/photos/foxbunny/
Registered Linux user #438078 (http://counter.li.org/)
I hang out on identi.ca: http://identi.ca/foxbunny

Gimp Brushmakers Guild
http://bit.ly/gbg-group

[Lorin Rivers]

The number of people that can write code better than I can is close to the number of people who CAN write code…

[Jason Brower]

You may be suprised how good you are.  Especially in such short time you can improve.  I went from barely scraping up web-pages to some pretty impressive intra-net sites in just a few months.
.Net will eventually teach you how not to code.  It makes you truly appriciate web2py.
It's tough to have persuade people that have a big bully behind them.  I know the feeling, but if you can develop the prototype and you do it right, you win.  Besides, I wonder if this consultant is coding at all.  His skills to code is directly relative to the statements he makes, at least to me.
BR,
Jason

[Jason Brower]

I did prototype in 2.5 weeks that would have taken months on Apache Tomcat (what they were currently using).
And the company was massive, KONE.  The content had to handle thousands of requests, work with 2 database times and lots of XML, and be easy to use and expand.  I am not there now, but to this day they have always been impress with how fast I was able to code. (I am still buddies with the guys.)
BR,
Jason

[mdipierro]

I second!

On Nov 29, 10:31 pm, Jason Brower <encomp...@gmail.com> wrote:
> You may be suprised how good you are.  Especially in such short time you
> can improve.  I went from barely scraping up web-pages to some pretty
> impressive intra-net sites in just a few months.
> .Net will eventually teach you how not to code.  It makes you truly
> appriciate web2py.
> It's tough to have persuade people that have a big bully behind them.  I
> know the feeling, but if you can develop the prototype and you do it
> right, you win.  Besides, I wonder if this consultant is coding at all.
> His skills to code is directly relative to the statements he makes, at
> least to me.
> BR,
> Jason
>
> On Mon, 2010-11-29 at 18:09 -0600, Lorin Rivers wrote:
> > The number of people that can write code better than I can is close to the number of people who CAN write code…
>
> > On Nov 29, 2010, at 17:08 , Branko Vukelic wrote:
>
> > > We know .NET will scale to thousands of nodes IF you write the .NET
> > > code right. If you write crappy code (and that's inevitable if you
> > > don't like .NET or you don't know .NET), it will not only NOT run on
> > > thousands of nodes, but will probably crash all of them.
>
> > > Having said that... if they can help you write better code on .NET
> > > than you currently write in web2py, the above argument turns on you.
>

> > >> <mailto:lriv...@mosasaur.com>

> > >> 512/203.3198 (m)
>
> > > --
> > > Branko Vukelić
>

> > > bg.bra...@gmail.com

[Michele Comitini]

We, people used to free software, keep forgetting some goodies of
things like web2py, since we give those for granted.

- full version, no demo or limited in time bull
- see how it is coded and know the *real* quality
- code written to solve problems and not sell licences
- support from the ones that designed the software

mic

2010/11/30 mdipierro <mdip...@cs.depaul.edu>:

[Branko Vukelic]

On Tue, Nov 30, 2010 at 1:09 AM, Lorin Rivers <lri...@mosasaur.com> wrote:
> The number of people that can write code better than I can is close to the number of people who CAN write code…

Let me share an anecdote from where I work. One of the IT staff here
fiddles with VB from time to time. He does consider himself an
intermediate programmer. So he comes to me from time to time to ask
for advice on how to do this or that. And each time, my first response
is: "Try to google out a library that does that." It almost always
turns out that there is (a) no such library shared on the net, or (b)
you have to pay for it. That's the kind of eco system he's in. We all
know how beautifully different that is with open-source software.

So, it's not just writing code. It's about "I'm not the brightest kid
on the block, and there might be others, who are smarter, and have
maybe shared their smarts with the rest of us." And you download that
lib, and you learn the API, read the source, and learn a thing or two.
So you don't always have to write better code than others.

Having said all of above... I've tried Ruby, Erlang, VB, Python, a
little bit of C. You always learn a few things by just trying to get
something working in any environment, and you can use that knowledge
to advance your understanding of the tool you chose. In other words, a
little experimenting with .NET won't hurt you a bit. If I'm not
mistaken, there's "express" version of whatever tool Microsoft offers,
and you can use that to educate yourself. See how it works and define
your arguments against it with something to back it up. And MSDN is a
fine piece of knowledge base, too, just to be fair. I've learned a lot
from it alone back when I didn't know how to google for stuff.

[Julio Schwarzbeck]

Even though I "get" your point, I ffeel that putting your eggs in the
MS basket is a really bad idea. Yes, one can "learn" from these
frameworks, C# has come a long way to be a relatively "cool" language
(which incidentally is because it has been "pythonized" during his
later incarnations, if anyone does not believe me just check C#'
"Dynamic" object and attribute declaration, heck, they even talk about
"duck typing" as a valid paradigm..

Here is the problem with the MS environment: it's *not* googling
"string methods", but "real" questions such as "instantiating a COM
component gives error 0x0a0b0d0e0f" (after you followed the same
procedure on a previous PC without problems), you will get the
wirdiest set of answers (including some classic ones such "reboot the
pc", or "make sure you have the latest service patch" - translation:
useless answers) but *rarely* you will get a right answer. With
python, and possibly other open source technologies, it is morelikely
you find you answer pretty quickly OR you just write your own solution
as opposed to having to end up calling "golden" or "platinum" or
whatever precious metal MS uses now as an excuse to get your money..

Stick with python (and web2py for that matter), you will do youself a
great favor on the long run.

Julio

On Nov 30, 5:31 am, Branko Vukelic <bg.bra...@gmail.com> wrote:

> bg.bra...@gmail.com

[VP]

I think .NET and web2py are not competitors. They are different ball
games. When things scale large, everything is important. A small
design issue might be magnified. I don't think web2py is proven yet,
in terms of scalability. What web2py needs is at least one real
example to show that it is scalable and what are the limits at the
extreme.

[Stef Mientki]

I'm not sure were session files are used for,
but isn't file access much slower than database access ?

Can session files be turned of ?

cheers,
Stef

[Kuba Kucharski]

> Can session files be turned of ?

you can turn off sessions by

session.forget()

and also store sessions in the database
that is how it works on GAE. In every default db.py you have:

if request.env.web2py_runtime_gae:
db = DAL('gae')
session.connect(request, response, db=db)

[VP]

This is an example -- I think -- where web2py still needs to be proven
in terms of scalability.

Yes, sessions can be stored in database, turned off in specific
functions, and regularly deleted. But I still think that there
should be a better systematic way of handling sessions in heavy
applications. I think the current set up is not effective when you
have large volume of users, which suddenly builds up many sessions.

Small things like this don't matter much until things get large
scaled.

===

To take web2py to the next level, I think Massimo might need to lay
out milestones or visions to be achieved in future releases of
web2py.

[mdipierro]

We should have a mechanism for storing sessions clientside but this is
a speed issue not a scalability issues. Storing sessions in encrypted
cookies makes a each request 1-2ms faster by trading bandhwidth
(required to communicate the session in cookie) with file IO (required
to locate and access the session in a file).

Scalability-wise sessions in files do not present a problem as long as
one uses a load balancer with support with sticky sessions.

On common problem arises when there are too many session files in the
same folder (the os becomes slow accessing the folder). This problem
has been solved in trunk last week by creating a sub-folder tree
structure for the session folder.

Session in files some advantages of sessions in cookies: they can
contain more data. They can be controller server-side (checked,
deleted, modified). They can be locked to force per-user serialization
of form processing(default in web2py).

Bottom line: current sessions are not a scalability issue but it would
be nice to have the option of sessions in signed cookies. It is not
difficult to do and we have it by the end of the year.

This will be a major speed up for system that do not have a filesystem
and currently require storing sessions on datastore, like on GAE.

Massimo

[Thadeus Burgess]

All of the problems I had with web2py were because of the sessions. Even storing the sessions in db caused web2py to crash.

Never really found a solution, there is something to the fact of having to pickle/unpickle the session that can't handle major loads.

--
Thadeus

[John Heenan]

The elephant in the room has not gone away Massimo.

Web2py is great for small projects.

DotNet is great for small and large projects.

The elephant in the room is not only the untested scalibility of
web2py but also the amount of resources that neeeds to be thrown at
web2py compared to DotNet and other frameworks as scale increases.

One of the glaring defciences in web frameworks that use Python is the
glaring engineering weakness of using thread per request web serving
instead of using event per request web serving. I think I have pointed
this out a number of times on this fourm, but it just does not sink
in. I even pointed out how Linux loast a PR war over this issue.

There is no need for Python based web frameworks to use thread per
request web serving.

John Heenan

On Nov 30, 4:05 am, mdipierro <mdipie...@cs.depaul.edu> wrote:
> You achieve scalability by replicating the web server behind a load
> balancer. This is documented in the book, chapter 11, using HAProxy.
> All frameworks work the same way in this respect. web2py has no
> intrinsic limitations. The bottle neck is the database connection. All
> frameworks have the same problem. You can replicate the database too
> and web2py supports multiple database clients with Round-Robin.
>
> On a small VPS, web2py in average, should execute one page in 20ms.
> Depending on how many requests/second you need you can determine how
> many servers you need.
>
> web2py apps run on Google App Engine and that means arbitrary
> scalability as long as you can live with the constraints imposed by
> the Google datastore (these limitations will go away as soon as Google
> releases MySQL in the cloud, which they announced some time ago).
>
> Please ask the consultant: which .NET feature makes it scale any
> better than web2py or Rails? If he explains we can address it more
> specifically.
>
> Massimo
>

[Branko Vukelic]

Erlang is for humongous, real-time, distributed, and highly-available apps.

Here's an example (maybe quoted one time too often):

http://www.sics.se/~joe/apachevsyaws.html

--
Branko Vukelić

bg.b...@gmail.com

[Thadeus Burgess]

Would that be the decision of the webserver? gevent uses a event-per-request and it is written completely in python.

--
Thadeus

[Branko Vukelic]

On Wed, Dec 1, 2010 at 8:10 PM, Thadeus Burgess <thad...@thadeusb.com> wrote:
> Would that be the decision of the webserver? gevent uses a event-per-request
> and it is written completely in python.

Anyway, they didn't say ASP.NET. They just said .NET.

http://ironpython.net/

Run web2py on that, and wahoo!

[mr.freeze]

I actually have run web2py on ironpython. It runs surprisingly well.
You have to remove references to signal, csv and logging modules but
Rocket actually runs. No GIL either! I'm experimenting with running
web2py on IIS with nwsgi (http://nwsgi.codeplex.com/). I would gladly
ditch ASP.NET for web2py.NET but the former is currently paying the
bills.


On Dec 1, 1:16 pm, Branko Vukelic <bg.bra...@gmail.com> wrote:

> On Wed, Dec 1, 2010 at 8:10 PM, Thadeus Burgess <thade...@thadeusb.com> wrote:
> > Would that be the decision of the webserver? gevent uses a event-per-request
> > and it is written completely in python.
>
> Anyway, they didn't say ASP.NET. They just said .NET.
>
> http://ironpython.net/
>
> Run web2py on that, and wahoo!
>
> --
> Branko Vukelić
>

> bg.bra...@gmail.com

[Kuba Kucharski]

Web2py is waiting for the first brave ONE to (be able to) GO and try
to SCALE. I believe this person will get a lot of attention and
resources from community and Massimo himself.

Real WORLD scalable implementations are VERY CUSTOM thing.

like for example:
http://python.mirocommunity.org/video/1886/djangocon-2010-scaling-the-wor

btw, they use apache.

My point is: let's focus on our killer products that will create good
enough tracking and attention to.. scale.

[Jorge Restrepo]

Yes.
I second you on that, Kuba.


I also want to procede with a large scale app, but I am waiting...

[Marcello Parra]

What is the largest application running web2py today?

[mdipierro]

These are the apps we have knowledge of:

http://web2py.com/poweredby

I have no idea how popular they are.

Massimo

On Dec 1, 1:34 pm, Marcello Parra <parro...@gmail.com> wrote:
> What is the largest application running web2py today?
>

> On Wed, Dec 1, 2010 at 5:28 PM, Kuba Kucharski <kuba.kuchar...@gmail.com>wrote:
>
> > Web2py is waiting for the first brave ONE to (be able to) GO and try
> > to SCALE. I believe this person will get a lot of attention and
> > resources from community and Massimo himself.
>
> > Real WORLD scalable implementations are VERY CUSTOM thing.
>
> > like for example:

> >http://python.mirocommunity.org/video/1886/djangocon-2010-scaling-the...

[Michele Comitini]

did anyone try to run web2py on this?

http://pypi.python.org/pypi/pastegevent/0.1

2010/12/1 Jorge Restrepo <jorg...@gmail.com>:

[Michele Comitini]

better yet: http://pypi.python.org/pypi/Spawning/

2010/12/1 Michele Comitini <michele....@gmail.com>:

[ctalley]

Nowhere in this thread is the word "risk" used. Bottom line, there's
probably a reluctance to bet on web2py when it's not a known quantity
in an application with the stated demands. Most people are risk
adverse because they don't want to get beat about the head and
shoulders if something turns out not to work. At least with .Net one
can always claim that it's used successfully in large scale systems.
Massimo's suggestion to quickly and cheaply build and test a prototype
is the best idea. Then if it works, it works. You're dealing with
facts and not theory.

[John Heenan]

The OP asked "What's the 'largest' scale web2py is known to perform
well on? "

Massimo evaded the question and no one else has provided an answer.

The core of Erlang is very simple. Erland does no

> bg.bra...@gmail.com

[mdipierro]

The fact is that there is not one number to measure scalability.
There are many parameters like:
number of requests/second (average)
number of requests/second (peak)
complexity of those requests/second (static pages, dynamic pages,
types of db queries, do they use caching).
number of servers used.
type of database used.
Is the database replicated? (how many clients? how many servers?).

I stand by my comment that it does not matter which framework you use.
It matters how the database is accessed and designed. If the db
scales, but the app is slow, you can replicate web2py (and the web
server) as much as you want.

I know there are users here who have web2py behind a load balancer (if
you do please let us know so the information is public).

There are frameworks (microframeworks) that are more lightweight than
web2py because they do less. That means on a single machine (without
replicating the db not the web serve) for simple apps that do not need
the extra web2py stuff, they will be faster and therefore hey will be
able to serve more pages/second before they start to fail. I am not
sure this is the case for apps that need session, authentication, etc.

web2py introduces a penalty when you have lots of tables and there are
some optimizations that need to be done in that case.

There are some tricks to speed-up web2py which are important to scale
it:

- Even if your app does not run on GAE, create an empty APP on GAE and
use it to server ALL the static files. Use your server only for
dynamic HTML pages.
- set migrate=False
- do not use {{import ...}} too much, use {{block}} if necessary.
- remove un-necessary {{...}} from the layout
- move as much code from models to modules and/or controllers
- make many small controllers as opposed to one large one
- bytecode compile the app
- cache as many selects as possible in ram.
- use sticky session in the load balancer
- use Field(...,'upload',uploadseparate=True)
- use session.connect(request,response,separate=True)
- try separate a complex app into smaller modular apps that share a
central authentication service.
- run a background process that clears expires sessions
- run a background process for time consuming tasks such as sending
emails



Massimo

[Marcello Parra]

Massimo,

Any doc/reference about this?

Thanks

[John Heenan]

The OP, Lorin, asked "What's the 'largest' scale web2py is known to

perform well on? "

Massimo evaded the question and no one else has provided an answer.

Hence the elephant in the room has been ignored.

Web2py is designed to use an OS 'thread per web' request as Web2py is
built on a WGSI infrastructure. 'Thread per request' web serving is an
approach that has been conclusively proven to scale very badly
compared to using events or message queues (such as Erlang). Even when
Web2py does not get its request through a thread, Web2py forces
requests into a thread in order to use the WGSI infrastructure, such
as requests passed through a single UNIX socket from Lightttpd.

As an aside, the core of Erlang is very simple. Erlang uses message
queues to simulate multiple processing. That is it. This is different
from using events because each so called 'process' in Erlang pumps or
'processes' its own message queue in its own time. In fact Ericsson,
the original sponsor of Erlang, at one stage dropped using Erlang in
favour of porting the Erlang approach to other languages.

There are other web2py 'elephants in the room', such as the dependence
and direction of web2py on the enormous talents of a single
individual: Massimo. As an example of just how big an issue this is,
The Linux Foundation compiles surveys that points out just how minor
the role of Linus Torvaldis is in both writing kernel code and and in
committing code and also just how much code comes from corporations as
opposed to what might be called lone geeks (called consultants in
surveys). I think Linus has even dropped out of the top 30 individual
contributors for writing kernel code. The message is very clear. If
Linus suddenly decides he has had enough or wants to wind down,
'offcial' Linux continues on.

A professional approach requires hard fact and a serious approach to
legitimate issues that at the very least acknowledges legitimate
issues and avoids propaganda. Hard facts requires real world tests,
not evasion, defensive speculation and evangelism.

John Heenan

[mdipierro]

Some of them are discussed here:

http://web2py.com/book/default/chapter/11#Efficiency-and-Scalability

Some are new features like:
session.connect(request,response,separate=True)

This was proposed by author of http://www.zhjtjyw.com/ which is made
with web2py and seems to have lots of users. So many that too many
session files/hour in one folder would slow down the file system.
Unfortunately I do not have statistics.

On Dec 1, 7:28 pm, Marcello Parra <parro...@gmail.com> wrote:
> Massimo,
>
> Any doc/reference about this?
>
> Thanks
>

[John Heenan]

With regard to comments below about elephants in the Web2py room, such
as vulnerability to individuals with comment Linus is being downplayed
by the LF (Linux foundation), following is a quote from
http://www.desktoplinux.com/news/NS7450801259.html in section 'Top
Linux Contributors' with regard to a 2008 study.

"Linus Torvalds, meanwhile, has fallen off the top 30 list, says the
LF. However, the study notes that the original Linux developer is
still a major contributor. It also notes that the list does not
include "merge commits," when one set of changes is merged into
another, where Torvalds is said to play a key role. In addition, the
growing acceleration of kernel development has increased the time
Torvalds spends reviewing other kernel changes."

John Heenan

[VP]

While he's very knowledgeable with web2py, I think John Heenan's
criticisms are unfair. From what I understand, it seems most other
recognizable Python frameworks employ WGSI. So this problem (if
there's a problem), is not unique to web2py. Comparing web2py to
Erlang is curious.

I am a newbie, but from what I see around here I don't think Massimo's
effort or zeal is a setback as John seems to suggest. I think any
project needs at least one person who puts his heart and soul into
it. If the issue is Massimo doesn't like or prevents other
contributors from taking a leadership role in the development of
web2py or its entities (books, slices, etc.?), that's a different
story. Is this what John is suggesting? If not, I don't see why
Massimo or anyone being a dependable leader of the project is a bad
thing.

I think it is clear that web2py is not at the level of Linux. It's
not at the level of Drupal. It's not even at the level of Django.
Probably there's currently no "killer" app developed with web2py yet.
I think it's acceptable, at this stage. It would be better to put
forward constructive solutions to get more developers/users into
web2py.

[weheh]

To the original post in this thread, I think the idea for distributing
session files across the file system is sensible. I use what I believe
is a similar web2py capability for distributing files and I know it
works and is scalable.

As for the rest of the thread, scalability is a moving target. Bottom
line -- somebody has to be smart or lucky enough to build a killer
web2py app and then the web2py community has to support efforts to
make sure it scales.

I'm encouraged to see this discussion.

[pbreit]

This is a good discussion as it is apparent that web2py is an
incredible framework for rapidly building well-designed applications
but it's not totally clear what happens in a high volume situation.
But let's not forget that pretty much every framework experiences this
exactly. While Massimo is with us and we hope he will be for a long
time, let's really push web2py. That's the best way to identify and
resolve issues.

Many projects like this begin with one visionary and it is
understandable to be concerned about the future of the project. But
with valuable projects such as web2py, they figure out a way to
endure. That's the beauty of open source.

One thing that would help (me, at least) is spec'ing out a solid
server configuration. It's so easy and inexpensive now to pick an
environment on Amazon or Rackspace ($0-8/month)(for US people, at
least). There are already some great docs and scripts to get web2py
running on Ubuntu and Apache. It would be great to see some
information on Nginx and Cherokee (on Ubuntu 10.04).

[Branko Vukelic]

How about we do something similar to what Blender foundation is doing
for Blender? Build an interesting web app every year as a community
project, and deploy it somewhere, get some publicity, have people try
it... While doing that, we could focus our efforts on improving web2py
etc.

The project will focus on a subset of features. Like "Let's do an app
that focuses on session handling." or "Let's do an app that focuses on
real-time communication", or "Let's do an app that deals with nodes.",
or "Let's do an app that runs on a mobile platform."

We could ask for sponsorship, and develop an app that a sponsor will
have vested interest in, etc.

--
Branko Vukelić

bg.b...@gmail.com

[Tom Atkins]

I'm assuming John Heenan's criticisms of web2py would apply equally to Django?

Please correct me if I'm wrong.

Django can scale pretty well - here's a presentation showing how Disqus have scaled their Django app to 250 million visitors a month and a peak of 17,000 requests per second to Django with Apache and mod_wsgi:

http://nosql.mypopescu.com/post/1409154668/disqus-scaling-the-worlds-largest-django-application

Presumably web2py could do the same?

[Tom Atkins]

I should have said "John Heenan's criticisms of web2py relating to scalability..."

[Kuba Kucharski]

> requests per second to Django with Apache and mod_wsgi:
> http://nosql.mypopescu.com/post/1409154668/disqus-scaling-the-worlds-largest-django-application

exactly and already mentioned in this thread

[Tom Atkins]

Very sorry - I missed the link you already posted - will make sure to read full thread in the future!

[mdipierro]

I thought experts4solutions.com would help with that.

> bg.bra...@gmail.com

[Branko Vukelic]

On Thu, Dec 2, 2010 at 3:18 PM, mdipierro <mdip...@cs.depaul.edu> wrote:
> I thought experts4solutions.com would help with that.

Well, it hasn't proposed any projects yet, no? What I had in mind is
something along the lines of:

"Ok, so Company XXX has a product that does A, B, and C. The goal of
this project is to do A, B and C, and maybe do C a bit differently, by
July next year."

Then you could look for a sponsor that would possibly use the product
later on, and you allow them exclusive access to WIP code, you give
them access to bug tracker, pre-release code, etc, maybe a special
exclusive non-GPL license for the code (the finished product is
released under GPL, but company, say, keeps the right to close-source
its modifications, etc). That way, you get access to money, and people
who have an interest in keeping the project alive on longer term.

It's a bit different than the classic investment.

--
Branko Vukelić

bg.b...@gmail.com

[Michele Comitini]

@John

I think the best argument to end this discussion, is that someone,
possibly you, write a simple
and sane example that brings down and locks web2py, so we will come
over it and there will be no excuse to fix.

mic

[John Heenan]

Let us be blunt about this. Web2py is out there megalomaniacally
presenting itself as all things to all people at a web framework
level. As it is now, Web2py is a very impressive achievement.

At a purely technical level Web2py is superior to Django in a large
number of important aspects. But that is not the point. Leaving aside
the irritating Django fan-boys, Django does not present itself as all
things to all people. At the top level Django presents itself as a
commercially backed niche product suitable for simple content
presentation of small organisations (such as regional newspapers).

If we take a straight comparison with a completely different web
framework that presents itself as all things to all people. such as
DotNet, then web2py must accept there are issues for comparison that
go way beyond pet issues found in the narrow confines of the academic
or fan-boy environments.

Here are some simple comparison points.

1) Corporate backing independent of lead developers
2) Proven scalability around simple metrics
3) The 'largest' projects

DotNet has been around for ten years now and has very wide
penetration. Where is Web2py going to be in ten years and what will
its penetration be?

Frankly I cannot imagine someone with Massimo's talents as remaining
interested in Web2py, despite his deep involvement now in Web2py.
Massimo in a prior life was a talented high energy physicist, why
shouldn't he change again? Who knows, maybe a vice chancellor of the
University of Chicago trying to keep a lid on embarrassing student/
staff scandals or fraudulent research, or lobbying for grants?

I am not being unfair to Web2py.

With regard to Django and Disqus, 30% of about 100 servers (about 30
servers) just appear to run Apache + mod_wsgi. This presumably is
where Django lives. The other 70% of servers are for databases,
caching, load balancing and for other Python scripts. This information
comes from http://www.slideshare.net/zeeg/djangocon-2010-scaling-disqus.
Since we can expect a lot of the requests are for the same
information, we can expect caching is very important.

Disqus claims to be able to reach 17,000 requests for seconds or about
570 requests per second per Apache server. Suppose each request lasts
no longer than a generous 200ms on average then Apache needs to be
able to maintain about 114 requests at once. Since Apache spawns or
maintains a thread for each request, this means that each server needs
to be able to maintain nearly 114 threads per server at once. I am not
impressed by this.

Consider that Lighttpd can handle 10,000 requests per second ON A
SINGLE SERVER using year 2000 technology.

John Heenan

On Dec 2, 9:46 pm, Tom Atkins <minkto...@gmail.com> wrote:
> I'm assuming John Heenan's criticisms of web2py would apply equally to
> Django?
>
> Please correct me if I'm wrong.
>
> Django can scale pretty well - here's a presentation showing how Disqus have
> scaled their Django app to 250 million visitors a month and a peak of 17,000
> requests per second to Django with Apache and mod_wsgi:
>

> http://nosql.mypopescu.com/post/1409154668/disqus-scaling-the-worlds-...

[Anthony]

Hi John,

You make some good points, but...

> Let us be blunt about this. Web2py is out there megalomaniacally
> presenting itself as all things to all people at a web framework
> level.

I don't think this is quite fair. web2py is presented as a general
purpose framework for building web applications (which covers a lot of
ground), but I think Massimo and everyone else recognize it's not
necessarily the right tool in every conceivable circumstance.

> ...Django does not present itself as all

> things to all people. At the top level Django presents itself as a
> commercially backed niche product suitable for simple content
> presentation of small organisations (such as regional newspapers).

That's not at all the impression I've gotten. Read the Django FAQ:
http://docs.djangoproject.com/en/1.2/faq/general/#why-does-this-project-exist

> Since Apache spawns or
> maintains a thread for each request, this means that each server needs
> to be able to maintain nearly 114 threads per server at once. I am not
> impressed by this.
>
> Consider that Lighttpd can handle 10,000 requests per second ON A
> SINGLE SERVER using year 2000 technology.

I think the point being made about Disqus was that your criticism was
not unique to web2py, though it sounded as if it was.

Anthony

[pbreit]

I don't think anyone is comparing web2py to .net (thank goodness). A
better comparison is Ruby on Rails and Massimo is a bit like DHH. I
get the impression that he is pretty passionate about web2py and his
position at the university would enable him to continue working on it
for awhile. And if that doesn't happen, hey, it's open source and
anyone can pick it up.

What we're missing is something like the 37signals product suite to
push the framework in real world use. Is anyone here working on a
public web site that might scale up? I'm about to give it a shot. The
risk seems manageable. I've never really heard of anyone actually
being unable to scale a popular service. The problem is usually the
opposite: premature optimization.

[Branko Vukelic]

On Fri, Dec 3, 2010 at 8:16 AM, pbreit <pbreit...@gmail.com> wrote:
> ...and Massimo is a bit like DHH...

Fortunately you are almost too wrong. ;)

[Michele Comitini]

The very great thing about web2py is that even if it is full of
features, Massimo
was able and willing to keep it small. Small in the sense that it
has little code,
because it leverages on Python language features and patterns. What
does that mean? It means that anyone can reach and understand any
part of web2py code
and fix/optimize it, so in the long run web2py will just become better
and better, closed to best.

Aside from that, web2py development speed is very addictive, for me it
is getting harder and harder to switch back to slowness :)

mic

2010/12/3 pbreit <pbreit...@gmail.com>:

[blackthorne]

hi,
I haven't commented this discussion but I've been reading it. I have
to tell you that you are bouncing way out of "the point" ironically
complaining that everyone else is talking about things other than the
elephant in the room.

It really doesn't matter Django (unless you bring something concrete
about it), it doesn't matter Massimo prior life or possible
alternatives into it, it doesn't matter disqus can reach 17,000
requests per second in a very specific context where caching is
allowed and the infrastructure used is absent nor if Lighthttpd can
handle 10,000 unknown type of requests in an unknown hardware. I make
stress testing for a living, and my reality ranges between 200 virtual
users being enough to turn a powerful cluster down and me looking for
more virtual machines to get enough virtual users to bring a simple
server down.

..you may have something to say but you are over-elaborating your
comments.

A bit of reality may help you understanding things outside the
enterprise corp.

No, web2py doesn't have many "Corporate backing independent of lead
developers",
No, web2py doesn't have proven scalability based on simple metrics,
No, web2py doesn't have the "largest projects".

What else doesn't web2py have?
It doesn't have a huge enterprise like Microsoft that owns the
software that runs in most machines from the booting process until the
office suite in all software layers.
It's not around for more than a decade, because even dotNet didn't
started from the ground when it comes to asp.net.
So it's more recent, it doesn't have the time of existence required
for things to happen.
It doesn't have other enterprises working with it and for it including
in Marketing and distribution.
Oh and It doesn't have a price tag that would be prohibitive for over
a half of the members of this group.

What does it have?
It's open source, you can use it, you can see how it works and change
it.
You can talk to main contributors and Massimo directly. The
(expensive) phone support of .Net doesn't give you that.
It doesn't depend on external tools other than web2py core itself. You
can run in on any webserver, you change it's DAL to SQLAlchemy, you
can run in Python, Jython, IronPython, code it with any editor with no
compromise or even with just a web browser.
It's open from its core with a transparent architecture that makes
web2py development more like plugging python power to some smart
helpers which you can manage with total flexibility. It's not a black
box where you throw modules for big tasks and you shall trust them in
every way (including security) so that you can make something useful
of it.
It's platform-independent. I run it on my Nokia N900, in my Mac, in my
Linux Machines or even Windows if I thought it was a good idea to use
it as a server.
It's free.

Now, to the "point":
Yes, there is no need to handle one request by thread but that's not
the strangulation point (yet). Storage is. A truly distributed, fault-
tolerant and scalable storage system that provides transactions and
consistency properties is in most times the point of strangulation.
Keeping the web2py way of doing things you could adopt solutions that
explored concurrency better. Check http://www.slideshare.net/ezmobius/erlangfactory
for more about this. This is cloud computing without the buzz part.
It's possible to do this keeping the same MVC programming approach but
the architecture has to change a lot.

You don't have this on web2py (for now) neither on .Net . Speaking of
scalability on topics other than these two, storage and distributed/
concurrent task, will always be a minor talk.

> comes fromhttp://www.slideshare.net/zeeg/djangocon-2010-scaling-disqus.

[Branko Vukelic]

I don't know why but Java hasn't been mentioned even once. Java has
seen lots of deployment, and I imagine some huge stuff might have been
built on it, and it's open-source-ish. :)

PHP? It's well capable of handling it's business, you know. Facebook's
front-end runs on PHP. And it's open-source. Ugly as hell, too.

Ruby on Rails was another option, right? Has all the benefits of an
open-source project with all the benefits of the 'proven to scale'
argument?

And finally, Erlang. Proven to massively scale in right hands, proven
ability to deal with hugely concurrent apps... If it's good enough for
Ericsson, right? I mean telecomm software: real-time, distributed, and
how many people use phones (including cells) around the world?
Facebook chat runs on it, Github has some components ported to Erlang,
and generally, wherever real-timeness is a priority, it's unbeatable.

So, it's not .NET vs web2py. It's proven-to-scale (they gave .NET and
Rails as examples, iirc), versus web2py. You have open-source
solutions other than Python/web2py. Frankly, I'd always bet on Erlang.
But I never learn enough of it to get started. :)

[mdipierro]

What do you people make of this:

http://muharem.wordpress.com/2007/07/31/erlang-vs-stackless-python-a-first-benchmark/

(web2py runs on stackless except that out server does not take
advantage of its features).

Massimo

On Dec 3, 1:32 pm, Branko Vukelic <bg.bra...@gmail.com> wrote:
> I don't know why but Java hasn't been mentioned even once. Java has
> seen lots of deployment, and I imagine some huge stuff might have been
> built on it, and it's open-source-ish. :)
>
> PHP? It's well capable of handling it's business, you know. Facebook's
> front-end runs on PHP. And it's open-source. Ugly as hell, too.
>
> Ruby on Rails was another option, right? Has all the benefits of an
> open-source project with all the benefits of the 'proven to scale'
> argument?
>
> And finally, Erlang. Proven to massively scale in right hands, proven
> ability to deal with hugely concurrent apps... If it's good enough for
> Ericsson, right? I mean telecomm software: real-time, distributed, and
> how many people use phones (including cells) around the world?
> Facebook chat runs on it, Github has some components ported to Erlang,
> and generally, wherever real-timeness is a priority, it's unbeatable.
>
> So, it's not .NET vs web2py. It's proven-to-scale (they gave .NET and
> Rails as examples, iirc), versus web2py. You have open-source
> solutions other than Python/web2py. Frankly, I'd always bet on Erlang.
> But I never learn enough of it to get started. :)
>
> --
> Branko Vukelić
>

> bg.bra...@gmail.com

[Branko Vukelic]

I found this:

http://code.google.com/p/stacklessexamples/wiki/StacklessWSGI

--
Branko Vukelić

bg.b...@gmail.com

[blackthorne]

I've read it while ago.
The "problem" with that test is the number of processors. It takes a
high number of CPUs to bring Erlang benefits.

And scalability performance is only one issue.
For example, how do you deal with deployment of a new version in large
distributed system while on production? With Erlang you can do it on
the fly with no ups and downs and you can even take it back
seamlessly.
What happens if you want to keep it ON for months? I have no clue
about Stackless Python.

That said, considering what I've seen about python and the differences
to Stackless, you bet on it as a great solution! I love it!

On Dec 3, 7:40 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
> What do you people make of this:
>

> http://muharem.wordpress.com/2007/07/31/erlang-vs-stackless-python-a-...

[Branko Vukelic]

On Sun, Dec 5, 2010 at 1:59 AM, blackthorne <franci...@gmail.com> wrote:
> I've read it while ago.
> The "problem" with that test is the number of processors. It takes a
> high number of CPUs to bring Erlang benefits.

Another 'problem' is that it's not about performance when it comes to
Erlang. It's about overall robustness. For example, Yaws HTTP server
may not be the fastest around, but you just cannot kill it. Even if it
drops a request, it will keep on running, and handling whatever
requests you throw at it. I guess I had that in mind when I said
scalability.

Also, Erlang has software threads, afaik, not hardware CPU threads,
and it manages those internally using a supervisor-worker
architecture. That's something built into the language, and you mostly
don't have to worry about it.

--
Branko Vukelić

bg.b...@gmail.com

[mdipierro]

I tried this:


# http://eventlet.net/doc/examples.html#wsgi-server
import eventlet
from eventlet import wsgi
from rocket import Rocket


def hello_world(env, start_response):
if env['PATH_INFO'] != '/':
start_response('404 Not Found', [('Content-Type', 'text/
plain')])
return ['Not Found\r\n']
start_response('200 OK', [('Content-Type', 'text/plain')])
return ['Hello, World!\r\n']


if __name__=='__main__':
if True:
r=Rocket(('127.0.0.1',8081),'wsgi', {'wsgi_app':hello_world})
r.start()
else:
wsgi.server(eventlet.listen(('127.0.0.1', 8081)), hello_world)

with

ab -n 10000 -c 10 http://127.0.0.1:8081/

rocket: 0.618 [ms]
eventlet: 0.443 [ms]

ab -n 10000 -c 100 http://127.0.0.1:8081/

rocket: 0.629 [ms]
eventlet:

Benchmarking 127.0.0.1 (be patient)
Completed 1000 requests
Test aborted after 10 failures
apr_socket_connect(): Connection reset by peer (54)
Total of 1998 requests completed




On Dec 4, 7:39 pm, Branko Vukelic <bg.bra...@gmail.com> wrote:

> On Sun, Dec 5, 2010 at 1:59 AM, blackthorne <francisco....@gmail.com> wrote:
> > I've read it while ago.
> > The "problem" with that test is the number of processors. It takes a
> > high number of CPUs to bring Erlang benefits.
>
> Another 'problem' is that it's not about performance when it comes to
> Erlang. It's about overall robustness. For example, Yaws HTTP server
> may not be the fastest around, but you just cannot kill it. Even if it
> drops a request, it will keep on running, and handling whatever
> requests you throw at it. I guess I had that in mind when I said
> scalability.
>
> Also, Erlang has software threads, afaik, not hardware CPU threads,
> and it manages those internally using a supervisor-worker
> architecture. That's something built into the language, and you mostly
> don't have to worry about it.
>
> --
> Branko Vukelić
>

> bg.bra...@gmail.com

[mdipierro]

This is supposed to be even better

# http://code.google.com/p/gevent/source/browse/examples/wsgiserver.py
from gevent import wsgi


but cannot install on snow leopard.

Massimo

On Dec 4, 9:19 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
> I tried this:
>
> #http://eventlet.net/doc/examples.html#wsgi-server

> import eventlet
> from eventlet import wsgi
> from rocket import Rocket
>
> def hello_world(env, start_response):
>     if env['PATH_INFO'] != '/':
>         start_response('404 Not Found', [('Content-Type', 'text/
> plain')])
>         return ['Not Found\r\n']
>     start_response('200 OK', [('Content-Type', 'text/plain')])
>     return ['Hello, World!\r\n']
>
> if __name__=='__main__':
>     if True:
>        r=Rocket(('127.0.0.1',8081),'wsgi', {'wsgi_app':hello_world})
>        r.start()
>     else:
>        wsgi.server(eventlet.listen(('127.0.0.1', 8081)), hello_world)
>
> with
>

> ab -n 10000 -c 10http://127.0.0.1:8081/

>
> rocket: 0.618 [ms]
> eventlet: 0.443 [ms]
>

> ab -n 10000 -c 100http://127.0.0.1:8081/

[Phyo Arkar]

This discussion becomes not about comparison of scalability of web2py
or scability of web2py vs other framework but web2py vs other
langauges??

>massimo wrote:
>ab -n 10000 -c 100 http://127.0.0.1:8081/
>
>rocket: 0.629 [ms]
>eventlet:

Massimo , does eventlet failed in that test?

[Phyo Arkar]

If this become a langauge war:

Erlang have many scability advantages , yes , but Langauge Looks
horrible!! almost as readable as C (in my opinion)

I will only choose python .It will not be Super Scalable as erlang
but still scalable as proven by google.

sorry for my bad english.

[Branko Vukelic]

On Sun, Dec 5, 2010 at 8:38 AM, Phyo Arkar <phyo.ar...@gmail.com> wrote:
> If this become a langauge war:
>
> Erlang have many scability advantages , yes , but Langauge Looks
> horrible!! almost as readable as C (in my opinion)

Erlang is not just a language. When I say Erlang, I mean Erlang/OTP,
and OTP is a huge general-purpose framework for networked
applications. So effectively you are still comparing frameworks.
Language itself is actually beautiful. I thought it was ugly at first,
but I obviously no longer think so. Once you get used to the syntax,
you get absorbed in how powerful the language is.

--
Branko Vukelić

bg.b...@gmail.com

[mdipierro]

Yes. eventlet failed the test. OSX 10.6.3 and python 2.6

On Dec 5, 1:34 am, Phyo Arkar <phyo.arkarl...@gmail.com> wrote:
> This discussion becomes not about comparison of scalability of  web2py
> or scability of web2py vs other framework but web2py vs other
> langauges??
>
> >massimo wrote:
> >ab -n 10000 -c 100http://127.0.0.1:8081/
>
> >rocket: 0.629 [ms]
> >eventlet:
>

> Massimo , does eventlet failed in that test?
>

> On 12/5/10, mdipierro <mdipie...@cs.depaul.edu> wrote:
>
> > This is supposed to be even better
>

> > #http://code.google.com/p/gevent/source/browse/examples/wsgiserver.py

[Albert Abril]

I've been researching about Erlang because Branko comments.

There's a few frameworks, and there is a compare chart: 

http://www.chicagoboss.org/compare.html

I get impressed by zotonic. It appears to be a good cms+framework.

[Branko Vukelic]

Zotonic is a framework with CMS included. It's actually quite powerful
(and large). It also packs lots of next-gen stuff like websockets and
comet. Comes with freeimage-based image processing capability etc.
Very cool stuff. Of course, my poor excuse for Erlang programming
didn't live up to the challenge of actually mastering Zotonic.

--
Branko Vukelić

bg.b...@gmail.com