web2py performance issue or rather my fault?

[Kuba Kucharski]

I had prepared some mini-portal, I have 1000 unique visits a day
but there is a performance issue

bandwidth is ok
memory is ok
processor load is ok
cache.ram is set for almost all of the queries, and is set for 1 hour

but

sometimes, once every few hits, it loads signifacantly slower
weirdest thing is when you re-click the link it loads instantly, when
you left it working to load on itself, it is slow.. like 4 to 8
seconds

what could it be? where to look for an answer? I think some of you had
to see this before..

--
Kuba

[Thadeus Burgess]

I have experienced this under high capacity on my web2py sites. I have
not found a solution to the issue as of yet. All I can say is I have
done AB testing comparing different python web frameworks with a basic
database IO, web2py just doesn't perform under high load.

Perhaps it is a coding issue with *how* you are caching your queries.
When caching in ram, *each* web2py process has to cache its own
version of the select. So if you reload a couple of times, apache( or
your web server ) might determine it needs to spawn another process,
which then in turn needs to cache all of the queries again.

Solution:

A) Use memcached
B) or cache on disk AND ram, so if a new process starts, doesn't have
it cached in ram, it will pull from the disk much quicker than
re-executing the query.

A is better.

--
Thadeus

[mdipierro]

Can you tell us more about the setting of this test. Did you use
apache or the built-in server? Did you use the same server in all
cases?

Massimo

On 7 Lug, 16:06, Thadeus Burgess <thade...@thadeusb.com> wrote:
> I have experienced this under high capacity on my web2py sites. I have
> not found a solution to the issue as of yet. All I can say is I have
> done AB testing comparing different python web frameworks with a basic
> database IO, web2py just doesn't perform under high load.
>
> Perhaps it is a coding issue with *how* you are caching your queries.
> When caching in ram, *each* web2py process has to cache its own
> version of the select. So if you reload a couple of times, apache(  or
> your web server ) might determine it needs to spawn another process,
> which then in turn needs to cache all of the queries again.
>
> Solution:
>
> A) Use memcached
> B) or cache on disk AND ram, so if a new process starts, doesn't have
> it cached in ram, it will pull from the disk much quicker than
> re-executing the query.
>
> A is better.
>
> --
> Thadeus
>

[Rahul]

Hi All,
I have experienced a similar issue with web2py. Issue:
[sometimes, once every few hits, it loads signifacantly slower

weirdest thing is when you re-click the link it loads instantly, when
you left it working to load on itself, it is slow.. like 4 to

8seconds ] Earlier I reported this kind of issue in a separate
thread.

Some findings ----
Office Env:
My web2py powered site will be up in production soon and I am awaiting
for getting in-house feedback. I am testing this on my local system
with the below config
OS: Win xp professional version 2002 with sp2 (old but works fine)
Machine: Intel P4, 2.8 GHz and 1 GB ram.
Web2py version latest (Version 1.79.2 (2010-06-08 22:45:26) )
I encounter the above said issue every time the system is left idle
for some minutes (Ex: If I start the web2py server and I am using my
web2py application continuously, it works well but if left idle for
some 5 mins or so, it becomes slow as mentioned above)

At home tough:
Home Env:
OS: Win xp professional version 2002 with sp2
Machine: Core2 Duo with 2 GB Ram
Web2py version 1.76.5 (prior to server change)
I deploy the same app and my site in the applications directory and I
never get such delays...

Conclusions:
1- May be its my system setup that could be the culprit
2- May be some thing is wrong with web2py or my code.

One more thing I noticed was (when using executesql the query or page
hit was a tad slower than when using the sql syntax provided by DAL)
so I changed all executesql statements to equivalent DAL statements
and these seemed to perform a bit faster.

Finally, I'll check the existing code with latest web2py version on my
home environment and get back if required.

I just thought that this could help in some way (since I am facing the
same issue) so jumped in between this conversation. Sorry for that
though.

Massimo, Thanks for a wonderful framework.. Everytime I see it and use
it, it keeps getting better and better..

Thanks, Rahul

[Rahul]

All, sorry forgot to mention previously we test this in our own
network (office intranet) and not the internet. The number of users
testing is max 2 to 3.

Thanks

[mdipierro]

Whatever the problem is it must be resolved. The strange thing is I do
not experience this on my production system (~50000 hits/day and
distinct 1600 visitors/day).

What could it be?
- cron when it runs spikes CPU usage. DO NOT USE. In production run
web2py cron in a separate process, not the one that runs the web
server.
- web2py session lock (the the same user can only access one page at
the time unless the lock is released via session._unlock())
- sqlite lock (depending on the query and the complexity of
transaction this may take long)
- db connection pooling locks (when look up for open connection)
- cache locks (it is a global lock, to ensure data integrity)
- cache.ram can cause memory leaks if the key depends on variables
- problem with web server (Rocket or the apache or mod_wsgi, etc.)

[Kuba Kucharski]

> What could it be?
let's rule things out

> - cron when it runs spikes CPU usage. DO NOT USE. In production run

not the case. no cron, no cpu spikes

> - web2py session lock (the the same user can only access one page at

I have session.forget..

> - sqlite lock (depending on the query and the complexity of

not the case, I use mysql now

> - db connection pooling locks (when look up for open connection)

how to check this? I even did pool_size=10 but I never have more than
3 users at once I think
it does not help

> - cache locks (it is a global lock, to ensure data integrity)

how to debug this?

> - cache.ram can cause memory leaks if the key depends on variables

no memory leaks noticed

> - problem with web server (Rocket or the apache or mod_wsgi, etc.)

I use Rocket now, I don't know when I will be able to migrate to wsgi,
probably soon

[Iceberg]

Me too experience similar issue as Rahul did. Issue:

> > [sometimes, once every few hits, it loads signifacantly slower
> > weirdest thing is when you re-click the link it loads instantly, when
> > you left it working to load on itself, it is slow.. like 4 to
> > 8seconds ]

... or even dozens of seconds. :-/

My findings, well, not really:

- Both my develop machine (windows xp home edition with latest ServicePack), and my production machine (a Linux), have such issue occasionally. Both environments uses web2py's built-in server, rocket or cherrypy. Both serve no more than 10 people or so.

- I could not find a pattern to reproduce the problem. On my development laptop, I just got an impression that it is more likely to happen for the first request after the web2py has been idle for long time due to no workload. So perhaps it is because the OS had swapped out all memory owned by web2py, therefore web2py need to "warm up" again. This is not web2py's fault, I think.

On my Linux production server, similar "pattern" exists (but I am not sure, did not dig into httpserver.log to confirm that).

Besides, the out-of-response is also likely to happen when user visit my app's statistics page with 10+ charts showed by flash. Perhaps the sudden burst of 10+ concurrent requests race each other?

@Massimo: You can check your production server's httpserver.log to see what is the longest (slowest) response time. If it is small enough, that is good, otherwise ... oh by the way, do you use web2py's built-in server (now rocket), or do you use apache-like frontend? That could make difference.

Best regards,
Iceberg, 2010-Jul-08, 20:50(PM), Thu



----------------------- Original Message -----------------------
From: mdipierro <mdip...@cs.depaul.edu> Sender: web...@googlegroups.com
To: web2py-users <web...@googlegroups.com>
Date: Thu, 8 Jul 2010 02:23:56 -0700 (PDT)
Subject: [web2py] Re: web2py performance issue or rather my fault?
-------------------

[Thadeus Burgess]

Setup:

Two applications that perform a simple db insert, db select, render a
template and return it. All in all this is a very small application.
The application has two views, one insert/select view and one select
view. Each of these views are tested separately using apache ab test.

The insert view, will insert a random 6 letter name into the database,
and return a select of the last 10 records inserted.

The ab test compose of several of the recommended tests that can
easily be found in a google search. Different tests compose of
concurrency, number of connections, etc etc. Each tested individually.
The web server was restarted and the database reset in between each ab
test to prove that everything is starting with a fresh slate.

The db is in postgresql, and uses the following schema:

TABLE friends
COL id
COL name

Each application has their own blank database in postgres to start.

Web Server: Cherokee with uwsgi running on a 256MB VPS with slicehost.

During the testing, all other virtual servers were DISABLED except for
the one being tested. This was to ensure that nothing would interfere
with the application.

First framework tested:
Name: web2py
Configuration
-------------------
db: DAL
cron: disabled
app: compiled
cache: none
pool_size: 10

The end result was about 68% throughput. There were 32% 500 internal
server errors. Error logs reveal IOErrors and the such as has been
duly noted in my previous google group posts.

Second framework:
name: Flask / werkzeug
Configuration
--------------------
db: sqlalchemy
cron: n/a
app: python source
cache: none
pool_size: 10

The end result was 100% throughput, no 500 internal server errors, no
error logs.

--
Thadeus

[Kuba Kucharski]

> - Both my develop machine (windows xp home edition with latest ServicePack), and my production machine (a Linux), have such issue occasionally. Both environments uses web2py's built-in server, rocket or cherrypy. Both serve no more than 10 people or so.

> @Massimo: You can check your production server's httpserver.log to see what is the longest (slowest) response time. If it is small enough, that is good, otherwise ...  oh by the way, do you use web2py's built-in server (now rocket), or do you use apache-like frontend? That could make difference.

I moved to apache/WSGI from Rocket. For me the problem disappears.

[Kuba Kucharski]

> The web server was restarted and the database reset in between each ab
> test

but AFAIK ab tends to be unreliable.. it is also my conclusion from
benchmarking rocket vs cherrypy
try httperf?

[mdipierro]

This is very interesting. I have never used uwsgi and cherokee with
web2py. I will try reproduce this.

Massimo

[Thadeus Burgess]

I get similar results with httperf. Web2py has over 15% 500 internal
server errors, flask has none.

--
Thadeus

[mdipierro]

I use apache + mod_wsgi and therefore there is no httpserver.log
logging. I will enable it.

[mdipierro]

Last week I run some tests. I can confirm I have NO dropped requests
on web2py.com running with the recommended configuration (apache
+mod_wsgi 3.1).

Here is the httpserver.log for 3 days of running (I removed the IPs
from the logs and requests for static files served directly by
apache).

http://www.web2py.com/examples/static/logs.txt

You can see that most pages are served in about 50 millisecond and all
of them return a 200 OK.

At 2010-07-08 16:58:22 I also run a test and you can see the logs for
that. No propped request from ab.

Clearly something is wrong with Thadues setup. I cannot help debug
this if I cannot reproduce the problem.
As soon as I have some time I will install uwsgi and try it.

@Thadeus. Can you post the application you used for testing?

Massimo

[Thadeus Burgess]

Massimo, web2py.com is not a valid application to be testing this on. Most of what web2py.com does is just render a template and display it, it hardly does any kind of strenuous dbio.

As you know I have a few separate servers running different configurations.

I mainly decided to do this testing as to the problems I have had with my application at work (which runs apache + mod_wsgi). I wondered if my blog had the same issue (which when I ran ab tests on it and spiked its usage, it did). Interesting thing is my blog ran on cherokee + uwsgi. I doubt it is a configuration issue since I still have the same problem on two completely different apps running completely different setups all different except web2py. They both use postgres as well.

I am attaching the two apps that I tested, as well as a text file that contains the raw ab tests that I collected to run

--
Thadeus

[MikeEllis]

I was just about to create new topic, but thought I'd start by
reporting here since it could be related.

The short description is:

1. Serve web2py (rocket.py, sqlite3, etc) over a lan connection from a
laptop
2. Open a page from my app on another laptop.
3. Also open the same page on the server laptop.
4. Start clicking reload alternately between the two laptops at about
1Hz
5. All goes well for an indeterminate period of time, typically about
1 minute.
6. Occasionally a reload will take about 20 seconds. Can happen on
either box.

Other Info
* db is very small - only 2 tables with only a few entries each.
* App is not doing anything intensive and the pages are relatively
simple.
* web2py 1.92
* OS X 10.6 on both laptops
* Browser = Chrome

Observations
* No evidence of high CPU usage.
* Happens even with pages that don't write to the db.
* I even managed once to produce the problem with an unmodified
Welcome app, but it took a really long time to make it happen and the
latency wasn't as bad.

When I reproduce the problem with Chrome DevTools open, the delay
corresponds to an incredibly long latency, ie 20 seconds, fetching one
or more .js or .css modules. Usually it's jquery.timers-1.2.js or
base.css but can occur with other modules. It's never the page html,
that always arrives in milliseconds.

Here's the Header info from the long fetch on base.css:

Request URL:http://192.168.253.105:8000/init/static/base.css
Request Method:GET
Status Code:304 Not Modified

Request Headers
Accept:text/css,*/*;q=0.1
Cache-Control:max-age=0
If-Modified-Since:Mon, 14 Jun 2010 23:30:00 GMT
Referer:http://192.168.253.105:8000/init/editproblem/index
User-Agent:Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; en-US)
AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.99 Safari/533.4

Response Headers
Connection:keep-alive
Content-Type:text/html; charset=UTF-8
Date:Sun, 18 Jul 2010 22:32:09 GMT
Server:Rocket 1.0.5 Python/2.6.1


One other oddity: I've been seeing (and ignoring) the following
warning from DevTools when js/css files are loaded:

"Resource interpreted as stylesheet but transferred with MIME type
text/html."

Someone on stackoverflow.com suggested putting

<meta http-equiv="content-script-type" content="text/javascript">

into the head of the document. I tried that (in layout.html) but it
had no effect on the warnings.

Anyone seen similar problems or have an idea what could be causing the
long fetch times?

Thanks,
Mike

>  AB_WEB2PY_FLASK.tar.lzma
> 1011KViewDownload

[Thadeus Burgess]

I am wondering if this is a web2py issue or a DAL issue.

I would like to re-run my tests just rendering a simple template with no dbio and see if I still get the same results. I love the DAL and there is nothing else quite like it, and would hate for it to be the cause.

I would like to compare the DAL standalone, without web2py to determine it is not the cause.

--
Thadeus

[David Marko]

I can see the same problems with failed requests on many my machines.
(WinXP, Debian 5) . The simple test using default web2py examples
shows me, that template rendering is OK
e.g. http://localhost:8000/examples/template_examples/test_for but
when using form example like in http://localhost:8000/examples/form_examples/form
causes request failures. Constantly, when I did test repeatedly,
template rendering example caused no problem, but form based example
claimed 13-15% of failed requests for me. I have use apache ab. As
stated above, I could see the same results despite of platform.

I tested today on latest version 1.81.4 with default Rocket web server.
(I remember I could see the same issues with CherryPy as well)

David

On 19 čnc, 06:22, Thadeus Burgess <thade...@thadeusb.com> wrote:
> I am wondering if this is a web2py issue or a DAL issue.
>
> I would like to re-run my tests just rendering a simple template with no
> dbio and see if I still get the same results. I love the DAL and there is
> nothing else quite like it, and would hate for it to be the cause.
>
> I would like to compare the DAL standalone, without web2py to determine it
> is not the cause.
>
> --
> Thadeus
>

[Thadeus Burgess]

I am taking http://techspot.zzzeek.org/?p=17 and making a benchmark_dal.py and testing the results vs sqlalchemy.

So far it looks promising, the DAL seems to be much more efficient than SQLAlchemy, and brick tons faster too.

This is good news for me, as it proves that the DAL is valuable... if only we could make it its own package instead of apart of web2py... that will be my next project.

Here I have attached to aggregate screenshots, and the file used to test DAL. I compared both SQLAlchemy and the DAL on my VPS server, each got their own postgresql database.

I am using a slightly modified DAL (I removed validators completely). However, after I ran the tests I realized migrate was still set to True.... so If I were to re-run the tests I expect the DAL to outperform SQLAlchemy even more.

Refer to the link at techspot.zzzeeek for more information about the tests.

http://static.thadeusb.com/total_time.jpg
http://static.thadeusb.com/call_count.jpg
http://static.thadeusb.com/benchmark_dal.py

I think it is safe to say that the DAL is not what is causing these 500 server errors. I think David is on to something. On my production sites, I always get errors related to the pages with SQLFORM on them, never on regular pages that just render a template.

--
Thadeus

[David Marko]

One can even user examples hosted on web2py.com with same results as I
tested on my local machines.

#### simple temple rendering - no failed requests ##############
c:\apache\bin>ab -n 100 -c 5 http://web2py.com/examples/template_examples/test_def
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking web2py.com (be patient).....done

Server Software: Apache/2.2.8
Server Hostname: web2py.com
Server Port: 80

Document Path: /examples/template_examples/test_def
Document Length: 7311 bytes

Concurrency Level: 5
Time taken for tests: 18.656 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 775042 bytes
HTML transferred: 731100 bytes
Requests per second: 5.36 [#/sec] (mean)
Time per request: 932.813 [ms] (mean)
Time per request: 186.563 [ms] (mean, across all concurrent
requests)
Transfer rate: 40.57 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 172 179 9.3 172 203
Processing: 391 721 64.0 734 781
Waiting: 188 436 111.7 391 703
Total: 563 901 64.6 906 953

#### rendering form example - many failed requests apears constantly
##############
c:\apache\bin>ab -n 100 -c 5 http://web2py.com/examples/form_examples/form
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking web2py.com (be patient).....done


Server Software: Apache/2.2.8
Server Hostname: web2py.com
Server Port: 80

Document Path: /examples/form_examples/form
Document Length: 23849 bytes

Concurrency Level: 5
Time taken for tests: 23.203 seconds
Complete requests: 100
Failed requests: 95
(Connect: 0, Receive: 0, Length: 95, Exceptions: 0)
Write errors: 0
Total transferred: 2427908 bytes
HTML transferred: 2383978 bytes
Requests per second: 4.31 [#/sec] (mean)
Time per request: 1160.156 [ms] (mean)
Time per request: 232.031 [ms] (mean, across all concurrent
requests)
Transfer rate: 102.18 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 156 174 7.2 172 203
Processing: 859 947 84.9 922 1375
Waiting: 203 284 100.0 234 703
Total: 1031 1121 85.6 1094 1547

[Thadeus Burgess]

Now the question is "why?"

I don't see how it can just be the generation of the form, I am willing to bet it is the session and CSRF tokens.

If someone has time, we need to test a couple of scenarios.

A) display a simple FORM
B) display a simple FORM, but do NOT include a form.accepts in the controller
C,D) Repeat a and b but with SQLFORM
E) Move sessions to database and repeat C,D

I still expect that even with sessions in the database you will still run into issues, as I do run into these on one of my production sites that does have session in the database. I would like to see if a implementation of a SecureSignedCookie for storing the session would solve the issue (this is what flask uses). Also, large sites such as twitter, facebook, google, etc all use SecureCookies to store session, since storing as many sessions as they use in a database would overtax their resources.

--
Thadeus

[Jonathan Lundell]

On Jul 18, 2010, at 4:10 PM, MikeEllis wrote:

> Here's the Header info from the long fetch on base.css:
>
> Request URL:http://192.168.253.105:8000/init/static/base.css
> Request Method:GET
> Status Code:304 Not Modified
>
> Request Headers
> Accept:text/css,*/*;q=0.1
> Cache-Control:max-age=0
> If-Modified-Since:Mon, 14 Jun 2010 23:30:00 GMT
> Referer:http://192.168.253.105:8000/init/editproblem/index
> User-Agent:Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; en-US)
> AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.99 Safari/533.4
>
> Response Headers
> Connection:keep-alive
> Content-Type:text/html; charset=UTF-8
> Date:Sun, 18 Jul 2010 22:32:09 GMT
> Server:Rocket 1.0.5 Python/2.6.1

Notice that this is Rocket's cached response. I think you'll see the original file being served as text/css.

By comparison, I don't see a Content-Type in Apache's response (this is from Safari's debugger). Apache does include a couple of headers that Rocket does not.

• Request URL:http://web2py.com/examples/static/styles.css
• Request Method:GET
• Status Code:

[Timbo]

It's defaulting to text/html since it's not actually sending a file.
This is a section of the HTTP spec that I missed.
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html says that no
entity headers (Content-* should be sent in a 304 instance)

I think I can have a patch by this evening.

However, I've never seen this behavior before so I'll need for someone
to be able to test this to see if this is actually the cause of the
problem. Have we confirmed that the delay behavior happens in
different browsers? I only see chrome above.

-tim

>  warningOrangeDot.png
> < 1KViewDownload

>
> 304 Not Modified
>         • Request Headers
>                 • Accept:text/css,*/*;q=0.1
>                 • Cache-Control:max-age=0

>                 • If-Modified-Since:Sat, 17 Jul 2010 15:14:16 GMT
>                 • If-None-Match:"42f90-d44-48b96c9edc200"
>                 • Referer:http://web2py.com/
>                 • User-Agent:Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-us) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16
>         • Response Headers
>                 • Cache-Control:max-age=86400
>                 • Connection:Keep-Alive
>                 • Date:Mon, 19 Jul 2010 14:47:39 GMT
>                 • Etag:"42f90-d44-48b96c9edc200"
>                 • Expires:Tue, 20 Jul 2010 14:47:39 GMT
>                 • Keep-Alive:timeout=2, max=15
>                 • Server:Apache/2.2.8 (Ubuntu) mod_wsgi/3.2-BRANCH Python/2.5.2 mod_ssl/2.2.8 OpenSSL/0.9.8g

[Jonathan Lundell]

On Jul 19, 2010, at 10:05 AM, Timbo wrote:

> It's defaulting to text/html since it's not actually sending a file.
> This is a section of the HTTP spec that I missed.
> http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html says that no
> entity headers (Content-* should be sent in a 304 instance)
>
> I think I can have a patch by this evening.
>
> However, I've never seen this behavior before so I'll need for someone
> to be able to test this to see if this is actually the cause of the
> problem. Have we confirmed that the delay behavior happens in
> different browsers? I only see chrome above.

I'm doubtful that the content-type is related to the delay; it seems more like a cosmetic issue. Still, since Chrome complains about it....

[Timbo]

Whoa back up. Rocket doesn't set the content-type at all, web2py
takes care of that (streamer.py, line 63). Actually, Rocket will send
on whatever headers it gets, I think Cherrypy was a little more
selective, which may be one reason the Rocket switch would let this
show up.

W.R.T. the slow requests, Rocket has some pretty verbose logging built-
in that web2py doesn't expose (http://packages.python.org/rocket/
usage.html#logging) Can one of you who is experiencing this problem
run web2py from this script:

-------------------
import os
import sys
import logging
import logging.handlers
log = logging.getLogger('Rocket')
log.setLevel(logging.DEBUG)
log.addHandler(logging.handlers.FileHandler('rocket.log'))

try:
path = os.path.dirname(os.path.abspath(__file__))
except NameError:
path=os.getcwd() # Seems necessary for py2exe

if not path in sys.path:
sys.path.append(path)
os.chdir(path)

import gluon.import_all
import gluon.widget

# Start Web2py and Web2py cron service!
gluon.widget.start(cron=True)

-------------------------

Please use this to create a debug log of a delayed request (the more
the merrier) and email it to me. As a side-note, some of you have
requested some features for Rocket. I'm limiting development to
essentials right now because of a case of dual-arm tendinitis.

Thanks.

On Jul 19, 12:11 pm, Jonathan Lundell <jlund...@pobox.com> wrote:
> On Jul 19, 2010, at 10:05 AM, Timbo wrote:
>
> > It's defaulting to text/html since it's not actually sending a file.
> > This is a section of the HTTP spec that I missed.

> >http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.htmlsays that no

[Jonathan Lundell]

On Jul 19, 2010, at 4:28 PM, Timbo wrote:

> Whoa back up. Rocket doesn't set the content-type at all, web2py
> takes care of that (streamer.py, line 63). Actually, Rocket will send
> on whatever headers it gets, I think Cherrypy was a little more
> selective, which may be one reason the Rocket switch would let this
> show up.

Something else is going on. For one thing, I don't see content-type via Apache. For another, I tried this:

headers['Content-Type'] = contenttype(static_file)
headers['Last-Modified'] = mtime
headers['Pragma'] = 'cache'
headers['Cache-Control'] = 'private'

if request and request.env.http_if_modified_since == mtime:
headers['Content-Type'] = 'text/cached' <<<<<<<<<<<<<<<<<<<<<<<<
raise HTTP(304)
elif request and request.env.http_range:
start_items = regex_start_range.findall(request.env.http_range)
if not start_items:

And I don't see it: I still see text/html.

[Iceberg]

On Jul 20, 1:11am, Jonathan Lundell <jlund...@pobox.com> wrote:
> On Jul 19, 2010, at 10:05 AM, Timbo wrote:
>
> > It's defaulting to text/html since it's not actually sending a file.
> > This is a section of the HTTP spec that I missed.

> >http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.htmlsays that no

> > entity headers (Content-* should be sent in a 304 instance)
>
> > I think I can have a patch by this evening.
>
> > However, I've never seen this behavior before so I'll need for someone
> > to be able to test this to see if this is actually the cause of the
> > problem.  Have we confirmed that the delay behavior happens in
> > different browsers? I only see chrome above.
>
> I'm doubtful that the content-type is related to the delay; it seems more like a cosmetic issue. Still, since Chrome complains about it....
>

Tim,

I think Chrome is more sensitive than others to complain incorrect
content-types. It is easy to see it in a Chrome (mine is 5.0.379.99 on
WinXP). Just start your local web2py server, then visit http://127.0.0.1:8000/welcome
and right-click at the background and choose the last item to pop-up a
"Developer Tools" window, then refresh the welcome page, now you can
see those Warnings such as "Resource interpreted as image but

transferred with MIME type text/html."

It should not be harmful to satisfy chrome by Tim's coming patch. Even
it might not be relevant to the delay behavior.

With respect to the delay behavior, I think it is cross browser,
albeit I don't have direct evidence. Most of my app's end users use
IE6/7/8, sometimes they complain some page take long time to load, but
usually a manually refresh (retry) can bypass the problem. :-/

Regards,
Iceberg

[Timbo]

@Jon,

Notice that HTTP(304) call, that doesn't read response.headers, it
only takes in what headers you give it. In this case it's none. Flip
over to http.py line 77...there's your answer.

@Iceberg,

As I've noted above, this is a web2py issue. I'm not going to have
rocket remove headers that the application supplies. This goes back
to the nanny web-server thing we discussed in another thread.

@Everyone,

So anyone wanna plugin that logging code I posted earlier and send me
a log? Please.

-tim

On Jul 19, 10:34 pm, Iceberg <iceb...@21cn.com> wrote:
> On Jul 20, 1:11am, Jonathan Lundell <jlund...@pobox.com> wrote:
>
>
>
>
>
> > On Jul 19, 2010, at 10:05 AM, Timbo wrote:
>
> > > It's defaulting to text/html since it's not actually sending a file.
> > > This is a section of the HTTP spec that I missed.

> > >http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.htmlsaysthat no

> > > entity headers (Content-* should be sent in a 304 instance)
>
> > > I think I can have a patch by this evening.
>
> > > However, I've never seen this behavior before so I'll need for someone
> > > to be able to test this to see if this is actually the cause of the
> > > problem.  Have we confirmed that the delay behavior happens in
> > > different browsers? I only see chrome above.
>
> > I'm doubtful that the content-type is related to the delay; it seems more like a cosmetic issue. Still, since Chrome complains about it....
>
> Tim,
>
> I think Chrome is more sensitive than others to complain incorrect
> content-types. It is easy to see it in a Chrome (mine is 5.0.379.99 on

> WinXP). Just start your local web2py server, then visithttp://127.0.0.1:8000/welcome

[Jonathan Lundell]

On Jul 20, 2010, at 4:27 PM, Timbo wrote:

> @Jon,
>
> Notice that HTTP(304) call, that doesn't read response.headers, it
> only takes in what headers you give it. In this case it's none. Flip
> over to http.py line 77...there's your answer.

Ah, right. We shouldn't be adding content-type headers to statuses without content. Ideally there wouldn't be a default at all (leave it to the caller), but that might go and break something. I'm guessing that a dict with statuses that take content would be fairly short.

Massimo, there's a typo in the defined_status dict at the top of http.py:

411: 'LENGHT REQUIRED',

[mdipierro]

Hi Thadeus,

I looked at the code you used for comparing web2py and Flask.

This is NOT A FAIR COMPARISON

Here is why:

1) you did not set migrate=False in web2py. This means at every
request you lock the entire database table metadata to check whether
the table has migrated. You do this while you have an open transaction
with the postgresql database. For such short code, this essentially
serializes all your requests. You should run it once with migrate=True
and then run the tests with migrate=False.

2) in the case of Flask you use raw SQL for the queries. In the case
of web2py you use DAL syntax. On such a short program I cannot say
what the DAL overhead may be but certainly you have it in one case and
not the other.

3) The web2py program has sessions enables (has to parse cookies,
create and parse session files, store the session, files, generate new
cookies and lock session files - at every request). You did not even
set session.forget() which means a new session file is saved at every
http request. The Flask program is doing nothing of this.

Of course the Flask program under this conditions performs better.

A more fair test would be:
1) set migrate=False in the web2py code
2) use SQLAchemy syntax for database queries in the flask code
3) enable serverside session in Flask code

Massimo

On Jul 18, 12:20 pm, Thadeus Burgess <thade...@thadeusb.com> wrote:
> Massimo, web2py.com is not a valid application to be testing this on. Most
> of what web2py.com does is just render a template and display it, it hardly
> does any kind of strenuous dbio.
>
> As you know I have a few separate servers running different configurations.
>
> I mainly decided to do this testing as to the problems I have had with my
> application at work (which runs apache + mod_wsgi). I wondered if my blog
> had the same issue (which when I ran ab tests on it and spiked its usage, it
> did). Interesting thing is my blog ran on cherokee + uwsgi. I doubt it is a
> configuration issue since I still have the same problem on two completely
> different apps running completely different setups all different except
> web2py. They both use postgres as well.
>
> I am attaching the two apps that I tested, as well as a text file that
> contains the raw ab tests that I collected to run
>
> --
> Thadeus
>

> On Sun, Jul 18, 2010 at 4:39 AM, mdipierro <mdipie...@cs.depaul.edu> wrote:
> > Last week I run some tests. I can confirm I have NO dropped requests
> > on web2py.com running with the recommended configuration (apache
> > +mod_wsgi 3.1).
>
> > Here is the httpserver.log for 3 days of running (I removed the IPs
> > from the logs and requests for static files served directly by
> > apache).
>
> >    http://www.web2py.com/examples/static/logs.txt
>
> > You can see that most pages are served in about 50 millisecond and all
> > of them return a 200 OK.
>
> > At  2010-07-08 16:58:22 I also run a test and you can see the logs for
> > that. No propped request from ab.
>
> > Clearly something is wrong with Thadues setup. I cannot help debug
> > this if I cannot reproduce the problem.
> > As soon as I have some time I will install uwsgi and try it.
>
> > @Thadeus. Can you post the application you used for testing?
>
> > Massimo
>
>
>

>  AB_WEB2PY_FLASK.tar.lzma
> 1011KViewDownload

[mdipierro]

One more thing.... because of design flask with generate and reuse pyc
file. In the case of web2py python code is executed. You have to
explicitly "bytecode compile" via admin. I am not sure if you have
done that.

Massimo

[Iceberg]

Hi Massimo, do you hint that all these measure are needed to minimize (if not get rid of) the occasional slow response?

1) set migrate=False in the web2py code

2) ... (N/A)
3) disable session feature as long as your app does not need it. (But how to? Is session.forget() enough?)
4) bytecode compile the app to avoid execfile() call

I have to admit that my apps do not follow any of above suggestion because I thought they were not a must. Tell me if I was wrong.

On the other hand, Kuba, who started this thread, later said "I moved to apache/WSGI from Rocket. For me the problem disappears." So I guess there is a standalone rule #5: Use apache/WSGI etc. instead of the built-in Rocket ?

Best regards,
Iceberg, 2010-Jul-21, 18:01(PM), Wed

----------------------- Original Message -----------------------
From: mdipierro <mdip...@cs.depaul.edu> Sender: web...@googlegroups.com
To: web2py-users <web...@googlegroups.com>
Date: Wed, 21 Jul 2010 01:55:27 -0700 (PDT)
Subject: [web2py] Re: web2py performance issue or rather my fault?
-------------------

[mdipierro]

The issue is that every web app eventually starts dropping requests
when the server exceeds its capacity.
That capacity depends on the web framework, the details of the app,
the ram and speed of the server. It is more likely if the web app an
exclusive lock on some file and keeps it locked for some time.

When comparing web frameworks it important to try compare apples with
apples. On a 256MB server on app may hit the threshold and start
dropping request while another not. If a web2py app has migrate=True
than lots of code is in effect serialized and it is more likely to hit
the threshold on heavy load.

In production you should
1) set migrate=false
2) bytecode compile your app
3) use cache (although it can be a memory hog)
4) move as much logic as possible (but table definitions) from models
to controller functions so that it is executed ONLY when the action
that needs it called and not per every request.

Usually 1+2 give you a factor 2x-3x speed-up.

This is not saying that there are not other problems. If there are
they must be investigated.

Massimo

[mdipierro]

> On the other hand, Kuba, who started this thread, later said "I moved to apache/WSGI from Rocket. For me the problem disappears."  So I guess there is a standalone rule #5: Use apache/WSGI etc. instead of the built-in Rocket ?

I do not think it is a web server issue as much as an issue with
limited available resources. 9/10 of requests are for static pages. If
you use apache+mod_wsgi these are served by apache (which is coded in
C) and not by web2py (which is coded in Python). This means that 9/10
of http requests suddenly run 10 faster and reduce cpu usage.

Massimo

[Kuba Kucharski]

still, there must be some webserver issue, you are ruling it out to
fast, Massimo. the reason is problems described by me happens even
under MINIMAL load - one person clicking on the webpage. the others
will tell you the same.

--
Kuba

[mdipierro]

I am not ruling it out. I believe some people are having the problem
but not everybody has having this problem. I am trying help isolate
the possible causes.

You said: "I moved to apache/WSGI from Rocket. For me the problem
disappears." therefore your problem is not the same that some other
problems experienced with apache/WSGI. You also said you cache.ram all
requests. Can you try remove the caching? Any improvement?

Massimo



On Jul 21, 7:57 am, Kuba Kucharski <kuba.kuchar...@gmail.com> wrote:

[Kuba Kucharski]

> You said: "I moved to apache/WSGI from Rocket. For me the problem
> disappears." therefore your problem is not the same that some other
> problems experienced with apache/WSGI.

look at Iceberg, Rahul, MikeEllis in this thread

>You also said you cache.ram all
> requests. Can you try remove the caching? Any improvement?

I think there are two or more problems. One is Thadeus having. The
second was described by me. They can be the same but they seem not to
be. As I investigated in this thread people talking about "my kind of
the problem" report using rocket/cherrypy. Removing caching gives no
improvement.

[Kuba Kucharski]

>> therefore your problem is not the same that some other
>> problems experienced with apache/WSGI.

exactly,

> look at Iceberg, Rahul, MikeEllis in this thread

I was just pointing out that I am not the only one seeing
"rocket-specific-problem"

[mdipierro]

I agree with your assessment. The problem you, Iceberg, Rahul, and
MikeEllis are having with Rocket appears distinct from the problem
Thadeus is having uwsgi+cherokee.

In my previous email I tried to suggest changes in Thadeus code to
isolate the cause of his problem.

Going back to your problem. I am still suspicious that this is a cron
issue. Can you try change cron=True into cron=False in web2py.py?

Massimo

[Thadeus Burgess]

I will make said changes, and add my Flask-DAL extension to the flask app, so at least the database layer will be the exact same.

Flask handles sessions too. I shouldn't disable web2py sessions while letting flask use sessions, that would be an unfair test too.

The good news is, that even with migrate=True the DAL still outperforms sqlalchemy (yay). So using sqlalchemy would result in a slightly slower flask.

--
Thadeus

[Timbo]

Can you get me a debug log of the problem as I described earlier?

[mdipierro]

Thank you Thadeus for your help with these tests.

I did not see a open_session(request) in your Flask code.

Where does Flask save sessions? filesystem or client? If they are
saved on the client that is closer to session.forget() in web2py since
filling the filesystem with one question file for every http request
when running ab may cause problems.

Massimo

On Jul 21, 9:44 am, Thadeus Burgess <thade...@thadeusb.com> wrote:
> I will make said changes, and add my Flask-DAL extension to the flask app,
> so at least the database layer will be the exact same.
>
> Flask handles sessions too. I shouldn't disable web2py sessions while
> letting flask use sessions, that would be an unfair test too.
>
> The good news is, that even with migrate=True the DAL still outperforms
> sqlalchemy (yay). So using sqlalchemy would result in a slightly slower
> flask.
>
> --
> Thadeus
>

[mdipierro]

I propose closing this thread and opening two different threads:

Rocket performance issues (Tim/Kuba please open it)
uWSGI performance issues (Thadeus please open it)

Otherwise this is getting more confused that needs to be.

Massimo

[Vidul Petrov]

I am in doubt that web2py is the bottleneck.
My recent tests on a modest cloud server (2GHz CPU, 1G RAM):
1_000, 5_000, and 10_000 requests in total, 100 concurrent requests -
16 milliseconds per request with no HTTP server errors.
The RDBMS and the HTTP server in question were Postgres and Cherokee.

On Jul 7, 11:04 pm, Kuba Kucharski <kuba.kuchar...@gmail.com> wrote:
> I had prepared some mini-portal, I have 1000 unique visits a day
> but there is a performance issue
>
> bandwidth is ok
> memory is ok
> processor load is ok
> cache.ram is set for almost all of the queries, and is set for 1 hour
>
> but
>
> sometimes, once every few hits, it loads signifacantly slower
> weirdest thing is when you re-click the link it loads instantly, when
> you left it working to load on itself, it is slow.. like 4 to 8
> seconds
>
> what could it be? where to look for an answer? I think some of you had
> to see this before..
>
> --
> Kuba

[mdipierro]

Which os? fcgi?

[Thadeus Burgess]

I try.

I have several copies of web2py core that I have modified in attempt to fix these issues, attempting to narrow down the cause, to no avail yet. I want to do two more tests. The first one, make flask use the web2py DAL so we can compare core to core. Make sure web2py has all of the fancy optimizations in place.

The second thing I want to do is develop a very minimalistic WSGI application that is not based on any framework. I want to execute the controllers in two different ways, I want to test a import based controller, and then an execfile based controller. Very simple, no templates, no database. This small test will verify if it is an issue that can be fixed (such as too much overhead processing the request), or if it is an issue that cannot be fixed (ie: a design flaw in using execfile).

Once the question of execfile vs import on performance is answered will depend on what should be tested next to figure out what is causing these issues.

--
Thadeus

[mdipierro]

Thank you Thadeus. This seems a good approach.

Massimo

On Jul 21, 4:33 pm, Thadeus Burgess <thade...@thadeusb.com> wrote:
> I try.
>
> I have several copies of web2py core that I have modified in attempt to fix
> these issues, attempting to narrow down the cause, to no avail yet. I want
> to do two more tests. The first one, make flask use the web2py DAL so we can
> compare core to core. Make sure web2py has all of the fancy optimizations in
> place.
>
> The second thing I want to do is develop a very minimalistic WSGI
> application that is not based on any framework. I want to execute the
> controllers in two different ways, I want to test a import based controller,
> and then an execfile based controller. Very simple, no templates, no
> database. This small test will verify if it is an issue that can be fixed
> (such as too much overhead processing the request), or if it is an issue
> that cannot be fixed (ie: a design flaw in using execfile).
>
> Once the question of execfile vs import on performance is answered will
> depend on what should be tested next to figure out what is causing these
> issues.
>
> --
> Thadeus
>

[mdipierro]

One more thing... in your tests, you may want to save the
httpserver.log. If you still have problems, it will be useful to
compare cherokee logs vs httpserver logs.

Massimo

On Jul 21, 4:33 pm, Thadeus Burgess <thade...@thadeusb.com> wrote:
> I try.
>
> I have several copies of web2py core that I have modified in attempt to fix
> these issues, attempting to narrow down the cause, to no avail yet. I want
> to do two more tests. The first one, make flask use the web2py DAL so we can
> compare core to core. Make sure web2py has all of the fancy optimizations in
> place.
>
> The second thing I want to do is develop a very minimalistic WSGI
> application that is not based on any framework. I want to execute the
> controllers in two different ways, I want to test a import based controller,
> and then an execfile based controller. Very simple, no templates, no
> database. This small test will verify if it is an issue that can be fixed
> (such as too much overhead processing the request), or if it is an issue
> that cannot be fixed (ie: a design flaw in using execfile).
>
> Once the question of execfile vs import on performance is answered will
> depend on what should be tested next to figure out what is causing these
> issues.
>
> --
> Thadeus
>