Google App Engine with web2py and high cpu cycles / slow response time

[justinjas]

I just started playing around with web2py and setup a very basic app
that merely displays the ip address of the visitor
(ipaddresstoday.com). The problem I'm having is that every request is
going over the allotted cpu cycles for an request in app engine.

For example in my logs on app engine I always see this.

10-17 11:58AM 01.591 / 200 817ms 2765mcycles 0kb

With the mcycles highlight in red because they are over allocation.
Also as you can see the response time is fairly slow at 817ms. I know
in general GAE has some restrictive limits but is this the norm for
web2py in app engine? Anyone else running on this platform and have
some good numbers to show.

I just want to make sure before I pick a framework to go with that it
will work in app engine without issue.

Thanks.

[Robin B]

Web2py 'works' on appengine, but it is reading, parsing and compiling
the models, controllers, and views on *every* request causing all the
wasted CPU cycles. Normally, web2py caches code as .pyc files, but
you cannot write the filesystem on appengine so nothing gets cached by
default. It is trivial to cache the compiled code in a dict and reuse
it on the next request. Another option is to add a virtual filesystem
on appengine, and cache the code as .pyc files.

Robin

[mdipierro]

On Oct 18, 9:51 am, Robin B <robi...@gmail.com> wrote:
> Web2py 'works' on appengine, but it is reading, parsing and compiling
> the models, controllers, and views on *every* request causing all the
> wasted CPU cycles. Normally, web2py caches code as .pyc files, but
> you cannot write the filesystem on appengine so nothing gets cached by
> default. It is trivial to cache the compiled code in a dict and reuse
> it on the next request.

Could you explain more?

[Robin B]

In compileapp.py, after reading, parsing, and compiling a model, view,
or controller, if a .pyc cannot be written to disk, instead store the
compiled code, by file name/function, in a global dict so that next
request you can simply load the precompiled code directly from RAM.

Since web2py does not have an environment (does not distinguish
between development and production etc), the only way to update the
cached code is to check the mtime of each file on every request which
is wasteful in production where the code does not change, but not
nearly as wasteful as repeatedly reading, parsing and compiling.

Robin

[yarko]

Sorry - I don't know about *.pyc files; I assume they're not platform
dependent, so I'm assuming that uploading *.pyc files with your app
doesn't work (?).

How is the Django support on GAE handling this issue?

[mdipierro]

Robin,

is it sufficient to byte-code compile the app and make sure the pyc
files (including gluon/*.pyc) are uploaded too or do we need to modify
web2py?

Massimo

[Robin B]

The uploaded *.pyc will not load on app engine for security reasons.

I am not sure how Django handles it, does Django use import or exec?

Modules that are imported get compiled and reused, but web2py uses
exec to mixin symbols which bypasses the regular import mechanism.

Putting the compiled source into a dict will cut the read, parse,
compile churn.

Robin

[mdipierro]

As far as I know Django imports.

When you say "The uploaded *.pyc will not load on app engine for
security reasons." do you refer to the gluon modules or also to the
app/compiled/*.pyc?

The latter should work, exactly because we bypass the normal import.

Massimo

[Robin B]

When I uploaded compiled *.pyc and tried to run it, an error was
raised, something about imp.get_magic not existing. It might be
possible to avoid imp.get_magic and get it loading, but I have not
tried that. I did try putting the code into a dict, and that sped
things up.

Robin

On Oct 18, 5:55 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:

[mdipierro]

Robin,

If you could send me the exact error I will ask Guido. This must have
a very easy fix.

Massimo

[mdipierro]

I see they are not exposing imp.get_magic() because they use a custom
interpreter.

Could you also try comment the following lines in compileapp.py and
see what happens?

if data[:4]!=imp.get_magic():
raise SystemError, "compiled code is incompatible"

It is unsafe but it is worth a try.

Massimo

[mdipierro]

This is interesting...
http://code.google.com/p/appengine-monkey/issues/detail?id=8

[Robin B]

I compiled the app locally using the web2py admin, edited the app.yaml
to upload *.pyc, then I uploaded the app to see if the compiled/*.pyc
worked and got the error. I assumed that Google removed get_magic for
some reason, but I did not pursue it.

It would be interesting to skip get_magic and see if it loads.

Robin

[Robin B]

These are the benefits of storing the compiled code in RAM (in a
dict):

1. guaranteed compatible bytecode (its compiled on the server).
2. user does not need to remember to (re)compile the app before
uploading.
3. user does not need to hassle with any .pyc what so ever.
4. fastest possible code access (the entire goal is performance)
5. gets around the fact that you cannot write the file system.

When trying to upload compiled .pyc, the only 100% guaranteed way to
get compatible bytecode on the server would be to compile the bytecode
on the server itself. Since appengine does not allow writing the pyc
to the file system, instead store the compiled bytecode in RAM, which
should always be as fast or faster than storing the bytecode on disk,
making no assumptions about disk caching.

Robin



On Oct 18, 7:04 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
> This is interesting...http://code.google.com/p/appengine-monkey/issues/detail?id=8

[mdipierro]

I agree and I will be waiting for a patch. ;-)

Anyway from
http://code.google.com/appengine/docs/whatisgoogleappengine.html

"The Python runtime environment uses Python version 2.5.2."
This means that just commenting the two lines I suggested above should
do the trick, until nobody upgrades to 2.6.

Massimo

[mdipierro]

Hi Justin,

could you do a test for us:

1) edit gluon/compileapp.py and comment

if data[:4]!=imp.get_magic():
raise SystemError, "compiled code is incompatible"

2) run web2py locally, without appengine and "compile" your app

3) deply on the appengine and see if it works and tell us what timing
you get.

Massimo

On Oct 17, 8:26 pm, justinjas <j...@justinjas.com> wrote:

[justinjas]

Hey Massimo,

I still get the same 2000mcycles+ with 800ms+ times.

10-18 08:48PM 27.724 / 200 912ms 2130mcycles 0kb

Of interest though is if I refresh the page quickly I can get fast
responses back from app engine.

10-18 08:48PM 30.007 / 200 16ms 22mcycles 0kb

As if app engine is doing their own code caching separate from pyc
files.

Another thing I've notice is that on the slow requests I will see
these warnings thrown in the logs:

no file locking
no sqlite3 or dbapi2 driver
no MySQLdb driver
no psycopg2 driver
no cx_Oracle driver
no MSSQL driver
no kinterbasdb driver
unable to import dbhash
unable to import py_compile
unable to import wsgiserver
URL rewrite is on. configuration in route.py
no cache.disk

But when the fast load happens on refresh there is only one error:

no cache.disk

--
Justin

[mdipierro]

Justin, this is fantastic,

I think we are learning something here:

1) GAE takes 912-16=896ms to import and bytecode compile the gluon/
*.py modules and takes 16ms to execute the models/views/controllers.

2) GAE can cache the former pyc files when requests/seconds is above
some threshold.

I think this is good news. We may not need to change anything.

Robin, what do you think?

Massimo

[Robin B]

Make sure you have edited app.yaml skipfiles section to upload .pyc
(it will skip upoading .pyc by default).

On the most trivial app it takes 12ms with no caching, and using RAM
caching I can get 3ms.

Robin

[mdipierro]

So I am not the only not sleeping tonight. ;-)

12ms is fantastic already. We should have a page with these numbers!

Massimo

[Robin B]

The 12ms to 3ms speedup was measured using the @logstats decorator,
and is after the first request once the framework is warmed up
(loaded) and a very simple controller/function that prints "hello
world" with no view/layout and a 0.py model that defines one
function. I wanted to eliminate as many variables as possible and see
how fast I could reload a model on an appengine server.

Robin

[Boris Manojlovic]

Hello,
 
please look into this, as this is what GAE really can do for us and code caching
http://code.google.com/appengine/docs/python/appcaching.html

Boris

--
"Only two things are infinite, the universe and human stupidity, and I'm not sure about the former."-Albert Einstein