Zip Importer High CPU every request

[Anthony]

Hi,

Is it normal for the zipimporter to take 500-1000ms every request -
should it be caching internally?

I'm importing like so:

# templates.py
sys.path.insert(0, 'lib/jinja2.zip')
from jinja2 import Environment, FunctionLoader , TemplateNotFound ,
MemcachedBytecodeCache

# main.py
import templates

Will it still be caching if im not importing the zip dircetly in the
main.py?

Anthony

[Rodrigo Moraes]

On Wed, Jan 7, 2009 at 2:16 PM, Anthony wrote:
> Is it normal for the zipimporter to take 500-1000ms every request -
> should it be caching internally?
>
> I'm importing like so:
>
> # templates.py
> sys.path.insert(0, 'lib/jinja2.zip')
> from jinja2 import Environment, FunctionLoader , TemplateNotFound ,
> MemcachedBytecodeCache
>

> Will it still be caching if im not importing the zip dircetly in the
> main.py?

Not an answer to your question (which I'd like to know :), just a
warning: you won't be able to use MemcachedBytecodeCache from Jinja2,
because bytecode usage is not allowed in App Engine.

The example using MemcachedBytecodeCache in the App Engine Cookbook
only works in the dev server, not in production. :-/

Apart from this, Jinja2 implementation is very smooth. :)

-- rodrigo

[Anthony]

It looks like the info in the logs could be down to the whole page
processing, but very simple pages (no db) consistently taking 500ms+
Ill give it a try without the zipimport.

I'm just using the loader cache from the cookbook (removed the
bytecode cache), jinja works great, although I did need to force
caching off in dev or the internal caches do not update after template
changes...


import config
import os
import sys
from google.appengine.api import memcache

sys.path.insert(0, 'lib/jinja2.zip')
from jinja2 import Environment, FunctionLoader , TemplateNotFound

def jinja2_template_loader(templatename):
templatepath = os.path.abspath(os.curdir+'/templates/'+templatename)
template = memcache.get(templatepath+config.VERSION)
if template is None:
try:
template = file(templatepath).read()
if config.LIVE:
memcache.set(templatepath+config.VERSION,template)
except:
template = None
return template

if config.LIVE:
jinja2_environment = Environment(loader = FunctionLoader
(jinja2_template_loader),cache_size=50)
else:
jinja2_environment = Environment(loader = FunctionLoader
(jinja2_template_loader),cache_size=0)


def render(template_name,context):
template = jinja2_environment.get_template(template_name)
context["stats"]="LIVE:" +str(config.LIVE) + " DEBUG:" +str
(config.DEBUG)
return template.render(context)

[Guido van Rossum]

One question -- are you defining a main() function? If not, your
entire app may be reloaded each time. Try putting logging statements
(or prints to sys.stderr) at the top-level of your module, and check
in the logs for these -- they should only be logged for the first
request. If they are logged for each request, then somehow your main
is being reloaded each time.

[Anthony]

Yes I have a main(), but the zipimporter is not in the .py file
containing the main() it is in an imported file.

The zipimporter is writing out its own info into the log.. for every
request - should this happen if it is cached?

[Ross]

On Jan 8, 2:46 pm, Anthony <acorc...@gmail.com> wrote:
> Yes I have a main(), but the zipimporter is not in the .py file
> containing the main() it is in an imported file.
>
> The zipimporter is writing out its own info into the log.. for every
> request - should this happen if it is cached?

I'm suffering from this problem, too. I'm using the Django Google App
Engine helper and it seems to be reloading the zip file for every
request. I stuck in a logging call after the sys.path insertion and
it seems to be running for every request. I am on SDK 1.1.8, but that
doesn't seem to change anything.

[Chris Johnson]

On Feb 1, 2:02 pm, Ross <rlig...@gmail.com> wrote:
> I'm suffering from this problem, too.  I'm using the Django Google App
> Engine helper and it seems to be reloading the zip file for every
> request.

Could it be that Google App Engine is caching apps for a very short
length of time for low-use apps? I find that if I make a bunch of
requests to my app in rapid succession, then I don't get the high CPU
warning for requests after the first. If I allow a few seconds between
requests, however, I see the warning about zipimporter taking a lot of
CPU time (though, strangely, not always).

I'm using the latest SVN version of the Google App Engine Helper for
Django.

[Ross Light]

I'm thinking now that that's probably the case.  After researching the docs, it would seem as though the Engine distributes the load over different web servers, which would cause the cache to cease to be.  I'd still like to know for sure, because it is rather annoying to have a lot of 2000ms requests.

[Sarath]

I see this too.. If i simply extract the django 1.0.2 into source
folder and upload the app, it gets ridiculously fast.. ~2000ms ->
~84ms..

Google should host django 1.0 in a different path on web servers and
give some configurable way either in app.yaml or in code. so we dont
have to upload it along with our app.

-Sarath.

On Feb 14, 9:07 pm, Ross Light <rlig...@gmail.com> wrote:
> I'm thinking now that that's probably the case.  After researching the docs,
> it would seem as though the Engine distributes the load over different web
> servers, which would cause the cache to cease to be.  I'd still like to know
> for sure, because it is rather annoying to have a lot of 2000ms requests.
>

[johnP]

The only reason to use zipimport is because Django does not easily fit
into a 1000 file limit. It seems a no-brainer to relax the 1000 file
limit...

[johnP]

What should be the new limit? People will always want more... But
I'd propose "whatever the number of files in the current version of
django + 1000" (So I can also use ReportLab)

:)

[Anthony]

I found that this helps:

if not 'shared/jinja2.zip' in sys.path:
sys.path.insert(0, 'shared/jinja2.zip')

ZipImport does not get called every request now.

[johnP]

Nice:)