jinja2 environment loading on each request to /

93 views
Skip to first unread message

pedepy

unread,
Mar 12, 2009, 7:38:12 PM3/12/09
to Google App Engine
hey so i use jinja2 templates because they are better [no citation
needed], but from what i understand, loading the jinja2 environment
can be quite an expensive task. Through logging, I have found out that
each request to / loads the environment over and over.. I have those
routines in a seperate python script (not the one that app.yaml
fowards requests to for /).

anything im missing here ? I think this could be adding also an extra
1000 cpu ms on every requests ...

(by the way, i have tried the cookbook's memcache technique and it
only works locally, it fails on the google servers..)

Brandon Thomson

unread,
Mar 12, 2009, 9:14:00 PM3/12/09
to Google App Engine
I've been trying to figure this out too... See also this thread:

http://stackoverflow.com/questions/618827/optimizing-jinja2-environment-creation

Tim Hoffman

unread,
Mar 12, 2009, 10:38:30 PM3/12/09
to Google App Engine
HI

One thing I noticed when hacking together a zope3 stack, was that any
z3 setup/loading of zcml
would execute on every request if the setup call was part of my main.

By putting all of the setup into a seperate import and not calling the
setup explicitly
would mean it would be cached.

Rgds

T

Jarek Zgoda

unread,
Mar 13, 2009, 7:34:35 AM3/13/09
to Google App Engine
If you move Environment creation routine to another module it will be
cached by Google, but only for few seconds (2-3?), so if your
application does not get enough traffic you'd gain nothing from this.

In my attempts to speedup things I discovered that only loader can be
cached, but it has to be invalidated upon each new deployment so its
key has to have application version added (this is how I did key
versioning in my caching tools). Unfortunately, the gain from this
caching approaches 0.

And the last thing: Armin Ronacher, the guy behind Jinja, is aware of
the problem and has some ideas on how to improve things on GAE,
hopefully including template bytecode caching (this would be really
super-nice!).

P. Hausel

unread,
Mar 13, 2009, 9:47:40 AM3/13/09
to Google App Engine
this is what i use:
===================
from jinja2 import Environment, FunctionLoader , TemplateNotFound
from google.appengine.api import memcache
import os,logging
VERSION = os.environ['CURRENT_VERSION_ID']

logging.warn ("COLD START")
def jinja2_template_loader(templatename):
templatepath = os.path.abspath(os.curdir+'/
template/'+templatename)
template = memcache.get(templatepath+VERSION)
logging.debug( "......INSIDE JINJA....")
if template is None:
try:
logging.debug( "......READ....")
template = file(templatepath).read()
memcache.set(templatepath+VERSION,template,time=500)
except:
template = None
return template
SERVER = os.environ.get('SERVER_SOFTWARE','').lower()
if SERVER.startswith('devel'):
logging.debug("...DEVEL JINJA....")
load_this = Environment(cache_size=0,loader = FunctionLoader
(jinja2_template_loader))
else:
load_this = Environment(loader= FunctionLoader
(jinja2_template_loader))

def render(template_name,context):
template = load_this.get_template(template_name)
content = template.render(context)
return content

=========================

if your app handlers are cached (ie via "if __name__ == '__main__':main
()") this should be imported only once per server instance. It also
caches templates both in memcache (for 8 minutes) and in instance
memory

(in case of local testing, templates are not cached in memory, so you
can make changes and see the changes immediately)

HTH
Peter

hoben

unread,
Mar 27, 2009, 8:11:21 AM3/27/09
to Google App Engine
How would a beginner implement this, Peter H.? Only loading it once
every eight minutes sounds great to me, and I have no idea where to
put your code.

On Mar 13, 9:47 am, peterh <peter.hau...@gmail.com> wrote:
> this is what i use:
> ===================
> fromjinja2import Environment, FunctionLoader , TemplateNotFound
> > > hey so i usejinja2templates because they are better [no citation

Gopal Patel

unread,
Mar 27, 2009, 8:38:55 AM3/27/09
to Google App Engine
that cookbook code was written by me. ....sorry for that...but
cookbook does not allow to re-edit once you post it.

I actually use , multiple loader for productin and testing,
the production uses memcache -- which not allow frequent changes to
template. { you have to return the expire date function in order to do
that } ,
so i use filesystemloader for testing purpose ....

as the marshal module is not available , i did an ugly hack and remove
the marshal code from bcccache ( conditionally via debug ) ....and
replace with my own string routine ... it is working good so far... my
response time are normally along 200-300ms with two-three datastore
read/write. i will paste it once i figured out how to do it without
touching the jinja2 code.

nonetheless , i think , jinja2 itself caching the template bytecode if
the instance is not restarted. so , jinja2 is good for a high traffic
site.

Bill

unread,
Mar 27, 2009, 2:19:35 PM3/27/09
to Google App Engine
On Mar 27, 5:38 am, gops <patelgo...@gmail.com> wrote:
> that cookbook code was written by me. ....sorry for that...but
> cookbook does not allow to re-edit once you post it.

I thought that as well, but it turns out you can edit your recipes.
After you login, click on the "Edit My Recipes" link at the top right
corner. (Yeah, they really should add an edit link right on the
recipe page.)
Reply all
Reply to author
Forward
0 new messages