Templetor template compilation indentation error
47 views
Skip to first unread message
Emiliano Martinez Contreras
Jun 15, 2010, 8:10:58 AM6/15/10
to we...@googlegroups.com
Hi guys,
We've been having serious problems with the performance of a high traffic site. Finally we discovered that most of the time is spent in template compilation. I am using templetor and I like it a lot, I would not like to change, moreover, given the amount of templates I have it would be a killer to change.
I installed python profiler and profiled the site, url visits that require template compilation take more or less 0,4s of cpu, on the other hand json serving for example takes only 0,01s, which is more or less coherent. I would like to reduce that 0,4s and I read that compiling templetor templates and using GAE render (I am not using GAE btw) may reduce that time by an order of magnitude.
So I've followed these instructions:
http://webpy.org/cookbook/templates_on_gae
The thing is that template.py compile does not produce a correct result, it muddles up the indentation of for loops and if/else conditions inside the templates, making it impossible to import the outcome due to indentation erros.
Any clue?
thanks
We've been having serious problems with the performance of a high traffic site. Finally we discovered that most of the time is spent in template compilation. I am using templetor and I like it a lot, I would not like to change, moreover, given the amount of templates I have it would be a killer to change.
I installed python profiler and profiled the site, url visits that require template compilation take more or less 0,4s of cpu, on the other hand json serving for example takes only 0,01s, which is more or less coherent. I would like to reduce that 0,4s and I read that compiling templetor templates and using GAE render (I am not using GAE btw) may reduce that time by an order of magnitude.
So I've followed these instructions:
http://webpy.org/cookbook/templates_on_gae
The thing is that template.py compile does not produce a correct result, it muddles up the indentation of for loops and if/else conditions inside the templates, making it impossible to import the outcome due to indentation erros.
Any clue?
thanks
Anand Chitipothu
Jun 15, 2010, 8:36:47 AM6/15/10
to we...@googlegroups.com
2010/6/15 Emiliano Martinez Contreras <e.martinez...@gmail.com>:
Compiling templates should not be performance problem because it is
done only once at the beginning. If it is recompiling the templates
every time, then there is something wrong with your settings.
What is the version of web.py that you are using? Can you provide a
sample template that is failing?
Emiliano Martinez Contreras
Jun 15, 2010, 10:15:38 AM6/15/10
to we...@googlegroups.com
Hi Anand,
Thanks for the quick reply. In my main sitemanager app what I do is use a cookie to find out the language of the visitor, according to that cookie a locale template path is chosen, so what I have is a hook which I then add as a preprocessor:
+++
def render_hook():
cookies = web.cookies(lang="en")
lang = cookies.lang
if lang in settings.VALID_LANGS:
locales_path = os.path.join(settings.TEMPLATES_PATH, "%s/"%(lang))
else:
locales_path = os.path.join(settings.TEMPLATES_PATH,"en/")
web.ctx.render = web.template.render(locales_path, globals={'session': web.ctx.session})
++++
As I said the function is loaded by a preprocessor:
+++
app.add_processor(web.loadhook(render_hook))
+++
I then return the webs like this:
+++
return web.ctx.render.base(content=web.ctx.render.search(...))
+++
Perhaps there is something wrong in doing this and the preprocessor forces a recompilation on each request?
Thanks for the quick reply. In my main sitemanager app what I do is use a cookie to find out the language of the visitor, according to that cookie a locale template path is chosen, so what I have is a hook which I then add as a preprocessor:
+++
def render_hook():
cookies = web.cookies(lang="en")
lang = cookies.lang
if lang in settings.VALID_LANGS:
locales_path = os.path.join(settings.TEMPLATES_PATH, "%s/"%(lang))
else:
locales_path = os.path.join(settings.TEMPLATES_PATH,"en/")
web.ctx.render = web.template.render(locales_path, globals={'session': web.ctx.session})
++++
As I said the function is loaded by a preprocessor:
+++
app.add_processor(web.loadhook(render_hook))
+++
I then return the webs like this:
+++
return web.ctx.render.base(content=web.ctx.render.search(...))
+++
Perhaps there is something wrong in doing this and the preprocessor forces a recompilation on each request?
2010/6/15 Anand Chitipothu <anand...@gmail.com>
--
You received this message because you are subscribed to the Google Groups "web.py" group.
To post to this group, send email to we...@googlegroups.com.
To unsubscribe from this group, send email to webpy+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/webpy?hl=en.
andrei
Jun 15, 2010, 10:45:14 AM6/15/10
to web.py
I think you'd better define web.ctx.render for each language in the
very begininning, not in every request.
By the way, you can set up only one render, and call render["en"] for
specific locale.
On Jun 15, 6:15 pm, Emiliano Martinez Contreras
> 2010/6/15 Anand Chitipothu <anandol...@gmail.com>
>
> > 2010/6/15 Emiliano Martinez Contreras <e.martinez.contre...@gmail.com>:
> > webpy+un...@googlegroups.com <webpy%2Bunsu...@googlegroups.com>.
very begininning, not in every request.
By the way, you can set up only one render, and call render["en"] for
specific locale.
On Jun 15, 6:15 pm, Emiliano Martinez Contreras
>
> > 2010/6/15 Emiliano Martinez Contreras <e.martinez.contre...@gmail.com>:
Anand Chitipothu
Jun 15, 2010, 11:41:14 AM6/15/10
to we...@googlegroups.com
2010/6/15 Emiliano Martinez Contreras <e.martinez...@gmail.com>:
> Hi Anand,
>
> Thanks for the quick reply. In my main sitemanager app what I do is use a
> cookie to find out the language of the visitor, according to that cookie a
> locale template path is chosen, so what I have is a hook which I then add as
> a preprocessor:
>
> +++
>
> def render_hook():
> cookies = web.cookies(lang="en")
> lang = cookies.lang
>
> if lang in settings.VALID_LANGS:
> locales_path = os.path.join(settings.TEMPLATES_PATH, "%s/"%(lang))
>
> else:
> locales_path = os.path.join(settings.TEMPLATES_PATH,"en/")
>
> web.ctx.render = web.template.render(locales_path, globals={'session':
> web.ctx.session})
>
> Thanks for the quick reply. In my main sitemanager app what I do is use a
> cookie to find out the language of the visitor, according to that cookie a
> locale template path is chosen, so what I have is a hook which I then add as
> a preprocessor:
>
> +++
>
> def render_hook():
> cookies = web.cookies(lang="en")
> lang = cookies.lang
>
> if lang in settings.VALID_LANGS:
> locales_path = os.path.join(settings.TEMPLATES_PATH, "%s/"%(lang))
>
> else:
> locales_path = os.path.join(settings.TEMPLATES_PATH,"en/")
>
> web.ctx.render = web.template.render(locales_path, globals={'session':
> web.ctx.session})
Try changing it to:
def render_hook():
cookies = web.cookies(lang="en")
lang = cookies.lang
if lang not in settings.VALID_LANGS:
lang = "en"
web.ctx.render = getattr(render, lang)
render = web.template.render(settings.TEMPLATES_PATH)
Emiliano Martinez Contreras
Jun 15, 2010, 1:25:37 PM6/15/10
to we...@googlegroups.com
@Anand I did what you said, which effectively is also what @Andrei said, however the CPU time does not improve, look at the profiling output:
+++
As you can see most of the time is spent in transformer.py which I believe has a lot to do with the compilation process, so even when the render is declared just after the url paths declaration it seems to be compiling. 0,4s per page rendering is absolutely not acceptable since, making *very rough* calculations it would mean that I can serve 2,5 users per seconds per core, If I have 2 cores I would be able to serve 5 users per second, once I have more than 5 concurrent users per second I would start to build up a queue.
P.D.: @Greg Milby, indeed cookies for languages is just a quick solution which I will probably change in a near future. I am not that worried about their volatility, my main concern is that search engines will not index languages other than english since the robots will not be using any cookie when visiting the site. I think the best is to actually have something in the path that reveals the language, either something like /en/rest_of_path or something like en.domain.com.
Thanks for all the replies being received.
+++
took 0.402858018875 seconds+++
71906 function calls (41723 primitive calls) in 0.403 CPU seconds
Ordered by: internal time, call count
List reduced from 437 to 40 due to restriction <40>
ncalls tottime percall cumtime percall filename:lineno(function)
5703/342 0.041 0.000 0.239 0.001 transformer.py:1060(com_binary)
8597/4 0.036 0.000 0.277 0.069 transformer.py:787(com_node)
6484/677 0.033 0.000 0.033 0.000 transformer.py:73(extractLineNo)
2447/2 0.030 0.000 0.048 0.024 template.py:1115(visit)
709 0.020 0.000 0.022 0.000 transformer.py:751(decode_literal)
8806 0.017 0.000 0.017 0.000 transformer.py:784(lookup_node)
1074/678 0.009 0.000 0.148 0.000 transformer.py:699(factor)
1018/2 0.008 0.000 0.278 0.139 transformer.py:1072(com_stmt)
1074/678 0.008 0.000 0.142 0.000 transformer.py:712(power)
2 0.007 0.004 0.285 0.143 transformer.py:121(transform)
709 0.007 0.000 0.030 0.000 transformer.py:762(atom_string)
2 0.007 0.003 0.375 0.188 template.py:862(compile_template)
2 0.006 0.003 0.291 0.146 transformer.py:127(parsesuite)
455 0.006 0.000 0.008 0.000 application.py:605(check)
342/336 0.006 0.000 0.112 0.000 transformer.py:1209(com_call_function)
1074/678 0.006 0.000 0.155 0.000 transformer.py:681(term)
1072/676 0.006 0.000 0.227 0.000 transformer.py:579(test)
1074/678 0.005 0.000 0.161 0.000 transformer.py:669(arith_expr)
1074/678 0.005 0.000 0.167 0.000 transformer.py:656(shift_expr)
676/2 0.005 0.000 0.278 0.139 transformer.py:1079(com_append_stmt)
1072/676 0.005 0.000 0.221 0.000 transformer.py:593(or_test)
1074/677 0.005 0.000 0.203 0.000 transformer.py:604(not_test)
1073/677 0.005 0.000 0.198 0.000 transformer.py:611(comparison)
1072/676 0.004 0.000 0.212 0.000 transformer.py:600(and_test)
2447 0.004 0.000 0.004 0.000 template.py:1117(classname)
1074/678 0.004 0.000 0.176 0.000 transformer.py:652(and_expr)
1074/678 0.004 0.000 0.184 0.000 transformer.py:648(xor_expr)
1074/1071 0.004 0.000 0.038 0.000 transformer.py:725(atom)
1074/678 0.004 0.000 0.192 0.000 transformer.py:644(expr)
1018/2 0.004 0.000 0.278 0.139 transformer.py:305(stmt)
334 0.004 0.000 0.266 0.001 transformer.py:312(simple_stmt)
698 0.004 0.000 0.006 0.000 ast.py:18(flatten_nodes)
334 0.003 0.000 0.011 0.000 template.py:167(readline)
703/698 0.003 0.000 0.003 0.000 ast.py:7(flatten)
334 0.003 0.000 0.243 0.001 transformer.py:410(yield_expr)
334 0.002 0.000 0.248 0.001 transformer.py:406(yield_stmt)
341/312 0.002 0.000 0.017 0.000 template.py:86(read_section)
361 0.002 0.000 0.003 0.000 transformer.py:768(atom_name)
378/377 0.002 0.000 0.004 0.000 utils.py:217(safeunicode)
334 0.002 0.000 0.003 0.000 template.py:558(emit)
As you can see most of the time is spent in transformer.py which I believe has a lot to do with the compilation process, so even when the render is declared just after the url paths declaration it seems to be compiling. 0,4s per page rendering is absolutely not acceptable since, making *very rough* calculations it would mean that I can serve 2,5 users per seconds per core, If I have 2 cores I would be able to serve 5 users per second, once I have more than 5 concurrent users per second I would start to build up a queue.
P.D.: @Greg Milby, indeed cookies for languages is just a quick solution which I will probably change in a near future. I am not that worried about their volatility, my main concern is that search engines will not index languages other than english since the robots will not be using any cookie when visiting the site. I think the best is to actually have something in the path that reveals the language, either something like /en/rest_of_path or something like en.domain.com.
Thanks for all the replies being received.
2010/6/15 Anand Chitipothu <anand...@gmail.com>
--
Roman Yakovenko
Jun 15, 2010, 2:28:39 PM6/15/10
to web.py
On Jun 15, 8:25 pm, Emiliano Martinez Contreras
problem?
<e.martinez.contre...@gmail.com> wrote:
> @Anand I did what you said, which effectively is also what @Andrei said,
> however the CPU time does not improve, look at the profiling output:
Can you create a small & complete application, which will show your
> @Anand I did what you said, which effectively is also what @Andrei said,
> however the CPU time does not improve, look at the profiling output:
problem?
Emiliano Martinez Contreras
Jun 15, 2010, 2:44:39 PM6/15/10
to we...@googlegroups.com
Ok, I read the Template.py code and noticed the fact that I forgot to use cache=True in the render statement, hence, templates are reloaded on every visit. It is the actual reloading (which I called compiling before) what is calling the transform.py so many times and consuming most of the cpu time, once the cache is activated the cpu time drops drastically:
And transform.py is no longer called. However, I noticed that the cache seems to be cleaned every so often, if I make very close in time request to the site the templates are not reloaded, but after a while of inactivity they do get reloaded.
took 0.0281779766083 seconds
2826 function calls (2809 primitive calls) in 0.028 CPU seconds
Ordered by: internal time, call count
List reduced from 96 to 40 due to restriction <40>
And transform.py is no longer called. However, I noticed that the cache seems to be cleaned every so often, if I make very close in time request to the site the templates are not reloaded, but after a while of inactivity they do get reloaded.
2010/6/15 Roman Yakovenko <roman.y...@gmail.com>
--
Anand Chitipothu
Jun 15, 2010, 8:55:41 PM6/15/10
to we...@googlegroups.com
2010/6/15 Emiliano Martinez Contreras <e.martinez...@gmail.com>:
> @Anand I did what you said, which effectively is also what @Andrei said,
> however the CPU time does not improve, look at the profiling output:
> however the CPU time does not improve, look at the profiling output:
Did you run this test using dev server? If so, try again after setting
web.config.debug = False.
Emiliano Martinez Contreras
Jun 16, 2010, 1:42:00 AM6/16/10
to we...@googlegroups.com
Nope, the test was done with apache+mod_wsgi+webpy and I did have web.config.debug = False, but actually specifying cache=True makes it work correctly.
2010/6/16 Anand Chitipothu <anand...@gmail.com>
--
Greg Milby
Jun 16, 2010, 8:13:38 AM6/16/10
to we...@googlegroups.com
is that, "web.config.cache=True"?
Escipion
Jun 16, 2010, 4:00:32 PM6/16/10
to web.py
Greg,
I do not know whether your statement would have the same effect, what
I did was:
render = web.template.render(loc=settings.TEMPLATES_PATH,
globals={'session': session}, cache=True)
On Jun 16, 2:13 pm, Greg Milby <gmi...@gmail.com> wrote:
> *is that, "web.config.cache=True"?
> *
> >> .
I do not know whether your statement would have the same effect, what
I did was:
render = web.template.render(loc=settings.TEMPLATES_PATH,
globals={'session': session}, cache=True)
On Jun 16, 2:13 pm, Greg Milby <gmi...@gmail.com> wrote:
> *is that, "web.config.cache=True"?
> *
>
> On Wed, Jun 16, 2010 at 1:42 AM, Emiliano Martinez Contreras <
>
> On Wed, Jun 16, 2010 at 1:42 AM, Emiliano Martinez Contreras <
>
> e.martinez.contre...@gmail.com> wrote:
> > Nope, the test was done with apache+mod_wsgi+webpy and I did have
> > web.config.debug = False, but actually specifying cache=True makes it work
> > correctly.
>
> > 2010/6/16 Anand Chitipothu <anandol...@gmail.com>
> > Nope, the test was done with apache+mod_wsgi+webpy and I did have
> > web.config.debug = False, but actually specifying cache=True makes it work
> > correctly.
>
> >> > @Anand I did what you said, which effectively is also what @Andrei said,
> >> > however the CPU time does not improve, look at the profiling output:
>
> >> Did you run this test using dev server? If so, try again after setting
> >> web.config.debug = False.
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "web.py" group.
> >> To post to this group, send email to we...@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> webpy+un...@googlegroups.com <webpy%2Bunsu...@googlegroups.com>
> >> > however the CPU time does not improve, look at the profiling output:
>
> >> Did you run this test using dev server? If so, try again after setting
> >> web.config.debug = False.
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "web.py" group.
> >> To post to this group, send email to we...@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> .
> >> For more options, visit this group at
> >>http://groups.google.com/group/webpy?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "web.py" group.
> > To post to this group, send email to we...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > webpy+un...@googlegroups.com <webpy%2Bunsu...@googlegroups.com>.
> >>http://groups.google.com/group/webpy?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "web.py" group.
> > To post to this group, send email to we...@googlegroups.com.
> > To unsubscribe from this group, send email to
Greg Milby
Jun 16, 2010, 4:02:57 PM6/16/10
to we...@googlegroups.com
that answers my question - wondered if it was part of the app, or if it was in the render.
thank you for answering,
greg
-----------------------
Visit
http://www.superantispyware.com/superantispyware.html?rid=3971 Remove All The Spyware - Not Just The Easy Ones!
http://1-4-u.info | Don't send insanely long links!
Need a Pick-Me-Up? http://quotes.feedtheguru.com
-----------------------
Visit
http://www.superantispyware.com/superantispyware.html?rid=3971 Remove All The Spyware - Not Just The Easy Ones!
http://1-4-u.info | Don't send insanely long links!
Need a Pick-Me-Up? http://quotes.feedtheguru.com
To unsubscribe from this group, send email to webpy+un...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages