pyTenjin Feature Requests
28 views
Skip to first unread message
Steve
Jun 5, 2009, 2:37:17 PM6/5/09
to kuwata-lab-products
Hi,
I've been working over the past couple days to port my Google App
Engine website from using the built-in django template system to
using pyTenjin. The porting has been difficult primarily because my
django based templates use extensively inline conditionals, like:
<div id="chart">{% if not use_swf %}<img src="{{ img_src }}"
alt="chart">{% endif %}</div>
I think getting inline conditionals in Tenjin is too much to hope
for. My first feature request though is to allow Tenjin <?py
statement ?> tags to ignore initial indentation as long as they are
consistently indented. For example, now in my html template I must do
this:
<body>
<div>
<p>
<?py if boolean: ?>
some conditional text
<?py #endif ?>
</p>
</div>
</body>
The indentation levels of the <?py ?> statements are mandatory to be
separate from my indentation of the html content. Instead, I would
like to be able to use whatever indentation is appropriate for the
html as long as it is constant between <?py ?> statements. For
example:
<body>
<div>
<p>
<?py if boolean: ?>
some conditional text
<?py #endif ?>
</p>
</div>
</body>
This would make pyTenjin statements similar to Mako's control
structures ( http://www.makotemplates.org/docs/syntax.html#syntax_control
) and python blocks ( http://www.makotemplates.org/docs/syntax.html#syntax_python
).
Another problem I have is that my django templates get pre-processed
with an html compressor which strips out redundant whitespace. Since
I indent my html liberally and use 4 spaces per indentation, there is
a 30% reduction with this processing. Yes I do know this is minimized
with html gzip transfer encoding. Howerver some browsers ( *cough* IE
*cough* ) have caching and etag bugs with respect to gzip'd content.
Anyway, since I lose the ability to easily strip whitespace with
Tenjin (because <?py ?> statement indentation is significant) I am
trying to reduce some of my whitespace. I would like to be able to do
the following:
<?py if boolean: ?>one set of <strong>important</strong> text
<?py else: ?>one set of <em>other</em> text
<? #endif ?>final unconditional text
Right now (pyTenjin 0.7) the first two lines work as expected.
However the text after the <? #endif ?> does not get output to the
html stream. I think making it work after any <?py ?> tag would be
nice for consistency. And I think it would be as easy as automatically
adding a \n newline to the stream after parsing any ?> closing mark.
The only place that might be too simplistic would be inside <pre>
blocks.
Preliminary benchmarks show pyTenjin to compare extremely favorably to
the GAE built-in django templates. I have several inline conditionals
to try to translate to Tenjin helper functions. After that I will be
sure to post some concrete performance numbers.
Thank you for your work on this viable templating system.
Regards,
Steve
I've been working over the past couple days to port my Google App
Engine website from using the built-in django template system to
using pyTenjin. The porting has been difficult primarily because my
django based templates use extensively inline conditionals, like:
<div id="chart">{% if not use_swf %}<img src="{{ img_src }}"
alt="chart">{% endif %}</div>
I think getting inline conditionals in Tenjin is too much to hope
for. My first feature request though is to allow Tenjin <?py
statement ?> tags to ignore initial indentation as long as they are
consistently indented. For example, now in my html template I must do
this:
<body>
<div>
<p>
<?py if boolean: ?>
some conditional text
<?py #endif ?>
</p>
</div>
</body>
The indentation levels of the <?py ?> statements are mandatory to be
separate from my indentation of the html content. Instead, I would
like to be able to use whatever indentation is appropriate for the
html as long as it is constant between <?py ?> statements. For
example:
<body>
<div>
<p>
<?py if boolean: ?>
some conditional text
<?py #endif ?>
</p>
</div>
</body>
This would make pyTenjin statements similar to Mako's control
structures ( http://www.makotemplates.org/docs/syntax.html#syntax_control
) and python blocks ( http://www.makotemplates.org/docs/syntax.html#syntax_python
).
Another problem I have is that my django templates get pre-processed
with an html compressor which strips out redundant whitespace. Since
I indent my html liberally and use 4 spaces per indentation, there is
a 30% reduction with this processing. Yes I do know this is minimized
with html gzip transfer encoding. Howerver some browsers ( *cough* IE
*cough* ) have caching and etag bugs with respect to gzip'd content.
Anyway, since I lose the ability to easily strip whitespace with
Tenjin (because <?py ?> statement indentation is significant) I am
trying to reduce some of my whitespace. I would like to be able to do
the following:
<?py if boolean: ?>one set of <strong>important</strong> text
<?py else: ?>one set of <em>other</em> text
<? #endif ?>final unconditional text
Right now (pyTenjin 0.7) the first two lines work as expected.
However the text after the <? #endif ?> does not get output to the
html stream. I think making it work after any <?py ?> tag would be
nice for consistency. And I think it would be as easy as automatically
adding a \n newline to the stream after parsing any ?> closing mark.
The only place that might be too simplistic would be inside <pre>
blocks.
Preliminary benchmarks show pyTenjin to compare extremely favorably to
the GAE built-in django templates. I have several inline conditionals
to try to translate to Tenjin helper functions. After that I will be
sure to post some concrete performance numbers.
Thank you for your work on this viable templating system.
Regards,
Steve
Steve
Jun 5, 2009, 8:42:33 PM6/5/09
to kuwata-lab-products
Hello again,
This may be a bug and not a feature request. I was reading through
the tenjin.py to make sure that you were handling gaememcache
invalidation. I do not understand why you are marking the templates
with a timestamp from time.time() when it is created instead of with
getmtime(). I believe this opens up a scenario where old templates
can be served.
GAE allows you to have multiple live versions. So you can have a v1
and v2 or dev and production. Live versions share the same datastore
and memcache though. So it is possible you could have this scenario
at a slow point in the middle of the night:
12:00 Update v1 (production) with new templates
12:05 Update v2 (development) with new templates
12:08 User connects to v1 enviornment and creates & memcaches template
with timestamp 12:08
12:10 Make v2 the live (production) enviornemnt
12:15 User connects to v2 (now production) and gets from memcache v1
template marked 12:08 timestamp
BUG: Should be serving v2 template from disk with 12:05
timestamp
Because you can shift back to older versions (like in case of
discovering an error) not only should you be using the mtime to
compare, but I believe you should be doing an == compare, not just
>=. Continuing the earlier example:
12:20 Memcache somehow expires v1 template (not likely, but it can
happen)
12:25 User cocnnects to v2 on new process, creates and memcaches v2
template with timestamp 12:05
12:30 Admin notices problem in v2 application and switches back to v1
12:35 User connects and gets memcached v2 template timestamp 12:05
BUG: Should be serving v1 template with timestamp 12:00
Thanks for your consideration,
Steve
This may be a bug and not a feature request. I was reading through
the tenjin.py to make sure that you were handling gaememcache
invalidation. I do not understand why you are marking the templates
with a timestamp from time.time() when it is created instead of with
getmtime(). I believe this opens up a scenario where old templates
can be served.
GAE allows you to have multiple live versions. So you can have a v1
and v2 or dev and production. Live versions share the same datastore
and memcache though. So it is possible you could have this scenario
at a slow point in the middle of the night:
12:00 Update v1 (production) with new templates
12:05 Update v2 (development) with new templates
12:08 User connects to v1 enviornment and creates & memcaches template
with timestamp 12:08
12:10 Make v2 the live (production) enviornemnt
12:15 User connects to v2 (now production) and gets from memcache v1
template marked 12:08 timestamp
BUG: Should be serving v2 template from disk with 12:05
timestamp
Because you can shift back to older versions (like in case of
discovering an error) not only should you be using the mtime to
compare, but I believe you should be doing an == compare, not just
>=. Continuing the earlier example:
12:20 Memcache somehow expires v1 template (not likely, but it can
happen)
12:25 User cocnnects to v2 on new process, creates and memcaches v2
template with timestamp 12:05
12:30 Admin notices problem in v2 application and switches back to v1
12:35 User connects and gets memcached v2 template timestamp 12:05
BUG: Should be serving v1 template with timestamp 12:00
Thanks for your consideration,
Steve
makoto kuwata
Jun 5, 2009, 11:33:01 PM6/5/09
to kuwata-lab-products
Hi Steve, thank you for trying pyTenjin.
On 2009-06-06 3:37, Steve <unetright.thebas...@xoxy.net> wrote:
> The indentation levels of the <?py ?> statements are mandatory to be
> separate from my indentation of the html content. Instead, I would
> like to be able to use whatever indentation is appropriate for the
> html as long as it is constant between <?py ?> statements. For
> example:
>
> <body>
> <div>
> <p>
> <?py if boolean: ?>
> some conditional text
> <?py #endif ?>
> </p>
> </div>
> </body>
What I can do is to extend Template class to add dummy 'if' statement.
main.py:
import tenjin
from tenjin.helpers import *
class MyTemplate(tenjin.Template):
def convert(self, input, filename=None):
self._is_first_stmt = True
return tenjin.Template.convert(self, input, filename)
def add_stmt(self, buf, code):
if self._is_first_stmt:
if buf and buf[-1] != "\n" and buf[-1].isspace():
indent = buf.pop()
tenjin.Template.add_stmt(self, buf, "if True: ##
dummy\n")
buf.append(indent)
self._is_first_stmt = False
tenjin.Template.add_stmt(self, buf, code)
print(MyTemplate('ex.pyhtml').script)
#tenjin.Engine.templateclass = MyTemplate
#engine = tenjin.Engine()
#print engine.get_template('ex.pyhtml').script
ex.pyhtml:
<html>
<body>
<ul>
<?py if items: ?>
<?py for item in items: ?>
<li>#{item}</li>
<?py #endfor ?>
<?py #endif ?>
</ul>
</body>
</html>
result:
$ python main.py
_buf.extend(('''<html>
<body>
<ul>\n''', ));
if True: ## dummy
if items:
for item in items:
_buf.extend((''' <li>''', to_str(item), '''</li>
\n''', ));
#endfor
#endif
_buf.extend((''' </ul>
</body>
</html>\n''', ));
This hack will work well, but notice that it changes line number
of converted script, so you will get '+1' line number when you
have error reported.
> Another problem I have is that my django templates get pre-processed
> with an html compressor which strips out redundant whitespace. Since
> I indent my html liberally and use 4 spaces per indentation, there is
> a 30% reduction with this processing. Yes I do know this is minimized
> with html gzip transfer encoding. Howerver some browsers ( *cough* IE
> *cough* ) have caching and etag bugs with respect to gzip'd content.
'30% reduction' is very impressive.
Hmm, it is difficult for Tenjin to strip spaces automatically and
correctly.
I don't have a good solution for this issue, but I'll try to think it.
--
regards,
makoto kuwata
On 2009-06-06 3:37, Steve <unetright.thebas...@xoxy.net> wrote:
> The indentation levels of the <?py ?> statements are mandatory to be
> separate from my indentation of the html content. Instead, I would
> like to be able to use whatever indentation is appropriate for the
> html as long as it is constant between <?py ?> statements. For
> example:
>
> <body>
> <div>
> <p>
> <?py if boolean: ?>
> some conditional text
> <?py #endif ?>
> </p>
> </div>
> </body>
main.py:
import tenjin
from tenjin.helpers import *
class MyTemplate(tenjin.Template):
def convert(self, input, filename=None):
self._is_first_stmt = True
return tenjin.Template.convert(self, input, filename)
def add_stmt(self, buf, code):
if self._is_first_stmt:
if buf and buf[-1] != "\n" and buf[-1].isspace():
indent = buf.pop()
tenjin.Template.add_stmt(self, buf, "if True: ##
dummy\n")
buf.append(indent)
self._is_first_stmt = False
tenjin.Template.add_stmt(self, buf, code)
print(MyTemplate('ex.pyhtml').script)
#tenjin.Engine.templateclass = MyTemplate
#engine = tenjin.Engine()
#print engine.get_template('ex.pyhtml').script
ex.pyhtml:
<html>
<body>
<ul>
<?py if items: ?>
<?py for item in items: ?>
<li>#{item}</li>
<?py #endfor ?>
<?py #endif ?>
</ul>
</body>
</html>
result:
$ python main.py
_buf.extend(('''<html>
<body>
<ul>\n''', ));
if True: ## dummy
if items:
for item in items:
_buf.extend((''' <li>''', to_str(item), '''</li>
\n''', ));
#endfor
#endif
_buf.extend((''' </ul>
</body>
</html>\n''', ));
This hack will work well, but notice that it changes line number
of converted script, so you will get '+1' line number when you
have error reported.
> Another problem I have is that my django templates get pre-processed
> with an html compressor which strips out redundant whitespace. Since
> I indent my html liberally and use 4 spaces per indentation, there is
> a 30% reduction with this processing. Yes I do know this is minimized
> with html gzip transfer encoding. Howerver some browsers ( *cough* IE
> *cough* ) have caching and etag bugs with respect to gzip'd content.
Hmm, it is difficult for Tenjin to strip spaces automatically and
correctly.
I don't have a good solution for this issue, but I'll try to think it.
--
regards,
makoto kuwata
makoto kuwata
Jun 6, 2009, 1:58:23 AM6/6/09
to kuwata-lab-products
Hi Steve,
I read your post and I agree with you.
I must fix timestamp issue.
Please give me time to change code.
Thank you very much.
--
regards,
makoto kuwata
I read your post and I agree with you.
I must fix timestamp issue.
Please give me time to change code.
Thank you very much.
--
regards,
makoto kuwata
Steve
Jun 6, 2009, 9:01:04 AM6/6/09
to kuwata-lab-products
I wanted:
>> <body>
>> <div>
>> <p>
>> <?py if boolean: ?>
>> some conditional text
>> <?py #endif ?>
>> </p>
>> </div>
>> </body>
You suggested:
straight-forward to me because I am unfamiliar with the inner workings
of the template class. In your example you used <?py for
item ... ?> indentation. Can you tell me if it will also work with
the indentation outside the py tag style? Like this:
<html>
<body>
<ul>
<?py if items: ?>
<?py for item in items: ?>
<li>#{item}</li>
<?py #endfor ?>
<?py #endif ?>
</ul>
</body>
</html>
The reason I would like to make sure this will still work relates to
my efforts to eliminate whitespace.
>> Another problem I have is that my django templates get pre-processed
>> with an html compressor which strips out redundant whitespace. Since
own simple compressor which will combine all whitespace except when it
matches "^\w*<?py". Since I have rewritten most of my inline
conditionals to use helper functions, I have relatively few <?py ?>
statements so this scheme would get me 95% of the way to my original
file sizes.
[From your reply to my email]
>> to list them all in the arguments line. What I would really like is
>> simply every ${var} and #{var} to automatically translate to
>> _context.get('var', None) instead of raising exceptions when var is
>> not passed in through the context.
> How to create your own Template class and convert "@var"
> into "_context.get('var', None)" automatically?
> def _expand(code):
> """expand '@var' into '_context.get("var", None)'"""
> return re.sub(r"@(\w+)", r"_context.get('\1', None)", code)
>
> class MyTemplate(tenjin.Template):
>
> def add_stmt(self, buf, code):
> tenjin.Template.add_stmt(self, buf, _expand(code))
>
> def add_expr(self, buf, code, flag_escape=None):
> tenjin.Template.add_expr(self, buf, _expand(code), flag_escape)
> <p>Hello #{@user or 'guest'}!<p>
> <?py if @message: ?>
> <p>#{@message}</p
Thank you again! Because of the strict indentation requirements of <?
py ?> statements I have tried to avoid them for conditionals whenever
possible. So I have a lot of constructs like:
#{(_context.get(conditional, False) and 'true text') or 'false text'}
These "conditionals" are like optional flags to the template. They
are defined true when used but undefined when not used. I don't want
to make all the callers have to know and pass in False for every
unused option. With your proposed subclass, I can at least simplify
this down to:
#{(@condition and 'true text') or 'false text'}
For me at least, that small reduction brings it down to the point
where the whole expression fits in my head. Being able to also do <p>#
{@message}</p> is a bonus!
>> Preliminary benchmarks show pyTenjin to compare extremely
>> favorably to the GAE built-in django templates. I have several
>> inline conditionals to try to translate to Tenjin helper
>> functions. After that I will be sure to post some concrete
>> performance numbers.
I have finished my initial port of my templates to pyTenjin. I am
happy to report that running on my GAE dev environment requests which
result in uncached template creation are running 10-15% quicker.
Requests which result use in memory (not memcache) cached templates
are running around 30% quicker than their in memory cached django
counterparts. It is not easily possible to measure the performance of
the gaememcached scenario on the single-threaded dev server. Since I
am reluctant to turn on profiling of the deployed application I cannot
judge that scenarios performance with respect to uncached django
performance (uncached since django does not memcache).
I would also like to point out that the 10-15% / 30% improvements are
in total request processing time. This includes significant non-
template related activities. Consequently the improvement in template
processing alone is actually some higher percentage. I chose to only
compare total request time for two reasons. First, isolating the
template processing time would be more difficult. Second, some
template frameworks incur more overhead in import processing. Because
GAE creates new runtime instances with little hesitation, I wanted to
make sure that import overhead of the template system was not
overlooked.
All in all I am quite pleased with how pyTenjin is performing. With
some of the template subclassing you have suggested, it will make it
less difficult to manage my templates. My only concern is that some
of the processing required to support #{@var} and the arbitrary
initial indentation of <?py ?> statements will erode some of the
performance gains.
Thank you very much for all your help in trying to make this work for
my situation.
Regards,
Steve
>> <body>
>> <div>
>> <p>
>> <?py if boolean: ?>
>> some conditional text
>> <?py #endif ?>
>> </p>
>> </div>
>> </body>
> What I can do is to extend Template class to add dummy 'if' statement.
> class MyTemplate(tenjin.Template):
>
> def convert(self, input, filename=None):
> self._is_first_stmt = True
> return tenjin.Template.convert(self, input, filename)
>
> def add_stmt(self, buf, code):
> if self._is_first_stmt:
> if buf and buf[-1] != "\n" and buf[-1].isspace():
> indent = buf.pop()
> tenjin.Template.add_stmt(self, buf, "if True: ##
> dummy\n")
> buf.append(indent)
> self._is_first_stmt = False
> tenjin.Template.add_stmt(self, buf, code)
>
> def convert(self, input, filename=None):
> self._is_first_stmt = True
> return tenjin.Template.convert(self, input, filename)
>
> def add_stmt(self, buf, code):
> if self._is_first_stmt:
> if buf and buf[-1] != "\n" and buf[-1].isspace():
> indent = buf.pop()
> tenjin.Template.add_stmt(self, buf, "if True: ##
> dummy\n")
> buf.append(indent)
> self._is_first_stmt = False
> tenjin.Template.add_stmt(self, buf, code)
> <html>
> <body>
> <ul>
> <?py if items: ?>
> <?py for item in items: ?>
> <li>#{item}</li>
> <?py #endfor ?>
> <?py #endif ?>
> </ul>
> </body>
> </html>
Thank you! That's great. The code you use to support this is not
> <body>
> <ul>
> <?py if items: ?>
> <?py for item in items: ?>
> <li>#{item}</li>
> <?py #endfor ?>
> <?py #endif ?>
> </ul>
> </body>
> </html>
straight-forward to me because I am unfamiliar with the inner workings
of the template class. In your example you used <?py for
item ... ?> indentation. Can you tell me if it will also work with
the indentation outside the py tag style? Like this:
<html>
<body>
<ul>
<?py if items: ?>
<?py for item in items: ?>
<li>#{item}</li>
<?py #endfor ?>
<?py #endif ?>
</ul>
</body>
</html>
my efforts to eliminate whitespace.
>> Another problem I have is that my django templates get pre-processed
>> with an html compressor which strips out redundant whitespace. Since
> '30% reduction' is very impressive.
> Hmm, it is difficult for Tenjin to strip spaces automatically and
> correctly.
> I don't have a good solution for this issue, but I'll try to think it.
I think this will not be too hard to fix on my end. I can write my
> Hmm, it is difficult for Tenjin to strip spaces automatically and
> correctly.
> I don't have a good solution for this issue, but I'll try to think it.
own simple compressor which will combine all whitespace except when it
matches "^\w*<?py". Since I have rewritten most of my inline
conditionals to use helper functions, I have relatively few <?py ?>
statements so this scheme would get me 95% of the way to my original
file sizes.
[From your reply to my email]
>> to list them all in the arguments line. What I would really like is
>> simply every ${var} and #{var} to automatically translate to
>> _context.get('var', None) instead of raising exceptions when var is
>> not passed in through the context.
> How to create your own Template class and convert "@var"
> into "_context.get('var', None)" automatically?
> def _expand(code):
> """expand '@var' into '_context.get("var", None)'"""
> return re.sub(r"@(\w+)", r"_context.get('\1', None)", code)
>
> class MyTemplate(tenjin.Template):
>
> def add_stmt(self, buf, code):
> tenjin.Template.add_stmt(self, buf, _expand(code))
>
> def add_expr(self, buf, code, flag_escape=None):
> tenjin.Template.add_expr(self, buf, _expand(code), flag_escape)
> <p>Hello #{@user or 'guest'}!<p>
> <?py if @message: ?>
> <p>#{@message}</p
Thank you again! Because of the strict indentation requirements of <?
py ?> statements I have tried to avoid them for conditionals whenever
possible. So I have a lot of constructs like:
#{(_context.get(conditional, False) and 'true text') or 'false text'}
These "conditionals" are like optional flags to the template. They
are defined true when used but undefined when not used. I don't want
to make all the callers have to know and pass in False for every
unused option. With your proposed subclass, I can at least simplify
this down to:
#{(@condition and 'true text') or 'false text'}
For me at least, that small reduction brings it down to the point
where the whole expression fits in my head. Being able to also do <p>#
{@message}</p> is a bonus!
>> Preliminary benchmarks show pyTenjin to compare extremely
>> favorably to the GAE built-in django templates. I have several
>> inline conditionals to try to translate to Tenjin helper
>> functions. After that I will be sure to post some concrete
>> performance numbers.
happy to report that running on my GAE dev environment requests which
result in uncached template creation are running 10-15% quicker.
Requests which result use in memory (not memcache) cached templates
are running around 30% quicker than their in memory cached django
counterparts. It is not easily possible to measure the performance of
the gaememcached scenario on the single-threaded dev server. Since I
am reluctant to turn on profiling of the deployed application I cannot
judge that scenarios performance with respect to uncached django
performance (uncached since django does not memcache).
I would also like to point out that the 10-15% / 30% improvements are
in total request processing time. This includes significant non-
template related activities. Consequently the improvement in template
processing alone is actually some higher percentage. I chose to only
compare total request time for two reasons. First, isolating the
template processing time would be more difficult. Second, some
template frameworks incur more overhead in import processing. Because
GAE creates new runtime instances with little hesitation, I wanted to
make sure that import overhead of the template system was not
overlooked.
All in all I am quite pleased with how pyTenjin is performing. With
some of the template subclassing you have suggested, it will make it
less difficult to manage my templates. My only concern is that some
of the processing required to support #{@var} and the arbitrary
initial indentation of <?py ?> statements will erode some of the
performance gains.
Thank you very much for all your help in trying to make this work for
my situation.
Regards,
Steve
makoto kuwata
Jun 6, 2009, 9:49:38 AM6/6/09
to kuwata-lab-products
Steve,
> Can you tell me if it will also work with
> the indentation outside the py tag style? Like this:
> <html>
> <body>
> <ul>
> <?py if items: ?>
> <?py for item in items: ?>
> <li>#{item}</li>
> <?py #endfor ?>
> <?py #endif ?>
> </ul>
> </body>
> </html>
Yes, you can write in this style. There is no problem.
I'll include this feature in the next release.
> My only concern is that some
> of the processing required to support #{@var} and the arbitrary
> initial indentation of <?py ?> statemen
Very little. Don't worry.
This cost raised only when template compilation and
no cost when template object is already cached.
--
regards,
makoto kuwata
> Can you tell me if it will also work with
> the indentation outside the py tag style? Like this:
> <html>
> <body>
> <ul>
> <?py if items: ?>
> <?py for item in items: ?>
> <li>#{item}</li>
> <?py #endfor ?>
> <?py #endif ?>
> </ul>
> </body>
> </html>
I'll include this feature in the next release.
> My only concern is that some
> of the processing required to support #{@var} and the arbitrary
> initial indentation of <?py ?> statemen
This cost raised only when template compilation and
no cost when template object is already cached.
--
regards,
makoto kuwata
Steve
Jun 6, 2009, 11:31:17 AM6/6/09
to kuwata-lab-products
Makoto,
> > <html>
> > <body>
> > <ul>
> > <?py if items: ?>
> > <?py for item in items: ?>
> > <li>#{item}</li>
> > <?py #endfor ?>
> > <?py #endif ?>
> > </ul>
> > </body>
> > </html>
>
> Yes, you can write in this style. There is no problem.
I have confirmed in my template that the above does work, and thank
you. This get's me 80% of what I was looking for. Is there anyway
your subclass technique might be modified to support the following:
<html>
<body>
<?py if bool1: ?>
<div class="a">
<div class="b">
<p>
text line A
<?py if bool2: ?>
text line B
<? #endif bool2 ?>
text line C
</p>
</div>
<?py if bool3: ?>
<p>
text line D
</p>
<?py #endif bool3 ?>
</div>
<?py #endif bool1 ?>
</body>
</html>
Notice in this example that bool2 and bool3 conditionals appear and
different indentations because of the structure of the html. But in
python, they must appear at equal indentation levels. Is there any
way to handle that?
> > My only concern is that some
> > of the processing required to support #{@var} and the arbitrary
> > initial indentation of <?py ?> statemen
>
> Very little. Don't worry.
> This cost raised only when template compilation and
> no cost when template object is already cached.
You are correct I profiled my same template taking advantage of both
the more flexible <?py ?> indentation and the @var expansion. This
uncached case showed approximately 1000 more function calls but the
resulting time increase was almost negligible.
You are correct that the additional cost would only be applicable when
not cached, however in a app engine environment, if you're not using
the gae memcache, the uncached case can be seen extremely frequently
because of the way GAE creates and tears down instances very often.
One other small note. In an earlier post I said I thought I could
easily build a simple whitespace compressor that would detect lines
with significant indentation. I used the wrong character class. It
should have been:
^\s*<?py"
Thank you again for all your help,
Steve
> > <html>
> > <body>
> > <ul>
> > <?py if items: ?>
> > <?py for item in items: ?>
> > <li>#{item}</li>
> > <?py #endfor ?>
> > <?py #endif ?>
> > </ul>
> > </body>
> > </html>
>
> Yes, you can write in this style. There is no problem.
you. This get's me 80% of what I was looking for. Is there anyway
your subclass technique might be modified to support the following:
<html>
<body>
<?py if bool1: ?>
<div class="a">
<div class="b">
<p>
text line A
<?py if bool2: ?>
text line B
<? #endif bool2 ?>
text line C
</p>
</div>
<?py if bool3: ?>
<p>
text line D
</p>
<?py #endif bool3 ?>
</div>
<?py #endif bool1 ?>
</body>
</html>
Notice in this example that bool2 and bool3 conditionals appear and
different indentations because of the structure of the html. But in
python, they must appear at equal indentation levels. Is there any
way to handle that?
> > My only concern is that some
> > of the processing required to support #{@var} and the arbitrary
> > initial indentation of <?py ?> statemen
>
> Very little. Don't worry.
> This cost raised only when template compilation and
> no cost when template object is already cached.
the more flexible <?py ?> indentation and the @var expansion. This
uncached case showed approximately 1000 more function calls but the
resulting time increase was almost negligible.
You are correct that the additional cost would only be applicable when
not cached, however in a app engine environment, if you're not using
the gae memcache, the uncached case can be seen extremely frequently
because of the way GAE creates and tears down instances very often.
One other small note. In an earlier post I said I thought I could
easily build a simple whitespace compressor that would detect lines
with significant indentation. I used the wrong character class. It
should have been:
^\s*<?py"
Thank you again for all your help,
Steve
Steve
Jun 6, 2009, 11:39:52 AM6/6/09
to kuwata-lab-products
I forgot to mention two small modifications that I used:
> > How to create your own Template class and convert "@var"
> > into "_context.get('var', None)" automatically?
> > def _expand(code):
> > """expand '@var' into '_context.get("var", None)'"""
> > return re.sub(r"@(\w+)", r"_context.get('\1', None)", code)
>
> > class MyTemplate(tenjin.Template):
>
> > def add_stmt(self, buf, code):
> > tenjin.Template.add_stmt(self, buf, _expand(code))
>
> > def add_expr(self, buf, code, flag_escape=None):
> > tenjin.Template.add_expr(self, buf, _expand(code), flag_escape)
> > <p>Hello #{@user or 'guest'}!<p>
> > <?py if @message: ?>
> > <p>#{@message}</p>
I actually used this instead:
_expand_match = re.compile(r'@(\w+)')
def _expand(code):
"""expand '@var' into '_context.get("var", '')'"""
return _expand_match.sub(r"_context.get('\1', '')", code)
The first difference is that the match pattern is compiled once
instead of on every invocation. The second difference is that when
the context key isn't found, I return the emptry string '' instead of
None. This is a little less efficient for boolean evaluation but it
makes <p>#{@message}</p> work cleaner when message doesn't exist.
Thanks,
Steve
> > How to create your own Template class and convert "@var"
> > into "_context.get('var', None)" automatically?
> > def _expand(code):
> > """expand '@var' into '_context.get("var", None)'"""
> > return re.sub(r"@(\w+)", r"_context.get('\1', None)", code)
>
> > class MyTemplate(tenjin.Template):
>
> > def add_stmt(self, buf, code):
> > tenjin.Template.add_stmt(self, buf, _expand(code))
>
> > def add_expr(self, buf, code, flag_escape=None):
> > tenjin.Template.add_expr(self, buf, _expand(code), flag_escape)
> > <p>Hello #{@user or 'guest'}!<p>
> > <?py if @message: ?>
> > <p>#{@message}</p>
_expand_match = re.compile(r'@(\w+)')
def _expand(code):
"""expand '@var' into '_context.get("var", '')'"""
return _expand_match.sub(r"_context.get('\1', '')", code)
The first difference is that the match pattern is compiled once
instead of on every invocation. The second difference is that when
the context key isn't found, I return the emptry string '' instead of
None. This is a little less efficient for boolean evaluation but it
makes <p>#{@message}</p> work cleaner when message doesn't exist.
Thanks,
Steve
makoto kuwata
Jun 6, 2009, 11:18:57 PM6/6/09
to kuwata-lab-products
On 2009-06-07 12:31, Steve <unetright.thebas...@xoxy.net> wrote:
> Is there anyway
> your subclass technique might be modified to support the following:
(...snip...)
> Is there anyway
> your subclass technique might be modified to support the following:
> Notice in this example that bool2 and bool3 conditionals appear and
> different indentations because of the structure of the html. But in
> python, they must appear at equal indentation levels. Is there any
> way to handle that?
Here you are:
> different indentations because of the structure of the html. But in
> python, they must appear at equal indentation levels. Is there any
> way to handle that?
main.py:
import tenjin # 0.7.0 or 0.8.0
from tenjin.helpers import *
class MyTemplate(tenjin.Template):
def add_stmt(self, buf, code):
s = buf[-1]
if s and s.isspace() and s.find("\n") < 0 and len(s) >
len(self._spaces):
buf[-1:-1] = (self._spaces, "if True: ## dummy
\n", )
if self.newline == "\r\n":
code = code.replace("\r\n", "\n")
buf.append(code)
if __name__ == '__main__':
tenjin.Engine.templateclass = MyTemplate
engine = tenjin.Engine()
template = engine.get_template("ex.pyhtml")
print(template.script)
ex.pyhtml:
<html>
<body>
<?py if bool1: ?>
<div class="a">
<div class="b">
<p>
text line A
<?py if bool2: ?>
text line B
text line C
</p>
</div>
<?py if bool3: ?>
<p>
text line D
</p>
<?py #endif bool3 ?>
</div>
<?py #endif bool1 ?>
</body>
</html>
</p>
</div>
<?py if bool3: ?>
<p>
text line D
</p>
<?py #endif bool3 ?>
</div>
<?py #endif bool1 ?>
</body>
</html>
result:
$ python main.py
_buf.extend(('''<html>
<body>\n''', ));
$ python main.py
_buf.extend(('''<html>
if True: ## dummy
if bool1:
_buf.extend((''' <div class="a">
<div class="b">
<p>
text line A\n''', ));
if True: ## dummy
if bool2:
_buf.extend((''' text line B
\n''', ));
#endif bool2
_buf.extend((''' text line C
</p>
</div>\n''', ));
if bool3:
_buf.extend((''' <p>
text line D
</p>\n''', ));
#endif bool3
_buf.extend((''' </div>\n''', ));
#endif bool1
_buf.extend((''' </body>
</html>\n''', ));
You can see that dummy if-statement is inserted before
'if bool1:' and 'if bool2:'.
Notice that this makes line numbers changed.
For exaple if you have an error on line 10, it may be
reported as line 11 or 12.
I hope this helps you.
--
regards,
makoto kuwata
makoto kuwata
Jun 7, 2009, 10:19:09 AM6/7/09
to kuwata-lab-products
Steve,
On 2009-06-07 12:18, makoto kuwata <kwa...@gmail.com> wrote:
>
> Here you are:
>
I'm sorry that MyTemlate class in previous mail has a bug.
For example, the following template reveals a bug.
ex2.pyhtml:
<div>
<div>
<?py if foo: ?>
foo
<?py #end ?>
</div>
<?py if bar: ?>
bar
<?py #end ?>
</div>
resutl:
$ python main.py ex2.pyhtml | cat -n
1 _buf.extend(('''<div>
2 <div>\n''', ));
3 if True: ## dummy
4 if foo:
5 _buf.extend((''' foo\n''', ));
6 #end
7 _buf.extend((''' </div>\n''', ));
8 if bar:
9 _buf.extend((''' bar\n''', ));
10 #end
11 _buf.extend(('''</div>\n''', ));
12
$ python main.py ex2.pyhtml | python
File "<stdin>", line 8
if bar:
^
IndentationError: unindent does not match any outer indentation
level
I hacked MyTempalte class again to solve this bug.
main2.py:
import tenjin
> len(self._spaces):
# buf[-1:-1] = (self._spaces, "if True: ## dummy
\n", )
if s and s.isspace() and s.find("\n") < 0:
spc = self._spaces
indent = ' ' * self.indent
while len(s) > len(spc):
buf[-1:-1] = (spc, "if True: ## dummy\n", )
spc += indent
if len(sys.argv) < 2: raise "template name is required."
template = MyTemplate(sys.argv[1], indent=2) # change
indent width to 2
#tenjin.Engine.templateclass = MyTemplate
#engine = tenjin.Engine(cache=False, indent=2) # change
indent width to 2
#template = engine.get_template(sys.argv[1])
print(template.script)
result:
$ python main2.py ex2.pyhtml
_buf.extend(('''<div>
<div>\n''', ));
if True: ## dummy
if True: ## dummy
if foo:
_buf.extend((''' foo\n''', ));
#end
_buf.extend((''' bar\n''', ));
#end
a number of dummy if-statements.
So you may not prefer this hack.
I think that it is very hard to control indents accurately
in Tenjin approach. To controll them, it is necessary to
parse Python statements and detect block correctly, like
Mako or Django does. But it makes Tenjin more complex and
heavy, so I wouldn't do that.
I'm sorry but I recommend to compromise at some level.
--
regards,
makoto kuwata
On 2009-06-07 12:18, makoto kuwata <kwa...@gmail.com> wrote:
>
> Here you are:
>
I'm sorry that MyTemlate class in previous mail has a bug.
For example, the following template reveals a bug.
ex2.pyhtml:
<div>
<div>
<?py if foo: ?>
foo
<?py #end ?>
</div>
<?py if bar: ?>
bar
<?py #end ?>
</div>
resutl:
$ python main.py ex2.pyhtml | cat -n
1 _buf.extend(('''<div>
2 <div>\n''', ));
3 if True: ## dummy
4 if foo:
5 _buf.extend((''' foo\n''', ));
6 #end
7 _buf.extend((''' </div>\n''', ));
8 if bar:
9 _buf.extend((''' bar\n''', ));
10 #end
11 _buf.extend(('''</div>\n''', ));
12
$ python main.py ex2.pyhtml | python
File "<stdin>", line 8
if bar:
^
IndentationError: unindent does not match any outer indentation
level
I hacked MyTempalte class again to solve this bug.
main2.py:
import tenjin
from tenjin.helpers import *
class MyTemplate(tenjin.Template):
def add_stmt(self, buf, code):
if buf:
s = buf[-1]
#if s and s.isspace() and s.find("\n") < 0 and len(s)
class MyTemplate(tenjin.Template):
def add_stmt(self, buf, code):
if buf:
s = buf[-1]
> len(self._spaces):
# buf[-1:-1] = (self._spaces, "if True: ## dummy
\n", )
if s and s.isspace() and s.find("\n") < 0:
spc = self._spaces
indent = ' ' * self.indent
while len(s) > len(spc):
buf[-1:-1] = (spc, "if True: ## dummy\n", )
spc += indent
if self.newline == "\r\n":
code = code.replace("\r\n", "\n")
buf.append(code)
if __name__ == '__main__':
import sys
code = code.replace("\r\n", "\n")
buf.append(code)
if __name__ == '__main__':
if len(sys.argv) < 2: raise "template name is required."
template = MyTemplate(sys.argv[1], indent=2) # change
indent width to 2
#tenjin.Engine.templateclass = MyTemplate
#engine = tenjin.Engine(cache=False, indent=2) # change
indent width to 2
#template = engine.get_template(sys.argv[1])
print(template.script)
result:
$ python main2.py ex2.pyhtml
_buf.extend(('''<div>
<div>\n''', ));
if True: ## dummy
if True: ## dummy
if foo:
_buf.extend((''' foo\n''', ));
#end
_buf.extend((''' </div>\n''', ));
if bar:
_buf.extend((''' bar\n''', ));
#end
_buf.extend(('''</div>\n''', ));
This solves the bug, but notice that this hack inserts
a number of dummy if-statements.
So you may not prefer this hack.
I think that it is very hard to control indents accurately
in Tenjin approach. To controll them, it is necessary to
parse Python statements and detect block correctly, like
Mako or Django does. But it makes Tenjin more complex and
heavy, so I wouldn't do that.
I'm sorry but I recommend to compromise at some level.
--
regards,
makoto kuwata
Steve
Jun 7, 2009, 6:17:29 PM6/7/09
to kuwata-lab-products
Hi Makoto,
Thank you for the updated code. I had an implmentation question about
one part:
> s = buf[-1]
did some debug logging and it looked like buf[-1] was always some
number of spaces or '\n'. Would there ever be a newline inside the
middle of the string? If not, I'd like to use:
if s and s.isspace() and s[-1] != '\n':
> I think that it is very hard to control indents accurately
> in Tenjin approach. To controll them, it is necessary to
> parse Python statements and detect block correctly, like
> Mako or Django does. But it makes Tenjin more complex and
> heavy, so I wouldn't do that.
> I'm sorry but I recommend to compromise at some level.
I know it is a selfish perspective, but I would really like it if
Tenjin would detect blocks correctly similar to Mako. Having moved
from django to pytenjin for performance reasons, I can understand not
wanting slow tenjin down too much. I agree that finding a compromise
would be a good thing.
May I suggest three aspects of what I would consider an ideal
compromise:
1) Improve block detection so that blocks could be arbitrarily
indented. So this is okay:
<?py if ?>
<?py if ?>
<?py else ?>
<?py #end ?>
<?py if ?>
<?py else ?>
<?py #end ?>
<?py end?>
2) Disallow inconsistent indenting of a single block level control
structure. For instance, this wound NOT be okay:
<?py if ?>
<?py else ?>
<?py #end ?>
3) Generate python code with normalized indentation instead of using
"if True:" dummy blocks.
I think these three principals could be accomplished relatively easily
with the following sort of pseudo-code algorithm:
py_parse = re.compile(r'(\s*)<\?py (\s*)(.*?)\?>')
output = []
new_blocks = set(('if', 'while', 'for', 'try', 'with', 'def',
'class'))
cont_blocks = set(('elif', 'else', 'except', 'finally'))
indent_spaces = 4
indent = ' ' * indent_spaces
depth = -1
for line in py_lines:
pre_space, in_space, statement = py_parse.match(line).groups()
if statement in new_blocks:
depth += 1
output.append(''.join((indent * depth, statement)))
elif statement in cont_blocks:
output.append(''.join((indent * depth, statement)))
elif statement.startswith('#end'):
depth -= 1
else:
output.append(''.join((indent * (depth+1), statement)))
output = '\n'.join(output)
I admit that this only handles single-line <?py> statements with no
trailing text. But, I think trailing text is easy and mutli-line <?py
\n?> statements could be handled with an additional regex.
I would personally like to see this functionality built-in to tenjin
because I think it makes sense to better be able to combined html
indentation levels with python statement indentation. However, if
this is not feasible it's not a very big problem for me. I
implemented a slightly simpler algorithm as a pre-process python
script that strips out almost all leading whitespace. It uses a
similar technique to preserve <?py > indentation levels but only
outputs a single space per indentation depth. With this done, it is
not hard to see and hand-edit inconsistent <?py> indentations.
In a previous email I had mentioned that I used an HTML "compressor"
to reduce file size by ~30%. With the leading-space elimination pre-
process script I managed to reduce the file size by ~25%. That's
enough that I don't feel bad about missing the last 5%.
Thanks,
Steve
Thank you for the updated code. I had an implmentation question about
one part:
> s = buf[-1]
> if s and s.isspace() and s.find("\n") < 0:
I usually try to avoid str.find() in algorithms whenever possible. I
did some debug logging and it looked like buf[-1] was always some
number of spaces or '\n'. Would there ever be a newline inside the
middle of the string? If not, I'd like to use:
if s and s.isspace() and s[-1] != '\n':
> I think that it is very hard to control indents accurately
> in Tenjin approach. To controll them, it is necessary to
> parse Python statements and detect block correctly, like
> Mako or Django does. But it makes Tenjin more complex and
> heavy, so I wouldn't do that.
> I'm sorry but I recommend to compromise at some level.
Tenjin would detect blocks correctly similar to Mako. Having moved
from django to pytenjin for performance reasons, I can understand not
wanting slow tenjin down too much. I agree that finding a compromise
would be a good thing.
May I suggest three aspects of what I would consider an ideal
compromise:
1) Improve block detection so that blocks could be arbitrarily
indented. So this is okay:
<?py if ?>
<?py if ?>
<?py else ?>
<?py #end ?>
<?py if ?>
<?py else ?>
<?py #end ?>
<?py end?>
2) Disallow inconsistent indenting of a single block level control
structure. For instance, this wound NOT be okay:
<?py if ?>
<?py else ?>
<?py #end ?>
3) Generate python code with normalized indentation instead of using
"if True:" dummy blocks.
I think these three principals could be accomplished relatively easily
with the following sort of pseudo-code algorithm:
py_parse = re.compile(r'(\s*)<\?py (\s*)(.*?)\?>')
output = []
new_blocks = set(('if', 'while', 'for', 'try', 'with', 'def',
'class'))
cont_blocks = set(('elif', 'else', 'except', 'finally'))
indent_spaces = 4
indent = ' ' * indent_spaces
depth = -1
for line in py_lines:
pre_space, in_space, statement = py_parse.match(line).groups()
if statement in new_blocks:
depth += 1
output.append(''.join((indent * depth, statement)))
elif statement in cont_blocks:
output.append(''.join((indent * depth, statement)))
elif statement.startswith('#end'):
depth -= 1
else:
output.append(''.join((indent * (depth+1), statement)))
output = '\n'.join(output)
I admit that this only handles single-line <?py> statements with no
trailing text. But, I think trailing text is easy and mutli-line <?py
\n?> statements could be handled with an additional regex.
I would personally like to see this functionality built-in to tenjin
because I think it makes sense to better be able to combined html
indentation levels with python statement indentation. However, if
this is not feasible it's not a very big problem for me. I
implemented a slightly simpler algorithm as a pre-process python
script that strips out almost all leading whitespace. It uses a
similar technique to preserve <?py > indentation levels but only
outputs a single space per indentation depth. With this done, it is
not hard to see and hand-edit inconsistent <?py> indentations.
In a previous email I had mentioned that I used an HTML "compressor"
to reduce file size by ~30%. With the leading-space elimination pre-
process script I managed to reduce the file size by ~25%. That's
enough that I don't feel bad about missing the last 5%.
Thanks,
Steve
makoto kuwata
Jun 13, 2009, 12:07:15 PM6/13/09
to kuwata-lab-products
Hi Steve,
I wrote my_template.py which parses template recogniginz
'#endfor', '#endif', and so on.
http://gist.github.com/129297
ex.pyhtml:
<html>
<body>
<?py if items: ?>
<table>
<?py i = 0 ?>
<?py klass = i % 2 and 'odd' or 'even' ?>
<tr class="#{klass}">
<td>#{i}</li>
<td>${item}</li>
</tr>
<?py else: ?>
<p>nothing.</p>
<?py #endfor ?>
</table>
<?py else: ?>
<p>Not found.</p>
<?py #endif ?>
</body>
</html>
example:
import tenjin
from tenjin.helpers import *
from my_template import MyTemplate
engine = tenjin.Engine(templateclass=MyTemplate, cache=False)
print(engine.get_template("ex.pyhtml").script)
output:
_buf.extend(('''<html>
<body>\n''', ));
if items:
_buf.extend((''' <table>\n''', ));
i = 0
for item in items:
i += 1
klass = i % 2 and 'odd' or 'even'
_buf.extend((''' <tr class="''', to_str(klass), '''">
<td>''', to_str(i), '''</li>
<td>''', escape(to_str(item)), '''</li>
</tr>\n''', ));
else:
_buf.extend((''' <p>nothing.</p>\n''', ));
#endfor
_buf.extend((''' </table>\n''', ));
else:
_buf.extend((''' <p>Not found.</p>\n''', ));
#endif
If you found any bugs, let me know.
--
regards,
makoto kuwata
I wrote my_template.py which parses template recogniginz
'#endfor', '#endif', and so on.
http://gist.github.com/129297
ex.pyhtml:
<html>
<body>
<?py if items: ?>
<table>
<?py i = 0 ?>
<?py for item in items: ?>
<?py i += 1 ?>
<?py klass = i % 2 and 'odd' or 'even' ?>
<tr class="#{klass}">
<td>#{i}</li>
<td>${item}</li>
</tr>
<?py else: ?>
<p>nothing.</p>
<?py #endfor ?>
</table>
<?py else: ?>
<p>Not found.</p>
<?py #endif ?>
</body>
</html>
example:
import tenjin
from tenjin.helpers import *
engine = tenjin.Engine(templateclass=MyTemplate, cache=False)
print(engine.get_template("ex.pyhtml").script)
output:
_buf.extend(('''<html>
<body>\n''', ));
_buf.extend((''' <table>\n''', ));
i = 0
for item in items:
i += 1
klass = i % 2 and 'odd' or 'even'
_buf.extend((''' <tr class="''', to_str(klass), '''">
<td>''', to_str(i), '''</li>
<td>''', escape(to_str(item)), '''</li>
</tr>\n''', ));
else:
_buf.extend((''' <p>nothing.</p>\n''', ));
#endfor
_buf.extend((''' </table>\n''', ));
else:
_buf.extend((''' <p>Not found.</p>\n''', ));
#endif
_buf.extend((''' </body>
</html>\n''', ));
This is an example and not tested enough.
</html>\n''', ));
If you found any bugs, let me know.
--
regards,
makoto kuwata
Steve
Jun 16, 2009, 8:24:25 PM6/16/09
to kuwata-lab-products
Thank you very much! It will take me several days to try to
understand this. Currently I am trying to migrate my data to a new
schema and then I will be able to upgrade to the latest pyTenjin
release. And then I'll have some time to test this new template
class. It looks really good.
Thanks,
Steve
understand this. Currently I am trying to migrate my data to a new
schema and then I will be able to upgrade to the latest pyTenjin
release. And then I'll have some time to test this new template
class. It looks really good.
Thanks,
Steve
Reply all
Reply to author
Forward
0 new messages