Remove Whitespace on Compile ?

[Jonathan Vanasco]

I couldn't find an answer to this anywhere in the docs ( or find this question asked online before , which I thought was odd )

All the suggestions I've discovered to strip whitespace in Mako implementations have revolved around Filtering functions and WSGI Middleware filtering.

Considering that Mako templates are compiled into Python... has anyone figured out a strategy to strip the whitespace from Templates on compile... so this would only be run when templates are generated ?

[Michael Bayer]

i suppose if you're looking to automate it, and i guess not use a preprocessor (one way to go), you could get at the lexer structure and work from that level.  considering a preprocessor would benefit from the tokenized structure anyway.   not something I've tried though, though I do think we have custom lexer hooks available in Template now.

--
You received this message because you are subscribed to the Google Groups "Mako Templates for Python" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mako-discuss...@googlegroups.com.
To post to this group, send email to mako-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/mako-discuss?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jonathan Vanasco]

I thought about preprocessing, but got worried about whitespace issues as they relate to code blocks.

I should open up the mako internals this week and see what you're doing. I'm just really surprised no one has brought this up before... It seemed to me as the point where a whitespace strip would be optimal.

[Jonathan Vanasco]

I have an idea on how to pull this off....

After combing through, and testing different scenarios, all the relevant stuff to be optimized happens in:

   codegen.py

   _GenerateRenderMethod().visitText

The original function is :

    class _GenerateRenderMethod(...

    ....

        def visitText(self, node):

            self.write_source_comment(node)

            self.printer.writeline("__M_writer(%s)" % repr(node.content) )

If we were to hack it... 

    WHITESPACE_OPTIMIZE = True

    DUPLICATE_NEWLINE = re.compile("(\\\\n)+")

    DUPLICATE_TAB = re.compile("(\\\\t)+")

    DUPLICATE_SPACE = re.compile("( )+")

    def optimize_whitespace( text_repr ):

        text_repr = DUPLICATE_NEWLINE.sub( '\\\\n' , text_repr )

        text_repr = DUPLICATE_TAB.sub( '\\\\t' , text_repr )

        text_repr = DUPLICATE_SPACE.sub( ' ' , text_repr )

        text_repr = text_repr.replace("\\n\\t","\\n")

        return text_repr

    ....

    class _GenerateRenderMethod(...

    ....

        def visitText(self, node):

            self.write_source_comment(node)

            node_content = repr(node.content)

            if WHITESPACE_OPTIMIZE:

                node_content = optimize_whitespace( node_content )

            self.printer.writeline("__M_writer(%s)" % node_content)

If this looks good, the next question would be -- how/where to store a configuration setting to strip whitespace.  Then I could offer a patch.  

[Michael Bayer]

the way this will work is to alter Lexer structure like an AST structure in place, then handed off to the compiler.   The compiler itself is not meant to be subclassed.

This effect can be produced right now, with a little more boilerplate than should ideally be necessary, by passing in lexer_cls to the Template, which calls the original Lexer(), alters it, and returns it.  This is in 0.7.4 not released yet.  A final feature would just be a callable that receives the Lexer structure and returns a new one (which may be the same structure altered in place).

Since we're dealing with text here you could just find all the TextNodes, update the .content attribute, and you're set.

[Jonathan Vanasco]

ok. i'll just wait until 0.7.4 is released.

i'm in no rush for this right now.

[Michael Bayer]

OK though I'd want to add this new feature and if you wanted to maybe plug that in for me and test, it would be a great help.   There's not much else going on with Mako these days and 0.7.4 has kind of been sitting in the queue for several months.

On Mar 6, 2013, at 2:23 PM, Jonathan Vanasco <jona...@findmeon.com> wrote:

ok. i'll just wait until 0.7.4 is released.

i'm in no rush for this right now.

[Jonathan Vanasco]

ok i'm in.

i'll carve out time in the next few days.

[Jonathan Vanasco]

sorry.

i've been busy, and trying to figure out how to specify when this
should and shouldn't happen.