Newlines are preserverd in jinja

[pankaj ghadge]

Hi all,

I'm referring Apache formula for virtual hosting.

https://github.com/saltstack-formulas/apache-formula/blob/master/apache/vhosts/standard.tmpl

New line are preserved even if I add " removes trailing newlines" i.e. "-"

...

   {% if site.get('CustomLog') != False -%}CustomLog {{ vals.CustomLog }} {{ vals.LogFormat }}{%- endif %}

    {% if False -%} False {%- endif %}

    {% if False -%} False {%- endif %}

    {% if site.get('DocumentRoot') != False -%}DocumentRoot {{ vals.DocumentRoot }}{%- endif %}

... 

Here as this " {% if False -%} False {%- endif %}" is false still it will add newline here.  

So here 2 extra new line will get added here, even if it's false. 

I have referred this URL:  

http://jinja.pocoo.org/docs/dev/templates/#whitespace-control

My example:

standard.tmpl

{# Define default values here so the template below can just focus on layout -#}

{% set sitename = site.get('ServerName', id) -%}

{% set vals = {

    'interface': site.get('interface', '*'),

    'port': site.get('port', '80'),

    'ServerName': sitename,

    'ServerAlias': site.get('ServerAlias', 'www.{0}'.format(sitename)),

    'ServerAdmin': site.get('ServerAdmin', 'webmaster@{0}'.format(sitename)),

    'DirectoryIndex': site.get('DirectoryIndex'),

    'UseCanonicalName': site.get('UseCanonicalName'),

    'LogLevel': site.get('LogLevel', 'warn'),

    'ErrorLog': site.get('ErrorLog', '{0}/{1}-error.log'.format('/var/log/httpd', sitename)),

    'LogFormat': site.get('LogFormat', '"%h %l %u %t \\\"%r\\\" %>s %b \\\"%{Referer}i\\\" \\\"%{User-Agent}i\\\""'),

    'CustomLog': site.get('CustomLog', '{0}/{1}-access.log'.format('/var/log/httpd', sitename)),

    'DocumentRoot': site.get('DocumentRoot', '{0}/{1}'.format('/var/www/html', sitename)),

    'VirtualDocumentRoot': site.get('VirtualDocumentRoot'),

    'Directory_default': '{0}/{1}'.format('/var/www/html', sitename),

    'Directory': {

        'Options': '-Indexes FollowSymLinks',

        'Order': 'allow,deny',

        'Allow': 'from all',

        'AllowOverride': 'None',

    },

} -%}

<VirtualHost {{ vals.interface }}:{{ vals.port }}>

    ServerName {{ vals.ServerName }}

    {% if site.get('ServerAlias') != False -%}ServerAlias {{ vals.ServerAlias }}{%- endif %}

    {% if site.get('ServerAdmin') != False -%}ServerAdmin {{ vals.ServerAdmin }}{%- endif %}

    {% if site.get('DirectoryIndex') -%}DirectoryIndex {{ vals.DirectoryIndex }}{%- endif %}
    # extra newline will be added as "DirectoryIndex" is not present in pillar, it returns False.
       # Here I don't want extra newline if statement is false.

    {% if site.get('UseCanonicalName') -%}UseCanonicalName {{ vals.UseCanonicalName }}{%- endif %}
     # Here also extra newline will be added as "UseCanonicalName" is not present in pillar, it returns False.     

    {% if site.get('LogLevel') != False -%}LogLevel {{ vals.LogLevel }}{%- endif %}

    {% if site.get('ErrorLog') != False -%}ErrorLog {{ vals.ErrorLog }}{%- endif %}

    {% if site.get('CustomLog') != False -%}CustomLog {{ vals.CustomLog }} {{ vals.LogFormat }}{%- endif %}

     {% if site.get('DocumentRoot') != False -%}DocumentRoot {{ vals.DocumentRoot }}{%- endif %}

    {% if site.get('VirtualDocumentRoot') != False -%}VirtualDocumentRoot {{ vals.VirtualDocumentRoot }}{%- endif %}

    {%- for path, dir in site.get('Directory', {}).items() %}

    {% set dvals = {

        'Options': dir.get('Options', vals.Directory.Options),

        'Order': dir.get('Order', vals.Directory.Order),

        'Allow': dir.get('Allow', vals.Directory.Allow),

        'AllowOverride': dir.get('AllowOverride', vals.Directory.AllowOverride),

    } -%}

    {% if path == 'default' -%}{% set path = vals.Directory_default -%}{%- endif %}

    <Directory "{{ path }}">

        {% if dir.get('Options') -%}Options {{ dvals.Options }}{%- endif %}

        {% if dir.get('Order') -%}Order {{ dvals.Order }}{%- endif %}

        {% if dir.get('Allow') -%}Allow {{ dvals.Allow }}{%- endif %}

        {% if dir.get('AllowOverride') -%}AllowOverride {{ dvals.AllowOverride }}{%- endif %}

        {% if dir.get('Formula_Append') -%}

        {{ dir.Formula_Append|indent(8) }}

        {%- endif %}

    </Directory>

    {%- endfor %}

    {% if site.get('Formula_Append') -%}

    {{ site.Formula_Append|indent(4) }}

    {%- endif %}

</VirtualHost>

I have also checked this syntax, but salt returns error.

<div>
        {%+ if something %}yay{% endif %}
</div>

Minion version: salt-call 2014.1.5

[Jeffrey Ollie]

On Tue, Sep 30, 2014 at 9:25 AM, pankaj ghadge <ghadge...@gmail.com> wrote:

I'm referring Apache formula for virtual hosting.

https://github.com/saltstack-formulas/apache-formula/blob/master/apache/vhosts/standard.tmpl

New line are preserved even if I add " removes trailing newlines" i.e. "-"

...

   {% if site.get('CustomLog') != False -%}CustomLog {{ vals.CustomLog }} {{ vals.LogFormat }}{%- endif %}

    {% if False -%} False {%- endif %}

    {% if False -%} False {%- endif %}

    {% if site.get('DocumentRoot') != False -%}DocumentRoot {{ vals.DocumentRoot }}{%- endif %}

... 

Here as this " {% if False -%} False {%- endif %}" is false still it will add newline here.  

So here 2 extra new line will get added here, even if it's false. 

Yeah, the whitespace control in Jinja can be difficult to use/understand.  Mostly, I've learned to ignore my OCD impulses when it doesn't really matter.  Most configuration files can deal with extra whitespace without any problems.  Where it does actually matter, it helps to look at the problem the way Jinja does.  For example:

{% if False -%} False {%- endif %}

{% if False -%} False {%- endif %}

looks to the Jinja parser like "{% if False -%} False {%- endif %}\n{% if False -%} False {%- endif %}\n".  The "-" that gets added to the Jinja templating only removes whitespace in a certain direction.  So the above is equivalent to "{% if False %}False{% endif %}\n{% if False %}False{% endif %}\n".  You can see that only the whitespace around the words "False" was removed.  If you want the newlines removed after the you'd have to write
"{% if False -%} False {%- endif -%}\n{% if False -%} False {%- endif -%}\n".  Notice the additional "-" in the endif.  That would be equivalent to "{% if False %}False{% endif %}{% if False %}False{% endif %}".  Note however that adding a '-' inside a template instruction will remove ALL white space up to the next non-whitespace character, not just a single newline.

My recommendation - unless it actually makes a difference, don't worry so much about whitespace in configuration files generated with Jinja.

--
Jeff Ollie

[C. R. Oldham]

> New line are preserved even if I add " removes trailing newlines" i.e. "-"

Just FYI there are 2 config options to help manage this--jinja_trim_blocks, and ninja_lstrip_blocks.  See the Salt sample config file for more details.

https://github.com/saltstack/salt/blob/develop/conf/master#L295-306

 -- 
C. R. Oldham, Platform Engineer, SaltStack, Inc.
801-564-4673 / c...@saltstack.com / https://github.com/cro

On Sep 30, 2014, at 8:48 AM, Jeffrey Ollie <je...@ocjtech.us> wrote:

On Tue, Sep 30, 2014 at 9:25 AM, pankaj ghadge <ghadge...@gmail.com> wrote:

I'm referring Apache formula for virtual hosting.

https://github.com/saltstack-formulas/apache-formula/blob/master/apache/vhosts/standard.tmpl

...

   {% if site.get('CustomLog') != False -%}CustomLog {{ vals.CustomLog }} {{ vals.LogFormat }}{%- endif %}

    {% if False -%} False {%- endif %}

    {% if False -%} False {%- endif %}

    {% if site.get('DocumentRoot') != False -%}DocumentRoot {{ vals.DocumentRoot }}{%- endif %}

... 

Here as this " {% if False -%} False {%- endif %}" is false still it will add newline here.  

So here 2 extra new line will get added here, even if it's false. 

Yeah, the whitespace control in Jinja can be difficult to use/understand.  Mostly, I've learned to ignore my OCD impulses when it doesn't really matter.  Most configuration files can deal with extra whitespace without any problems.  Where it does actually matter, it helps to look at the problem the way Jinja does.  For example:

{% if False -%} False {%- endif %}

{% if False -%} False {%- endif %}

looks to the Jinja parser like "{% if False -%} False {%- endif %}\n{% if False -%} False {%- endif %}\n".  The "-" that gets added to the Jinja templating only removes whitespace in a certain direction.  So the above is equivalent to "{% if False %}False{% endif %}\n{% if False %}False{% endif %}\n".  You can see that only the whitespace around the words "False" was removed.  If you want the newlines removed after the you'd have to write
"{% if False -%} False {%- endif -%}\n{% if False -%} False {%- endif -%}\n".  Notice the additional "-" in the endif.  That would be equivalent to "{% if False %}False{% endif %}{% if False %}False{% endif %}".  Note however that adding a '-' inside a template instruction will remove ALL white space up to the next non-whitespace character, not just a single newline.

My recommendation - unless it actually makes a difference, don't worry so much about whitespace in configuration files generated with Jinja.

--
Jeff Ollie

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jeffrey Ollie]

On Tue, Sep 30, 2014 at 10:19 AM, C. R. Oldham <c...@saltstack.com> wrote:

> New line are preserved even if I add " removes trailing newlines" i.e. "-"

Just FYI there are 2 config options to help manage this--jinja_trim_blocks, and ninja_lstrip_blocks.  See the Salt sample config file for more details.

https://github.com/saltstack/salt/blob/develop/conf/master#L295-306

Yes, but I would consider that a "nuclear option" as it's going to affect every template.  If you write all your own templates it's not a problem, but if you copy templates from other sources the templates won't work they way you expect.
 

--
Jeff Ollie

[pankaj ghadge]

Hi ,

Thanks for reply,

But when it is true it merge with the above line,

 

<VirtualHost {{ vals.interface }}:{{ vals.port }}>

    ServerName {{ vals.ServerName }}

    {% if site.get('ServerAlias') != False -%}ServerAlias {{ vals.ServerAlias }}{%- endif %}

    {% if site.get('ServerAdmin') != False -%}ServerAdmin {{ vals.ServerAdmin }}{%- endif %}

    {% if site.get('DirectoryIndex') -%}DirectoryIndex {{ vals.DirectoryIndex }}{%- endif -%}

    {% if site.get('UseCanonicalName') -%}UseCanonicalName {{ vals.UseCanonicalName }}{%- endif -%}
    For above both statement it condition is true then it merge with above line like.  

    {% if site.get('LogLevel') != False -%}LogLevel {{ vals.LogLevel }}{%- endif %}

    {% if site.get('ErrorLog') != False -%}ErrorLog {{ vals.ErrorLog }}{%- endif %}

    {% if site.get('CustomLog') != False -%}CustomLog {{ vals.CustomLog }} {{ vals.LogFormat }}{%- endif %}

Output of jinja:

<VirtualHost *:80>

    ServerName example.net

    ServerAlias www.example.net

    ServerAdmin webm...@example.com

    DirectoryIndex index.html index.php index.htmUseCanonicalName OffLogLevel warn
   # Above UseCanonical and Loglevel merges with Directory index,  

    ErrorLog /var/log/httpd/sgp.it3ds.net-error.log

    CustomLog /var/log/httpd/sgp.it3ds.net-access.log "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""

    DocumentRoot /data/www

    VirtualDocumentRoot None

    AddOutputFilterByType DEFLATE text/plain

    AddOutputFilterByType DEFLATE text/html

    AddOutputFilterByType DEFLATE text/xml

    AddOutputFilterByType DEFLATE text/css

    AddOutputFilterByType DEFLATE application/xml

    AddOutputFilterByType DEFLATE application/xhtml+xml

    AddOutputFilterByType DEFLATE application/rss+xml

</VirtualHost>

If I remove this {%- endif -%} and add {%- endif %}

output is :

<VirtualHost *:80>

    ServerName example.net

    ServerAlias www.example.net

    ServerAdmin webm...@example.net

    
    # Here 4 new line got added as below both statement return false.
    # {% if site.get('DirectoryIndex') -%}DirectoryIndex {{ vals.DirectoryIndex }}{%- endif -%}

    # {% if site.get('UseCanonicalName') -%}UseCanonicalName {{ vals.UseCanonicalName }}{%- endif -%}

    LogLevel warn

    ErrorLog /var/log/httpd/sgp.it3ds.net-error.log

    CustomLog /var/log/httpd/sgp.it3ds.net-access.log "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""

    DocumentRoot /data/www

    VirtualDocumentRoot None

    AddOutputFilterByType DEFLATE text/plain

    AddOutputFilterByType DEFLATE text/html

    AddOutputFilterByType DEFLATE text/xml

    AddOutputFilterByType DEFLATE text/css

    AddOutputFilterByType DEFLATE application/xml

    AddOutputFilterByType DEFLATE application/xhtml+xml

    AddOutputFilterByType DEFLATE application/rss+xml

</VirtualHost>

[pankaj ghadge]

One more thing I forgot to add, about salt-master config file, I have made changes in it and made jinja_trim_blocks: True, like below, but salt has started returning me an error in Pillar file.

	# See http://jinja.pocoo.org/docs/api/#high-level-api
	#
	# If this is set to True the first newline after a Jinja block is removed
	# (block, not variable tag!). Defaults to False, corresponds to the Jinja
	# environment init variable "trim_blocks".
	# jinja_trim_blocks: False jinja_trim_blocks: True
	# If this is set to True leading spaces and tabs are stripped from the start
	# of a line to a block. Defaults to False, corresponds to the Jinja
	# environment init variable "lstrip_blocks".
	# jinja_lstrip_blocks: False

[Jeffrey Ollie]

On Wed, Oct 1, 2014 at 3:22 AM, pankaj ghadge <ghadge...@gmail.com> wrote:

Hi ,

Thanks for reply,

But when it is true it merge with the above line,

Yes, adding "-" to a Jinja block removes ALL whitespace, INCLUDING newlines. That will merge lines together. 

 

If I remove this {%- endif -%} and add {%- endif %}

Yes,

"{% if False %}Hello{% endif %}\n{% if False %}Goodbye{% endif %}\n"

results in:

"Hello\nGoodbye\n"

Newlines (and other whitespace) is not modified outside of template blocks unless the "-" is added.  This may be somewhat different from other template languages/scripting environments.

--
Jeff Ollie

[Jeffrey Ollie]

On Wed, Oct 1, 2014 at 4:06 AM, pankaj ghadge <ghadge...@gmail.com> wrote:

One more thing I forgot to add, about salt-master config file, I have made changes in it and made jinja_trim_blocks: True, like below, but salt has started returning me an error in Pillar file.

As I said, the "jinja_trim_blocks" is a kind of "nuclear option" that affects every template in your system.  Most templates, and especially templates that you copy from other sources will be written expecting jinja_trim_blocks to be False.  Some whitespace in YAML is significant, so applying jinja_trim_blocks to a template that doesn't expect it has resulted in invalid YAML.

--
Jeff Ollie

[pankaj ghadge]

So there is no solution for this, it's bad.

Also tried below but no luck.

<VirtualHost {{ vals.interface }}:{{ vals.port }}>

    ServerName {{ vals.ServerName }}

    {% if site.get('ServerAlias') != False -%}ServerAlias {{ vals.ServerAlias }}{%- endif %}

    {% if site.get('ServerAdmin') != False -%}ServerAdmin {{ vals.ServerAdmin }}{%- endif %}

    {% if site.get('DirectoryIndex') -%}

        DirectoryIndex {{ vals.DirectoryIndex }}

    {%- else -%}

        {% if False -%}

        {%- endif -%}

    {%- endif %}

    {% if site.get('UseCanonicalName') -%}

        UseCanonicalName {{ vals.UseCanonicalName }}

    {%- else -%}

        {% if False -%}

        {%- endif -%}

    {%- endif %}

    # I thought this will work but still it doesn't give me proper solution,
    # not removing extra 4 new lines added.
   

    {% if site.get('LogLevel') != False -%}LogLevel {{ vals.LogLevel }}{%- endif %}

    {% if site.get('ErrorLog') != False -%}ErrorLog {{ vals.ErrorLog }}{%- endif %}

    {% if site.get('CustomLog') != False -%}CustomLog {{ vals.CustomLog }} {{ vals.LogFormat }}{%- endif %}

[Jeffrey Ollie]

On Thu, Oct 2, 2014 at 2:01 AM, pankaj ghadge <ghadge...@gmail.com> wrote:

So there is no solution for this, it's bad.

Sure, there's a solution, it's just not pretty to look at.

{% if False %}hello

{% endif %}{% if False %}goodbye

{% endif %}

You've got to get the newlines inside the Jinja blocks.

Which is why I've learned to stop being OCD about extra newlines when it doesn't matter.  Apache configs don't care about a few extra newlines here and there.  This is probably more what you are looking for, but I find this harder to read when I'm creating templates, and since it doesn't matter to Apache, I keep my templates easier to read and don't worry about a few extra newlines in the resulting file.

{% if site.get('LogLevel') != False %}LogLevel {{ vals.LogLevel }}
{% endif %}{% if site.get('ErrorLog') != False %}ErrorLog {{ vals.ErrorLog }}
{% endif %}{% if site.get('CustomLog') != False %}CustomLog {{ vals.CustomLog }} {{ vals.LogFormat }}
{% endif %}
 

--
Jeff Ollie

[Dmitry Malinovsky]

Hi Pankaj, you can try a different template engine - Mako, for example.

[pankaj ghadge]

Thanks for your replies, 

For now I will go with few extra newlines, as in Apache it doesn't matter. I will also try Mako engine later on. Thanks a lot again.