undef / nil / empty in template

[Helmut Schneider]

Hi,

openvpn.yaml:
[...]
profiles:
vpn:
openvpn:
defaults:
client:
dev: 'tun'
proto: 'udp'
resolv-retry: 'infinite'
nobind:
user: 'nobody'
group: 'nogroup'
persist-key:
persist-tun:

init.pp:
[...]
$openvpnConf = $profiles['vpn']['openvpn']['defaults']['client']
[...]

In the template:

### <%= @openvpnConf['dev'] %> ###
### <%= @openvpnConf['nobind'] %> ###

The result is

### tun ###
### undef ###

but I would expect

### tun ###
### ###

The problem is that testing for defined?, .nil? and also != 'undef' all
fail.

How can I test if a key has a value withn the template?

Thank you!

[Christopher Wood]

Have you considered switching to an EPP template? You can limit the data passed in to only valid types (otherwise catalog compilation failure), it's quite useful.

https://puppet.com/docs/puppet/5.5/lang_template_epp.html

In the example below, you might do something like:

$content = epp('modulename/template.epp', {
'dev' => $openvpnConf['dev'],
})

file { '/path/to/file':
content => $content,
}

With your template:

----------
<% |
String $dev,
| -%>
### <%= $dev %> ###
----------

If $dev inside the template ends up as anything but a string, splat goes your catalog with a helpful error message.

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/xn0lcerxlvsqnut003%40news.gmane.org.
> For more options, visit https://groups.google.com/d/optout.

[Helmut Schneider]

Christopher Wood wrote:

> Have you considered switching to an EPP template? You can limit the
> data passed in to only valid types (otherwise catalog compilation
> failure), it's quite useful.

Not yet. And I'm not sure if that will help. In my case there are
commands with and without parameters:

proto udp
dev tun
persist-tun
nobind

So even if I pass only specific ones I still have to check if there is
a corresponding value for the key, otherwise

<%= key %> <%= value %>

will fail.

[Christopher Wood]

On Fri, Jul 13, 2018 at 03:44:04PM +0000, Helmut Schneider wrote:
> Christopher Wood wrote:
>
> > Have you considered switching to an EPP template? You can limit the
> > data passed in to only valid types (otherwise catalog compilation
> > failure), it's quite useful.
>
> Not yet. And I'm not sure if that will help. In my case there are
> commands with and without parameters:
>
> proto udp
> dev tun
> persist-tun
> nobind

This still sounds like a data validation item quite doable with types.

https://puppet.com/docs/puppet/5.5/lang_data_hash.html#the-hash-data-type

Hash[Enum['proto', 'dev'], String]
Hash[Enum['proto', 'dev'], Variant[String, Undef]]

> So even if I pass only specific ones I still have to check if there is
> a corresponding value for the key, otherwise
>
> <%= key %> <%= value %>
>
> will fail.

However the odd thing is that I am unable to reproduce what you are seeing with a plain undef in a very simple case. The undef is not stringified for me in puppet 5.4.0.


$ cat /tmp/x.pp
$x = { 'a' => undef }
$c = template('/tmp/t.erb')
notice($c)
$ cat /tmp/t.erb
a is <%= @x[0] %>
$ puppet apply /tmp/x.pp
Notice: Scope(Class[main]): a is

Notice: Compiled catalog for cwl in environment production in 0.03 seconds
Notice: Applied catalog in 0.16 seconds


Possibly is_a? might help in this case if you need the erb for flexibility. Very simplistically and untested:

<%= key %><%= if value.is_a? String then " = #{value}" end %>
<%= key %><%= if value.is_a? Array then " = #{value.sort.join(' ')}" end %>

Then if you see the undef appear in your output file you will know it exists in the yaml as one of these specific data types.

[Helmut Schneider]

Christopher Wood wrote:

> On Fri, Jul 13, 2018 at 03:44:04PM +0000, Helmut Schneider wrote:
> > Christopher Wood wrote:
> >
> > > Have you considered switching to an EPP template? You can limit
> > > the data passed in to only valid types (otherwise catalog
> > > compilation failure), it's quite useful.
> >
> > Not yet. And I'm not sure if that will help. In my case there are
> > commands with and without parameters:
> >
> > proto udp
> > dev tun
> > persist-tun
> > nobind
>
> This still sounds like a data validation item quite doable with types.
>
>
https://puppet.com/docs/puppet/5.5/lang_data_hash.html#the-hash-data-type
>
> Hash[Enum['proto', 'dev'], String]
> Hash[Enum['proto', 'dev'], Variant[String, Undef]]
>
> > So even if I pass only specific ones I still have to check if there
> > is a corresponding value for the key, otherwise
> >
> > <%= key %> <%= value %>
> >
> > will fail.
>
> However the odd thing is that I am unable to reproduce what you are
> seeing with a plain undef in a very simple case. The undef is not
> stringified for me in puppet 5.4.0.

I changed the template to output value.class:

proto String
dev String
persist-tun Symbol
nobind Symbol
resolv-retry String
comp-lzo String
user String
group String
persist-key Symbol
cert String
key String
ca String
ns-cert-type String
verb String
log-append String
script-security String
plugin String
up String
down String

After further investigation this happend with deep_merge, because
without:

proto String
dev String
persist-tun NilClass
nobind NilClass
resolv-retry String
compress NilClass
comp-lzo String
user String
group String
persist-key NilClass
cert String
key String
ca String
ns-cert-type String
verb String
log-append String
script-security String
plugin NilClass
up String
down String

Without the deep_merge "if @openvpnConf[parameter]" works as expected.

helmut@h2786452:~$ puppet -V
4.10.12
helmut@h2786452:~$

[Christopher Wood]

Nice catch, wouldn't have figured on that.

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/xn0lcew6gvygcft005%40news.gmane.org.

[Helmut Schneider]

Christopher Wood wrote:

> Nice catch, wouldn't have figured on that.

You gave the hint with "if value.is_a? String" ;)

And now? Is that expected? What can I do, "if
@openvpnConf[parameter].is_a? Symbol"?! And what is a symbol and how do
I check if it empty?

Even more questionmark now...

[Henrik Lindberg]

You may be getting the symbol :undef which is used in some parts of
puppet to represent puppet undef. In puppet 4x we changed a lot around
undef/nil.

Which version of puppet are you on?

- henrik

--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

[Helmut Schneider]

Henrik Lindberg wrote:

> On 2018-07-13 19:39, Helmut Schneider wrote:
> > Christopher Wood wrote:
> >
> > > Nice catch, wouldn't have figured on that.
> >
> > You gave the hint with "if value.is_a? String" ;)
> >
> > And now? Is that expected? What can I do, "if
> > @openvpnConf[parameter].is_a? Symbol"?! And what is a symbol and
> > how do I check if it empty?
> >
> > Even more questionmark now...
> >
>
> You may be getting the symbol :undef which is used in some parts of
> puppet to represent puppet undef. In puppet 4x we changed a lot
> around undef/nil.
>
> Which version of puppet are you on?

[Johan Fleury]

Le 13 juillet 2018 17:44:04 GMT+02:00, Helmut Schneider <jump...@gmx.de> a écrit :
> In my case there are commands with and without parameters:
>
> proto udp
> dev tun
> persist-tun
> nobind
>

What I usualy do in this case is to define the variable-without-param as a Boolean and use str2bool from stdlib to produce the correct value

e.g.

- in Puppet : $bind = False
- in the erb : str2bool(@bind, 'bind', 'nobind')
--
Johan Fleury
PGP Key ID : 0x5D404386805E56E6

[Johan Fleury]

I meant bool2str, sorry.

https://github.com/puppetlabs/puppetlabs-stdlib/blob/master/README.md#bool2str

[Henrik Lindberg]

On 2018-07-14 13:47, Johan Fleury wrote:
> I meant bool2str, sorry.
>
> https://github.com/puppetlabs/puppetlabs-stdlib/blob/master/README.md#bool2str
>

Helmut, before digging yourself deeper into problems by using work
around on top of problems - do consider using EPP since it protects you
from the issues of needing to know how puppet represents things in Ruby;
which is complicated as puppet handles things differently in different
parts of the code base for backwards compatibility reasons.

Many of the functions in stdlib are smelly as they are sometimes quite
imprecise and not always correct. The "bool2str" however, does what it
is supposed to, but will error if not given a boolean true or false.
That is, it will error if given empty string, undef, or the ruby symbol
:undef. (Thus, in your case, you may get another surprise/error if you
try to use that function).

In puppet language (in EPP) you can do this:

$result = if $val { 'it is truthy' } else { 'it is falsey' }

To get the truthy string if value is neither false nor undef, and
otherwise the falsey value. (An empty string is truthy).

In EPP (as in the rest of puppet), if parameters are declared with
a data type then you will get automatic parameter checking
and get an error message that explains the difference (expected type
vs. actual data type of given value). In an ERB template you need to
do that all by yourself. As you have seen, not doing that can give you
problems somewhere deep in the middle of the template instead of much
closer to where the problem actually is.

So, best (as also suggested earlier by Christopher Wood), is to use EPP
and to declare the data types of the parameters the template expects.
If you also declare the data types of the class parameters in the class
where you are using the template you move the issue of argument value
correctness even closer to the source.

Hope this helps.

Best,

[Helmut Schneider]

Henrik Lindberg wrote:

> On 2018-07-14 13:47, Johan Fleury wrote:
> > I meant bool2str, sorry.
> >
> >
https://github.com/puppetlabs/puppetlabs-stdlib/blob/master/README.md#bool2str
> >
>
> Helmut, before digging yourself deeper into problems by using work
> around on top of problems - do consider using EPP since it protects
> you from the issues of needing to know how puppet represents things
> in Ruby; which is complicated as puppet handles things differently in
> different parts of the code base for backwards compatibility reasons.
>
> Many of the functions in stdlib are smelly as they are sometimes quite
> imprecise and not always correct. The "bool2str" however, does what
> it is supposed to, but will error if not given a boolean true or
> false. That is, it will error if given empty string, undef, or the
> ruby symbol :undef. (Thus, in your case, you may get another
> surprise/error if you try to use that function).
>
> In puppet language (in EPP) you can do this:
>
> $result = if $val { 'it is truthy' } else { 'it is falsey' }

I find the existing documentation in the net very confusing so I
havent' used epp yet. E.g. according to
https://puppet.com/docs/puppet/5.4/lang_template_epp.html this should
work:

content => epp("openvpn/etc/openvpn/config.epp", { openvpnConf =>
$openvpnConf, openvpnMode => $openvpnMode, instance => $instance }),

<%- | Hash $openvpnConf,
String $openvpnMode,
String $category,
String $parameters,
String $instance
| -%>

<% ({
'Misc' => [
'script-security',
'plugin',
'up',
'down',
],
}).each |$category, $parameters| { -%>
<%= $category %>
<% } -%>

Error: Could not retrieve catalog from remote server: Error 500 on
SERVER: Server Error: Evaluation Error: Error while evaluating a
Function Call, epp(): Invalid EPP: Ambiguous EPP parameter expression.
Probably missing '<%-' before parameters to remove leading whitespace
at
/etc/puppetlabs/code/modules/openvpn/templates/etc/openvpn/config.epp:6:
6 at /etc/puppetlabs/code/modules/openvpn/manifests/init.pp:28:22 on
node h2786452.stratoserver.net

The same documentation uses different sysntax, once without '$'

$servers.each |server|

and then with '$'

$ntp::restrict.flatten.each |$restrict|

What is correct? And what is wrong with the code above? Do I need to
declare only variables that I pass to the epp or also those I create
within the epp?

[Henrik Lindberg]

It is important that there is no text before the opening <%-
Not sure if you have a blank line there. If so you will get a syntax
error because of the text output before the declaration of the parameters.

That is wrong, cannot work, should be $server

>
> and then with '$'
>
> $ntp::restrict.flatten.each |$restrict|
>

That is correct.

> What is correct? And what is wrong with the code above? Do I need to
> declare only variables that I pass to the epp or also those I create
> within the epp?
>

You do not need to declare those that you create inside of the EPP.
They are like local variables inside the EPP.

Hope that helps.

[Helmut Schneider]

Henrik Lindberg wrote:

> On 2018-07-15 15:45, Helmut Schneider wrote:
> > <%- | Hash $openvpnConf,
>
> It is important that there is no text before the opening <%-
> Not sure if you have a blank line there. If so you will get a syntax
> error because of the text output before the declaration of the
> parameters.

Thank you.

I managed to get it work with the following code:

<%- | Hash $openvpnConf, String $openvpnMode, String $instance | -%>
<% ({
'Mode' => [
"$openvpnMode",

],
}).each |$category, $parameters| { -%>

### <%= $category %> ###
<% $parameters.each |$parameter| { -%>
<% if $parameter == 'remote' { -%>
<%= $parameter %> <%= $openvpnConf['server'] %> <%=
$openvpnConf['port'] %>
<% } else { -%>
<%= $parameter %> <%= $openvpnConf[$parameter] %>
<% } -%>
<% } %>
<% } -%>

What does not work yet is to add something after <%- | [...] | -%>,
everything I add (here <%= $openvpnMode %>) gives an error. E.g.:

<%- | Hash $openvpnConf, String $openvpnMode, String $instance | -%>
<%= $openvpnMode %>
<% ({
'Mode' => [
"$openvpnMode",

],
}).each |$category, $parameters| { -%>

### <%= $category %> ###
<% $parameters.each |$parameter| { -%>
<% if $parameter == 'remote' { -%>
<%= $parameter %> <%= $openvpnConf['server'] %> <%=
$openvpnConf['port'] %>
<% } else { -%>
<%= $parameter %> <%= $openvpnConf[$parameter] %>
<% } -%>
<% } %>
<% } -%>

fails with

Error: Could not retrieve catalog from remote server: Error 500 on
SERVER: Server Error: Evaluation Error: Error while evaluating a
Function Call, epp(): Invalid EPP: Ambiguous EPP parameter expression.
Probably missing '<%-' before parameters to remove leading whitespace
at

/etc/puppetlabs/code/modules/openvpn/templates/etc/openvpn/config.epp:2:
20 at /etc/puppetlabs/code/modules/openvpn/manifests/init.pp:29:22 on
node h2786452

How can I fix this?

[Henrik Lindberg]

It is a bug, please file a ticket in puppet's Jira for project PUP.

Looks like it is confused by the use of ( ) around your hash.

I tried this as a workaround:

$template = @(END)
<%-| $x | -%>
<%= $x -%>
<%- $y = {
'Mode' => [ $x, ],
}
$y.each |$category, $parameters| { -%>
<%= $category %> = <%= $parameters %>
<%- } -%>
END
notice inline_epp($template, x => 'testing')

See you I use an assignment to a local variable $y to get a hash to work
with.

What epp does is to translate <%= ... %> to a print-string expression.
When that is followed by a left parenthesis it seems to think it is a
call i.e. as if you had written the following in puppet:

print($x)({...})

Hope that helps you work around the problem.

[Helmut Schneider]

Henrik Lindberg wrote:

> On 2018-07-16 10:21, Helmut Schneider wrote:
> > How can I fix this?
> >
>
> It is a bug, please file a ticket in puppet's Jira for project PUP.

https://tickets.puppetlabs.com/browse/PUP-9005

Thank you!