Puppet 4: defined resource types and epp template

[Martin Alfke]

Hi,

I recently had an issue with epp template within a defined resource type.
Let’s assume the following code snippets:

# modules/test/manifests/init.pp
class test {
::test::files { 'test':
param1 => 'value',
}
}

# modules/test/manifests/files.pp
define test::files (
$param1 = '',
){
file { "/tmp/${title}":
ensure => file,
content => epp('test/files.epp'),
}
}

# modules/test/templates/files.epp
<%= $param1 %>

The parameter Param1 will not get the data provided within define declaration.

Is this desired behaviour?
Should I open a bug?
Is this a known bug?

When passing the param1 data via hash to epp function, data will get added to the template.

Best,
Martin

[Henrik Lindberg]

That is exactly what you should do. An external (file based epp) when
called, does not get to see variables in the scope from which it was
called/used. This design is deliberate. Think of the template as a
function you are calling, and you have to give it its arguments.

Contrast this with the inline_epp, which you can think of as a
lambda/code-block. Here the code block gets to see the variables in
scope, since it is itself in that scope (part of the same piece of code).

This design makes the code more maintainable and templates more
reusable. It is also easier to test the templates (there is a command
line utility (puppet epp IIRC) that allows you to feed values into a
templates and render the result).

Hope that helps
- henrik

>
> Best,
> Martin
>


--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

[John Bollinger]

On Monday, February 1, 2016 at 12:15:12 PM UTC-6, henrik lindberg wrote:

Contrast this with the inline_epp, which you can think of as a
lambda/code-block. Here the code block gets to see the variables in
scope, since it is itself in that scope (part of the same piece of code).

Hmmm.  The docs cast that behavior of inline_epp() as a special exception, and claim it's applicable only when the template declares no parameters and you don't pass any (and only with inline_epp()).  Are the docs wrong?  If they are correct, then surely it is best to view that exception as an exception, not as some sort of natural consequence of the scope in which the template or the inline_epp() call appears.

In fact, inasmuch as the template might be presented in the form of a variable defined in a different scope, drawing its actual value from who-knows-where, I really think it's confusing to assert that such an exception is a natural extension of Puppet's scoping rules.  Call it what it is: an ease-of-use aid applicable only to cases where the stricter ordinary scoping rules of EPP templates are of little advantage.

John

[Martin Alfke]

On 01 Feb 2016, at 19:15, Henrik Lindberg <henrik....@cloudsmith.com> wrote:

>
> That is exactly what you should do. An external (file based epp) when called, does not get to see variables in the scope from which it was called/used. This design is deliberate. Think of the template as a function you are calling, and you have to give it its arguments.

In this case the documentation on the website is wrong:
https://docs.puppetlabs.com/puppet/latest/reference/lang_template_epp.html#example-template

The example just switches from template() to epp() function.
According to your writing you must pass all parameters to the epp() function.

>
> Contrast this with the inline_epp, which you can think of as a lambda/code-block. Here the code block gets to see the variables in scope, since it is itself in that scope (part of the same piece of code).
>

Got it.
To be honest: this is ugly.
I always thought of epp() being a full replacement for template() without the scope issue.
This especially feels bad, when we will have performance improvement for epp in the future.

I need to rethink whether inline_epp() is the proper replacement for template() function.

> This design makes the code more maintainable and templates more reusable. It is also easier to test the templates (there is a command line utility (puppet epp IIRC) that allows you to feed values into a templates and render the result).
>
> Hope that helps
> - henrik
>
>>
>> Best,
>> Martin
>>
>
>
> --
>
> Visit my Blog "Puppet on the Edge"
> http://puppet-on-the-edge.blogspot.se/
>

> --
> You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/n8o7b5%24k5j%241%40ger.gmane.org.
> For more options, visit https://groups.google.com/d/optout.

[Henrik Lindberg]

On 2016-04-02 11:46, Martin Alfke wrote:
>
> On 01 Feb 2016, at 19:15, Henrik Lindberg <henrik....@cloudsmith.com> wrote:
>
>>
>> That is exactly what you should do. An external (file based epp) when called, does not get to see variables in the scope from which it was called/used. This design is deliberate. Think of the template as a function you are calling, and you have to give it its arguments.
>
> In this case the documentation on the website is wrong:
> https://docs.puppetlabs.com/puppet/latest/reference/lang_template_epp.html#example-template
>

The long example is wrong in that it does not show that arguments must
be given in a hash.

The documetation for epp has unfortunately been boiled down to just the
bare minimum and the important distinction that variables must be given
has been lost in the series of rewrites that were made.

The implementation clearly restricts epp() to only see global scope.

The specification is also clear about scoping:
https://github.com/puppetlabs/puppet-specifications/blob/master/language/templates.md#visibility-of-scoped-variables

I filed a documentation ticket.

> The example just switches from template() to epp() function.
> According to your writing you must pass all parameters to the epp() function.
>

yes, epp() requires that arguments are given in the call. epp()
templates only have access to global variables (i.e. $::osfamily and such).

>>
>> Contrast this with the inline_epp, which you can think of as a lambda/code-block. Here the code block gets to see the variables in scope, since it is itself in that scope (part of the same piece of code).
>>
> Got it.
> To be honest: this is ugly.
> I always thought of epp() being a full replacement for template() without the scope issue.

The ERB based template() function is for templates in files. The epp()
function is also for files. The big difference between them is that all
variables used by the template processed by epp() must be given to the
function.

The inline_epp() may in the special case, when it does not declare
parameters have access to all variables in scope. This was deemed to be
harmless as all of the logic is in one place, and there is never a
question of reuse. It is also seen as an easy way to transition to
inline_epp() from inline_template().

With that, I am not sure I understand your comments. What is it that you
think is ugly?

> This especially feels bad, when we will have performance improvement for epp in the future.
>

I have not seen tickets with reports of performance problems. How bad is it?

Best
- henrik

[Henrik Lindberg]

On 2016-02-02 21:15, John Bollinger wrote:
>
> On Monday, February 1, 2016 at 12:15:12 PM UTC-6, henrik lindberg wrote:
>
> Contrast this with the inline_epp, which you can think of as a
> lambda/code-block. Here the code block gets to see the variables in
> scope, since it is itself in that scope (part of the same piece of
> code).
>
>
>
> Hmmm. The docs

The docs are wrong. I have filed a ticket.

The behavior is specified here
https://github.com/puppetlabs/puppet-specifications/blob/master/language/templates.md#visibility-of-scoped-variables

The implementation works as specified.

Best
- henrik

> <https://docs.puppetlabs.com/puppet/latest/reference/lang_template_epp.html#special-scope-rule-for-inlineepp>

> cast that behavior of inline_epp() as a special exception, and claim
> it's applicable only when the template declares no parameters and you
> don't pass any (and only with inline_epp()). Are the docs wrong? If

> they are correct, then surely it is best to view that exception *as* an

> exception, not as some sort of natural consequence of the scope in which
> the template or the inline_epp() call appears.
>
> In fact, inasmuch as the template might be presented in the form of a
> variable defined in a different scope, drawing its actual value from
> who-knows-where, I really think it's confusing to assert that such an
> exception is a natural extension of Puppet's scoping rules. Call it
> what it is: an ease-of-use aid applicable only to cases where the
> stricter ordinary scoping rules of EPP templates are of little advantage.
>
>
> John
>

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-dev+...@googlegroups.com

> <mailto:puppet-dev+...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-dev/02150181-1b5d-47a6-b2a4-4ebb3f8a6f98%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-dev/02150181-1b5d-47a6-b2a4-4ebb3f8a6f98%40googlegroups.com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.

[Martin Alfke]

On 04 Feb 2016, at 14:30, Henrik Lindberg <henrik....@cloudsmith.com> wrote:

> On 2016-04-02 11:46, Martin Alfke wrote:
>>
>> On 01 Feb 2016, at 19:15, Henrik Lindberg <henrik....@cloudsmith.com> wrote:
>>
>>>
>>> That is exactly what you should do. An external (file based epp) when called, does not get to see variables in the scope from which it was called/used. This design is deliberate. Think of the template as a function you are calling, and you have to give it its arguments.
>>
>> In this case the documentation on the website is wrong:
>> https://docs.puppetlabs.com/puppet/latest/reference/lang_template_epp.html#example-template
>>
> The long example is wrong in that it does not show that arguments must be given in a hash.
>
> The documetation for epp has unfortunately been boiled down to just the bare minimum and the important distinction that variables must be given has been lost in the series of rewrites that were made.
>
> The implementation clearly restricts epp() to only see global scope.
>
> The specification is also clear about scoping: https://github.com/puppetlabs/puppet-specifications/blob/master/language/templates.md#visibility-of-scoped-variables

Yes. the written description is clear about that.

>
> I filed a documentation ticket.
>
>> The example just switches from template() to epp() function.
>> According to your writing you must pass all parameters to the epp() function.
>>
>
> yes, epp() requires that arguments are given in the call. epp() templates only have access to global variables (i.e. $::osfamily and such).
>
>
>>>
>>> Contrast this with the inline_epp, which you can think of as a lambda/code-block. Here the code block gets to see the variables in scope, since it is itself in that scope (part of the same piece of code).
>>>
>> Got it.
>> To be honest: this is ugly.
>> I always thought of epp() being a full replacement for template() without the scope issue.
>
> The ERB based template() function is for templates in files. The epp() function is also for files. The big difference between them is that all variables used by the template processed by epp() must be given to the function.
>
> The inline_epp() may in the special case, when it does not declare parameters have access to all variables in scope. This was deemed to be harmless as all of the logic is in one place, and there is never a question of reuse. It is also seen as an easy way to transition to inline_epp() from inline_template().
>
> With that, I am not sure I understand your comments. What is it that you think is ugly?
>
>> This especially feels bad, when we will have performance improvement for epp in the future.
>>
>
> I have not seen tickets with reports of performance problems. How bad is it?

You mentioned it somewhere at PuppetConf that in the future the epp() function will have a performance benefit over template() function due to template() needing the ruby stack whereas epp() is running in the java based parser (as far as I have understood this).

>
> Best
> - henrik
>
> --
>
> Visit my Blog "Puppet on the Edge"
> http://puppet-on-the-edge.blogspot.se/
>

> --
> You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/n8vjpn%244u0%241%40ger.gmane.org.

[Henrik Lindberg]

On 2016-04-02 14:40, Martin Alfke wrote:
> On 04 Feb 2016, at 14:30, Henrik Lindberg <henrik....@cloudsmith.com> wrote:
>> On 2016-04-02 11:46, Martin Alfke wrote:
>>> This especially feels bad, when we will have performance improvement for epp in the future.
>>>
>> I have not seen tickets with reports of performance problems. How bad is it?
>
> You mentioned it somewhere at PuppetConf that in the future the epp() function will have a performance benefit over template() function due to template() needing the ruby stack whereas epp() is running in the java based parser (as far as I have understood this).
>

It has a performance benefit already! What I was referring to is that in
order to construct the ERB context we need to iterate over all variables
in all visible scopes and create a temporary class/context where each of
those variables are added as instance variables. This may mean hundreds
or possible thousands of variables that must be copied over. This is
repeated for each an every call to template() and inline_template().

The epp() and inline_epp() does not have to do this. It simply uses the
scope (global or local depending on epp/inline_epp), then variable
access goes directly against the scope.

I noticed a performance improvement when manually playing around with
epp. But this was for small templates, and I did not write any
benchmarks for epp. However, .pp logic is slower than ruby in general,
so it depends on what you are doing.

We have ideas for further improvements.

A big improvement will come in the future when we have a native
compiler. The EPP based templates will then run much faster, and most
likely faster than ERB since ERB means calling Ruby in an external
process and serializing the 100-1000 variables for each and every
invocation.

So, unless you are using very large templates or have complex logic,
iterate over massive data structures, or having the need to use external
ruby code, I would absolutely recommend using EPP as soon as possible.

If anyone have real performance problems with EPP I would like to hear
about it so we can optimize.

Best,