Puppet Templates (ARM-3) Submitted

[Henrik Lindberg]

Hi,
I have now finished the work on "ARM-3 Puppet Templates" - the proposal
is here:
https://github.com/puppetlabs/armatures/blob/master/arm-3.puppet_templates/puppet_templates.md

There is also a full implementation available from my branch here:
https://github.com/hlindberg/puppet/tree/feature/master/templates

(this implementation also includes ARM-4 Heredoc).

In short Embedded Puppet Templates (EPP) is an ERB replacement that
works the same way as ERB, but the logic is written in the Puppet Language.

In addition the ERB tags <% %> <%- %->, <%=, <%#, <%% %%> the
implementation also support declaring required and optional parameters
(i.e. parameters with default values), and allows passing/setting named
arguments when evaluating the template.

You have seen templates before, but here is an epp template example:

$x = 'Puppet'
notice inline_epptemplate('Hello <%= $x %> Templates!')

You can comment on this ARM here, do a pull request to the armatures
repo, or comment directly on a commit.

If you want to try out the implementation, you need to pull it from my
branch (shown above) and then run with the setting --parser future

Regards
- henrik

[Henrik Lindberg]

On 2013-03-05 23:46, Henrik Lindberg wrote:
> Hi,
> I have now finished the work on "ARM-3 Puppet Templates" - the proposal
> is here:
> https://github.com/puppetlabs/armatures/blob/master/arm-3.puppet_templates/puppet_templates.md
>
>
> There is also a full implementation available from my branch here:
> https://github.com/hlindberg/puppet/tree/feature/master/templates
>
> (this implementation also includes ARM-4 Heredoc).
>
> In short Embedded Puppet Templates (EPP) is an ERB replacement that
> works the same way as ERB, but the logic is written in the Puppet Language.
>

Validation Option
-------
One thing I forgot to add is that it may be a good idea to be able to
declare that validation is wanted inside of the template. Although a
file extension like .html.epp states that this is supposed to result in
html, the same does not exist when using an inline template.

It is also diffult to know if the template is just a snippet that is
later merged with other content.

In ARM-4 Heredoc, the same is achieved by using ':syntax' in the opening
heredoc tag (this will trigger validation of the prouced content). The
same could be done for EPP. e.g.

File: atemplate.epp
---
<% ($x,$y) : json %>
This is supposed to be valid json text
---

Since this is implemented for heredoc, it is trivial to add validation
also of the result produced by a template.

Do you think it is of value to be able to validate the produced result?
(To get errors immediately as opposed to when the content of some file
is much later used by an agent).

Restricted Scope
-------
Another idea I did not explore in the proposal is if it is of value to
provide a restricted scope where access is only given to variables that
are passed to the template. If so, this should be something that is
specified in the call to epptemplate(), or inline_epptemplate(), but
their signatures modeled after the corresponding erb based functions
does not easily handle this.

Do you think it is a good idea to be able to restrict the template to
only see the given variables?

Regards
- henrik

[Eric Sorenson]

On Friday, May 3, 2013 2:46:27 PM UTC-7, henrik lindberg wrote:

Hi,
I have now finished the work on "ARM-3 Puppet Templates" - the proposal
is here:
https://github.com/puppetlabs/armatures/blob/master/arm-3.puppet_templates/puppet_templates.md

Hi Henrik, I am pretty excited about this change but I have a couple of questions, for you or for the general audience.

Do you think it will be desirable to phase out/deprecate ERB templating? Or would Puppet benefit from supporting two templating languages?

Seems like on the one hand it's nice to be more consistent language-wise and not force users to shift mentally (`$var` in Puppet vs `var` in ERB has caused me much head-scratching in the past!) but if there are things that people need to do in templates that aren't in the DSL's vocabulary it would be a loss.

 

If you want to try out the implementation, you need to pull it from my
branch (shown above) and then run with the setting --parser future

 I will try it out and let you know how it goes.

-eric0

[Henrik Lindberg]

On 2013-08-05 1:49, Eric Sorenson wrote:
> On Friday, May 3, 2013 2:46:27 PM UTC-7, henrik lindberg wrote:
>
> Hi,
> I have now finished the work on "ARM-3 Puppet Templates" - the proposal
> is here:
> https://github.com/puppetlabs/armatures/blob/master/arm-3.puppet_templates/puppet_templates.md
> <https://github.com/puppetlabs/armatures/blob/master/arm-3.puppet_templates/puppet_templates.md>
>
> Hi Henrik, I am pretty excited about this change but I have a couple of
> questions, for you or for the general audience.
>

Great. Everyone I talked to about this have been positive too.

> Do you think it will be desirable to phase out/deprecate ERB templating?
> Or would Puppet benefit from supporting two templating languages?
>

Don't see a phase out happening near term since there are many modules
that contains ERB templates as well as inline templates to evaluate Ruby
code.

Also EPP does not offer an alternative to inline ERB template being used
as a kind of quick "embedded ruby function" (because it is too much work
to write a real puppet function and they all live in one global name space).

So - as far as templating goes (produce text), yes, the ERB support
should be phased out. It will take time, and I think more work is needed
to support functions/lambdas/logic in a good/convenient way (Have no
concrete proposal in this area yet).

> Seems like on the one hand it's nice to be more consistent language-wise
> and not force users to shift mentally (`$var` in Puppet vs `var` in ERB
> has caused me much head-scratching in the past!) but if there are things
> that people need to do in templates that aren't in the DSL's vocabulary
> it would be a loss.
>

There are several benefits: consistent language / idioms, protection
against "executing inside the puppet runtime", portability (at some
point the underlying parser/evaluator may be written in some other
language).

I have a hard time imagining that EPP makes anything impossible to do
since you can call out to functions, and functions can do anything, and
we now support iteration. It is just a better separation of concerns.
EPP templating is about producing text, not a way to sneak in functions.

>
> If you want to try out the implementation, you need to pull it from my
> branch (shown above) and then run with the setting --parser future
>
>
> I will try it out and let you know how it goes.
>

cool.
- henrik

[Dustin Mitchell]

Picking up discussion of this ARM, now that I've found time to look deeply at it.

This may be silly, but "Extended Puppet Templates" seems more logically abbreviated "EPT" - why "EPP"?

Could you include a link to the Puppet DSL in the text?  It's easy to think that that means Puppet itself (e.g., resource declarations), when really it is a form of Ruby (I think).  The link will add context.

The paragraph beginning with "Also note that it is not possible" is confusing since template parameters have not yet been introduced at that point in the text - perhaps move it down to the next section?

"If someone chooses to use these questionable expressions inside a template, there is no real harm; only poor design." -- I think we will want to validate for this at some point, and it will be much easier to do so if the documentation said "DO NOT DO THIS" from the beginning, rather than "Go ahead, but it's poor design".

As for @ vs. $ -- how difficult would it be to write erb2epp.rb to do the search-and-replace in a syntax-sensitive fashion?  If that's just an hour's hacking, then it might be a very useful addition, especially when ERB is deprecated.

As for validation of filetypes, I think this is a good idea, but I don't like the <%- ($x, $y) :html -%> syntax for this.  I'd prefer something a little more explicit and expandable, perhaps
<%- ($x, $y) syntax => html -%>

As for scope, I think that the fact that ERBs inherit their invoker's scope is a source of bugs and unintentional use of scoped variables.  However, restricted scope would make it more difficult to convert ERBs.  Ideally, this would be an option to the EPP functions, defaulting to restricted scope, so that conversion from ERB to EPP involves explicitly adding that option and only removing it when all variables are passed.  Your comment indicates that's difficult, though, so I'd leave it to you to measure the effort vs. future-proofing tradeoff.

Finally, it'd be nice to see a fully worked example with some iteration over lists, conditionals, and maybe some small Ruby tricks like shuffling lists or mapping.  The only lengthy code example is illegal.

Most of these suggestions concern the text of the ARM itself.  I haven't looked at the implementation, but assuming it works as advertised, I'm 100% in favor of this.

Dustin

[Henrik Lindberg]

Dustin, thanks for the review!
(more comments inline)
Regards
- henrik

On 2013-23-08 3:11, Dustin Mitchell wrote:
> Picking up discussion of this ARM, now that I've found time to look
> deeply at it.
>
> This may be silly, but "Extended Puppet Templates" seems more logically
> abbreviated "EPT" - why "EPP"?
>

Does it say "Extended Puppet Templates" somewhere? The idea is to do the
same as in Ruby, i.e. "Embedded Ruby" = ERP, "Embedded Puppet" = EPP.

> Could you include a link to the Puppet DSL in the text? It's easy to
> think that that means Puppet itself (e.g., resource declarations), when
> really it is a form of Ruby (I think). The link will add context.
>

Sorry, but I don't get your question. EPP is to Puppet what ERB is to
ruby. The EPP content is in a file, typically with extension .epp, but
can be anything except .pp (which is naturally "not embedded puppet code").

But, I think you are asking something else...

> The paragraph beginning with "Also note that it is not possible" is
> confusing since template parameters have not yet been introduced at that
> point in the text - perhaps move it down to the next section?
>
> "If someone chooses to use these questionable expressions inside a
> template, there is no real harm; only poor design." -- I think we will
> want to validate for this at some point, and it will be much easier to
> do so if the documentation said "DO NOT DO THIS" from the beginning,
> rather than "Go ahead, but it's poor design".
>

Yes, agree.

> As for @ vs. $ -- how difficult would it be to write erb2epp.rb to do
> the search-and-replace in a syntax-sensitive fashion? If that's just an
> hour's hacking, then it might be a very useful addition, especially when
> ERB is deprecated.
>

True, but it can be "anything" in the ruby logic.
Using a tool like Geppetto it is trivial to search replace and also
review what is being changed.

BTW, I think the ARM will receive a round of edits as preparation for
writing the real documentation. So thanks for identifying where
improvements can be made.

> As for validation of filetypes, I think this is a good idea, but I don't
> like the <%- ($x, $y) :html -%> syntax for this. I'd prefer something a
> little more explicit and expandable, perhaps
> <%- ($x, $y) syntax => html -%>
>

The idea was to mimic the same specification in ARM-4 Heredoc, where the
syntax is ':syntax'.

I am not super happy with the somewhat magic <%- () -%> construct. It
allowed me to reuse parameter parsing. Did not spend too much time on
figuring out alternatives, but the ones that came to mind I thought were
worse.

> As for scope, I think that the fact that ERBs inherit their invoker's
> scope is a source of bugs and unintentional use of scoped variables.
> However, restricted scope would make it more difficult to convert ERBs.
> Ideally, this would be an option to the EPP functions, defaulting to
> restricted scope, so that conversion from ERB to EPP involves explicitly
> adding that option and only removing it when all variables are passed.
> Your comment indicates that's difficult, though, so I'd leave it to you
> to measure the effort vs. future-proofing tradeoff.
>

Very good point. It is currently a bit of a mess since the original
template functions performs join of results if more than one file is
given. Lots of parameter overloading to make the function to many
different things is not good either. But, I totally agree that it should
be the caller's responsibility to give access to the current scope or not.

Whether it should be restricted scope of not by default I don't know. It
could be a chore to have to write a hash with name to value associations.

There is a Redmine issue where it is discussed if it is good or bad that
the template functions can join results from multiple templates. I would
prefer if it was just one, and instead call a join function if that
behavior is needed.

> Finally, it'd be nice to see a fully worked example with some iteration
> over lists, conditionals, and maybe some small Ruby tricks like
> shuffling lists or mapping. The only lengthy code example is illegal.
>

Yes, more real world examples is always good. Maybe take some existing
in ERB and convert them. Will certainly be made for the finished
documentation. (The ARM is after all input to implementation and
documentation).

> Most of these suggestions concern the text of the ARM itself. I haven't
> looked at the implementation, but assuming it works as advertised,

Yes, it works as advertized; the syntax for syntax checking is not
implemented, and (naturally, it does not yet call out to perform any
such checks).

> I'm
> 100% in favor of this.
>

Super!

- henrik

[Dustin J. Mitchell]

On Sun, Aug 25, 2013 at 5:45 PM, Henrik Lindberg
<henrik....@cloudsmith.com> wrote:
>> This may be silly, but "Extended Puppet Templates" seems more logically
>> abbreviated "EPT" - why "EPP"?
>>
> Does it say "Extended Puppet Templates" somewhere? The idea is to do the
> same as in Ruby, i.e. "Embedded Ruby" = ERP, "Embedded Puppet" = EPP.

I should have said "Embedded", not "Extended". And, fair enough -
Embedded RuBy templates are not "ERT", they are "ERB". So Embedded
PuPpet templates can be "EPP".

>> Could you include a link to the Puppet DSL in the text? It's easy to
>> think that that means Puppet itself (e.g., resource declarations), when
>> really it is a form of Ruby (I think). The link will add context.
>>
> Sorry, but I don't get your question. EPP is to Puppet what ERB is to ruby.
> The EPP content is in a file, typically with extension .epp, but can be
> anything except .pp (which is naturally "not embedded puppet code").
>
> But, I think you are asking something else...

I'm asking, what language do I use between <% %>? Can I write

<%
file {
'/etc/motd':
content => "hello, world\n"
}
%>

which is what "Embedded Puppet" suggests to me. That's not the case,
and "Puppet DSL" is only slightly less unclear, unless the reader is
enough of a Rubyist to realize that "DSL" in the Ruby community refers
to a modified Ruby environment, and not an entirely different
language. Basically, please add a link so I know unambiguously what I
can and cannot do between <% .. %>.

>> As for validation of filetypes, I think this is a good idea, but I don't
>> like the <%- ($x, $y) :html -%> syntax for this. I'd prefer something a
>> little more explicit and expandable, perhaps
>> <%- ($x, $y) syntax => html -%>
>>
> The idea was to mimic the same specification in ARM-4 Heredoc, where the
> syntax is ':syntax'.

I could make the same criticism there (I just haven't gotten to that ARM yet).

> I am not super happy with the somewhat magic <%- () -%> construct. It
> allowed me to reuse parameter parsing. Did not spend too much time on
> figuring out alternatives, but the ones that came to mind I thought were
> worse.

In the absence of other data, explicit is better than implicit:

<%- params => ($x, $y),
syntax => :html -%>

> Whether it should be restricted scope of not by default I don't know. It
> could be a chore to have to write a hash with name to value associations.

EPP doesn't need to inherit the warts of ERB, at least not by default.
If backward-compatibility is easy but not the defalut, then new users
will not accidentally use the deprecated feature - at least not
without careful thought.

In Python, if you wanted to pass your local variables to a called
function, you could use something like

called_function(**locals())

Which is explicit (and very rare). I don't necessarily think that EPP
should use the ** syntax, but certainly using an explicit option for
the behavior is possible.

> There is a Redmine issue where it is discussed if it is good or bad that the
> template functions can join results from multiple templates. I would prefer
> if it was just one, and instead call a join function if that behavior is
> needed.

I think the sticking point on that bug is that the change would be
backward-incompatible. The same is not the case for EPP. If the EPP
functions *never* concatenate by default, then users converting from
ERB templates will need to manually apply the concatenation during
conversion.

Dustin

[Henrik Lindberg]

On 2013-26-08 24:07, Dustin J. Mitchell wrote:
> On Sun, Aug 25, 2013 at 5:45 PM, Henrik Lindberg
> <henrik....@cloudsmith.com> wrote:
>>> This may be silly, but "Extended Puppet Templates" seems more logically
>>> abbreviated "EPT" - why "EPP"?
>>>
>> Does it say "Extended Puppet Templates" somewhere? The idea is to do the
>> same as in Ruby, i.e. "Embedded Ruby" = ERP, "Embedded Puppet" = EPP.
>
> I should have said "Embedded", not "Extended". And, fair enough -
> Embedded RuBy templates are not "ERT", they are "ERB". So Embedded
> PuPpet templates can be "EPP".
>

i.e. it is using the original file extensions .rb, and .pp and prepends
an 'e' for embedded.

>>> Could you include a link to the Puppet DSL in the text? It's easy to
>>> think that that means Puppet itself (e.g., resource declarations), when
>>> really it is a form of Ruby (I think). The link will add context.
>>>
>> Sorry, but I don't get your question. EPP is to Puppet what ERB is to ruby.
>> The EPP content is in a file, typically with extension .epp, but can be
>> anything except .pp (which is naturally "not embedded puppet code").
>>
>> But, I think you are asking something else...
>
> I'm asking, what language do I use between <% %>? Can I write
>
> <%
> file {
> '/etc/motd':
> content => "hello, world\n"
> }
> %>
>

At the technical parsing level, yes. This is then validated to only
include what we want to allow. The first implementation is very permissive.

> which is what "Embedded Puppet" suggests to me. That's not the case,
> and "Puppet DSL" is only slightly less unclear, unless the reader is
> enough of a Rubyist to realize that "DSL" in the Ruby community refers
> to a modified Ruby environment, and not an entirely different
> language. Basically, please add a link so I know unambiguously what I
> can and cannot do between <% .. %>.
>

All Puppet DSL expressions that do not require "top level" (i.e. class
and define) can be used.

>>> As for validation of filetypes, I think this is a good idea, but I don't
>>> like the <%- ($x, $y) :html -%> syntax for this. I'd prefer something a
>>> little more explicit and expandable, perhaps
>>> <%- ($x, $y) syntax => html -%>
>>>
>> The idea was to mimic the same specification in ARM-4 Heredoc, where the
>> syntax is ':syntax'.
>
> I could make the same criticism there (I just haven't gotten to that ARM yet).
>

Fair. It should be the same/similar though - and in the heredoc case
there are additional constraints since the construct is "parsed" by the
lexer.

>> I am not super happy with the somewhat magic <%- () -%> construct. It
>> allowed me to reuse parameter parsing. Did not spend too much time on
>> figuring out alternatives, but the ones that came to mind I thought were
>> worse.
>
> In the absence of other data, explicit is better than implicit:
>
> <%- params => ($x, $y),
> syntax => :html -%>
>

Yeah, but then a different tag is required since the text is not valid
Puppet DSL syntax, or new Puppet DSL syntax is required that is specific
for a "template function" which this essentially is.

Will look at it again.

>> Whether it should be restricted scope of not by default I don't know. It
>> could be a chore to have to write a hash with name to value associations.
>
> EPP doesn't need to inherit the warts of ERB, at least not by default.

Totally agree.

> If backward-compatibility is easy but not the defalut, then new users
> will not accidentally use the deprecated feature - at least not
> without careful thought.
>
> In Python, if you wanted to pass your local variables to a called
> function, you could use something like
>
> called_function(**locals())
>
> Which is explicit (and very rare). I don't necessarily think that EPP
> should use the ** syntax, but certainly using an explicit option for
> the behavior is possible.
>

Internally there is a way to get a hash map of all variables in scope; a
simple function could be implemented if this is needed.
Have to think more about a restricted scope feature.

>> There is a Redmine issue where it is discussed if it is good or bad that the
>> template functions can join results from multiple templates. I would prefer
>> if it was just one, and instead call a join function if that behavior is
>> needed.
>
> I think the sticking point on that bug is that the change would be
> backward-incompatible. The same is not the case for EPP. If the EPP
> functions *never* concatenate by default, then users converting from
> ERB templates will need to manually apply the concatenation during
> conversion.
>

With future parser and iteration it is quite easy.
['file1', 'file2', 'file3'].collect |$file| {
epp_template($file)
}.join('')

Or some variation thereof.
I can also imagine passing an array of files (instead of just one file)
with the meaning "please join the result of these". This was not
convenient in the old parser as you had to assign the array to a
variable first.

- henrik

[Erik Dalén]

They don't require top level in puppet manifests, this should be legal:

class foo {

  class bar {

    ...

  }

}

just that it declares the classes foo and foo::bar, not foo and bar.

Is this somehow disallowed in epp?

node statements requires top level though afaik.

in puppet version 2.6.8+ anonymous arrays and hashes should be allowed, so that worked in recent versions of the old parser as well.

--
Erik Dalén

[Henrik Lindberg]

On 2013-26-08 1:07, Erik Dal�n wrote:
> On 25 August 2013 15:47, Henrik Lindberg <henrik....@cloudsmith.com

I am using imprecise language, sorry.
The statements 'node', 'class xxx{}, and 'define xxx {}' are not allowed
in "conditional logic" which is the same as "nested block expression"
which is in fact an EPP is evaluated in.

> just that it declares the classes foo and foo::bar, not foo and bar.
>
> Is this somehow disallowed in epp?
>

Do you want to define classes and user defined types as a side effect
while producing template text? That seems odd.
The statement I made is only about Puppet DSL logic inside a template.

> node statements requires top level though afaik.
>

yes.

>
> in puppet version 2.6.8+ anonymous arrays and hashes should be allowed,
> so that worked in recent versions of the old parser as well.
>

oh, I was not aware of that. Have to check then. I started looking at
the grammar around mid 2.7.x, and when I compared what worked and what
did not against fairly recent versions on 2.7, and then for all 3.x
versions. Anyway, needs to be checked so not spreading false information
about problems that are non-existing.

Thanks for pointing it out.
- henrik

[Erik Dalén]

On 25 August 2013 16:34, Henrik Lindberg <henrik....@cloudsmith.com> wrote:

On 2013-26-08 1:07, Erik Dalén wrote:

On 25 August 2013 15:47, Henrik Lindberg <henrik.lindberg@cloudsmith.com

No, I don't want to, just seemed like a odd exception as it uses the same parser afaik. But the conditional logic/nested block thing explains it, thanks.

 

--
Erik Dalén

[Dustin J. Mitchell]

On Sun, Aug 25, 2013 at 6:47 PM, Henrik Lindberg
<henrik....@cloudsmith.com> wrote:
> At the technical parsing level, yes. This is then validated to only include
> what we want to allow. The first implementation is very permissive.

Oh! Then, my reading of the ARM was wrong - I thought that "Puppet
DSL" referred to the Puppet Ruby DSL. Now that I re-read the ARM, I
see that I was thrown off-track by the word "DSL", and read the rest
of the document based on that premise. Some examples will probably
prevent others from making this mistake -- particularly examples using
iteration (since that's the most common application of ruby
expressions in ERB).

Dustin

[Henrik Lindberg]

I find it is difficult to refer to "The Puppet Language", it does not
really have a name "Puppet DSL" is what it is called most of the time...

- henrik

[Dustin J. Mitchell]

Yes - examples will definitely clear up confusion for anyone who has
the same experience as me (which may be very few people!)

Dustin