Puppet Heredoc for Puppet 3.5.0 - RFC

[Henrik Lindberg]

In Puppet 3.5.0, there is an implementation of ARM-4 Puppet Heredoc,
(available when turning on --parser future).

The original ARM text has been copied to a google doc to make it easier
to comment. This version is found here:
https://docs.google.com/document/d/1vcYTFCah3zc1n86FTWqSfOLyoeAqcQ3FDzlHX-x0fyM/edit?usp=sharing

If you want a shorter introduction, you find my blog post about heredoc
here http://puppet-on-the-edge.blogspot.se/2014/03/heredoc-is-here.html

Looking forward to receiving feedback and questions.

The idea is that heredoc is made available as a regular feature in
Puppet 4.0.

Regards
- henrik

[Andy Parker]

There hasn't been any discussion on this as far as I can see. The functionality is in the 3.5.0 version that we had to pull, and it will also be in 3.5.1 that we should be releasing soon. If anyone has any comments please let us know. The design of this is still open for change until we reach puppet 4, where the new language system will become the default and any more changes to this will become more difficult.

- henrik

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/lg7t66%24m08%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

--

Andrew Parker

an...@puppetlabs.com

Freenode: zaphod42

Twitter: @aparker42

Software Developer

Join us at PuppetConf 2014, September 22-24 in San Francisco

Register by May 30th to take advantage of the Early Adopter discount —save $349!

[Joshua Hoblitt]

I haven't tried it (I just started testing 3.5.1-rc1 this morning) so
these are quick comments from only having looked at the ARM.

* I really like the perl5 style interpolation control.

* The End Marker/indentation control is a great feature.

* Is it possible to escape the endtag so that it can appear inside the
heredoc string?

* I feel it's a bit confusing for an operator to have a function call
like syntax. I'm sure it's already been discussed at length but why
wasn't the shell/perl5/ruby << heredoc operator reused?

-Josh

--

>> email to puppet-dev+...@googlegroups.com.

[Andy Parker]

On Tue, Apr 15, 2014 at 11:15 AM, Joshua Hoblitt <jhob...@cpan.org> wrote:

I haven't tried it (I just started testing 3.5.1-rc1 this morning) so
these are quick comments from only having looked at the ARM.

* I really like the perl5 style interpolation control.

* The End Marker/indentation control is a great feature.

* Is it possible to escape the endtag so that it can appear inside the
heredoc string?

I don't believe so. In fact I don't know of any heredoc system that allows that, since you can just select a different end marker.

 

* I feel it's a bit confusing for an operator to have a function call
like syntax.  I'm sure it's already been discussed at length but why
wasn't the shell/perl5/ruby << heredoc operator reused?

Actually, there wasn't much discussion. Henrik will have to explain the choice. I always assumed it had to do with parsing, possibly since << is used to introduce the collection query (a complete guess).

 

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/534D7752.9020402%40cpan.org.

For more options, visit https://groups.google.com/d/optout.

[Joshua Hoblitt]

On 04/16/2014 01:05 PM, Andy Parker wrote:
> On Tue, Apr 15, 2014 at 11:15 AM, Joshua Hoblitt <jhob...@cpan.org> wrote:
>>
>> * Is it possible to escape the endtag so that it can appear inside the
>> heredoc string?
>>
>>
> I don't believe so. In fact I don't know of any heredoc system that allows
> that, since you can just select a different end marker.

Nor do I; nor am I requesting this feature. :) I would be nice for this
to be mentioned in the eventual docs.

The only solid use case that occurs to me is being able to put complete
code examples into a heredoc. I imagine supporting such a feature would
require the parser to backtrack from the endtag to look for an escape
sequence, which may not be worth the cost.

>> * I feel it's a bit confusing for an operator to have a function call
>> like syntax. I'm sure it's already been discussed at length but why
>> wasn't the shell/perl5/ruby << heredoc operator reused?
>>
>
> Actually, there wasn't much discussion. Henrik will have to explain the
> choice. I always assumed it had to do with parsing, possibly since << is
> used to introduce the collection query (a complete guess).

That was my thought to but it looks like the DSL has a left shift
operator I was ignorant of:

https://github.com/puppetlabs/puppet/blob/master/lib/puppet/parser/lexer.rb#L168

The total extent of knowledge about racc is that it exists. Is there a
goal / technical reason to try to keep the grammar as context free as
possible?

-Josh

--

[Henrik Lindberg]

On 2014-16-04 23:41, Joshua Hoblitt wrote:
> On 04/16/2014 01:05 PM, Andy Parker wrote:
>> On Tue, Apr 15, 2014 at 11:15 AM, Joshua Hoblitt <jhob...@cpan.org> wrote:
>>>
>>> * Is it possible to escape the endtag so that it can appear inside the
>>> heredoc string?
>>>
>>>
>> I don't believe so. In fact I don't know of any heredoc system that allows
>> that, since you can just select a different end marker.
>
> Nor do I; nor am I requesting this feature. :) I would be nice for this
> to be mentioned in the eventual docs.
>

yes. good point. (The very purpose of using heredoc is to avoid escapes,
and it misses to point this out :-)

> The only solid use case that occurs to me is being able to put complete
> code examples into a heredoc. I imagine supporting such a feature would
> require the parser to backtrack from the endtag to look for an escape
> sequence, which may not be worth the cost.
>

As Andy said, the idea is to pick something that does not exist in the
text as the end tag, allowing it to be escaped creates a horrible mess,
since you can use a multi word string, it can be pretty much anything.

>>> * I feel it's a bit confusing for an operator to have a function call
>>> like syntax. I'm sure it's already been discussed at length but why
>>> wasn't the shell/perl5/ruby << heredoc operator reused?
>>>
>>
>> Actually, there wasn't much discussion. Henrik will have to explain the
>> choice. I always assumed it had to do with parsing, possibly since << is
>> used to introduce the collection query (a complete guess).
>
> That was my thought to but it looks like the DSL has a left shift
> operator I was ignorant of:
>
> https://github.com/puppetlabs/puppet/blob/master/lib/puppet/parser/lexer.rb#L168
>
> The total extent of knowledge about racc is that it exists. Is there a
> goal / technical reason to try to keep the grammar as context free as
> possible?
>

The decision was based on several things:

* << is already left shift, although <<- would be unique (but then mean
trim if we were to use Ruby semantics. (There is no clash with <|, or <||)
* there are issues with the Ruby syntax (see the ARM - things you cannot do)
* I wanted to be able to control the escapes individually, and therefore
needed something that marks the end lexically. (now the closing
parenthesis in @(END) ).
* Same reason for the syntax checking support, yet another parameter
that is hard to handle if there is no end.
* All of the work is done in the lexer and it needs to be context free.
The lexer delivers tokens based on source code that is "out of band".

To me it is not so strange to think about @(END) as a function call to
the special function @() - a function that sucks up the source until the
first argument (the end tag), and then hides it from the grammar, and
then producing that text as the result of the "call".

Ruby newbies struggle with the Heredoc syntax in Ruby afaict, and the
various languages that has heredoc all do it slightly differently, so
there was no clear winning syntax to adopt (at least that was my
conclusion).

- henrik

[Jakov Sosic]

Hi, I'm forwarding my question from puppet-users because it didn't get
many answers there...



I'm developing some of my custom types, and some of them share same
params, with exactly the same methods...

For example, I have these params in three of my types and they are same:

lib/puppet/type/mytype1.rb
lib/puppet/type/mytype2.rb
lib/puppet/type/mytype3.rb

newproperty(:comment) do
defaultto ''
end


Is there a way I can maybe extract that code into some external library
and reuse it in all of my three types? Now, this definition is not a
problem, but when you start to include various checks, and override
methods like insync? and your param code grows to 30-40 lines, and you
use it in multiple types, it gets usefull to extract that code to some
external class and include it into these types.



I have same problem with methods in providers, for example:

lib/puppet/provider/mytype1/default.rb
lib/puppet/provider/mytype2/default.rb
lib/puppet/provider/mytype3/default.rb

all have for example:

# sets comment
def comment=(value)
@property_hash[:comment]=(value)
end

Is there a way to extract method provider into some external file and
just include it somehow?


If it's possible I would like to escape a route of creating another
custom type, and add its value to param of another custom type :-/

[Felix Frank]

On 04/17/2014 02:27 PM, Jakov Sosic wrote:
> Hi, I'm forwarding my question from puppet-users because it didn't get
> many answers there...

Dude. That was yesterday. Evening. Impatient much? ;-)

Also you apparently sent this in response to an answer in the Heredoc
RFC thread, which is very bad form.

I shall respond on the users' list when I get a chance. Let's not derail
this thread any further.

Cheers,
Felix

[Jakov Sosic]

On 04/17/2014 03:11 PM, Felix Frank wrote:
> On 04/17/2014 02:27 PM, Jakov Sosic wrote:
>> Hi, I'm forwarding my question from puppet-users because it didn't get
>> many answers there...
>
> Dude. That was yesterday. Evening. Impatient much? ;-)

Sorry, I'm not impatient but since I was off the list for half a year, I
though that the volume of traffic is somehow lower, so just in case I
reposted here :)


PS. Sorry for the RFC breakage :)

[Nan Liu]

On Thu, Apr 17, 2014 at 7:27 AM, Jakov Sosic <jso...@gmail.com> wrote:

Hi, I'm forwarding my question from puppet-users because it didn't get many answers there...



I'm developing some of my custom types, and some of them share same params, with exactly the same methods...

For example, I have these params in three of my types and they are same:

lib/puppet/type/mytype1.rb
lib/puppet/type/mytype2.rb
lib/puppet/type/mytype3.rb

  newproperty(:comment) do
    defaultto ''
  end


Is there a way I can maybe extract that code into some external library and reuse it in all of my three types? Now, this definition is not a problem, but when you start to include various checks, and override methods like insync? and your param code grows to 30-40 lines, and you use it in multiple types, it gets usefull to extract that code to some external class and include it into these types.

You can write a custom property to change how it behaves (by writing a custom insync, should_to_s, is_to_s)[1]. But I don't think there's a way to share newproperty, newparam across multiple types. There are examples of auto generated types[2], but there's so much metaprogramming I doubt it's worth while for a simple resource. 

 

I have same problem with methods in providers, for example:

lib/puppet/provider/mytype1/default.rb
lib/puppet/provider/mytype2/default.rb
lib/puppet/provider/mytype3/default.rb

all have for example:

  # sets comment
  def comment=(value)
    @property_hash[:comment]=(value)
  end

Is there a way to extract method provider into some external file and just include it somehow?


If it's possible I would like to escape a route of creating another custom type, and add its value to param of another custom type :-/

The easiest thing to do for providers is have the common code in a parent provider[3]. You can also override the method in the child provider.

Nan

1. https://github.com/vmware/vmware-vmware_lib/blob/master/lib/puppet/property/vmware_array.rb

2. https://github.com/vmware/vmware-vcenter/blob/master/lib/puppet/type/esx_vmknic.rb#L51-L92
3. parent: https://github.com/vmware/vmware-vcenter/blob/master/lib/puppet/provider/vcenter.rb
Puppet::Type.type(...).provide(:default, :parent => Puppet::Provider::Vcenter) do

[Jakov Sosic]

On 04/17/2014 04:32 PM, Nan Liu wrote:

> The easiest thing to do for providers is have the common code in a
> parent provider[3]. You can also override the method in the child provider.

Thank you Nan Liu.

I gave you credis on bitbucket in my module:

https://bitbucket.org/jsosic/puppet-cobbler/issue/20/copy-pase-of-methods-in-various-providers