Puppet 3.0 and Hiera

2315 views
Skip to first unread message

Pieter van de Bruggen

unread,
May 3, 2012, 1:05:14 PM5/3/12
to puppet...@googlegroups.com

Greetings,

As many of you may be aware, Hiera will be tightly integrated into Puppet in the upcoming release of Puppet 3.0. As a final sanity check of this work, I’d like to open our current plan for integration up for feedback. This is particularly for feedback from existing Hiera users, but I hope to solicit good feedback from those not using Hiera as well.

The problem, as it currently exists, is that Puppet (core) has no good first-class mechanism for separating configuration data from manifests. Everything from $faked_namespaces__in__variable_names to specialty “params” classes have been tried, with varying degrees of success.

Hiera came along as another solution to this problem, and provided a useful abstraction for hierarchical data lookup, but life as a plugin meant that certain integrations were still difficult. Our aim in Puppet 3.0 is to tighten up these integrations (making Hiera more useful), provide first-class data separation solution (for those not already using Hiera), and to provide a simple and safe migration for those currently using Hiera.

Here’s what’s new:

  • Hiera data keys can be namespaced
    • (e.g. 'dns::nameserver': '8.8.8.8')
  • Namespaced data will automatically populate class parameters
    • (e.g. 'dns::nameserver' will be automatically looked up for the $nameserver parameter when you include dns)

Here’s what’s changed:

  • The hiera-puppet module will no longer be required
    • It should, however, still continue to work
  • The heira() function (from the hiera-puppet module) will be superceded by the lookup() function (in Puppet core)
  • hiera_include() will not be ported; include now properly accept arrays, making this redundant

Here’s what were still wondering about:

  • How should we integrate hiera_array() and hiera_hash()?
  • How should we integrate hiera’s “default” and “override” parameters?
  • How should we handle overlaps between data supplied by Hiera and data supplied in a parameterized class include?

If you’re interested in test-driving the new functionality for yourself, checkout out the master branch of the Puppet repository on Github, install Hiera (with gem install hiera) and make the following configuration changes:

  • Set data_binding_terminus to hiera
  • If you have a Hiera configuration file, set hiera_config to point to it.
  • If you don’t, create a file in ${confdir}/hiera.yaml with these contents:

     :backend:

       - yaml

     :yaml:

       :datadir: $confdir/data

     :hierarchy:

       - %{certname}

       - %{environment}

       - global

This sets up a default hierarchy that looks for values in ${confdir}/data, first in the ${certname}.yml file, then in the ${environment}.yml file, then in global.yml.

That should be it!  Please, let us know if you're having trouble getting started, or if you have questions, concerns, thoughts, or opinions about any of this.

Thanks!


Craig Dunn

unread,
May 4, 2012, 4:40:33 AM5/4/12
to puppet...@googlegroups.com, Pieter van de Bruggen

All sounds very good...

> Here�s what�s new:
>
> * Hiera data keys can be namespaced
> o (e.g. 'dns::nameserver': '8.8.8.8')

Namespace scoping is defiantly a massive improvement to Hiera - Im
curious how this affects back ends though... Will lookup("nameserver")
called from Puppet in a class called DNS be always mapped to
dns::nameserver and that value passed as the lookup key to Hiera and on
to whatever back end you're using? If so is that going to make existing
Hiera setups incompatible?

> * Namespaced data will automatically populate class parameters
> o (e.g. 'dns::nameserver' will be automatically looked up for the
> $nameserver parameter when you include dns)

I currently try and achieve this with using blah/%{calling_module} in my
hierarchy and therefore class dns ( $nameserver=hiera("nameserver" )
will look for nameserver: in blah/dns.yaml which also adds some nice
grouping/separation for setups with many Hiera enabled modules - will
the use of %{calling_module} still be supportable?

Also - is there any indication that tying a lookup to a particular back
end may be supported at some point?
(http://projects.puppetlabs.com/issues/12951)

Sounds like a lot of great new functionality though, looking forward to
testing it out!

Craig


--
Craig Dunn | http://www.craigdunn.org
Yahoo/Skype: craigrdunn | Twitter: @crayfishX

R.I.Pienaar

unread,
May 4, 2012, 5:05:58 AM5/4/12
to puppet...@googlegroups.com


----- Original Message -----
> From: "Pieter van de Bruggen" <pie...@puppetlabs.com>

<snip>

>
> * How should we integrate hiera_array() and hiera_hash() ?
> * How should we integrate hiera ’s “default” and “override”
> parameters?
> * How should we handle overlaps between data supplied by Hiera
> and data supplied in a parameterized class include?

Given:

class foo($something="default") { }

I think the plan was that there would be a priority order as below:

- someone wrote in a manifest: class{"foo": something => something}
- an ENC supplied the values for something on the class foo
- someone did "include foo" or class{"foo": } this would consult hiera
- if hiera does not have an answer it would default to "default"

This gives people with more complex needs than those matched by hiera
the ability to use ENCs and get exactly the data model they desire as
well as people who do not want any magical data lookup for some class
the ability to hard code values.

And at the same time it lets module authors provide out of the box
defaults that work without placing a load on module users where they
might have to read every class and set hiera values for every argument
before they can use a class if all they wanted was the default out of
the box behaviour.

Does this sound sane?

> If you’re interested in test-driving the new functionality for
> yourself, checkout out the master branch of the Puppet repository on
> Github , install Hiera (with gem install hiera ) and make the
> following configuration changes:
>
> * Set data_binding_terminus to hiera
> * If you have a Hiera configuration file, set hiera_config to
> point to it.
> * If you don’t, create a file in ${confdir}/hiera.yaml with these
> contents:
>
>
> :backend:
>
> - yaml
>
> :yaml:
>
> :datadir: $confdir/data
>
> :hierarchy:
>
> - %{certname}
>
> - %{environment}
>
> - global
>
> This sets up a default hierarchy that looks for values in
> ${confdir}/data , first in the ${certname}.yml file, then in the
> ${environment}.yml file, then in global.yml .

These should probably %{::certname} etc right?

Ashley Penney

unread,
May 4, 2012, 8:47:01 AM5/4/12
to puppet...@googlegroups.com
On Fri, May 4, 2012 at 5:05 AM, R.I.Pienaar <r...@devco.net> wrote:

I think the plan was that there would be a priority order as below:

 - someone wrote in a manifest: class{"foo": something => something}
 - an ENC supplied the values for something on the class foo
 - someone did "include foo" or class{"foo": } this would consult hiera
 - if hiera does not have an answer it would default to "default"

This gives people with more complex needs than those matched by hiera
the ability to use ENCs and get exactly the data model they desire as
well as people who do not want any magical data lookup for some class
the ability to hard code values.

And at the same time it lets module authors provide out of the box
defaults that work without placing a load on module users where they
might have to read every class and set hiera values for every argument
before they can use a class if all they wanted was the default out of
the box behaviour.

Does this sound sane?

I can't speak for anyone else but this sounds fantastic to me.  I've already
ran into this problem where I want a bunch of defaults to be used unless
they need to specifically overwrite them in Hiera and I don't always want
to add those into Hiera (because of the lack of namespacing it can get
messy).  All of the changes mentioned so far will be of very big benefit
and really improve things. :) 

Josh

unread,
May 4, 2012, 9:00:15 AM5/4/12
to puppet...@googlegroups.com
I'm not sure if its in there but one thing that would be *amazing* would be support for empty arrays/hashes. E.g.

$hash1 = hiera_hash('hash1', {})
$hash1 = hiera_hash('hash2', {})

This means that now I can do a:

merge_hash($hash1, $hash2)

This will now work regardless of whether I have declared hash1 and hash2 for a given node or not, where as at the moment if you don't declare both of them in hiera the merge fails because the empty hash is seen as a string.

Native support for the merging would be handy as well.

Josh

Jeremy Carroll

unread,
May 4, 2012, 12:19:37 PM5/4/12
to puppet...@googlegroups.com
We use Hiera backed Puppet here at Klout. Mainly for the reasons of separating configuration data from puppet modules. We have had great success, and I welcome this change to puppet core. Reminds me of some of Chef's "Deep Merge" functionality. Here are a few of my thoughts.
  • Name spacing variable names is a welcome improvement. It will help with variable name collisions we have experienced (Ex: module1::hosts, module2::hosts). We currently use the %{calling_class}, %{calling_module} method of scoping hiera variables.
  • I currently like the flexibility of defining my own order for merging of variables in hiera.yaml. But I would like Puppet to formalize the merge order so everybody can use the same style when creating modules for reuse. Also have some methods to specify overrides instead of merging from top to bottom for hashes / arrays. Helps for configurations for systems that don't want to inherit everything. Chef has a lot of work for Deep Merging ordering that I believe Hiera would benefit from.
  • hiera_hash would be improved by sorting the hash natively. We currently repeat this pattern every time we use a hiera hash. We create a sorted array of each hash key, then iterate over the array to lookup values in the hash. If we fail to do this, puppet run re-orders the file every run as hash orders are not guaranteed. This causes a lot of un-necessary work and change notifications. Example below.
hieradata
--- 
hiera_var:
  user:
    value:  'my_value'
module/params.pp
$hiera_var   = hiera_hash('hiera_var')
$sorted_var  = sort(keys($hiera_var))
module/template/sometemplate.pp
<% scope.lookupvar('module::params::sorted_var').each do |item| -%><% scope.lookupvar('hadoop::params::hiera_var')[item].each do |key,value| -%>
<%= user %> <%= key %> <%= value %>
<% end -%><% end -%>

R.I.Pienaar

unread,
May 4, 2012, 12:27:48 PM5/4/12
to puppet...@googlegroups.com


----- Original Message -----
> From: "Jeremy Carroll" <jer...@klout.com>
> To: puppet...@googlegroups.com
> Sent: Friday, May 4, 2012 5:19:37 PM
> Subject: [Puppet Users] Re: Puppet 3.0 and Hiera
>
> We use Hiera backed Puppet here at Klout. Mainly for the reasons of
> separating configuration data from puppet modules. We have had great
> success, and I welcome this change to puppet core. Reminds me of
> some of Chef's "Deep Merge" functionality. Here are a few of my
> thoughts.
>
>
> * Name spacing variable names is a welcome improvement. It will
> help with variable name collisions we have experienced (Ex:
> module1::hosts, module2::hosts). We currently use the
> %{calling_class}, %{calling_module} method of scoping hiera
> variables.
> * I currently like the flexibility of defining my own order for
> merging of variables in hiera.yaml. But I would like Puppet to
> formalize the merge order so everybody can use the same style
> when creating modules for reuse. Also have some methods to
> specify overrides instead of merging from top to bottom for
> hashes / arrays. Helps for configurations for systems that don't
> want to inherit everything. Chef has a lot of work for Deep
> Merging ordering that I believe Hiera would benefit from.

Do you mean you want the hierarchy to be standardised?

can you expand this with examples please?


> * hiera_hash would be improved by sorting the hash natively. We
> currently repeat this pattern every time we use a hiera hash. We
> create a sorted array of each hash key, then iterate over the
> array to lookup values in the hash. If we fail to do this,
> puppet run re-orders the file every run as hash orders are not
> guaranteed. This causes a lot of un-necessary work and change
> notifications. Example below.

Unfortunately ruby hashes has no order so we cant sort the hashes for you :(

Things are improving in 1.9, but I do not think there is much we can do there
> * Hiera data keys can be namespaced
>
>
> * (e.g. 'dns::nameserver': '8.8.8.8' )
> * Namespaced data will automatically populate class parameters
>
> * (e.g. 'dns::nameserver' will be automatically looked up for
> the $nameserver parameter when you include dns )
>
>
> Here’s what’s changed:
>
> * The hiera-puppet module will no longer be required
>
>
> * It should, however, still continue to work
> * The heira() function (from the hiera-puppet module) will be
> superceded by the lookup() function (in Puppet core)
> * hiera_include() will not be ported; include now properly accept
> arrays, making this redundant
>
>
> Here’s what were still wondering about:
>
> * How should we integrate hiera_array() and hiera_hash() ?
> * How should we integrate hiera ’s “default” and “override”
> parameters?
> * How should we handle overlaps between data supplied by Hiera
> and data supplied in a parameterized class include?
>
>
> If you’re interested in test-driving the new functionality for
> yourself, checkout out the master branch of the Puppet repository on
> Github , install Hiera (with gem install hiera ) and make the
> following configuration changes:
>
> * Set data_binding_terminus to hiera
> * If you have a Hiera configuration file, set hiera_config to
> point to it.
> * If you don’t, create a file in ${confdir}/hiera.yaml with these
> contents:
>
>
> :backend:
>
> - yaml
>
> :yaml:
>
> :datadir: $confdir/data
>
> :hierarchy:
>
> - %{certname}
>
> - %{environment}
>
> - global
>
> This sets up a default hierarchy that looks for values in
> ${confdir}/data , first in the ${certname}.yml file, then in the
> ${environment}.yml file, then in global.yml .
>
> That should be it! Please, let us know if you're having trouble
> getting started, or if you have questions, concerns, thoughts, or
> opinions about any of this.
>
> Thanks!
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/GxVliXSX5u0J .
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
Message has been deleted

Ohad Levy

unread,
May 4, 2012, 3:05:45 PM5/4/12
to puppet...@googlegroups.com
On Fri, May 4, 2012 at 12:05 PM, R.I.Pienaar <r...@devco.net> wrote:


----- Original Message -----
> From: "Pieter van de Bruggen" <pie...@puppetlabs.com>

<snip>

>
>     * How should we integrate hiera_array() and hiera_hash() ?
>     * How should we integrate hiera ’s “default” and “override”
>     parameters?
>     * How should we handle overlaps between data supplied by Hiera
>     and data supplied in a parameterized class include?

Given:

class foo($something="default") { }

I think the plan was that there would be a priority order as below:

 - someone wrote in a manifest: class{"foo": something => something}
 - an ENC supplied the values for something on the class foo
 - someone did "include foo" or class{"foo": } this would consult hiera
 - if hiera does not have an answer it would default to "default"

 Would be possible to define which function is called in this case? by default that should be heira, but in case someone wants something else, I think we should allow that?

Ohad

Daniel Pittman

unread,
May 4, 2012, 4:01:23 PM5/4/12
to puppet...@googlegroups.com
The implementation of the automatic lookup is through an indirection
and terminus - so you can (theoretically) run with `none` as the
back-end, and you can supply your own Ruby code that implements the
lookup in some other way.

--
Daniel Pittman
⎋ Puppet Labs Developer – http://puppetlabs.com
♲ Made with 100 percent post-consumer electrons

Trevor Vaughan

unread,
May 5, 2012, 12:26:08 PM5/5/12
to puppet...@googlegroups.com, Pieter van de Bruggen
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

With the integration of Hiera, is the extlookup functionality being deprecated?

If so, I would like to ask that the team add a Hiera backend for the existing extlookup data sources.

I know that it won't support hashes but it would help to break the view that Puppet routinely breaks existing functionality on upgrades.

Thanks,

Trevor

On 05/03/2012 01:05 PM, Pieter van de Bruggen wrote:
> Greetings,
>
> As many of you may be aware, Hiera will be tightly integrated into Puppet
> in the upcoming release of Puppet 3.0. As a final sanity check of this
> work, I�d like to open our current plan for integration up for feedback.
> This is particularly for feedback from existing Hiera users, but I hope to
> solicit good feedback from those not using Hiera as well.
>
> The problem, as it currently exists, is that Puppet (core) has no good
> first-class mechanism for separating configuration data from manifests.
> Everything from $faked_namespaces__in__variable_names to specialty �params�
> classes have been tried, with varying degrees of success.
>
> Hiera came along as another solution to this problem, and provided a useful
> abstraction for hierarchical data lookup, but life as a plugin meant that
> certain integrations were still difficult. Our aim in Puppet 3.0 is to
> tighten up these integrations (making Hiera more useful), provide
> first-class data separation solution (for those not already using Hiera),
> and to provide a simple and safe migration for those currently using Hiera.
>
> Here�s what�s new:
>
> - Hiera data keys can be namespaced
> - (e.g. 'dns::nameserver': '8.8.8.8')
> - Namespaced data will automatically populate class parameters
> - (e.g. 'dns::nameserver' will be automatically looked up for the
> $nameserver parameter when you include dns)
>
> Here�s what�s changed:
>
> - The hiera-puppet module will no longer be required
> - It should, however, still continue to work
> - The heira() function (from the hiera-puppet module) will be superceded
> by the lookup() function (in Puppet core)
> - hiera_include() will not be ported; include now properly accept
> arrays, making this redundant
>
> Here�s what were still wondering about:
>
> - How should we integrate hiera_array() and hiera_hash()?
> - How should we integrate hiera�s �default� and �override� parameters?
> - How should we handle overlaps between data supplied by Hiera and data
> supplied in a parameterized class include?
>
> If you�re interested in test-driving the new functionality for yourself,
> checkout out the master branch of the Puppet repository on
> Github<https://github.com/puppetlabs/puppet>,
> install Hiera (with gem install hiera) and make the following configuration
> changes:
>
> - Set data_binding_terminus to hiera
> - If you have a Hiera configuration file, set hiera_config to point to
> it.
> - If you don�t, create a file in ${confdir}/hiera.yaml with these
> contents:
>
> :backend:
>
> - yaml
>
> :yaml:
>
> :datadir: $confdir/data
>
> :hierarchy:
>
> - %{certname}
>
> - %{environment}
>
> - global
>
> This sets up a default hierarchy that looks for values in ${confdir}/data,
> first in the ${certname}.yml file, then in the ${environment}.yml file,
> then in global.yml.
>
> That should be it! Please, let us know if you're having trouble getting
> started, or if you have questions, concerns, thoughts, or opinions about
> any of this.
>
> Thanks!
>

- --
Trevor Vaughan
Vice President, Onyx Point, Inc.
email: tvau...@onyxpoint.com
phone: 410-541-ONYX (6699)
pgp: 0x6C701E94

- -- This account not approved for unencrypted sensitive information --
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQEcBAEBAgAGBQJPpVScAAoJECNCGV1OLcypcesIAJT9gARlmQNxR7ftUo4soR1r
R2aCh/k2B+Wylj/xfF3o9j3gZh/1WrOLogMGThNgYT9Vcv50zCcbeFjbCTVr2bBo
18gtlcoJhTBeUebBk6CsFsG+sIApwja3UDbpSm49FRquCK/KI+fBV8gHhAQt9fgm
1P0ZXB1KU61nhqrLEpHNRmJVKBvLCwDuqEMJltMuGoysczWEv0uVROtYari1T3g+
tNqov2EkSj5NV+Au5DeNNKviToEDn+wR7JvtXEqRdwSc142U9PnZwp8MvM+/i2+W
RGKqg8HZuMMLEbbSnimdRnS6pCwKdgVJDPwBb+CoiZJJ0ftWCF2F9lNDjgUNjcI=
=/h12
-----END PGP SIGNATURE-----
tvaughan.vcf

R.I.Pienaar

unread,
May 6, 2012, 6:26:52 AM5/6/12
to puppet...@googlegroups.com, puppet...@googlegroups.com, Pieter van de Bruggen

On 5 May 2012, at 17:26, Trevor Vaughan <tvau...@onyxpoint.com> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> With the integration of Hiera, is the extlookup functionality being deprecated?
>
> If so, I would like to ask that the team add a Hiera backend for the existing extlookup data sources.
>
> I know that it won't support hashes but it would help to break the view that Puppet routinely breaks existing functionality on upgrades.


The hiera-puppet project provides a script to convert extlookup CSV to json or yaml, using that to migrate would be my preferred way with the extlookup function logging deprecation warnings

If this reflects the actual plan I don't know, I did expect the people working on the merging project to engage on this thread more since they know best and there seems to be a few points for discussion here already that needs clarification


>
> Thanks,
>
> Trevor
>
> On 05/03/2012 01:05 PM, Pieter van de Bruggen wrote:
>> Greetings,
>>
>> As many of you may be aware, Hiera will be tightly integrated into Puppet
>> in the upcoming release of Puppet 3.0. As a final sanity check of this
>> work, I’d like to open our current plan for integration up for feedback.
>> This is particularly for feedback from existing Hiera users, but I hope to
>> solicit good feedback from those not using Hiera as well.
>>
>> The problem, as it currently exists, is that Puppet (core) has no good
>> first-class mechanism for separating configuration data from manifests.
>> Everything from $faked_namespaces__in__variable_names to specialty “params”
>> classes have been tried, with varying degrees of success.
>>
>> Hiera came along as another solution to this problem, and provided a useful
>> abstraction for hierarchical data lookup, but life as a plugin meant that
>> certain integrations were still difficult. Our aim in Puppet 3.0 is to
>> tighten up these integrations (making Hiera more useful), provide
>> first-class data separation solution (for those not already using Hiera),
>> and to provide a simple and safe migration for those currently using Hiera.
>>
>> Here’s what’s new:
>>
>> - Hiera data keys can be namespaced
>> - (e.g. 'dns::nameserver': '8.8.8.8')
>> - Namespaced data will automatically populate class parameters
>> - (e.g. 'dns::nameserver' will be automatically looked up for the
>> $nameserver parameter when you include dns)
>>
>> Here’s what’s changed:
>>
>> - The hiera-puppet module will no longer be required
>> - It should, however, still continue to work
>> - The heira() function (from the hiera-puppet module) will be superceded
>> by the lookup() function (in Puppet core)
>> - hiera_include() will not be ported; include now properly accept
>> arrays, making this redundant
>>
>> Here’s what were still wondering about:
>>
>> - How should we integrate hiera_array() and hiera_hash()?
>> - How should we integrate hiera’s “default” and “override” parameters?
>> - How should we handle overlaps between data supplied by Hiera and data
>> supplied in a parameterized class include?
>>
>> If you’re interested in test-driving the new functionality for yourself,
>> checkout out the master branch of the Puppet repository on
>> Github<https://github.com/puppetlabs/puppet>,
>> install Hiera (with gem install hiera) and make the following configuration
>> changes:
>>
>> - Set data_binding_terminus to hiera
>> - If you have a Hiera configuration file, set hiera_config to point to
>> it.
>> - If you don’t, create a file in ${confdir}/hiera.yaml with these
> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>
> <tvaughan.vcf>

Nigel Kersten

unread,
May 6, 2012, 5:54:41 PM5/6/12
to puppet...@googlegroups.com, Pieter van de Bruggen
On Sat, May 5, 2012 at 9:26 AM, Trevor Vaughan <tvau...@onyxpoint.com> wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

With the integration of Hiera, is the extlookup functionality being deprecated?

If so, I would like to ask that the team add a Hiera backend for the existing extlookup data sources.

I know that it won't support hashes but it would help to break the view that Puppet routinely breaks existing functionality on upgrades.

We'll deprecate it, but not *break* it.

We'd go through a whole release cycle before actually removing the extlookup function, and as RI pointed out, we can migrate the backend stores from extlookup CSV to straight up JSON/YAML.


Derek J. Balling

unread,
May 7, 2012, 10:22:53 AM5/7/12
to puppet...@googlegroups.com
On May 3, 2012, at 1:05 PM, Pieter van de Bruggen wrote:

As many of you may be aware, Hiera will be tightly integrated into Puppet in the upcoming release of Puppet 3.0.

What will this mean for sites that have specifically avoided using Hiera? I would presume that this should just mean "business as usual", with no significant impact?




Nigel Kersten

unread,
May 7, 2012, 11:54:00 AM5/7/12
to puppet...@googlegroups.com
Yes. If you're not planning to externalize data at all, the only difference you'll see is that you'll now have Hiera installed on your system when you install Puppet 3.0.


Derek J. Balling

unread,
May 7, 2012, 12:26:32 PM5/7/12
to puppet...@googlegroups.com

On May 7, 2012, at 11:54 AM, Nigel Kersten wrote:
> Yes. If you're not planning to externalize data at all, the only difference you'll see is that you'll now have Hiera installed on your system when you install Puppet 3.0.

Cool. Was really panicking there for a moment.

I kinda like having "human readable" manifests as opposed to "machine readable" YAML :-)

Cheers,
D


Nigel Kersten

unread,
May 7, 2012, 12:45:21 PM5/7/12
to puppet...@googlegroups.com
Externalizing data actually helps things be much more human readable imho, as you end up having semantically useful labels in the puppet manifests rather than naked data.

But yes, if you're not interested in externalizing data, so you haven't put anything in the Hiera store, and continue to embed data in your manifests, everything will still work as before.




--
Nigel Kersten | http://puppetlabs.com | @nigelkersten

Christopher Wood

unread,
May 7, 2012, 12:47:56 PM5/7/12
to puppet...@googlegroups.com
Wrapper script (similar concept for anywhere with a yaml reader):

#!/usr/bin/perl
use warnings;
use strict;
use YAML::Syck;
use File::Slurp;
use Data::Dumper;
my $file = $ARGV[0];
my $text = read_file($file);
my $yaml = Load($text);
print Dumper($yaml);


Out of nosiness, how do you handle the edgier cases in your manifests? For example, how everything is a standard syslog client, except for this host which is loghost, and these couple of clients which all do different things with syslog?



> Cheers,
> D

Derek J. Balling

unread,
May 8, 2012, 6:00:07 AM5/8/12
to puppet...@googlegroups.com

On May 7, 2012, at 12:47 PM, Christopher Wood wrote:
> Wrapper script (similar concept for anywhere with a yaml reader):

Still have to *write* the YAML files.

> Out of nosiness, how do you handle the edgier cases in your manifests? For example, how everything is a standard syslog client, except for this host which is loghost, and these couple of clients which all do different things with syslog?

We would have the syslog module have all the "common" files/packages/services/etc. declared, and then simply do something like:

if ($fqdn =~ /^syslog/)
{
..... log host specific stuff .....
}
else
{
..... NON-log host specific stuff .....
}

doesn't seem like rocket science, and makes it clear, all in one place, what's "common", and what's "special".

Puppet provides some wonderful decision-making capabilities in the manifests themselves, I've yet to see any need to have some ugly machine-readable formatted file to solve a problem.

But I *have* seen the parts of our organization that DO use Hiera complaining loudly that they can't easily figure out which of a slurry of YAML files is changing some setting/variable to a value they're not expecting.

Cheers,
D

John Warburton

unread,
May 8, 2012, 7:14:48 AM5/8/12
to puppet...@googlegroups.com
On 8 May 2012 20:00, Derek J. Balling <dr...@megacity.org> wrote:

On May 7, 2012, at 12:47 PM, Christopher Wood wrote:
> Wrapper script (similar concept for anywhere with a yaml reader):

Still have to *write* the YAML files.

Oh yeah. This is my big motivator sticking with extlookup and not converting to Hiera - anyone in the team can do CSV. But YAML is starting to push well into the dev camp and far away from ops (IMHO)

John

Craig Dunn

unread,
May 8, 2012, 7:22:11 AM5/8/12
to puppet...@googlegroups.com

> Still have to *write* the YAML files.
>
> Oh yeah. This is my big motivator sticking with extlookup and not
> converting to Hiera - anyone in the team can do CSV. But YAML is
> starting to push well into the dev camp and far away from ops (IMHO)

Unless you're doing rich data types such as hashes, which if you're
using CSV then it sounds like you're not, then a simple YAML
representation becomes "key: val" - YAML only starts pushing into dev
camp when and if you use some of the more advanced features like anchors
IMO.

Plus, when talking about Hiera we shouldn't be focusing too much on the
credits or pitfalls of YAML - there are other DB and file based back
ends available, and if one of those doesn't fit the bill it's very
possible to extend Hiera with a new backend that supports a different
format.

R.I.Pienaar

unread,
May 8, 2012, 7:43:01 AM5/8/12
to puppet...@googlegroups.com
If you want the same abilities that CSV has then this would be your YAML:

sysadmin: y...@your.com
nameservers: [1.2.3.4, 2.3.4.5]

just that simple, I think this is much clearer than CSV. If people really are
attached to CSV i guess it would be pretty trivial to write a CSV backend for
hiera but in the past one of the big complaints I had about extlookup was CSV
files.

When extlookup was written puppet did not have hashes and so CSV was a perfect
fit, now though it does and CSV just cant store all the kinds of data we needed
so rather than provide a crippled CSV implementation I decided to skip it.

John Warburton

unread,
May 8, 2012, 8:06:13 AM5/8/12
to puppet...@googlegroups.com
Thanks R.I. & Craig Dunn

On 8 May 2012 21:43, R.I.Pienaar <r...@devco.net> wrote:

If you want the same abilities that CSV has then this would be your YAML:

sysadmin: y...@your.com
nameservers: [1.2.3.4, 2.3.4.5]

just that simple, I think this is much clearer than CSV.  If people really are

That's great - an easy migration then

Cheers

John

Christopher Wood

unread,
May 8, 2012, 9:39:41 AM5/8/12
to puppet...@googlegroups.com
On Tue, May 08, 2012 at 06:00:07AM -0400, Derek J. Balling wrote:
>
> On May 7, 2012, at 12:47 PM, Christopher Wood wrote:
> > Wrapper script (similar concept for anywhere with a yaml reader):
>
> Still have to *write* the YAML files.

I don't think I've ever actually opened a text editor and written YAML. (Tweaked, sure, but not written.) This concept is useful across any number of scripting languages:

#!/usr/bin/perl
use warnings;
use strict;
use YAML::Syck;
my %opts = (
'key1' => 'value1',
'key2' => [ 'list2a', 'list2b' ],
'key3' => { 'key3a' => 'value3a', 'key3b' => [ 'list3b1', 'list3b2' ], },
);
$YAML::Syck::SortKeys = 1;
my $yaml = Dump( \%opts );
print $yaml;

It's probably easier to think in your preferred language's data structure and then translate that to YAML. As discussed in other branches, most of our data structure requirements really aren't that complex.

> > Out of nosiness, how do you handle the edgier cases in your manifests? For example, how everything is a standard syslog client, except for this host which is loghost, and these couple of clients which all do different things with syslog?
>
> We would have the syslog module have all the "common" files/packages/services/etc. declared, and then simply do something like:
>
> if ($fqdn =~ /^syslog/)
> {
> ..... log host specific stuff .....
> }
> else
> {
> ..... NON-log host specific stuff .....
> }
>
> doesn't seem like rocket science, and makes it clear, all in one place, what's "common", and what's "special".
>
> Puppet provides some wonderful decision-making capabilities in the manifests themselves, I've yet to see any need to have some ugly machine-readable formatted file to solve a problem.
>
> But I *have* seen the parts of our organization that DO use Hiera complaining loudly that they can't easily figure out which of a slurry of YAML files is changing some setting/variable to a value they're not expecting.
>
> Cheers,
> D
>

Pieter van de Bruggen

unread,
May 8, 2012, 6:01:04 PM5/8/12
to puppet...@googlegroups.com
As a courtesy update for our user community:

My earlier claims about the "lookup" function were exaggerated.  Instead, the "hiera", "hiera_array", and "hiera_hash" functions will all be available, provided by Hiera.

Also, as mentioned earlier, class parameters will be sourced from multiple locations, with the following priorities:
 - Hardcoded in the manifest (e.g. class { "foo": bar => "baz" })
 - Supplied by an ENC
 - Automatic namespaced data lookup in Hiera
 - Class-specified default value

Feel free to ask if you have any further questions.

Thanks,
  - Pieter
Reply all
Reply to author
Forward
0 new messages