PR hiera/137 - recursive hiera data item lookups

[Eric Sorenson]

Hi, there's been a lively discussion on an interesting pull request that came in:

https://github.com/puppetlabs/hiera/pull/137

In the interest of soliciting input from a wider audience I'd like to move the discussion on this request to the list.

Personally I have two issues, one major/semantic and one minor/syntactic.

The major problem is that this seems to reverse the entire goal of "separate data from code" by putting code back into data. In addition to the issues @ripienaar raised in [the previous recursion PR](http://projects.puppetlabs.com/issues/18030#note-7) around visibility/action-at-a-distance, this seems to make "more magic" and I feel pretty strongly that we need less magic even if it comes at the expense of more code -- in this case, a puppet manifest variable that does the stringified variable interpolation you're proposing in the data.

The minor problem is that, as Henrik notes, Hiera already has ONE syntax for variable interpolation that uses a different sigil to Puppet's. That's kind of weird, and again I'd like to see changes that bring the ecosystem closer together, not push it further apart.


Eric Sorenson - eric.s...@puppetlabs.com
#puppet irc: eric0

[Henrik Lindberg]

My thinking is...

Hiera allows interpolation of variables from scope. I would like to
restrict scope variable interpolation to only accept top-scope variables
set at the point where an ENC has done its job (i.e. facts + whatever
was set by the ENC). I see it as quite problematic that a lookup can
access whatever happens to be in context. It leads to hard to find
problems, leaking of information, and poor separation of concern. It
also leads to uncertainty after the fact where the interpolated value
came from, it makes it impossible to cache the result, and it is
impossible to do upfront validation.


[1,2,3].collect |$x| { lookup(mykey) }

where mykey is defined in hiera as '# %{x}' will produce

['# 1', '# 2', '# 3']

Surprise when user changes 'x' to 'y', or if there was an 'x' in the
outer scope that the user shadowed unknowingly of the interpolation of
'x' in mykey.

I am ok with interpolating global/topscope variables though since they
are bound at the start of the evaluation (and do not change).

I wonder how many that actually makes use of the feature to interpolate
scoped variables from the calling scope in their hiera-data?

W.r.t interpolation of other hiera keys I have (perhaps surprisingly)
fewer issues. Since it is known at the start of evluation what is bound
to the respective keys and the designer of the data layout is
responsible for both keys there is really no problem (i.e. it is the
same concern). One naturally has to check for circularity, but that is a
small implementation detail.

In the PR where the discussion started I made some proposals. I am
repeating them here with more motivation.

Since hiera uses %{} for interpolation of scope, and one can see this as
being the same as the ${} interpolation in Puppet DSL, it seems natural
to allow an expression. e.g. in Puppet DSL one would interpolate a
lookup by writing:

"the data is ${lookup(a_key)}"

Why not simply do the same in the hiera data? i.e.

'the data is %{lookup(a_key)}'

Hiera would not have to parse and handle all expressions, just the call
to the lookup function. (Later this can be expanded with use of the
future parser).

There are also other alternatives. When Puppet Templates are added, it
is possible to bind a template string containing puppet logic, and
evaluating that. The puppet templates can be parameterized (like
function) which enables them to be free of concern w.r.t the context
where they are evaluated.

The fact that hiera uses %{} instead of ${} is unfortunate, and probably
not something we can easily change without risking breaking peoples data
(containing ${} for other purposes).

More longer term, ARM-8 defines how data is expressed in Puppet DSL
language and the difference %{} ${} will be resolved (together with the
issues of supporting variable lookup as well as interpolation of other
keys. (The semantics for when things are bound are much tighter).

Finally, if you think it is a chore to write something like:

lookup(key)

and would like that to be baked into the language itself via syntax, I
may be persuaded ;-) even if lookup(key) is more descriptive than a
syntactic marker (we are starting to run out of them btw).

Looking forward to hearing everyone's opinion.

Regards
- henrik

[Erik Dalén]

A big +1 from me for having the feature. I've needed it a couple of times, and the easiest workaround when not having it is simply data duplication, which is really bad.

In some cases I've worked around that data duplication using more code, but usually a bunch of class inheritances in puppet, which IMO is even more magical and non obvious.

Also I think this is a much better thing than the Puppet scope variable interpolation, for example I want other systems than puppet to be able to query the hiera data which this allows but the %{} interpolation doesn't.

As to the exact syntax I'm quite indifferent, but I would like the hiera CLI or other things to be able to lookup values, which Henrik's %{lookup(key)} proposal might have a hard time doing.

--
Erik Dalén

[Ashley Penney]

On Wed, Jun 26, 2013 at 6:57 AM, Erik Dalén <erik.gus...@gmail.com> wrote:

A big +1 from me for having the feature. I've needed it a couple of times, and the easiest workaround when not having it is simply data duplication, which is really bad.

In some cases I've worked around that data duplication using more code, but usually a bunch of class inheritances in puppet, which IMO is even more magical and non obvious.

Also I think this is a much better thing than the Puppet scope variable interpolation, for example I want other systems than puppet to be able to query the hiera data which this allows but the %{} interpolation doesn't.

As to the exact syntax I'm quite indifferent, but I would like the hiera CLI or other things to be able to lookup values, which Henrik's %{lookup(key)} proposal might have a hard time doing.

This is closest to my opinion.  I am worried it's going to lead to more and more nested nightmares in production where you have to leap around all over the place to find out what ultimately feeds into Puppet (which is a probably today) but I've also done awful things in manifests to avoid duplication, and like Erik I've ended up with either a hodge-podge of functions or a bunch of class inheritance.

While I'm not a super fan of the idea I would prefer to keep this functionality in Hiera so that the manifests stay simple.

[Jeff McCune]

On Tue, Jun 25, 2013 at 6:06 PM, Eric Sorenson <eric.s...@puppetlabs.com> wrote:

Hi, there's been a lively discussion on an interesting pull request that came in:

https://github.com/puppetlabs/hiera/pull/137

In the interest of soliciting input from a wider audience I'd like to move the discussion on this request to the list.

Personally I have two issues, one major/semantic and one minor/syntactic.

The major problem is that this seems to reverse the entire goal of "separate data from code" by putting code back into data. In addition to the issues @ripienaar raised in [the previous recursion PR](http://projects.puppetlabs.com/issues/18030#note-7) around visibility/action-at-a-distance, this seems to make "more magic" and I feel pretty strongly that we need less magic even if it comes at the expense of more code -- in this case, a puppet manifest variable that does the stringified variable interpolation you're proposing in the data.

I don't think this major problem is a problem.  As you note we already support variable interpolation in Hiera.  It only took me a couple of hours of doing something "serious" with Hiera to land in a situation where I wanted to interpolate another hiera value.  I was genuinely surprised that I could interpolate scope variables but not other hiera variables.

The minor problem is that, as Henrik notes, Hiera already has ONE syntax for variable interpolation that uses a different sigil to Puppet's. That's kind of weird, and again I'd like to see changes that bring the ecosystem closer together, not push it further apart.

I don't particularly like new syntax, but I do think it's possible to make this work without introducing more magic or more syntax.

What if we just used the existing sigil and searched the hiera hierarchy before searching the Puppet scope?  Would this address the magic and syntax concerns?

-Jeff

[Jesse Hathaway]

W.r.t interpolation of other hiera keys I have (perhaps surprisingly)
fewer issues. Since it is known at the start of evluation what is bound
to the respective keys and the designer of the data layout is
responsible for both keys there is really no problem (i.e. it is the
same concern). One naturally has to check for circularity, but that is a
small implementation detail.

This has been our experience, that reasoning about hiera interpolation is actually easier than reasoning about scope lookups. Also, the addition of hiera interpolation support has allowed us to move more of our configuration data out of puppet node definitions and into hiera, which has allowed us to more clearly separate data from implementation as well as easily generate the yaml configuration files.

[Jesse Hathaway]

I don't particularly like new syntax, but I do think it's possible to make this work without introducing more magic or more syntax.

What if we just used the existing sigil and searched the hiera hierarchy before searching the Puppet scope?  Would this address the magic and syntax concerns?

We decided to introduce the new sigil rather than use the existing '%' because we found it confusing to not know how the interpolation was going to be performed. I think it is better to be explicit in the syntax and know how the the interpolation is going to be performed. I prefer the shorter ^{potto01_ip} syntax as it makes the yaml/json configuration easier to read and mirrors the scope interpolation syntax %{potto01_ip}.

[Trevor Vaughan]

I've been following this discussion because I have also quickly hit the point where I need to interpolate other hiera values.

Would it be possible to have % and ^ alias to a more descriptive nomenclature?

Something like % == scope and ^ == hiera? With the previous example: scope{potto01_ip} and hiera{potto01_ip}

Though I miss PERL (it's a joke people), I'm a big fan of readability and the constant sigil wrangling isn't very readable for people new to the language and makes maintenance from the unindoctrinated more difficult.

Thanks,

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To post to this group, send email to puppe...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-dev.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvau...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

[Dean Wilson]

On 26 June 2013 17:29, Trevor Vaughan <tvaughan@onyxpoint.com> wrote:

 

Something like % == scope and ^ == hiera? With the previous example: scope{potto01_ip} and hiera{potto01_ip}

And this fits nicely with the often proposed (and not yet accepted afaik) facts hash.

facts['osfamily']

 

  Dean

--
Dean Wilson               http://www.unixdaemon.net
Profanity is the one language all programmers understand
--- Anon

[Andy Parker]

On Tue, Jun 25, 2013 at 8:15 PM, Henrik Lindberg <henrik....@cloudsmith.com> wrote:

On 2013-26-06 3:06, Eric Sorenson wrote:

Hi, there's been a lively discussion on an interesting pull request that came in:

https://github.com/puppetlabs/hiera/pull/137

In the interest of soliciting input from a wider audience I'd like to move the discussion on this request to the list.

Personally I have two issues, one major/semantic and one minor/syntactic.

The major problem is that this seems to reverse the entire goal of "separate data from code" by putting code back into data. In addition to the issues @ripienaar raised in [the previous recursion PR](http://projects.puppetlabs.com/issues/18030#note-7) around visibility/action-at-a-distance, this seems to make "more magic" and I feel pretty strongly that we need less magic even if it comes at the expense of more code -- in this case, a puppet manifest variable that does the stringified variable interpolation you're proposing in the data.

The minor problem is that, as Henrik notes, Hiera already has ONE syntax for variable interpolation that uses a different sigil to Puppet's. That's kind of weird, and again I'd like to see changes that bring the ecosystem closer together, not push it further apart.

My thinking is...

Hiera allows interpolation of variables from scope. I would like to restrict scope variable interpolation to only accept top-scope variables set at the point where an ENC has done its job (i.e. facts + whatever was set by the ENC). I see it as quite problematic that a lookup can access whatever happens to be in context. It leads to hard to find problems, leaking of information, and poor separation of concern. It also leads to uncertainty after the fact where the interpolated value came from, it makes it impossible to cache the result, and it is impossible to do upfront validation.


[1,2,3].collect |$x| { lookup(mykey) }

where mykey is defined in hiera as '# %{x}' will produce

['# 1', '# 2', '# 3']

Surprise when user changes 'x' to 'y', or if there was an 'x' in the outer scope that the user shadowed unknowingly of the interpolation of 'x' in mykey.

I could see this working if you can also specify a collection of extra data. 

  lookup('mykey', { x => $my_x_value })

This is a very common way of controlling and making explicit what data is visible to the lookup function and gives a lot more control over it (and refactorings like the one you mention remain safe).

 

I am ok with interpolating global/topscope variables though since  they are bound at the start of the evaluation (and do not change).

I wonder how many that actually makes use of the feature to interpolate scoped variables from the calling scope in their hiera-data?

I know of one team where it being more that topscope caught them completely off guard (there was a parameter called $environment to a class and that completely changed the hiera lookups inside that class). I think more likely is that hiera data is depending on fully qualified variable names.

 

W.r.t interpolation of other hiera keys I have (perhaps surprisingly) fewer issues. Since it is known at the start of evluation what is bound to the respective keys and the designer of the data layout is responsible for both keys there is really no problem (i.e. it is the same concern). One naturally has to check for circularity, but that is a small implementation detail.

In the PR where the discussion started I made some proposals. I am repeating them here with more motivation.

Since hiera uses %{} for interpolation of scope, and one can see this as being the same as the ${} interpolation in Puppet DSL, it seems natural to allow an expression. e.g. in Puppet DSL one would interpolate a lookup by writing:

  "the data is ${lookup(a_key)}"

Why not simply do the same in the hiera data? i.e.

  'the data is %{lookup(a_key)}'

Expanding hiera interpolations to allow this would actually also allow another thing that is asked for: https://projects.puppetlabs.com/issues/16235

If the %{} syntax allowed more complex expressions then I could see this being used to specify a literal value in order to achieve "escaping". So in order to get %{SERVERNAME} in the value of a hiera lookup (which isn't possible right now) you could do something like:

  

  %{literal('%{SERVERNAME}')}

 

Hiera would not have to parse and handle all expressions, just the call to the lookup function. (Later this can be expanded with use of the future parser).

There are also other alternatives. When Puppet Templates are added, it is possible to bind a template string containing puppet logic, and evaluating that. The puppet templates can be parameterized (like function) which enables them to be free of concern w.r.t the context where they are evaluated.

The fact that hiera uses %{} instead of ${} is unfortunate, and probably not something we can easily change without risking breaking peoples data (containing ${} for other purposes).

More longer term, ARM-8 defines how data is expressed in Puppet DSL language and the difference %{} ${} will be resolved (together with the issues of supporting variable lookup as well as interpolation of other keys. (The semantics for when things are bound are much tighter).

Finally, if you think it is a chore to write something like:

lookup(key)

and would like that to be baked into the language itself via syntax, I may be persuaded ;-) even if lookup(key) is more descriptive than a syntactic marker (we are starting to run out of them btw).

Looking forward to hearing everyone's opinion.

Regards
- henrik

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.

To post to this group, send email to puppe...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-dev.
For more options, visit https://groups.google.com/groups/opt_out.

--

Andrew Parker

an...@puppetlabs.com

Freenode: zaphod42

Twitter: @aparker42

Software Developer

Join us at PuppetConf 2013, August 22-23 in San Francisco - http://bit.ly/pupconf13
Register now and take advantage of the Early Bird discount - save 25%!

[Henrik Lindberg]

On 2013-26-06 15:02, Ashley Penney wrote:
> On Wed, Jun 26, 2013 at 6:57 AM, Erik Dal�n <erik.gus...@gmail.com
> <mailto:erik.gus...@gmail.com>> wrote:
>
>
> A big +1 from me for having the feature.. I've needed it a couple of

> times, and the easiest workaround when not having it is simply data
> duplication, which is really bad.
> In some cases I've worked around that data duplication using more
> code, but usually a bunch of class inheritances in puppet, which IMO
> is even more magical and non obvious.
>
> Also I think this is a much better thing than the Puppet scope
> variable interpolation, for example I want other systems than puppet
> to be able to query the hiera data which this allows but the %{}
> interpolation doesn't.
>
> As to the exact syntax I'm quite indifferent, but I would like the
> hiera CLI or other things to be able to lookup values, which
> Henrik's %{lookup(key)} proposal might have a hard time doing.
>
>
> This is closest to my opinion. I am worried it's going to lead to more
> and more nested nightmares in production where you have to leap around
> all over the place to find out what ultimately feeds into Puppet (which
> is a probably today) but I've also done awful things in manifests to
> avoid duplication, and like Erik I've ended up with either a hodge-podge
> of functions or a bunch of class inheritance.
>

As we move forward, the idea is to be able to explain where everything
comes from (without having to evaluate) - this requires getting rid of
scoped variable interpolation (non top scope).

> While I'm not a super fan of the idea I would prefer to keep this
> functionality in Hiera so that the manifests stay simple.

- henrik

[Henrik Lindberg]

On 2013-26-06 15:29, Jeff McCune wrote:

> What if we just used the existing sigil and searched the hiera hierarchy
> before searching the Puppet scope? Would this address the magic and
> syntax concerns?
>

Bingo! if we also state that a name that starts with :: is a scope
variable and all others are hiera keys - then this explicitly blocks
relative variable references.

(Half victory as it is still possible to write ::foo::bar::x I assume)

- henrik

[Henrik Lindberg]

On 2013-26-06 18:35, Andy Parker wrote:
> On Tue, Jun 25, 2013 at 8:15 PM, Henrik Lindberg

> <henrik....@cloudsmith.com <mailto:henrik....@cloudsmith.com>>

> wrote:
> My thinking is...
>
> Hiera allows interpolation of variables from scope. I would like to
> restrict scope variable interpolation to only accept top-scope
> variables set at the point where an ENC has done its job (i.e. facts
> + whatever was set by the ENC). I see it as quite problematic that a
> lookup can access whatever happens to be in context. It leads to
> hard to find problems, leaking of information, and poor separation
> of concern. It also leads to uncertainty after the fact where the
> interpolated value came from, it makes it impossible to cache the
> result, and it is impossible to do upfront validation.
>
>
> [1,2,3].collect |$x| { lookup(mykey) }
>
> where mykey is defined in hiera as '# %{x}' will produce
>
> ['# 1', '# 2', '# 3']
>
> Surprise when user changes 'x' to 'y', or if there was an 'x' in the
> outer scope that the user shadowed unknowingly of the interpolation
> of 'x' in mykey.
>
>
> I could see this working if you can also specify a collection of extra
> data.
>
> lookup('mykey', { x => $my_x_value })
>
> This is a very common way of controlling and making explicit what data
> is visible to the lookup function and gives a lot more control over it
> (and refactorings like the one you mention remain safe).

Yes, that looks like the ability in Puppet Templates. There, the
template starts with a declaration of its parameters.

An idea is that the hiera strings are EPP :) either make hiera evaluate
them (magic), or lookup and evaluate.

inline_epptemplate(lookup(mykey), { x => $my_x })

The hiera string would then contain EPP e.g.

"<%- ($x = 'no x given') -%>
I was given an x with value <%= $x %>, that
when looked up became <%= lookup($x) %>
Bye"

>
> I am ok with interpolating global/topscope variables though since
> they are bound at the start of the evaluation (and do not change).
>
> I wonder how many that actually makes use of the feature to
> interpolate scoped variables from the calling scope in their hiera-data?
>
>
> I know of one team where it being more that topscope caught them
> completely off guard (there was a parameter called $environment to a
> class and that completely changed the hiera lookups inside that class).
> I think more likely is that hiera data is depending on fully qualified
> variable names.

I would like to not support them as they need to be evaluated
dynamically, but that is a different story. Absolute is fine since a
particular dynamic scope is then not needed and it is then not possible
to reference variables inside a define, or in a local scope in a lambda.

>
> W.r.t interpolation of other hiera keys I have (perhaps
> surprisingly) fewer issues. Since it is known at the start of
> evluation what is bound to the respective keys and the designer of
> the data layout is responsible for both keys there is really no
> problem (i.e. it is the same concern). One naturally has to check
> for circularity, but that is a small implementation detail.
>
> In the PR where the discussion started I made some proposals. I am
> repeating them here with more motivation.
>
> Since hiera uses %{} for interpolation of scope, and one can see
> this as being the same as the ${} interpolation in Puppet DSL, it
> seems natural to allow an expression. e.g. in Puppet DSL one would
> interpolate a lookup by writing:
>
> "the data is ${lookup(a_key)}"
>
> Why not simply do the same in the hiera data? i.e.
>
> 'the data is %{lookup(a_key)}'
>
>
> Expanding hiera interpolations to allow this would actually also allow
> another thing that is asked for:
> https://projects.puppetlabs.com/issues/16235
>
> If the %{} syntax allowed more complex expressions then I could see this
> being used to specify a literal value in order to achieve "escaping". So
> in order to get %{SERVERNAME} in the value of a hiera lookup (which
> isn't possible right now) you could do something like:
> %{literal('%{SERVERNAME}')}
>

hm, no escaping? I would expect \% to escape a % etc. like everywhere else.

- henrik

[Andy Parker]

       'the data is %{lookup(a_key)}'


Expanding hiera interpolations to allow this would actually also allow
another thing that is asked for:
https://projects.puppetlabs.com/issues/16235

If the %{} syntax allowed more complex expressions then I could see this
being used to specify a literal value in order to achieve "escaping". So
in order to get %{SERVERNAME} in the value of a hiera lookup (which
isn't possible right now) you could do something like:
   %{literal('%{SERVERNAME}')}

hm, no escaping? I would expect \% to escape a % etc. like everywhere else.

Not in hiera 1.0. If we started allowing \% to mean a literal % then existing hiera data we would hit the situation where a windows path in the data would suddenly stop interpolating. If we hold off for a hiera2 to do these kinds of changes, then I would rather have "\" escaping, which is much more understood and probably expected.

I was just pointing out that if we did my suggestion then we could have the literal % in hiera 1

 

- henrik

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.
To post to this group, send email to puppe...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-dev.
For more options, visit https://groups.google.com/groups/opt_out.

[Henrik Lindberg]

On 2013-26-06 22:37, Andy Parker wrote:

> Not in hiera 1.0.. If we started allowing \% to mean a literal % then

> existing hiera data we would hit the situation where a windows path in
> the data would suddenly stop interpolating.

Would that apply to \%{ as well? (i.e. is that a typical thing in a
windows path?

> If we hold off for a hiera2
> to do these kinds of changes,

I would like to hold off and do the changes in hiera2.

> then I would rather have "\" escaping,
> which is much more understood and probably expected.
>
> I was just pointing out that if we did my suggestion then we could have
> the literal % in hiera 1

ok, I did get that ;-)

- henrik

[Wil Cooley]

+1 for the feature; I like Jeff's idea about using the existing sigil, although I like Henrik's nested-function-like suggestion.

I think there is a particular case for this when it comes to lazy/busy sysadmins integrating 3rd party modules from Puppet Forge (and for encouraging writing internal modules with a general enough interface that they can easily be contributed):

Let's say I've got modules for configuring nss_pam_ldapd in EL6 and for configuring the native Solaris ldap_cachemgr. Both need, among other things, the list of LDAP server URIs. If I am not too lazy and thinking seriously about contributing them to PF, I'll write them to do a hiera lookup using the correct name for automatic parameter lookups (we're still on 2.7 & working towards 3 compatibility):

class nss_pam_ldapd::config {

  $server_uris = hiera('nss_pam_ldapd::config::server_uris', ...),

  ...

class ldap_cachemgr::conig {

  $server_uris = hiera('ldap_cachemgr::config::server_uris, ...),

  ...

and now I have made more work for myself because I have to duplicate (correctly) all of the info in hiera:

nss_pam_ldapd::config::server_uris:

  - ldaps://ldap1.example.com

  - ldaps://ldap2.example.com

 ...

ldap_cachemgr::config::server_uris:

  - ldaps://ldap1.example.com

  - ldaps://ldap2.example.com

 ...

whereas what I would like is:

ldap_login_server_uris:

 - ldaps://...

nss_pam_ldapd::config::server_uris: %{ldap_login_server_uris}

ldap_cachemgr::config::server_uris: %{ldap_login_server_uris}

but since I cannot do this and I am really lazy/busy, I'm going to be dumb and just lookup 'ldap_login_server_uris' and everyone else who might have benefited from the module can go stuff it:

class nss_pam_ldapd::config {

  $server_uris = hiera('ldap_login_server_uris')

  ...

And I am over-simplifying, because there are more like 8 parameters, almost all of which apply between both modules and all of which need to be duplicated.

Wil

[Henrik Lindberg]

On 2013-27-06 6:33, Wil Cooley wrote:
> +1 for the feature; I like Jeff's idea about using the existing sigil,
> although I like Henrik's nested-function-like suggestion.
>
> I think there is a particular case for this when it comes to lazy/busy
> sysadmins integrating 3rd party modules from Puppet Forge (and for
> encouraging writing internal modules with a general enough interface
> that they can easily be contributed):
>
> Let's say I've got modules for configuring nss_pam_ldapd in EL6 and for
> configuring the native Solaris ldap_cachemgr. Both need, among other
> things, the list of LDAP server URIs. If I am not too lazy and thinking
> seriously about contributing them to PF, I'll write them to do a hiera
> lookup using the correct name for automatic parameter lookups (we're
> still on 2.7 & working towards 3 compatibility):
>
> class nss_pam_ldapd::config {
> $server_uris = hiera('nss_pam_ldapd::config::server_uris', ...),
> ...
>
> class ldap_cachemgr::conig {
> $server_uris = hiera('ldap_cachemgr::config::server_uris, ...),
> ...
>
> and now I have made more work for myself because I have to duplicate
> (correctly) all of the info in hiera:
>
> nss_pam_ldapd::config::server_uris:

> - ldaps://ldap1.example.com <http://ldap1.example.com>
> - ldaps://ldap2.example.com <http://ldap2.example.com>
> ...
>
> ldap_cachemgr::config::server_uris:
> - ldaps://ldap1..example.com <http://ldap1.example.com>
> - ldaps://ldap2.example.com <http://ldap2.example.com>

> ...
>
> whereas what I would like is:
>
> ldap_login_server_uris:
> - ldaps://...
>
> nss_pam_ldapd::config::server_uris: %{ldap_login_server_uris}
> ldap_cachemgr::config::server_uris: %{ldap_login_server_uris}
>
> but since I cannot do this and I am really lazy/busy, I'm going to be
> dumb and just lookup 'ldap_login_server_uris' and everyone else who
> might have benefited from the module can go stuff it:
>
> class nss_pam_ldapd::config {
> $server_uris = hiera('ldap_login_server_uris')
> ...
>
> And I am over-simplifying, because there are more like 8 parameters,
> almost all of which apply between both modules and all of which need to
> be duplicated.
>

Then you will probably like the proposal in ARM-8 that allows mapping
the parameters of one type onto another. This was added to support
exactly the case you describe; i.e. binding the parameters for some main
"entry point" class and the reusing all/subset of them in other classes
(possible with mapping of names).

It is also much clearer that the intent is to use "the same set of
parameters", then cherry picking them individually using interpolation.

- henrik

[Ashley Penney]

On Thu, Jun 27, 2013 at 10:43 AM, Henrik Lindberg <henrik....@cloudsmith.com> wrote:

Then you will probably like the proposal in ARM-8 that allows mapping the parameters of one type onto another. This was added to support exactly the case you describe; i.e. binding the parameters for some main "entry point" class and the reusing all/subset of them in other classes (possible with mapping of names).

It is also much clearer that the intent is to use "the same set of parameters", then cherry picking them individually using interpolation.

I hate to ask this but could you give an example here?  I just made an attempt to read ARM-8 and maybe I'm just not that smart but I found it almost impossible to understand.  I couldn't find anything that looked like the reusing of a subnet in other classes.  I'm working my way through the examples document which is (so far) the only thing I've properly understood.  The ARM-8 itself is off talking about lambas and multi binding and consumers and so forth and I just stuggled to get any kind of overview/feeling for what's going on.

[Henrik Lindberg]

On 2013-27-06 18:28, Ashley Penney wrote:
> I hate to ask this but could you give an example here? I just made an
> attempt to read ARM-8 and maybe I'm just not that smart but I found it
> almost impossible to understand. I couldn't find anything that looked
> like the reusing of a subnet in other classes. I'm working my way
> through the examples document which is (so far) the only thing I've
> properly understood. The ARM-8 itself is off talking about lambas and
> multi binding and consumers and so forth and I just stuggled to get any

> kind of overview/feeling for what's going on..
>

ARM-8 is very technical and detailed; also unfinished...

The syntax for binding parameters in the proposed Puppet DSL syntax
is:

bind parameters myclass to { x => 'value', y => 'value' }

which sets the parameters x and y of class 'myclass'.
If you want to bind the parameters of myotherclass to the same values as
those for myclass you write

bind parameters myotherlcass to myclass

that is, if they both accept x and y as parameters. (If they don't there
is a way to bind a mapping of names).

The extension of this is that it will also be possible to define
structured typed/named data - in some cases it makes more sense to
define such a "data record" and use that for several classes.

And to follow on from there we have cross-node functionality where such
structured data may come from a different node.

The core idea in ARM.8 is that there should be one binding/injection
mechanism uniting the various ways data gets concretely bound to
parameters, etc.

We are currently working on a subset of ARM-8 for data in modules where
hiera is given a small facelift at the same time.

Hope that helps explain a bit about ARM-8 and the specific thing;
"reusing parameter bindings" that I pointed out earlier.

Regards
- henrik

[Jesse Hathaway]

If there are concerns with adding a new sigil '^', which is not obviously intuitive, then I would propose keeping only one sigil for interpolation '%' but being explicit on the type of lookup performed.

Examples:

1.  Scope lookup of `potto01_ip`:
   
1. webserver_ip: "%{scope("potto01_ip")}"
2. Hiera lookup of `potto01_ip`
1. webserver_ip: "%{hiera("potto01_ip")}"

This obviously breaks current scope lookups and would need to be deferred to a 2.0 version, but I think it is better to break compatibility rather than cluttering up the language in inconsistent ways.

[Eric Sorenson]

Hi Jesse -- I think this is getting pretty close, but you're right this would not work without a major version bump. Some comments/questions from the thread--

- The 'creeping perlism' of more sigils seems to be pretty unpopular, instead most people favour concrete syntax like your example below

- Erik had concerns about using hiera data outside of Puppet - is this a common use case? we're looking at moving hiera closer to puppet (i.e. for data in modules) and if we can move to a single codebase there it simplifies things a fair bit (i.e. relying on future parser) at the expense of hiera-without-puppet. Is it using hiera as a library that's important or specifically the command-line program?

- Exploring that line of thought further, what if the right-hand-side value could contain Puppet DSL expressions? Obviously this opens the door to some crazy stuff -- I already have concern with putting code back into data and this goes further down the path, but y'all have pretty convincing use cases for increasing sophistication in hiera, so it seems worth thinking about taking it to the logical extension.

- Reading through Wil's example is really interesting because it also touches on module reusability and module APIs. Are others on the thread in a similar situation, where you need to compose/abstract common config parameters to reduce duplication across your site data hierarchy? (And Wil, is there a reason you can't just do that 'ldap_login_server_uris' mapping at class declaration time rather than putting the lookups into the class definitions themselves?)

eric0

[Erik Dalén]

On 3 July 2013 00:27, Eric Sorenson <eric.s...@puppetlabs.com> wrote:

Hi Jesse -- I think this is getting pretty close, but you're right this would not work without a major version bump. Some comments/questions from the thread--

- The 'creeping perlism' of more sigils seems to be pretty unpopular, instead most people favour concrete syntax like your example below

- Erik had concerns about using hiera data outside of Puppet - is this a common use case? we're looking at moving hiera closer to puppet (i.e. for data in modules) and if we can move to a single codebase there it simplifies things a fair bit (i.e. relying on future parser) at the expense of hiera-without-puppet. Is it using hiera as a library that's important or specifically the command-line program?

It is mostly to be able to use it as a library.

We use hiera for node classification in some installations and want to be able to lookup what classes a node should get (and also see the value of any other hiera key).

At the moment we use the hiera node terminus for that: https://github.com/puppetlabs/puppet/pull/1414

And for looking up values we mainly use a little rest api I wrote: https://github.com/dalen/hieralookup

As long as there is a api where you can input a hiera key and a scope to get back the value I can modify it to fit that :)

 

- Exploring that line of thought further, what if the right-hand-side value could contain Puppet DSL expressions? Obviously this opens the door to some crazy stuff -- I already have concern with putting code back into data and this goes further down the path, but y'all have pretty convincing use cases for increasing sophistication in hiera, so it seems worth thinking about taking it to the logical extension.

I've kind of already done that with the puppetdbquery hiera backend. Could be nice to have that done in a more general way and not have a specific hiera backend for every type of lookup function you could want there.

I agree that it feels odd to put code inside data though. But it is a very effective way to reuse forge modules without having to modify them and not having to hardcode or duplicate a lot of data that can be looked up using some function.

--
Erik Dalén

[Henrik Lindberg]

On 2013-03-07 10:48, Erik Dal�n wrote:

> I agree that it feels odd to put code inside data though. But it is a
> very effective way to reuse forge modules without having to modify them
> and not having to hardcode or duplicate a lot of data that can be looked
> up using some function.
>

To me the distinction "data vs. code" is not as interesting as that of
"separation of concerns" - there are many sane use-cases for logic in
"data". It is more important to think about the various concerns; how
can a module be reused, how is it configured, how can data be reused
(not having to specify the same item over and over), how can a module
adapt (or be told to adapt) its behavior depending on the the presence
of other modules, etc.

Regards
- henrik

[Trevor Vaughan]

+1

Henrik has quite accurately described the use case that I'm looking for.

Trevor

On Wed, Jul 3, 2013 at 11:31 AM, Henrik Lindberg <henrik....@cloudsmith.com> wrote:

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.

To post to this group, send email to puppe...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-dev.
For more options, visit https://groups.google.com/groups/opt_out.

[Jesse Hathaway]

On Tuesday, July 2, 2013 5:27:56 PM UTC-5, Eric Sorenson wrote:

Hi Jesse -- I think this is getting pretty close, but you're right this would not work without a major version bump. Some comments/questions from the thread--

- The 'creeping perlism' of more sigils seems to be pretty unpopular, instead most people favour concrete syntax like your example below

I agree that using a single sigil for interpolation, '%', and using function notation is more explicit, I will update the pull request with that change. 

- Erik had concerns about using hiera data outside of Puppet - is this a common use case? we're looking at moving hiera closer to puppet (i.e. for data in modules) and if we can move to a single codebase there it simplifies things a fair bit (i.e. relying on future parser) at the expense of hiera-without-puppet. Is it using hiera as a library that's important or specifically the command-line program?

We have only recently moved a large percentage of our data into hiera, but we are already thinking of use cases where querying hiera outside of puppet would be useful. As an example since we use supply_drop to puppet our servers it would be useful to query our hiera configuration to create our capistrano roles. We would love to use Hiera as a general configuration store, which could then be queried from multiple sources. Also, given that hiera fills a niche that is under served, I think it would be great if other configuration tools picked it up, e.g. chef, cfengine, etc.

 

- Exploring that line of thought further, what if the right-hand-side value could contain Puppet DSL expressions? Obviously this opens the door to some crazy stuff -- I already have concern with putting code back into data and this goes further down the path, but y'all have pretty convincing use cases for increasing sophistication in hiera, so it seems worth thinking about taking it to the logical extension.

I think there would be possible value in this, we have already hand instances where it would help to interpolate values in hiera that are keys of a hash, i.e. %{hiera(ips["potto01_ip"])}

 

- Reading through Wil's example is really interesting because it also touches on module reusability and module APIs. Are others on the thread in a similar situation, where you need to compose/abstract common config parameters to reduce duplication across your site data hierarchy? (And Wil, is there a reason you can't just do that 'ldap_login_server_uris' mapping at class declaration time rather than putting the lookups into the class definitions themselves?)

Here I think Henrik's description of the separation of concerns provides a persuasive argument for having logic in data. Specifically having hiera interpolation support has helped us to move more of our configuration out of puppet and into hiera.

[Jesse Hathaway]

On Wednesday, July 3, 2013 5:06:36 PM UTC-5, Jesse Hathaway wrote:

On Tuesday, July 2, 2013 5:27:56 PM UTC-5, Eric Sorenson wrote:

Hi Jesse -- I think this is getting pretty close, but you're right this would not work without a major version bump. Some comments/questions from the thread--

- The 'creeping perlism' of more sigils seems to be pretty unpopular, instead most people favour concrete syntax like your example below

I agree that using a single sigil for interpolation, '%', and using function notation is more explicit, I will update the pull request with that change. 

I have updated the pull request with the single sigil syntax:

https://github.com/puppetlabs/hiera/pull/137#commits-pushed-b39db81

[Wil Cooley]

On Tue, Jul 2, 2013 at 3:27 PM, Eric Sorenson
<eric.s...@puppetlabs.com> wrote:

> - Reading through Wil's example is really interesting because it also
> touches on module reusability and module APIs. Are others on the thread in a
> similar situation, where you need to compose/abstract common config
> parameters to reduce duplication across your site data hierarchy? (And Wil,
> is there a reason you can't just do that 'ldap_login_server_uris' mapping at
> class declaration time rather than putting the lookups into the class
> definitions themselves?)

Let me make sure I understand what you are suggesting. When I declare
the class, I expressly use the hiera lookup to provide the appropriate
parameter for the class:

modules/nss_pam_ldapd/manifests/config.pp:
class nss_pam_ldapd::config {
$server_uris = hiera('nss_pam_ldapd::config::server_uris', ...), #
maybe kill explicit hiera() here for simplicity
...
}

manifests/site.pp:
node foo {
include nss_pam_ldapd
class { 'nss_pam_ldapd::config':
server_uris => hiera('ldap_login_server_uris')
}
}

Sure, you can do that, but declaring the class is a good deal messier
-- with the hiera-internal reference it would be just "include
nss_pam_ldapd". From what I understand, practices are evolving towards
simplifying node declaration by putting more of the complexity in
hiera. If so, then it seems better to appeal to people's
self-interest/laziness by making it easy to do it that way.

Wil

[Nick Fagerlund]

Hi all! Docs team here.

It looks like recursive Hiera data lookups were merged via pull request 152. However, we have no preliminary documentation for how this works and what the limitations are -- across the multiple discussion threads and multiple outdated implementations, it's looking like it would take a lot of detective work to figure out the final behavior.

In order to allow release of 1.3.0, the docs team needs a brief write-up of the final behavior and its implications. Jesse Hathaway, Adrien, and Henrik -- would one or several of you be able to write something up for us? Or can you give a pointer to something final and complete that already exists within the various threads?

[Henrik Lindberg]

The support is based on calling lookup functions in the interpolation.
Two such functions are supported:

* hiera('hiera_key')
* scope('variable_name')

They are used in hiera interpolation like this:

mykey: "I include %{hiera('otherkey')}"
otherkey: "the other key"

When looking up the key 'mykey', the result is "I include the other key".

The key argument in the function must be surrounded by ' or ".

The scope function is basically the same as the regular variable
interpolation, but it performs an extra lookup in the given 'extra data'
that can be supplied to hiera when doing lookup if the lookup in
the regular scope did not produce a value. (Someone else may have a
better explanation / examples of usage).

The implementation checks for endless recursion and issues an error if
an attempt is made to interpolate an already interpolated key.

Note that it is sensitive wrt. whitespace, users should not have any
whitepsace in the interpolation function. (e.g. this will not work
"I include %{ hiera( 'otherkey' ) }")

Hope that answers the questions.

Regards
- henrik