Announce: Hiera 2.0.0 available

[Kylo Ginsberg]

Hiera 2.0.0 is a new major version that includes several new features that expand Hiera's abilities. It also includes one breaking change so please read the release notes linked below.

New features include the ability to pass options to the deep merge gem, new sub-keys to allow lookups to index into data structures, and an `alias()` function to make a key an alias for another key. It also includes a fix for interpolation recursion loops and several other bugs. 

Release notes are available here: http://docs.puppetlabs.com/hiera/latest/release_notes.html

To see a complete list of issues fixed in this release: https://tickets.puppetlabs.com/issues/?filter=13825

We're tracking bugs people find in this release with the "Affected Version" field set to "HI 2.0.0": https://tickets.puppetlabs.com/issues/?filter=13824

--

Kylo Ginsberg | ky...@puppetlabs.com | irc: kylo | twitter: @kylog

PuppetConf 2015 is coming to Portland, Oregon! Join us October 5-9.

Register now to take advantage of the Early Adopter discount —save $349!

[Johan De Wit]

On 26/03/15 10:36, Stig Sandbeck Mathisen wrote:
> The search path to hiera.yaml starts with
> "/etc/puppetlabs/code/hiera.yaml", should this be
> "/etc/puppet/code/hiera.yaml"?

As of puppet 4 (and the AllInOne packaging), the config files are
changed from /etc/puppet/* to /etc/puppetlabs/*.

(http://docs.puppetlabs.com/puppet/pre4.0/reference/)

hth

grts




--
Johan De Wit

Open Source Consultant

Red Hat Certified Engineer (805008667232363)
Puppet Certified Professional 2013/2014 (PCP0000006)
Puppet Certified Intstructor
blog : http://johan.koewacht.net/
_________________________________________________________

Open-Future Phone +32 (0)2/255 70 70
Zavelstraat 72 Fax +32 (0)2/255 70 71
3071 KORTENBERG Mobile +32 (0)474/42 40 73
BELGIUM http://www.open-future.be
_________________________________________________________


Next Events:
Puppet Fundamentals | http://www.open-future.be/puppet-fundamentals-training-16th-till-18th-march
Puppet Architect | http://www.open-future.be/puppet-architect-training-19th-till-20th-march
Puppet Practitioner | http://www.open-future.be/puppet-practitioner-training-14th-till-16th-april
Linux Training | http://www.open-future.be/linux-training-20th-till-24th-april
Bacula Administrator 1 | http://www.open-future.be/bacula-administrator-i-training-28th-till-30th-april
Zabbix Certified Specialist | http://www.open-future.be/zabbix-certified-specialist-training-4th-till-6th-may
Zabbix Certified Professional | http://www.open-future.be/zabbix-certified-professional-training-7th-till-8th-may
Subscribe to our newsletter | http://eepurl.com/BUG8H

[John Bollinger]

On Wednesday, March 25, 2015 at 7:46:38 PM UTC-5, Kylo Ginsberg wrote:

Hiera 2.0.0 is a new major version that includes several new features that expand Hiera's abilities. It also includes one breaking change so please read the release notes linked below.

New features include the ability to pass options to the deep merge gem, new sub-keys to allow lookups to index into data structures, and an `alias()` function to make a key an alias for another key. It also includes a fix for interpolation recursion loops and several other bugs. 

Release notes are available here: http://docs.puppetlabs.com/hiera/latest/release_notes.html

To see a complete list of issues fixed in this release: https://tickets.puppetlabs.com/issues/?filter=13825

We're tracking bugs people find in this release with the "Affected Version" field set to "HI 2.0.0": https://tickets.puppetlabs.com/issues/?filter=13824

I'm pleased to see Hiera moving forward, and the changes almost all seem positive.

Would someone please explain a little more about HI-14, though, and especially about how the change implemented to fix that issue actually addresses it at all?  The issue description is about traversing structured data in interpolation tokens, with an especial focus on structured fact values, but the "fix" seems to have been to modify the interpretation of keys.  Either I'm misunderstanding something, or that doesn't address the issue at all.

As a secondary matter, the new behavior causes some valid keys to be interpreted differently in Hiera 2 than in Hiera 1, therefore it constitutes a breaking change.  If retained, it should be documented as such.  (It appears that a regression has already been reported against HI-14, in fact.)

Personally, though, I don't see much value in it.  Supposing that I'm reading everything correctly, I would like to see the change rolled back on the basis that it does not address the issue, and that the minor convenience it provides is not sufficient justification for a breaking change.  If this behavior change is in fact desired, then at minimum a separate ticket should be created to describe it, and HI-14 reopened for a solution that actually addresses it.

[Trevor Vaughan]

Is there going to be a native package drop of Hiera pre-AIO?

As John pointed out in another thread, it would be ideal to be able to update individual components without having to drop a monolithic package every time.

Thanks,

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CALsUZFFun_HsmnEOUF9%2BhdWznzfO4RPp%3DoJvxNp4RGwhRXrrYQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvau...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

[bert hajee]

Looking at the direction hiera is going, people might be interested in a yaml replacement for hiera. I've written a proof of concept (actually it's already a bit more ;-)). I'm interested in any feedback. You can find it here https://forge.puppetlabs.com/hajee/connect. A writeup of what you can do with it, you can find here: https://github.com/hajee/connect/blob/master/doc/nutshell.md

regards,

Bert

Op donderdag 26 maart 2015 01:46:38 UTC+1 schreef Kylo Ginsberg:

[Kylo Ginsberg]

On Thu, Mar 26, 2015 at 6:45 AM, Trevor Vaughan <tvau...@onyxpoint.com> wrote:

Is there going to be a native package drop of Hiera pre-AIO?

Not in the current plans.

Kylo

 

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoV%2BHX1EXMgr2jSzjV98MnoOAin3ZkZxF_UAqTrnMBUPbA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

[John Bollinger]

On Thursday, March 26, 2015 at 8:24:30 AM UTC-5, John Bollinger wrote:

Would someone please explain a little more about HI-14, though, and especially about how the change implemented to fix that issue actually addresses it at all?  The issue description is about traversing structured data in interpolation tokens, with an especial focus on structured fact values, but the "fix" seems to have been to modify the interpretation of keys.  Either I'm misunderstanding something, or that doesn't address the issue at all.

Kylo? Anybody? Bueller?


John

[Thomas Hallgren]

A key can be qualified and consist of several segments. Let's assume that the first segment is the key of some structured data. The remaining segments of the key will then navigate in that data. Example:

$ hiera user
{"name"=>"kim", "home"=>"/home/kim"}

$ hiera user.name
kim

So the key user.name, when used as an interpolation token, will lookup 'kim'.

Also, see example under the "Qualified Key Lookup" here: https://github.com/puppetlabs/hiera/blob/a008efc22f5907228d5c6f9952914b2a97bb8bf6/README.md

- thomas

[John Bollinger]

On Wednesday, April 1, 2015 at 9:01:52 AM UTC-5, Thomas Hallgren wrote:

On 2015-03-31 14:57, John Bollinger wrote:

On Thursday, March 26, 2015 at 8:24:30 AM UTC-5, John Bollinger wrote:

Would someone please explain a little more about HI-14, though, and especially about how the change implemented to fix that issue actually addresses it at all?  The issue description is about traversing structured data in interpolation tokens, with an especial focus on structured fact values, but the "fix" seems to have been to modify the interpretation of keys.  Either I'm misunderstanding something, or that doesn't address the issue at all.

Kylo? Anybody? Bueller?

A key can be qualified and consist of several segments. Let's assume that the first segment is the key of some structured data. The remaining segments of the key will then navigate in that data. Example:

$ hiera user
{"name"=>"kim", "home"=>"/home/kim"}

$ hiera user.name
kim

Yes, I follow that this is the new behavior that HI-14 implements.  My point is that it is not the feature that HI-14's description requests, nor that either of the issues marked as dupes of HI-14 requests.  As such,

1. HI-14 appears to have been closed inappropriately.  The issue it describes (and that at least two dupes of it *also* requested) has not been fixed.
2. To the best of my knowledge, the change to key interpretation that was actually implemented does not correspond to any accepted issue.
   

I have concerns here both about the process and about the result.  With respect to process, we appear to have had a behavior change implemented that was never accepted -- a breaking change, no less.  The change that was accepted for implementation is something different and as-yet unimplemented.  With respect to result, we have a change that breaks existing sites without warning, acknowledgement, or even any particularly good justification.  I cry "foul"!


John

[Thomas Hallgren]

John,

We're seem to be talking past each other here. The description in HI-14 mentions facts specifically and that too is available. I can declare this for instance:

days_up: "%{scope('system_uptime.days')}"

or simply:

days_up: "%{system_uptime.days}"

and it returns the 'days' item from the structured fact 'system_uptime'. IMO, that's "traversing structured data in interpolation tokens, with especial focus on structured fact values" but it's obviously not what you're looking for.

Can you be more specific in what way the issue description doesn't match the implementation?

Thanks,
- thomas

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/fc95b2f5-e5e0-402a-8c0d-1e9fcf0a82ab%40googlegroups.com.

[Trevor Vaughan]

I think perhaps if the issue could be pointed out that provides for *key* interpolation on dotted values without interpolation notation, that might clear things up.

Thomas' example does provide HI-14 as I read it *but* there may have been an unintended side effect that the following is a disaster for exsiting sites:

my.days_up: "%{system_uptime.days}"

my.days_up will try to traverse into the data structure of 'my' and will break. This is bad. Nothing should be interpolated that is not surrounded in "%{}"

John, did I explain this correctly? This appears to be the nature of the comment filed in HI-14 as well.

Thanks,

Trevor

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/551C6E8C.2080907%40puppetlabs.com.

For more options, visit https://groups.google.com/d/optout.

[Thomas Hallgren]

On 2015-04-02 01:29, Trevor Vaughan wrote:
> I think perhaps if the issue could be pointed out that provides for
> *key* interpolation on dotted values without interpolation notation,
> that might clear things up.
>
> Thomas' example does provide HI-14 as I read it *but* there may have
> been an unintended side effect that the following is a disaster for
> exsiting sites:
>
> my.days_up: "%{system_uptime.days}"
>
> my.days_up will try to traverse into the data structure of 'my' and
> will break. This is bad. Nothing should be interpolated that is not
> surrounded in "%{}"
>
>

I think that whether declaring a value with a dotted key is considered
legal or not is a separate discussion. If that's the case, then we can't
use dots to traverse. Not when looking up and not in interpolated
expressions (since that's an implicit lookup).

- thomas

[Trevor Vaughan]

Agreed, but as John points out, it *was* legal (intentionally or not) so this is a breaking change for some people.

Being a breaking change, I would vote for backing it out until something can be decided that will not break existing systems.

Is there any value that current is not valid in a key that would work instead of a dot?

Thanks,

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/551C860E.7030701%40puppetlabs.com.

For more options, visit https://groups.google.com/d/optout.

[Thomas Hallgren]

On 2015-04-02 02:12, Trevor Vaughan wrote:

Agreed, but as John points out, it *was* legal (intentionally or not) so this is a breaking change for some people.

Being a breaking change, I would vote for backing it out until something can be decided that will not break existing systems.

Is there any value that current is not valid in a key that would work instead of a dot?

Keep in mind though, that this is not a new Hiera 1.3.x. It's not even 1.x. It's 2.0.0. If there is any point in time when we want to make a statement on what characters that are legal when declaring a key in Hiera, that time is now.

- thomas

Thanks,

Trevor

On Wed, Apr 1, 2015 at 7:58 PM, Thomas Hallgren <thomas....@puppetlabs.com> wrote:

On 2015-04-02 01:29, Trevor Vaughan wrote:

I think perhaps if the issue could be pointed out that provides for *key* interpolation on dotted values without interpolation notation, that might clear things up.

Thomas' example does provide HI-14 as I read it *but* there may have been an unintended side effect that the following is a disaster for exsiting sites:

my.days_up: "%{system_uptime.days}"

my.days_up will try to traverse into the data structure of 'my' and will break. This is bad. Nothing should be interpolated that is not surrounded in "%{}"

I think that whether declaring a value with a dotted key is considered legal or not is a separate discussion. If that's the case, then we can't use dots to traverse. Not when looking up and not in interpolated expressions (since that's an implicit lookup).

- thomas

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/551C860E.7030701%40puppetlabs.com.

For more options, visit https://groups.google.com/d/optout.

--

Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvau...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoVtQhG34cFk6BADSnQAubrWhLx6ARbmKEmGDJ9ZMTwBtw%40mail.gmail.com.

[Trevor Vaughan]

Heh, for some reason my brain skipped that.

Ok, that makes sense I suppose as long as people are made aware that this is a breaking change.

Trevor

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/551C8AB3.4080000%40puppetlabs.com.

For more options, visit https://groups.google.com/d/optout.

[John Bollinger]

On Wednesday, April 1, 2015 at 5:17:54 PM UTC-5, Thomas Hallgren wrote:

John,

We're seem to be talking past each other here. The description in HI-14 mentions facts specifically and that too is available. I can declare this for instance:

days_up: "%{scope('system_uptime.days')}"

or simply:

days_up: "%{system_uptime.days}"

and it returns the 'days' item from the structured fact 'system_uptime'. IMO, that's "traversing structured data in interpolation tokens, with especial focus on structured fact values" but it's obviously not what you're looking for.

No, that's exactly what I'm looking for.  I am pleased and somewhat mollified to find that it's actually included.  However, it seems not to be described anywhere in the docs -- neither in the release notes, nor in the section of the main (2.0.0) docs that deals with interpolation.

The flip side, however, is that the altered behavior of key interpretation, although attributed to HI-14 in the release notes and in the git log, is no part of the feature actually requested in HI-14, or in any of its duplicates.  HI-14 has apparently been hijacked to implement an altogether different behavior than was requested and which acceptance of that ticket approved.  All of the documentation around this ticket, other than the ticket itself, focuses on the key interpretation side.  The bulk of the new tests focus on the key interpretation side.  Even the initial answer to my question here focused on key interpretation. That HI-14's actual requested behavior was implemented, too, as an apparent afterthought, is little consolation.

With the key interpretation difference being a breaking change, the change management process has failed if that behavior change indeed was nowhere requested or approved.  Therefore, as a matter of a disciplined approach to process, I would like to see the key interpretation part of the change backed out.  Realistically, I have no expectation that that will actually happen.  What needs to happen, though, is that the release notes be updated to document the key interpretation change as a breaking change, and that at least the main docs document the extended interpolation behavior (at all).  Were I in PL's management, I would have additional items for that list.


John

[Thomas Hallgren]

John,

The HI-14's requested behavior was not implemented as an afterthought. On the contrary. Since all facts have been available in the top scope since Puppet 3.5, there was simply no need for a specific handling of facts in Hiera and the approach taken was very deliberate. We knew that whatever we would put in place would be applicable for all variables in the scope. We also knew that since scope variables are not the only structured data out there, it would be somewhat congested to create solution that would work only with scope variables but not with everything else. I think that would have been a really bad decision and as far as I can remember it was never brought up to discussion.

The current solution works with:

  %{scope('some.dotted.key')}

and

  %{hiera('some.dotted.key')}

and

  Hiera.lookup('some.dotted.key')

The interpolated hiera lookup is implicitly doing the last one so they need to come as a package. The way I see it, this is all about key interpretation (interpolation is a lookup) which might explain my line of thought when first answering your question.

All current behavior is retained with one exception; sites that make use of dotted keys in their own Hiera data today will break when they start using Hiera 2.0. I totally agree that this must be explicitly stated in the documentation since we apparently have had no restriction on this in the past. An example retrieving data from inside a structured fact would be a good addition too.

- thomas

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/673143a1-1d76-448d-a402-a518135748c0%40googlegroups.com.

[John Bollinger]

On Thursday, April 2, 2015 at 10:33:02 AM UTC-5, Thomas Hallgren wrote:

John,

The HI-14's requested behavior was not implemented as an afterthought. On the contrary. Since all facts have been available in the top scope since Puppet 3.5, there was simply no need for a specific handling of facts in Hiera and the approach taken was very deliberate. We knew that whatever we would put in place would be applicable for all variables in the scope. We also knew that since scope variables are not the only structured data out there, it would be somewhat congested to create solution that would work only with scope variables but not with everything else. I think that would have been a really bad decision and as far as I can remember it was never brought up to discussion.

Am I misspelling of the word "interpolation", or something?  Is it too similar to the word "interpretation" that I have been using to talk about how hiera translates keys into corresponding values?  I'm getting frustrated here.

I have not suggested that the new feature should have been restricted to facts specifically or even to variables generally.  Although HI-14 uses facts as its examples, the text is not so restrictive, and I agree that such a solution would have been awkward.  No, I am saying that the change should have been restricted to the context of interpolation tokens, on account of that being the only context contemplated by the feature request.  Protestations notwithstanding, that aspect of the change has clearly gotten little love, even though it is both the actual intended target and by far the most important part.  It provides for doing things with Hiera that were difficult or impossible to do before.

The change to key interpretation, on the other hand, is largely a solution without a problem, and, moreover, without any request or change approval that I have seen.  I probably wouldn't be so on about it if it did not constitute a breaking change, but I find it pretty concerning that a breaking change is introduced without request, approval, or any particular justification other than possibly that it was more convenient to make the change than to avoid making it.  Mustering up some charity, I am prepared to believe that the changes were implemented based mostly on the ticket title, without an effective analysis of the actual problem as set forth in the description text.  That's only a possible explanation, though, not a justification.

The current solution works with:

  %{scope('some.dotted.key')}

Yes.
 

and

  %{hiera('some.dotted.key')}

Why?

 

and

  Hiera.lookup('some.dotted.key')

Why?

 

The interpolated hiera lookup is implicitly doing the last one so they need to come as a package.

Fine, the last two need to come as a package, but why did either one need to come at all?  What would have been the problem with, say,

%{hiera('some').dotted.key}

or

%{hiera('some')[dotted][key]}

or any of probably a dozen other alternatives?  Surely wherever Hiera.lookup('some.dotted.key') now returns an object, Hiera.lookup('some')['dotted']['key'] always did return that same object.

My apologies.  Please feel no obligation to respond to those implementation-directed comments.  With the interpolation angle actually covered (its total lack of documentation notwithstanding), my concern is now much more with how and why the change ended up exceeding its requested scope than with the details of change itself.  The merits of the change to key semantics aside, that change should not have been implemented under the auspices of HI-14 as it is written.

In the end, of course, PL's change management processes are none of my responsibility.  I am speaking from the perspective of a concerned bystander with a vested interest in Puppet and PuppetLabs remaining strong and effective, and playing fast and loose with the change management process is not supportive of continuing effectiveness.  Frankly, at this point I am even more more concerned that no one seems to want to acknowledge that there even was a problem with how this went down than I am with the fact that it did go down.  But if I am whistling in the wind, then so be it.


John

[Thomas Hallgren]

Agreed. We could have done:

 %{scope('some.dotted.key')}

and then

 %{hiera('some')[dotted][key]}

or limited the use of dotted keys to scope interpolation. Such a solution would have satisfied the requirements expressed in the description of HI-14 (albeit, as you mention, the title has a more general wording). If our objective was to deliver a Hiera 1.3.5 or even a Hiera 1.4.0, that's probably what we would have done.

With Hiera 2.0.0 we decided to go for a simple syntax that was applicable everywhere. It simple, consistent and will avoid future questions like; why is it only possible to traverse scope data and not other structured data? Why is a key interpreted different in this interpolation expression than in that? Why can't I use a dotted key in the lookup when I can use it in interpolation? Etc.

The change will only affect sites where dotted keys are used in Hiera data negatively and it's not the only breaking change in Hiera 2.0.0.

A side note:
Hiera.lookup('some')['dotted']['key'] will not always return the same object since the 'some' may return a Hash where 'dotted' is missing from the first backend but not in the next. The interpolation expression %{hiera('some')[dotted][key]} would, if implemented consistently, share that behavior. For the lookup, a possible remedy would be to use a 'hash' + deep merge but that's not possible when using interpolation expression. I'm well aware that it's a corner case but it does show the advantage of using a simple and consistent interpretation of the lookup key.

- thomas

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/c522c4a4-832b-4915-909d-877af185ef5f%40googlegroups.com.