Announce: Hiera 1.2.0 Available

[Matthaus Owens]

Hiera 1.2.0 is a feature release in the 1.x series with
new features and bug fixes.

Downloads are available at:
* Source: https://downloads.puppetlabs.com/hiera/hiera-1.2.0.tar.gz

RPMs are available at https://yum.puppetlabs.com/el or /fedora

Rubygem available at http://rubygems.org/gems/hiera

Debs are available at https://apt.puppetlabs.com

Mac package is available at
https://downloads.puppetlabs.com/mac/hiera-1.2.0.dmg

Please report feedback via the Puppet Labs Redmine site, using a
affected version of 1.2.0:
http://projects.puppetlabs.com/projects/hiera/

Fixes targeted at the final of this version in our bug tracker:
http://projects.puppetlabs.com/versions/332

========================
## Hiera 1.2.0 Release Notes ##
========================
# Features

Add deep-merge feature to backend lookups

- Config option :merge_behavior => :native|:deep|:deeper
- Add optional requirement on deep_merge gem to support
:deep and :deeper options
- Update Yaml backend to use Backend.merge_answer
- Update Json backend to use Backend.merge_answer

(#16644) Add a generic file cache

Add a general file cacher in Hiera::Filecache based on the work that was
done in the YAML backend.

Adjust the YAML and JSON backends to use this cache

(#18718) Create logger to handle fallback

Sometimes a logger has been configured, but is not suitable for being
used. An example of this is when the puppet logger has been configured,
but hiera is not being used inside puppet. This adds a FallbackLogger
that will choose among the provided loggers for one that is suitable.

# Bug Fixes

(#17434) Detect loops in recursive lookup

The recursive lookup functionality was vulnerable to infinite recursion
when the values ended up referring to each other. This keeps track of
the names that have been seen in order to stop a loop from occuring. The
behavior for this was extracted to a class so that it didn't clutter the
logic of variable interpolation. The extracted class also specifically
pushes and pops on an internal array in order to limit the amount of
garbage created during these operations. This modification should be
safe so long a new Hiera::RecursiveLookup is used for every parse that
is done and it doesn't get shared in any manner.

(#17434) Support recursive interpolation

The original code for interpolation had, hidden somewhere in its depths,
supported recursive expansion of interpolations. This adds that support
back in.

=====================
## Hiera 1.2.0 Changelog ##
=====================

Andrew Parker (13):
26311b7 (#18718) Load logger classes eagerly
2520aa3 (#18718) Create logger to handle fallback
074f5c8 (#18718) Enable console fallback when logger not suitable
8db2949 (#18718) Implement suitablity check for puppet logger
dc98e2d (#17434) Add YARD for #parse_string
06dcf8e (#17434) Clarify tests for #parse_string
dc6c538 (#17434) Add tests to exclude unwanted lookups
3a2660d (#17434) Stronger assertion about how keys are looked up
4d85f92 (Maint) Describe desired behavior in backend specs
023001d (#17434) Simplify string interpolation
9a3f1fd (#17434) Simplify logic around looking up values
453b489 (#17434) Support recursive interpolation
9a62bfd (#17434) Detect loops in recursive lookup

Jeff McCune (4):
b2623d9 (maint) Add Travis CI Support
fcecdbf (maint) Add Travis CI support to active branches
5262050 (maint) Add Ruby 2.0.0 to Travis build matrix
d9db368 Add contributing document to Hiera

Justen Walker (7):
4ac8372 Add deep-merge feature to backend lookups
3da83b2 Allow both symbols and strings when deciding behavior of
merge_answer
950076b Fix undefined method `[]' for nil:NilClass error in
yaml_backend_spec.rb
b317d10 Add deep-merge feature to backend lookups
13b79ef Allow both symbols and strings when deciding behavior of
merge_answer
d84cd11 Fix undefined method `[]' for nil:NilClass error in
yaml_backend_spec.rb
a4bbc76 Fix #19792 - cryptic error when missing deep_merge gem

Matthaus Owens (7):
5828192 (maint) Remove natty from build_defaults, changelog
2d9f4f1 (maint) Remove rc_mocks from build_defaults
7097b58 (maint) Add f18 mocks to build_defaults
0fd72a1 (packaging) Update lib/hiera.rb for Hiera 1.2.0-rc1
6414532 (packaging) Update lib/hiera.rb for 1.2.0-rc2
8f25616 (packaging) Update lib/hiera.rb for 1.2.0-rc3
636ad58 (packaging) Update lib/hiera.rb for 1.2.0

Moses Mendoza (1):
b8f77fb [packaging] Update mocks for rpmbuilder mock format

Patrick Carlisle (3):
ea485ab (#16089) Reduce scope lookups in parse_string
3c24788 (#16089) Reduce scope lookups in parse_string
bd2e3d3 (#17434) Revert "Merge pull request #29 from
timurbatyrshin/puppet-style"

R.I.Pienaar (2):
c8e3e90 16644 - should have a generic file cache
aceca79 (#16644) Add a generic file cache

Sharif Nassar (1):
ab7fc5a Spelling.

floatingatoll (1):
e3c8045 comment typo fix in lib/hiera/backend.rb

[Brian Lalor]

On Apr 3, 2013, at 8:39 PM, Matthaus Owens <matt...@puppetlabs.com> wrote:

> Hiera 1.2.0 is a feature release in the 1.x series with
> new features and bug fixes.

Can Hiera be upgraded independently of the version of Puppet being used?

--
Brian Lalor
bla...@bravo5.org

[Ellison Marks]

Since I can imagine a situation that I might want to use the deep merge functionality on some lookups but not on others, Is it currently possible to specify the type of merge to do in the hiera_hash call? If not, is that a planned feature?

On a slightly related note, is the function documentation going to get an update soon? All three hiera functions are still listed as "undocumented". They do have fairly complete documentation in http://docs.puppetlabs.com/hiera/1/index.html, so perhaps something could be borrowed from there.

[Matthaus Owens]

Brian,
Yes, Hiera can be upgraded independently of Puppet.

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
> To post to this group, send email to puppet...@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>



--
Matthaus Owens
Release Manager, Puppet Labs

[Erik Dalén]

On Thursday 4 April 2013 at 02:39, Matthaus Owens wrote:
> ========================
> ## Hiera 1.2.0 Release Notes ##
> ========================
> # Features
>
> Add deep-merge feature to backend lookups
>
> - Config option :merge_behavior => :native|:deep|:deeper
> - Add optional requirement on deep_merge gem to support
> :deep and :deeper options
> - Update Yaml backend to use Backend.merge_answer
> - Update Json backend to use Backend.merge_answer

So if I understood it correctly the difference is when the values are unmergeable and :deeper will give the value from the higher (more specific) hierarchy level while :deep will instead give the value from the lower (less specific) hierarchy level. Correct?

I don't really see the point of the :deep option then as the point of hiera is to give the most specific answer for the node in question.

But could be nice to at least log a warning or something when this happens.
And also document the two options a bit more :)

> (#17434) Detect loops in recursive lookup
>
> The recursive lookup functionality was vulnerable to infinite recursion
> when the values ended up referring to each other. This keeps track of
> the names that have been seen in order to stop a loop from occuring. The
> behavior for this was extracted to a class so that it didn't clutter the
> logic of variable interpolation. The extracted class also specifically
> pushes and pops on an internal array in order to limit the amount of
> garbage created during these operations. This modification should be
> safe so long a new Hiera::RecursiveLookup is used for every parse that
> is done and it doesn't get shared in any manner.
>
> (#17434) Support recursive interpolation
>
> The original code for interpolation had, hidden somewhere in its depths,
> supported recursive expansion of interpolations. This adds that support
> back in.

So, is this for expansion of a variable that gives another string that should do variable expansion?

Not recursive hiera lookups? (which would be really nice, especially in backends that don't support references)

--
Erik Dalén

[Nick Fagerlund]

On Wednesday, April 3, 2013 9:42:12 PM UTC-7, Ellison Marks wrote:

On a slightly related note, is the function documentation going to get an update soon? All three hiera functions are still listed as "undocumented". They do have fairly complete documentation in http://docs.puppetlabs.com/hiera/1/index.html, so perhaps something could be borrowed from there.

Yeah, those are getting real doc strings in Puppet 3.2, which will start getting preview releases very soon now (i.e. probably a week or two).

Now that I think about it, we could probably just patch the 3.1.1 reference to include the new strings in the meantime. I'll go do that.

[Jeff McCune]

On Thu, Apr 4, 2013 at 3:02 AM, Erik Dalén <erik.gus...@gmail.com> wrote:

On Thursday 4 April 2013 at 02:39, Matthaus Owens wrote:
> ========================
> ## Hiera 1.2.0 Release Notes ##
> ========================
> # Features
>
> Add deep-merge feature to backend lookups
>
> - Config option :merge_behavior => :native|:deep|:deeper
> - Add optional requirement on deep_merge gem to support
> :deep and :deeper options
> - Update Yaml backend to use Backend.merge_answer
> - Update Json backend to use Backend.merge_answer

So if I understood it correctly the difference is when the values are unmergeable and :deeper will give the value from the higher (more specific) hierarchy level while :deep will instead give the value from the lower (less specific) hierarchy level. Correct?

I'm not sure if this merge behavior has anything to do with the hierarchy levels.  The setting determines how the same value found in multiple backends, (e.g. JSON and YAML) are handled.  I agree, there's lots of room for improvement regarding the documentation of this.  I also regret letting the options be called "deep" and "deeper" since they really don't relate to depth, but instead they relate to the ordering of the back ends.

When using "deep," backends that are listed at the top in the `:backends:` configuration array will take precedence over later backends when multiple backends return a value and the values are of different types (can't be merged).

When using "deeper," backends that are listed after other backends that return a value for the same key will take precedence.

I guess this could be better said as, when using deep, the first backend to return a value will take precedence over other back ends that also return values that cannot be merged.  When using deeper, the last backend to return a value will take precedence over other back ends that return values that cannot be merged.

Ugh.  Clear as mud.

 

I don't really see the point of the :deep option then as the point of hiera is to give the most specific answer for the node in question.

yeah, deep and deeper don't really have anything to do with the hierarchy.  They only affect the value when multiple backends are being used and the values cannot be merged across backends.

 

But could be nice to at least log a warning or something when this happens.
And also document the two options a bit more :)

Agreed!  Are you interested in submitting a patch?  The code in question is at:

https://github.com/puppetlabs/hiera/blob/1.2.0/lib/hiera/backend.rb#L151-L160

and:

https://github.com/puppetlabs/hiera/blob/1.2.0/lib/hiera/backend.rb#L193

 

Hope this helps,

-Jeff

[Jeff McCune]

On Thu, Apr 4, 2013 at 10:50 AM, Jeff McCune <je...@puppetlabs.com> wrote:

On Thu, Apr 4, 2013 at 3:02 AM, Erik Dalén <erik.gus...@gmail.com> wrote:

On Thursday 4 April 2013 at 02:39, Matthaus Owens wrote:
> ========================
> ## Hiera 1.2.0 Release Notes ##
> ========================
> # Features
>
> Add deep-merge feature to backend lookups
>
> - Config option :merge_behavior => :native|:deep|:deeper
> - Add optional requirement on deep_merge gem to support
> :deep and :deeper options
> - Update Yaml backend to use Backend.merge_answer
> - Update Json backend to use Backend.merge_answer

So if I understood it correctly the difference is when the values are unmergeable and :deeper will give the value from the higher (more specific) hierarchy level while :deep will instead give the value from the lower (less specific) hierarchy level. Correct?

I'm not sure if this merge behavior has anything to do with the hierarchy levels.

Hrm, no...  I'm wrong.

The YAML and JSON backends use the same precedence behavior as well: 

https://github.com/puppetlabs/hiera/blob/1.2.0/lib/hiera/backend/yaml_backend.rb#L49

https://github.com/puppetlabs/hiera/blob/1.2.0/lib/hiera/backend/json_backend.rb#L45

So you're correct, deep will keep the first answer found in the hierarchy if later values are of different types.  Deeper will keep the last answer found in the hierarchy if later values are of different types.

-Jeff

[Chuck]

Hiera 1.2.0 is refusing to use the Puppet_logger  backend on my puppet masters and is dumping ALL of its logs into my HTTP error.log file.  Which then fills up my /var/log file system.

puppetdb-terminus-1.2.0-1.el6.noarch

puppet-3.1.1-1.el6.noarch

puppet-server-3.1.1-1.el6.noarch

hiera-1.2.0-1.el6.noarch

[Mon Apr 08 16:07:33 2013] [notice] Apache/2.2.15 (Unix) DAV/2 Phusion_Passenger/3.0.17 mod_ssl/2.2.15 OpenSSL/1.0.0-fips configured -- resuming normal operations

WARN: Mon Apr 08 16:07:37 +0000 2013: Not using Hiera::Puppet_logger. It does not report itself to be suitable.

DEBUG: Mon Apr 08 16:07:37 +0000 2013: Hiera YAML backend starting

[windowsrefund]

Personally, I'm not looking forward to dealing with this "deep merge" thing at all and would like to see it ripped out. When someone takes the time to properly organize their backend data along with a sane hierarchy, there's no need for this kind of overhead.