Jira (PUP-11621) Drop Hiera 3 Requirement

Issue Type:	Task
Assignee:	Unassigned
Created:	2022/08/29 4:15 PM
Fix Versions:	PUP 8.0.0
Priority:	Normal
Reporter:	Michael Hashizume

Hiera 5 has been a part of Puppet since Puppet 4.9 (released in February 2017). However, there are still Hiera functions that rely on the Hiera 3 gem.

In Puppet 8, we have the opportunity to remove the hard dependency on Hiera 3.

In broad strokes, this is what's required:

Modify puppet to call `Puppet.features.hiera?` instead of `require 'hiera'`
Update puppet's gemspec and ext/project_data.yaml to drop `hiera` as a runtime dependency
Remove hiera from install.rb

This message was sent by Atlassian Jira (v8.20.11#820011-sha1:0629dd8)

Josh Cooper (Jira)

unread,

Aug 30, 2022, 11:23:01 AM8/30/22

to puppe...@googlegroups.com

Josh Cooper updated an issue

Puppet /

Change By:	Josh Cooper

Hiera 5 has been a part of Puppet since Puppet 4.9 (released in February 2017). However, there are still Hiera functions that rely on the Hiera 3 gem.

In Puppet 8, we have the opportunity to remove the hard dependency on Hiera 3.

In broad strokes, this is what's required:

* Modify puppet to call `Puppet.features.hiera?` instead of `require 'hiera'`
* Update puppet's gemspec and ext/project_data.yaml to drop `hiera` as a runtime dependency
* Remove hiera from install.rb

Josh Cooper (Jira)

unread,

Aug 30, 2022, 11:37:02 AM8/30/22

to puppe...@googlegroups.com

Josh Cooper updated an issue

Puppet /

Change By:	Josh Cooper

Hiera 5 has been a part of Puppet since Puppet 4.9 (released in February 2017). However, there are still Hiera functions version 3 custom backends that rely on the Hiera 3 gem.

In Puppet 8, we have the opportunity to remove the hard dependency on Hiera 3.

In broad strokes, this is what's required:
* Modify puppet to call `Puppet.features.hiera?` instead of `require 'hiera'`
* Update puppet's gemspec and ext/project_data.yaml to drop `hiera` as a runtime dependency
* Remove hiera from install.rb

Note the {{{}hiera_include{}}}, {{{}hiera_hash{}}}, {{hiera_array}} 4.x functions (such as [https://github.com/puppetlabs/puppet/blob/main/lib/puppet/functions/hiera_include.rb)] should continue to work after this change, because those functions are implemented using puppet lookup.

Josh Cooper (Jira)

unread,

Aug 30, 2022, 11:38:02 AM8/30/22

to puppe...@googlegroups.com

Josh Cooper updated an issue

Puppet /

Change By:	Josh Cooper

Hiera 5 has been a part of Puppet since Puppet 4.9 (released in February 2017). However, there are still Hiera version 3 custom backends that rely on extend the base [Backend class|https://github.com/puppetlabs/hiera/blob/main/lib/hiera/backend.rb] and require the Hiera 3 gem to be installed .

In Puppet 8, we have the opportunity to remove the hard dependency on Hiera 3.

In broad strokes, this is what's required:
* Modify puppet to call `Puppet.features.hiera?` instead of `require 'hiera'`
* Update puppet's gemspec and ext/project_data.yaml to drop `hiera` as a runtime dependency
* Remove hiera from install.rb

Note the {{

{} hiera_include { }} } , {{ {} hiera_hash { }} } , {{hiera_array}} 4.x functions (such as [https://github.com/puppetlabs/puppet/blob/main/lib/puppet/functions/hiera_include.rb)] should continue to work after this change, because those functions are implemented using puppet lookup.

Morgan Rhodes (Jira)

unread,

Aug 30, 2022, 4:07:03 PM8/30/22

to puppe...@googlegroups.com

Morgan Rhodes updated an issue

Puppet /

Change By:	Morgan Rhodes
Epic Link:	PUP-11617

Ben Ford (Jira)

unread,

Aug 30, 2022, 6:13:02 PM8/30/22

to puppe...@googlegroups.com

Ben Ford commented on

https://docs.google.com/spreadsheets/d/1HmKxSkqk0bxbNNCmuoFLm2MslCbJhu8NtYgOLLN4rTU/edit#gid=278845657

Re: Drop Hiera 3 Requirement

There are only 13 modules on the Forge that implement Hiera 3 backends. Because of how Hiera 5 backends are implemented, I don't know how to programmatically identify them. If you know a way to do that, I can re-run the analysis and omit those that have a 5 backend too.

Henrik Lindberg (Jira)

unread,

Oct 2, 2022, 5:31:03 PM10/2/22

to puppe...@googlegroups.com

Henrik Lindberg commented on

Re: Drop Hiera 3 Requirement

Ben Ford Hiera 5 backends would most likely be written in ruby and be functions that operates on the lookup context passed into those functions - most likely having a call to context.not_found.

Something to look out for are modules with both hiera 3 and hira 5 implementations where the hiera 5 implementation just delegates to the hiera 3 implementation. In those cases, the hiera 3 implementation may (unfortunately) require the hiera 3 gem.

Morgan Rhodes (Jira)

unread,

Dec 1, 2022, 1:31:02 PM12/1/22

to puppe...@googlegroups.com

Morgan Rhodes updated an issue

Puppet /

Change By:	Morgan Rhodes
Sprint:	Phoenix 2022-12-21

Morgan Rhodes (Jira)

unread,

Dec 1, 2022, 1:40:02 PM12/1/22

to puppe...@googlegroups.com

Morgan Rhodes updated an issue

Puppet /

Change By:	Morgan Rhodes

Hiera 5 has been a part of Puppet since Puppet 4.9 (released in February 2017). However, there are still Hiera version 3 custom backends that extend the base [Backend class|https://github.com/puppetlabs/hiera/blob/main/lib/hiera/backend.rb] and require the Hiera 3 gem to be installed.

In Puppet 8, we have the opportunity to remove the hard dependency on Hiera 3.

In broad strokes, this is what's required:
* Modify puppet to call `Puppet.features.hiera?` instead of `require 'hiera'`
* Update puppet's gemspec and ext/project_data.yaml to drop `hiera` as a runtime dependency
* Remove hiera from install.rb

Note the {{hiera_include}}, {{hiera_hash}}, {{hiera_array}} 4.x functions (such as [https://github.com/puppetlabs/puppet/blob/main/lib/puppet/functions/hiera_include.rb)] should continue to work after this change, because those functions are implemented using puppet lookup.

We want to make sure we still exercise hiera in spec tests, so we may need to add hiera to the feature gem group in the gemfile [here|https://github.com/puppetlabs/puppet/blob/4f111f444166f5a7a32429592858b607923c0c0e/Gemfile#L21-L35]

Morgan Rhodes (Jira)

unread,

Dec 1, 2022, 1:41:01 PM12/1/22

to puppe...@googlegroups.com

Morgan Rhodes updated an issue

Puppet /

Change By:	Morgan Rhodes
Story Points:	3

Josh Cooper (Jira)

unread,

Dec 6, 2022, 12:01:03 PM12/6/22

to puppe...@googlegroups.com

Josh Cooper updated an issue

Puppet /

Change By:	Josh Cooper

Hiera 5 has been a part of Puppet since Puppet 4.9 (released in February 2017). However, there are still Hiera version 3 custom backends that extend the base [Backend class|https://github.com/puppetlabs/hiera/blob/main/lib/hiera/backend.rb] and require the Hiera 3 gem to be installed.

In Puppet 8, we have the opportunity to remove the hard dependency on Hiera 3.

In broad strokes, this is what's required:
* Modify puppet to call `Puppet.features.hiera?` instead of `require 'hiera'`
* Update puppet's gemspec and ext/project_data.yaml to drop `hiera` as a runtime dependency
* Remove hiera from install.rb

Note the {{ {} hiera_include { }} } , {{ {} hiera_hash { }} }

, {{hiera_array}} 4.x functions (such as [https://github.com/puppetlabs/puppet/blob/main/lib/puppet/functions/hiera_include.rb)] should continue to work after this change, because those functions are implemented using puppet lookup.

We want to make sure we still exercise hiera in spec tests, so we may need to add hiera to the feature gem group in the gemfile [here|https://github.com/puppetlabs/puppet/blob/4f111f444166f5a7a32429592858b607923c0c0e/Gemfile#L21-L35] .

Josh Cooper (Jira)

unread,

Dec 7, 2022, 1:06:02 PM12/7/22

to puppe...@googlegroups.com

Josh Cooper updated an issue

Puppet /

Change By:	Josh Cooper

Hiera 5 has been a part of Puppet since Puppet 4.9 (released in February 2017). However, there are still Hiera version 3 custom backends that extend the base [Backend class|https://github.com/puppetlabs/hiera/blob/main/lib/hiera/backend.rb] and require the Hiera 3 gem to be installed.

In Puppet 8, we have the opportunity to remove the hard dependency on Hiera 3.

In broad strokes, this is what's required:
* Modify puppet to call `Puppet.features.hiera?` instead of `require 'hiera'`
* Update puppet's gemspec and ext/project_data.yaml to drop `hiera` as a runtime dependency
* Remove hiera from install.rb

Note the {{{}hiera_include{}}}, {{{}hiera_hash{}}}, {{hiera_array}} 4.x functions (such as [https://github.com/puppetlabs/puppet/blob/main/lib/puppet/functions/hiera_include.rb)] should continue to work after this change, because those functions are implemented using puppet lookup (see lib/puppet/pops/lookup) .

1. The following "requires" do not need to be modified, because the files exist in puppet:

{noformat}
require 'hiera_puppet'
require 'hiera/scope'
require 'hiera/puppet_function'
{noformat}

2. The following "requires" should be converted to "Puppet.features.hiera?" or they can be kept as "require 'hiera'" provided the call site guards against hiera not being present and a LoadError being raised.

{noformat}
lib/hiera_puppet.rb:require 'hiera'
{noformat}

3. We want to make sure we still exercise hiera in spec tests and benchmarks , so we may need to add move the hiera gem in the Gemfile to the feature "features" gem group in the gemfile [here|https://github.com/puppetlabs/puppet/blob/4f111f444166f5a7a32429592858b607923c0c0e/Gemfile#L21-L35]. And add an rspec filter so tests and benchmarks only run when the feature is true, something like: describe "...", if: Puppet.features.hiera?. Make sure all tests pass when using `bundle install --without features` and `bundle install --with features`.

4. The eyaml_lookup_key supports Hiera 3, but guards against hiera-eyaml not being installed, so I don't think these sites need to be modified:

{noformat}
lib/puppet/functions/eyaml_lookup_key.rb: require 'hiera/backend/eyaml/encryptor'
lib/puppet/functions/eyaml_lookup_key.rb: require 'hiera/backend/eyaml/utils'
lib/puppet/functions/eyaml_lookup_key.rb: require 'hiera/backend/eyaml/options'
lib/puppet/functions/eyaml_lookup_key.rb: require 'hiera/backend/eyaml/parser/parser'
{noformat}

5. Pops supports loading a Hiera v3 config https://github.com/puppetlabs/puppet/blob/cb63661659a029d0a52033c07a61720c0abe1033/lib/puppet/pops/lookup/hiera_config.rb#L284-L286. Replace "require 'hiera'" with Puppet.features.hiera? and if it's false raise an error that indicates the hiera gem is not present in order to load hiera v3 configs.

6. Pops already guards against LoadError when trying to load a hiera v3 backend in https://github.com/puppetlabs/puppet/blob/cb63661659a029d0a52033c07a61720c0abe1033/lib/puppet/pops/lookup/data_dig_function_provider.rb#L65-L94, so I don't think anything is required there.

Morgan Rhodes (Jira)

unread,

Dec 7, 2022, 1:33:02 PM12/7/22

to puppe...@googlegroups.com

Morgan Rhodes updated an issue

Puppet /

Change By:	Morgan Rhodes
Sprint:	Phoenix 2022-12-21

Morgan Rhodes (Jira)

unread,

Jan 4, 2023, 1:10:02 PM1/4/23

to puppe...@googlegroups.com

Morgan Rhodes updated an issue

Puppet /

Change By:	Morgan Rhodes
Sprint:	Phoenix 2023-01-18

Michael Hashizume (Jira)

unread,

Jan 10, 2023, 5:39:02 PM1/10/23

to puppe...@googlegroups.com

Michael Hashizume assigned an issue to Michael Hashizume

Puppet /

Change By:	Michael Hashizume
Assignee:	Michael Hashizume

Josh Cooper (Jira)

unread,

Jan 16, 2023, 9:29:02 PM1/16/23

to puppe...@googlegroups.com

Josh Cooper commented on