Is this possible with Hiera - Puppet Module Development and using Hiera

[Peter J. Foley]

Hi Puppet Users,

 

I have just started playing with Puppet and Hiera.  I am curious if it is possible to control how hiera does variable lookup within a module I am creating to test it.

 

I have read through the documentation I can find on Hiera and Puppet however I cannot find any documentation suggesting a way to it.

 

For the record these are the versions that I am using (taken from the Puppetlabs Debian/Devel):

hiera (1.0.0-0.1rc3)

hiera-puppet (1.0.0-0.1rc1-1-g3e68ff0)

puppet-common (3.0.0-0.1rc3puppetlabs1)

puppet (3.0.0-0.1rc3puppetlabs1)

 

Essentially I would like to define a hiera.yaml file (just the hierarchy and puppet datasource location) within my module and use it to clearly separate Operating System differences to the default state.  For example:

 

------==== {modulepath}/hiera.yaml =======------

:hierarchy:

           - %{operatingsystem}

            - defaults

 

:puppet:

   :datasource: %{modulename}/config/

------===========------

 

So in my module directory I could have:

{modulepath}/config/debian.pp

{modulepath}/config/centos.pp

{modulepath}/config/defaults.pp

 

(It would be even better if in the file name somehow we could grip up similar OS types, ie debian_ubuntu.pp)

 

I know that I can use puppet logic statements (case statements, if, …) to set variables per OS but I feel that it would be cleaner and clearer to separate the OS specific stuff and it would be easier to update a module for a specific OS and override something set in the defaults.pp file.

 

So is this possible or is this something that could be considered as a feature request.

 

Thanks,

 

Peter.

[Steve Traylen]

On Monday, 18 June 2012 13:02:33 UTC+2, Peter wrote:

Hi Puppet Users, 

I have just started playing with Puppet and Hiera.  I am curious if it is possible to control how hiera does variable lookup within a module I am creating to test it.

 

I have read through the documentation I can find on Hiera and Puppet however I cannot find any documentation suggesting a way to it.

 

For the record these are the versions that I am using (taken from the Puppetlabs Debian/Devel):

hiera (1.0.0-0.1rc3)

hiera-puppet (1.0.0-0.1rc1-1-g3e68ff0)

puppet-common (3.0.0-0.1rc3puppetlabs1)

puppet (3.0.0-0.1rc3puppetlabs1)

 

Essentially I would like to define a hiera.yaml file (just the hierarchy and puppet datasource location) within my module and use it to clearly separate Operating System differences to the default state.  For example:

 

The hiera.yaml file is the global configuration for all paths, however it could contain a 

path of 

/etc/puppet/modules/%{module_name}/config/operatingsystem

/etc/puppet/modules/%{module_name}/config/defaults

 

------==== {modulepath}/hiera.yaml =======------

:hierarchy:

           - %{operatingsystem}

            - defaults

 

:puppet:

   :datasource: %{modulename}/config/

------===========------

 

So in my module directory I could have:

{modulepath}/config/debian.pp

{modulepath}/config/centos.pp

{modulepath}/config/defaults.pp

 

(It would be even better if in the file name somehow we could grip up similar OS types, ie debian_ubuntu.pp)

 

Look at the "osfamily" fact, this debian on ubuntu and redhat.

 

I know that I can use puppet logic statements (case statements, if, …) to set variables per OS but I feel that it would be cleaner and clearer to separate the OS specific stuff and it would be easier to update a module for a specific OS and override something set in the defaults.pp file.

 

So is this possible or is this something that could be considered as a feature request.

So it's possible but you can't distribute the hiera.yaml with your module since this is a site setting.

 

 

Thanks,

 

Peter.

[Peter]

On Tuesday, 19 June 2012 03:26:14 UTC+10, Steve Traylen wrote:

On Monday, 18 June 2012 13:02:33 UTC+2, Peter wrote:

Hi Puppet Users, 

Essentially I would like to define a hiera.yaml file (just the hierarchy and puppet datasource location) within my module and use it to clearly separate Operating System differences to the default state.  For example:

The hiera.yaml file is the global configuration for all paths, however it could contain a path of 

/etc/puppet/modules/%{module_name}/config/operatingsystem

/etc/puppet/modules/%{module_name}/config/defaults

Hi Steve,

Thanks for the reply.

Although this will work it is not something I believe we should be aiming for.  I am hoping to eventually make some of the modules I am working on available on the Puppet Forge.

Sorry you may have miss-understood me, I was not talking about over-riding the hiera.yaml puppet file, I was talking about a version that describes how hiera should interact with the puppet module it is currently interacting with.

Using the solution you propose for module users it would we require additional work updating and maintaining hiera.yaml settings that could be non-trivial (what if a module uses a sub-directory data instead of config, or OSFamily instead of OS, or other filters).  Additionally what if the module developer expects a particular ordering of the files but the module users has something specific for there environment.

If we could implement a way for module developers to signal to hiera information on how to interact with values specific to modules it provides the best of both worlds:

*  Module Users can specify the order of hiera values and control the values they set; and

*  Module Develoers can specify the location of hiera values and what is the order is all without the module user having to work it out themselves.

Obviously in the puppet data extension to hiera it has been hardcoded to look for a data.pp file so it could be configured to re-use some of the other hiera code to interrogate a module hiera.yaml file.

Thanks,

Peter

[jcbollinger]

I confess that I don't follow what you are suggesting.  Hiera already supports module developers providing data, and module users overlaying their own on top, in the order of their choice.  What more would your suggestion provide in that area?  And why do module developers need to configure specific locations for (the root of) their data on the filesystem?  That should be under the control of module users (== the administrators of the master).  What am I missing?


John

[Peter]

Hi Joh,

On Wednesday, 20 June 2012 03:18:46 UTC+10, jcbollinger wrote:

I confess that I don't follow what you are suggesting.  Hiera already supports module developers providing data, and module users overlaying their own on top, in the order of their choice.  What more would your suggestion provide in that area?  And why do module developers need to configure specific locations for (the root of) their data on the filesystem?  That should be under the control of module users (== the administrators of the master).  What am I missing?

Thanks for taking the time to reply.

Yes hiera provides support for module developers ... however I would argue that it is limited for example, the hiera-puppet plugin is hard-coded to only look for values in the module manifest directory for a file called data.pp.

This does allow for a very low threshold for module developers to start implementing hiera in their modules (rename their existing params.pp files).

I believe by extending the hiera-puppet plugin for module developers to follow a similar convention as hiera for module users would make certain things easier for module developers.

By providing the ability to signal the hiera-puppet plugin (using a module-hiera.yaml file (for lack of a better name)) a module developer could reduce complexity and use the DRY principle to setup sane defaults (either in a defaults.pp or in a defaults.yaml file) and than use layer on top specific settings for Operating systems or even Hardware Types (as examples).

Sorry I don't have time right now to provide something in depth but a brief example would be setting up ISCSI, the module I currently have has a complex params.pp file for detecting particular setups.

As a module developer I am interested in setting up a hierarchy such that from most specific to least specific:   

* Hardware_platform

* Operating System

* Defaults

By defining this in the module-hiera.yaml file it makes it very easy for me to signal to the module user what settings override which.

I am not suggesting taking any control from module users, and to be explicit user settings (if set) would override the module variable.

Hopefully I have done a better job of explaining what I am thinking.

Thanks,

Peter.

[Ashley Penney]

On Fri, Jun 22, 2012 at 7:02 PM, Peter <pe...@ifoley.id.au> wrote:

Thanks for taking the time to reply.

Yes hiera provides support for module developers ... however I would argue that it is limited for example, the hiera-puppet plugin is hard-coded to only look for values in the module manifest directory for a file called data.pp.

This does allow for a very low threshold for module developers to start implementing hiera in their modules (rename their existing params.pp files).

I believe by extending the hiera-puppet plugin for module developers to follow a similar convention as hiera for module users would make certain things easier for module developers.

By providing the ability to signal the hiera-puppet plugin (using a module-hiera.yaml file (for lack of a better name)) a module developer could reduce complexity and use the DRY principle to setup sane defaults (either in a defaults.pp or in a defaults.yaml file) and than use layer on top specific settings for Operating systems or even Hardware Types (as examples).

Sorry I don't have time right now to provide something in depth but a brief example would be setting up ISCSI, the module I currently have has a complex params.pp file for detecting particular setups.

As a module developer I am interested in setting up a hierarchy such that from most specific to least specific:   

* Hardware_platform

* Operating System

* Defaults

By defining this in the module-hiera.yaml file it makes it very easy for me to signal to the module user what settings override which.

I am not suggesting taking any control from module users, and to be explicit user settings (if set) would override the module variable.

Hopefully I have done a better job of explaining what I am thinking.

Thanks,

Peter.

If I understand your idea correctly then I too would like to see this.  We would be able to use a hierarchy within the module (when sharing on the forge) to provide various defaults without having to do a bunch of selectors on $operatingsystem etc and make it easier for someone to start providing overrides from their own hiera data.  Right now to provide defaults that differ for an operating system you would have to do a bunch of logic in params.pp that the person would want to strip out and replace with a call to hiera anyway.  By allowing us to create params.pp as a series of hiera calls in the first place it'll reduce the amount of modifications (hopefully to zero) when taking a module off the forge.

[jcbollinger]

On Friday, June 22, 2012 6:02:33 PM UTC-5, Peter wrote:

I believe by extending the hiera-puppet plugin for module developers to follow a similar convention as hiera for module users would make certain things easier for module developers.

By providing the ability to signal the hiera-puppet plugin (using a module-hiera.yaml file (for lack of a better name)) a module developer could reduce complexity and use the DRY principle to setup sane defaults (either in a defaults.pp or in a defaults.yaml file) and than use layer on top specific settings for Operating systems or even Hardware Types (as examples).

What I'm getting is that the central idea you propose is to give Hiera's Puppet plugin the ability to collect data not only from different namespaces, as it does now, but also from different classes in each namespace.  I also get the impression that you are reserving use of the YAML backend to module users.

I'm not prepared to discount your proposal, but I do think you are artificially limiting yourself if as a module developer you refuse to use the YAML backend.  If you want more flexibility today then that's where you should look.
 

John

[Kelsey Hightower]

I like where this idea is going. You would like a backend, hiera-puppet for now, to be able to lookup "default" data from the current module directory during parse time. Let's open a ticket (feature request) for this and workout the details.