[Hiera]: How to solve a yaml files getting large issue?

[Vaidas Jablonskis]

Hi People,

I use Puppet 3.0 with Hiera using Yaml backend. Everything works great. I am getting to a point where my yaml files a becoming really large, for instance '%{environment}/roles/myapp.yaml' is pretty huge, over a hundred of lines. So it's getting tricky to manage huge files. It contains various configuration parameters for 'myapp' which is built out of multiple modules and software components.

I believe it's pretty common to have an application which needs a ton of configuration parameters, but you want to keep everything in hiera.

So I am just wondering how would you solve this issue? 

My hierarchy:

:hierarchy:

  - '%{environment}/nodes/%{fqdn}'

  - '%{environment}/roles/%{role}'

  - '%{environment}/common'

Thanks in advance!

[jcbollinger]

On Thursday, November 29, 2012 7:13:33 PM UTC-6, Vaidas Jablonskis wrote:

Hi People,

I use Puppet 3.0 with Hiera using Yaml backend. Everything works great. I am getting to a point where my yaml files a becoming really large, for instance '%{environment}/roles/myapp.yaml' is pretty huge, over a hundred of lines. So it's getting tricky to manage huge files. It contains various configuration parameters for 'myapp' which is built out of multiple modules and software components.

I believe it's pretty common to have an application which needs a ton of configuration parameters, but you want to keep everything in hiera.

So I am just wondering how would you solve this issue? 

If you are storing a lot of data in your hiera data files, then perforce they will become large.  The only way to store the same data yet make individual files smaller is to spread the data over more files.  You can use %{calling_module} and / or %{calling_class} in your hierarchy declaration to add module- and class-specific levels.  In the end, however, that just produces a more complex set of data files in place of more data in each file.

My inclination would be to define default values in my Puppet manifests wherever it is sensible to do so, either for direct use by my classes or for use through hiera via its Puppet back-end.  In my YAML data files I would put only customization data fundamentally environment-specific data.  I'm not too keen on %{calling_module} or especially %{calling_class} (especially the latter), but I wouldn't altogether rule out their use.


John