Using exported resources as data containers?

[mpou...@afilias.info]

I have a difficult-to-manage application which does not implement a conf.d or include syntax in its configuration, but requires a bunch of config snippets that contain information only on groups of other servers.   I've been dealing with this by generating the config snippets from templates on some servers as exported resources, realizing them on the central server, and then executing an external script to "compile" these snippets into the final config.

This has a couple of drawbacks.  First, it requires puppet to stat nearly 15,000 little tiny config snippets every run that are not actually used directly, and shouldn't need to exist.  Second, the final config file, because it's compiled by an external script, isn't under the control of puppet, so it has no idea if that file gets modified by something outside.. so it can't know to update it. 

I've been mulling over a better way to manage this config file, and I think I've hit on an idea, but I have no idea if it will actually work, or what the syntax would look like if it could.

I'm thinking of replacing the @@file resources on the remote servers with a defined type .. say .. @@data_container.   Then, on the server where the data is needed I could use a collector to iterate over the exported resources reading data from them to use in the single template for the final config file.

Where the data is defined:

@@data_container { 'mydata':

   someparameter => 'foo'

}

And then in the template on the other host, somehow get a collection of those resources into an array, and make use of their parameters as variables to be referenced in the template.. 

<%- collection.each do |data| -%>

<%= data.someparameter %>

<%- end -%>

Would this work at all?  Is there syntax to support something like this?

[Daniel Siechniewicz]

Hi,

Sounds like a job for https://github.com/dalen/puppet-puppetdbquery potentially? pdbresourcequery or maybe even the hiera backend.


Regards,
Daniel

[Matthew Pounsett]

On 10 September 2014 07:30, Daniel Siechniewicz <dan...@siechniewicz.com> wrote:

Hi,

Sounds like a job for https://github.com/dalen/puppet-puppetdbquery potentially? pdbresourcequery or maybe even the hiera backend.

Hiera doesn't apply here, because it's data gathered from the servers (mostly from facts) where the config snippets are defined.  Putting that sort of thing in Hiera violates Don't Repeat Yourself.  I'll have a look at the others though.. I'm not familiar with them.  Thanks for the suggestions.

[Daniel Siechniewicz]

Hi,

This particular hiera backend, from what I understand, is extracting
values from puppetdb "on the fly", so you don't have to put any values
in hiera yaml/json files save for a relevant puppetdb query. This
would centralize data gathering and localize it to a "data component".
Otherwise you end up with puppetdb queries within your manifests which
could be not what you want. Then again, this really depends on your
preferences. Good luck and tell us if this worked for you.


Regards,
Daniel

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/puppet-users/VzW7ZODB7hQ/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/CAKZ22fLmwD0eqXs7DttKydpZjPBTJMJq0PcQY28vpnbO%3D7mxGg%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.

[Matthew Pounsett]

On 10 September 2014 08:08, Daniel Siechniewicz <dan...@siechniewicz.com> wrote:

Hi,

This particular hiera backend, from what I understand, is extracting
values from puppetdb "on the fly", so you don't have to put any values
in hiera yaml/json files save for a relevant puppetdb query. This
would centralize data gathering and localize it to a "data component".
Otherwise you end up with puppetdb queries within your manifests which
could be not what you want. Then again, this really depends on your
preferences. Good luck and tell us if this worked for you.

It looks like it's designed to deal with facts and classes only.  I really need something that can get data from resources (defined types).  It's crappy design, but I could move *some* of the global variables defined in my type's module into facts to make those available, but I don't see a decent way to get data out of the resources, so I don't think it solves the whole problem.

Thanks for the suggestion though.. this looks like it could be very useful to simplify a couple other things I've been looking at.

 

[José Luis Ledesma]

Perhaps you can use datacat_fragments exported resources and then collect them   and put them together with the datacat_collector.

https://github.com/richardc/puppet-datacat/

Regards,

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/7732833b-55f8-4e1b-8869-4ec231d36964%40googlegroups.com.

[Matthew Pounsett]

On 10 September 2014 12:57, José Luis Ledesma <joseluis...@gmail.com> wrote:

Perhaps you can use datacat_fragments exported resources and then collect them   and put them together with the datacat_collector.

https://github.com/richardc/puppet-datacat/

That looks really promising.  It'll take me a bit to wrap my head around it, because I think I'll need to refactor a couple of things to use it... but I think I can make it do what I need.

Thanks!

[jcbollinger]

On Wednesday, September 10, 2014 9:09:14 AM UTC-5, Matthew Pounsett wrote:

On 10 September 2014 08:08, Daniel Siechniewicz <dan...@siechniewicz.com> wrote:

Hi,

This particular hiera backend, from what I understand, is extracting
values from puppetdb "on the fly", so you don't have to put any values
in hiera yaml/json files save for a relevant puppetdb query. This
would centralize data gathering and localize it to a "data component".
Otherwise you end up with puppetdb queries within your manifests which
could be not what you want. Then again, this really depends on your
preferences. Good luck and tell us if this worked for you.

It looks like it's designed to deal with facts and classes only.  I really need something that can get data from resources (defined types).

Or maybe that's just how you've determined to do it.  And it is indeed a crappy design, because Puppet resources -- including exported ones -- are not intended to function as data vehicles within the master, and they do not conveniently do so.  Instead, they are intended to carry data from master to agent.

 

 It's crappy design, but I could move *some* of the global variables defined in my type's module into facts to make those available, but I don't see a decent way to get data out of the resources, so I don't think it solves the whole problem.

If you have to manage a site configuration wherein one machine's configuration needs to draw on details of many other machines', then it seems to me that the way to go would be to model all that data in hiera in such a way that each machine can get what it needs from there.  The main thing there is to avoid putting anything in a machine-specific hierarchy level if more than one node needs to draw on it.  You can use hashes keyed on the node identifier instead.

John