R10K in an existing infrastructure OR How to write yourself into a corner.

[Robert Davidson]

We're still running puppet 3.6.2, and are finally getting around to trying to implement R10K. For assorted reasons, we have not been able to do this before now, and have a very large stack of home-grown modules that are all sitting in a monolithic repo. For the most part, it's been straightforward. We are splitting up the modules into their own repos, tagging them with version numbers, etc. But I've now hit an issue that's got me blocked.

We have several modules that include important data files inside them. (Which is bad practice, I know, but many of these were written a while ago and we're slowly working to refactor.)
In our current, monolithic setup, they reside in paths like this:
git/puppet/environments/production_ng/modules/$MODULE/files/customers/$FILENAME

Each $FILENAME must be pushed out, with that name and it's contents, to a particular directory on the machine using the module. The files contain anywhere from ten to a hundred lines of config, varying per customer. At the moment, we push them using a recursive file resource into the directory.

Under r10k, we want to use tags to mark version numbers and pull them into environments. But these config files need to change rapidly, and would result in ridiculous version creep if we increment every time we had to adjust one of them. What is a good way to deal with data files like this (ones where putting the contents into hiera is not really viable)? How do I treat them under R10K?

--
Robert Davidson

[Rob Nelson]

Have you looked at including these modules inside your r10k controlrepo? From a workflow perspective, that might be easier (branch, tweak files, PR/merge to `production`), though there may be issues surrounding access and permissions, of course. Have a gander at http://garylarizza.com/blog/2015/11/16/workflows-evolved-even-besterer-practices/ for controlrepo theory and http://rnelson0.com/2015/04/15/improved-r10k-deployment-patterns/ which details the conversion from modules in other repos to a consolidated controlrepo.

If access rights are an issue, you could have a standalone module that is pushed out by r10k via cron on a regular basis, across all repositories. See https://github.com/puppetinabox/controlrepo/blob/production/Puppetfile#L51-L53 and https://github.com/puppetinabox/controlrepo/blob/production/dist/profile/manifests/puppet_master.pp#L25-L29 as an example of this. This would push the current version of that module to every environment that exists at the time. I believe that if you specify a certain ref/branch inside an environment, it will deploy that ref/branch, otherwise it deploys the default branch of the repo, but would certainly verify that if you go down that road.

Rob Nelson
rnel...@gmail.com

--
Robert Davidson

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1EE73329D6577F44A3C2FB0F7D4ACAE98D244374%40mbx-02.
For more options, visit https://groups.google.com/d/optout.

[Robert Davidson]

I would prefer not to have to treat these modules any differently than the rest - the only thing about the problem modules is that they include data files that need to be pushed intact. Putting them into the control repo would diverge them from the way we want to handle all other modules, which strikes me as sub-optimal. I'm mostly looking to see if anyone else has experience handling situations like this, and if they were able to come up with a way to handle it without changing how modules are handled.

One thing we did think about was to create a "data" module that contains files like these, and are referred to inside r10K by branch rather than tag, letting us continue to treat the bulk of the module correctly while isolating our config files off to their own space. It's ugly, but could theoretically work. I can't help but feel that there must be a better way to do this, though.

--
Robert Davidson

> send an email to puppet-users...@googlegroups.com
> <mailto:puppet-users%2Bunsu...@googlegroups.com> .

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-
> users/1EE73329D6577F44A3C2FB0F7D4ACAE98D244374%40mbx-02

> <https://groups.google.com/d/msgid/puppet-
> users/1EE73329D6577F44A3C2FB0F7D4ACAE98D244374%40mbx-02> .

> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout> .

>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to puppet-users...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/CAC76iT-
> urwaEHe8KiJf81Y5jcdPF7%3Dg%3D%2B-HDhy0VUSG-
> 2bTmJQ%40mail.gmail.com <https://groups.google.com/d/msgid/puppet-
> users/CAC76iT-urwaEHe8KiJf81Y5jcdPF7%3Dg%3D%2B-HDhy0VUSG-
> 2bTmJQ%40mail.gmail.com?utm_medium=email&utm_source=footer> .

[Rob Nelson]

The 'site_configs' module is pretty equivalent to the 'data' module you propose - odd config files, SSL certs, etc. I've been using that successfully at home and work for over a year with r10k without any issues. I'm not using 'data in modules' yet but that would let you combine hiera data with the files, as well.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1EE73329D6577F44A3C2FB0F7D4ACAE98D2478EF%40mbx-02.

For more options, visit https://groups.google.com/d/optout.

--

Rob Nelson
rnel...@gmail.com

[Rilindo Foster]

Hi Robert!

Is separating the customer data an option? That would let you manage the modules without having to increment them constantly, while pulling the customer data from another location - perhaps from a GIT repository using Puppet's vcsrepo module.

- Rilindo

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1EE73329D6577F44A3C2FB0F7D4ACAE98D2478EF%40mbx-02.

[John Warburton]

On Thursday, 29 September 2016, Rob Nelson <rnel...@gmail.com> wrote:

The 'site_configs' module is pretty equivalent to the 'data' module you propose - odd config files, SSL certs, etc. 

Rob

I can't find site_configs on the forge or google. Any pointers?

Thanks 

John 

--
John Warburton
Ph: 0417 299 600
Email: jwarb...@gmail.com

[Rob Nelson]

Whoops, it's called lab_config in that project (site_config is my private repo for the home lab) and can be found at https://github.com/puppetinabox/lab_config. There's not as much to that since it's a reference project, but for example DNS updates are pushed out every 15 minutes from it (agents check in every 30 minutes, though), so it's just a matter of pushing a change to 'master' in that repo and boom, deployed within the hour.

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAAJLFxXZSAxxUY-6sRR7f_3dZEAriMqzHgfrxzfcjYUM1XFvaw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Rob Nelson
rnel...@gmail.com

[Robert Davidson]

That would be messy, given the Very Interesting way in things are separated, network-wise. It's probably a better long-term solution, but in the short term, setting up a data or config module and making r10K track it by branch looks to be the most viable solution.

--
Robert Davidson


> -----Original Message-----
> From: puppet...@googlegroups.com [mailto:puppet-
> us...@googlegroups.com] On Behalf Of Rilindo Foster
> Sent: Wednesday, September 28, 2016 8:44 PM
> To: puppet...@googlegroups.com
> Subject: RE: [Puppet Users] R10K in an existing infrastructure OR How to
> write yourself into a corner.
>

> users/01f701d21a03%24bb6afa20%243240ee60%24%40gmail.com.

[John Warburton]

Thanks Rob

On Thursday, 29 September 2016, Rob Nelson <rnel...@gmail.com> wrote:

Whoops, it's called lab_config in that project (site_config is my private repo for the home lab) and can be found at https://github.com/puppetinabox/lab_config. There's not as much to that since it's a reference project, but for example DNS updates are pushed out every 15 minutes from it (agents check in every 30 minutes, though), so it's just a matter of pushing a change to 'master' in that repo and boom, deployed within the hour.

On Thursday, September 29, 2016, John Warburton <jwarb...@gmail.com> wrote:

On Thursday, 29 September 2016, Rob Nelson <rnel...@gmail.com> wrote:

The 'site_configs' module is pretty equivalent to the 'data' module you propose - odd config files, SSL certs, etc. 

Rob

I can't find site_configs on the forge or google. Any pointers?

Thanks 

John 

--
John Warburton
Ph: 0417 299 600
Email: jwarb...@gmail.com

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAAJLFxXZSAxxUY-6sRR7f_3dZEAriMqzHgfrxzfcjYUM1XFvaw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Rob Nelson
rnel...@gmail.com

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAC76iT-aiwZcD1-7B%3DYy0brzbRrpz0-Z6Ck9_ULsDJE7D%2B-CKw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--