Puppet control, Hiera data, puppetfile, and r10k and git merging woes

562 views
Skip to first unread message

Mike Sharpton

unread,
Aug 17, 2016, 4:21:45 PM8/17/16
to Puppet Users
Hey all,

We are coming up on an issue in our environment in where we have multiple Puppet environments that are backed by git branches in a puppet control repo.  Our Hiera data is stored inside these branches and changed frequently by our Operations teams.  Of which we then have them merge changes up the environment chain and r10k through our Puppet environments.  This is all fine.

Ex, dev -> test -> production, hiera data changes are moved up and tested each step of the way.

When things aren't fine is when we are testing code in our dev or test branch and we have changed the tags for modules/repos inside the Puppetfile of those branches that we don't want in production right away (dev/test).  This code only applies to dev environment, on purpose.  

Our operations team then comes along with their hiera changes and merges the puppetfile module/repo changes up the chain along with the hiera data.  Effectively moving our Puppetfile changes up the chain when we don't want to.  We have thought about splitting hiera data out our puppet control module like it was before Puppet 4, but this leaves us no room to test hiera data up our environment chain and also leaves us with some CI work to make this feasible.  Having the hieradata in each environment is too nice.  We also attempted to monkey with .gitignore, but this is not meant to do what we are trying to do.  Don't merge Puppetfile unless I want to. 

Has anyone ran into this and found a somewhat elegant solution?  Everything we are coming up with is either not easy to manage, or just doesn't make sense to do.  Perhaps we are missing something simple and are over thinking things.  Thanks in advance.

Mike

Christopher Wood

unread,
Aug 17, 2016, 4:52:31 PM8/17/16
to puppet...@googlegroups.com
It sounds like these might help:

https://puppet.com/blog/git-workflows-puppet-and-r10k

http://garylarizza.com/blog/categories/r10k/

Seems like you would benefit from having all teams work from branches of current production and merge back, rather than maintaining a semi-permanent dev branch shared by everybody. This is usually where I suggest that people review commits and talk to each other and figure out what's good, but sometimes that's like pulling teeth.
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [1]puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> [2]https://groups.google.com/d/msgid/puppet-users/9d9e18a4-a6e4-4d04-b0b3-377b848a8504%40googlegroups.com.
> For more options, visit [3]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. mailto:puppet-users...@googlegroups.com
> 2. https://groups.google.com/d/msgid/puppet-users/9d9e18a4-a6e4-4d04-b0b3-377b848a8504%40googlegroups.com?utm_medium=email&utm_source=footer
> 3. https://groups.google.com/d/optout

Mike Sharpton

unread,
Aug 18, 2016, 8:27:40 AM8/18/16
to Puppet Users, christop...@pobox.com
Thanks for your reply.  We based our initial design on shit Gary says.  This may be our only option as you say, to have hiera data changes made to each static branch/puppet environment by hand and not merge.  We need the static branches for separation of Puppet environments.  Problem with this approach is humans will make errors between each branch sometimes or always.  The branches/environments will eventually become snow flakes over time as far as Hieradata.  Perhaps we can possibly merge them weekly to lower this risk.  Assuming no code changes are in flight, which there most likely always will be.  The search continues. Thanks again,

Mike

Rob Nelson

unread,
Aug 18, 2016, 9:01:00 AM8/18/16
to puppet...@googlegroups.com
Mike, is there a reason that Puppetfile changes and hiera changes are being made in sync, when they aren't tied to each other? Perhaps those Puppetfile changes that are not ready to be merged should be in a branch called 'experimental' (or even a more-persistent-than-normal feature branch) and only merged to 'dev' when it's time to start promoting the code. You've probably already thought of that, but throwing it out there just in case.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/cfcde2dc-2904-4286-bd01-c934857c0ee5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--

Mike Sharpton

unread,
Aug 18, 2016, 9:58:41 AM8/18/16
to Puppet Users
Rob,

Thanks for your reply.  When our ops team makes a change in hieradata to the dev branch for example, all dev machines (in the dev puppet environment) will receive the new hiera data (we want this to happen).  We then ask them to merge that up to test, and finally to production.  If an engineer/developer is testing some new version of a repo/pupept module and has incremented the tag in puppetfile in the dev branch for this module, when the ops team merges the control module from dev to test, the puppetfile and tag are also merged as this is all included in a control repo.  Puppetfile is newer than where I am merging to, merge it.

We may have a fix of which we didn't think about.  Hot fix push direct to the branch you need to for hiera data then merge to all, even in the wrong direction of an upward flow.  dev -> test -> production is a logical flow.  In this case if ops doesn't change Puppetfile in their hot fix branch, it will see the puppetfile it is merging to is newer and not merge it.  I think this will work, we are testing it now.  If this works, I will post to this thread.  This may help people even though it goes against logical merge paths in my head.  I am not a developer by any means, perhaps this makes perfect sense to them.

Mike

Christopher Wood

unread,
Aug 18, 2016, 11:00:01 AM8/18/16
to puppet...@googlegroups.com
I'm missing why you need static branches. I'm picturing something more like:

git checkout production
git checkout -b ticket1234
# make changes, commit, push, test, repeat
git merge production # catch up on any prod changes, retest
git tag ticket.1234
git checkout production
git merge ticket1234
git branch -d ticket1234

That way everybody's changes are working pretty close to what production is right now.

The alternatives are curating your branches, periodically re-branching from production, or just accepting the current state, as near as I can tell off the cuff. If you want to maintain something it requires maintenance work no matter the tool you pick.


On Thu, Aug 18, 2016 at 05:27:40AM -0700, Mike Sharpton wrote:
> Thanks for your reply.  We based our initial design on shit Gary says.
>  This may be our only option as you say, to have hiera data changes made
> to each static branch/puppet environment by hand and not merge.  We need
> the static branches for separation of Puppet environments.  Problem with
> this approach is humans will make errors between each branch sometimes or
> always.  The branches/environments will eventually become snow flakes over
> time as far as Hieradata.  Perhaps we can possibly merge them weekly to
> lower this risk.  Assuming no code changes are in flight, which there most
> likely always will be.  The search continues. Thanks again,
> Mike
>
> On Wednesday, August 17, 2016 at 3:52:31 PM UTC-5, Christopher Wood wrote:
>
> It sounds like these might help:
>
> [1]https://puppet.com/blog/git-workflows-puppet-and-r10k
>
> [2]http://garylarizza.com/blog/categories/r10k/
> >    email to [1][3]puppet-users...@googlegroups.com.
> >    To view this discussion on the web visit
> >  
>  [2][4]https://groups.google.com/d/msgid/puppet-users/9d9e18a4-a6e4-4d04-b0b3-377b848a8504%40googlegroups.com.
> >    For more options, visit [3][5]https://groups.google.com/d/optout.
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [6]puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> [7]https://groups.google.com/d/msgid/puppet-users/cfcde2dc-2904-4286-bd01-c934857c0ee5%40googlegroups.com.
> For more options, visit [8]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. https://puppet.com/blog/git-workflows-puppet-and-r10k
> 2. http://garylarizza.com/blog/categories/r10k/
> 3. javascript:
> 4. https://groups.google.com/d/msgid/puppet-users/9d9e18a4-a6e4-4d04-b0b3-377b848a8504%40googlegroups.com
> 5. https://groups.google.com/d/optout
> 6. mailto:puppet-users...@googlegroups.com
> 7. https://groups.google.com/d/msgid/puppet-users/cfcde2dc-2904-4286-bd01-c934857c0ee5%40googlegroups.com?utm_medium=email&utm_source=footer
> 8. https://groups.google.com/d/optout

Mike Sharpton

unread,
Aug 18, 2016, 12:07:23 PM8/18/16
to Puppet Users, christop...@pobox.com
The static branches are basically Puppet environments in which nodes are bound/pointed to them in their puppet.conf.  This way we can open CR's per set of nodes and move up the chain.  Also, I may have found another option on Gary's site.  We could r10k our hiera data and split it from our control repo.  More to come.  Thanks again for thoughts.

Rob Nelson

unread,
Aug 18, 2016, 12:24:58 PM8/18/16
to puppet...@googlegroups.com
The term 'environment' is overloaded. In the context of puppet, I prefer to think of it as "a set of puppet code/data representing a branch of the controlrepo' (puppet environment), rather than 'an environment that nodes run in' (dev/qa/prod/etc) (node environment). Since you can make the latter part of your hiera hierarchy, the only puppet environment that needs to live forever is 'production'. Inside it, the hieradata has ALL the data for all of the node environments, so they actually all check into 'production'. The hierarchy value for the node environment can be a custom fact, calculated or a file on disk, so nodes can get the right node environment data from the puppet environment 'production'.

:hierarchy:
  - "nodes/%{::trusted.certname}"
  - "node_env/%{node_env}"
  - "common"
By differenting the various uses of the overloaded term 'environment' a bit, you can actually streamline your workflow quite a bit. Now all your data is in one place. When you create a feature branch for testing, you can then point the canary nodes to that branch (`puppet agent -t --environment ticket1234`, or putting `environment = ticket1234` in puppet.conf). Whether you're changing roles and profiles, hiera data, or the Puppetfile, it's contained in that branch, but you can actually have production, dev, qa nodes check into it and get the new results, so you aren't surprised when a Puppetfile change in dev trickles up to prod and suddenly things blow up. Of course, you need to have some canary nodes in each node environment (or place a LOT of trust in --noop mode) to get there, but I think that's a reasonable goal to work toward.



Aside: I know we have discussed workflows and the various types of environments on this list quite a bit this Spring/Summer. Does anyone have a good reference article for this already, or do we need to come up with one? I think this is an important gap to fill.


--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/b4d986ee-a5e4-4c40-9e0e-2d3d32f35b18%40googlegroups.com.

Alex Samad

unread,
Aug 18, 2016, 8:49:11 PM8/18/16
to Puppet Users
Hi

I have recently gone through this problem.

I had initial thought to you different branches for the different environments.
say 
prod
uat
sim
inf
dev

But was advised best to go with production and testing.


so I have and I have used a grouping in my ENC to put machines into the above groups. But I am about ready to go back to more branches again.  Why 

I have setup my profiles 

my_ssh profile this handles all the general setup of ssh server and client - standard across all machines


so my roles just include profile::my_ssh

My issue with this is if I want to now role out a change to my ssh setup I can't isolate which box gets its, pot luck, wait long enough and they all get it.

I am not sure its the best to go to each dev/uat machine and run puppet --environment <mytest> is the best solution plus my ENC sets the environment ..  don't want pesky dev guys change environments on me.

I don't want to have to create a new profile and attach it to the roles I want to test on.

So I am thinking ... multiple branches is sounding good.

But im keen to see what comes out of this.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

Rob Nelson

unread,
Aug 18, 2016, 11:20:47 PM8/18/16
to puppet...@googlegroups.com
The number of branches you have has almost nothing to do with how fast your nodes converge to a new desired state. If you want them to check in on command, use mcollective or another orchestration engine to make that happen.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/b4d986ee-a5e4-4c40-9e0e-2d3d32f35b18%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1527405f-dda2-4a6b-b35a-70cea5618021%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


--

Alex Samad

unread,
Aug 19, 2016, 12:05:10 AM8/19/16
to Puppet Users
Hi

Think you missed my point. in my example I want to limit where the updates go.

Presumably the puppet agent is set to some periodic period.  I believe 30min.

so when i push my test case, say using the puppet --environment command, 30min later it will be updated back.

A

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/b4d986ee-a5e4-4c40-9e0e-2d3d32f35b18%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1527405f-dda2-4a6b-b35a-70cea5618021%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--

Rich Burroughs

unread,
Aug 19, 2016, 10:22:17 AM8/19/16
to puppet...@googlegroups.com
I'd love to see you write an article about this if there's not one already. I think these are pretty common workflow issues people grapple with.


Rich

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/b4d986ee-a5e4-4c40-9e0e-2d3d32f35b18%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAC76iT-3u3Ara1zWde6YwtKCyHjk%3DhcAVcFhVQGXkJBdvCK8aQ%40mail.gmail.com.

Chadwick Banning

unread,
Aug 20, 2016, 8:50:12 AM8/20/16
to Puppet Users
This is an issue I run into pretty regularly. If your Puppet infrastructure is even moderately complex, I'd recommend NOT equating a Puppet environment to an operational environment, operational environment being the groups of machines known as dev, qa, staging, etc.

For instance, in my infrastructure we have 50+ different operational environments. If I equate each one of these to a Puppet environment, I'd need 50+ branches. While doable, this immediately becomes a nightmare if I have a change that applies to all or some of the operational environments -- say, changing something in my base profile. Now I have to a) hope all 50+ branches are somewhat in sync, and b) merge my change into *each* branch 50+ times. If the branches aren't in sync at all I very well might end up having to fix unique conflicts each time I merge.

This is *not* a place where you want to end up.

Christopher Wood

unread,
Aug 20, 2016, 12:20:18 PM8/20/16
to puppet...@googlegroups.com
Lots about hiera data in this thread, how about modules? Having a common environment for the common modules and using basemodulepath helps some, but it's not everything.
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [1]puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> [2]https://groups.google.com/d/msgid/puppet-users/8abde89d-dfec-486e-b0ba-7ced6b6e07d8%40googlegroups.com.
> For more options, visit [3]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. mailto:puppet-users...@googlegroups.com
> 2. https://groups.google.com/d/msgid/puppet-users/8abde89d-dfec-486e-b0ba-7ced6b6e07d8%40googlegroups.com?utm_medium=email&utm_source=footer
> 3. https://groups.google.com/d/optout

Chadwick Banning

unread,
Aug 20, 2016, 1:52:49 PM8/20/16
to puppet...@googlegroups.com

Can you explain more? When you say "Having a common environment for the common modules" that sounds like you would need to apply multiple puppet environments to a node to get the full config...one "common" environment and one with "non-common" configuration...and I don't think this is currently possible?


>    email to [1]puppet-users+unsubscribe@googlegroups.com.

>    To view this discussion on the web visit
>    [2]https://groups.google.com/d/msgid/puppet-users/8abde89d-dfec-486e-b0ba-7ced6b6e07d8%40googlegroups.com.
>    For more options, visit [3]https://groups.google.com/d/optout.
>
> References
>
>    Visible links
>    1. mailto:puppet-users+unsub...@googlegroups.com

>    2. https://groups.google.com/d/msgid/puppet-users/8abde89d-dfec-486e-b0ba-7ced6b6e07d8%40googlegroups.com?utm_medium=email&utm_source=footer
>    3. https://groups.google.com/d/optout

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/4YL6D4wwJww/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/20160820161935.GB18127%40iniquitous.heresiarch.ca.

Alex Samad

unread,
Aug 20, 2016, 6:39:03 PM8/20/16
to puppet...@googlegroups.com
On 20 August 2016 at 22:50, Chadwick Banning <chadwic...@gmail.com> wrote:
> This is an issue I run into pretty regularly. If your Puppet infrastructure
> is even moderately complex, I'd recommend NOT equating a Puppet environment
> to an operational environment, operational environment being the groups of
> machines known as dev, qa, staging, etc.

But how to you stage a roll out of an update. If you want it to go to
dev then uat then prod ... or through some logical steps.

presuming you have a common profile used by all.

>
> For instance, in my infrastructure we have 50+ different operational
> environments. If I equate each one of these to a Puppet environment, I'd
> need 50+ branches. While doable, this immediately becomes a nightmare if I
> have a change that applies to all or some of the operational environments --
> say, changing something in my base profile. Now I have to a) hope all 50+
> branches are somewhat in sync, and b) merge my change into *each* branch 50+
> times. If the branches aren't in sync at all I very well might end up having
> to fix unique conflicts each time I merge.
>
> This is *not* a place where you want to end up.

Yes agree sounds like it would be a nightmare

>
[snip]

Chadwick Banning

unread,
Aug 20, 2016, 9:04:34 PM8/20/16
to Puppet Users
As Rob Nelson mentioned above, you can differentiate between operational environments in Hiera as long as you have the appropriate facts available.

If you differentiate Puppet environments and operational environments, then it's easier to address staged rollouts in each appropriate context. Staged rollouts of changes across *operational* environments may be done through Hiera, and staged rollouts of Puppet code (usually common Puppet code that cuts across operational environments) can be done through *Puppet* environments.

If your environment is simple enough...such as a single app with dev, staging, and production operational environments, then equating a Puppet environment to an operational environment is that much of an issue. For more complex Puppet setups, equating them always creates issues IMHO.

This topic is really interesting to me since I've run into it multiple times, the last being very recent.

Chadwick Banning

unread,
Aug 20, 2016, 9:06:05 PM8/20/16
to Puppet Users
"If your environment is simple enough...such as a single app with dev, staging, and production operational environments, then equating a Puppet environment to an operational environment is that much of an issue."

that should read *isn't* that much of an issue".

Alex Samad

unread,
Aug 21, 2016, 11:21:08 PM8/21/16
to puppet...@googlegroups.com
On 21 August 2016 at 11:04, Chadwick Banning <chadwic...@gmail.com> wrote:
> As Rob Nelson mentioned above, you can differentiate between operational
> environments in Hiera as long as you have the appropriate facts available.
>
> If you differentiate Puppet environments and operational environments, then
> it's easier to address staged rollouts in each appropriate context. Staged
> rollouts of changes across *operational* environments may be done through
> Hiera, and staged rollouts of Puppet code (usually common Puppet code that
> cuts across operational environments) can be done through *Puppet*
> environments.
>
> If your environment is simple enough...such as a single app with dev,
> staging, and production operational environments, then equating a Puppet
> environment to an operational environment is that much of an issue. For more
> complex Puppet setups, equating them always creates issues IMHO.


Okay I get it you have another branch in hiera lets say env as you have above.

But they pull in the same profile class - lets say openssh.

So lets say ssh come out with v3, you want to slowly roll out.

Do you have a bit if then or case switch statement in your ssh profile
class or do you create a new ssh class call ssh-v3 and then assign it
to only the env you want to.

seems like every time you want to make a change to a profile/class you
need to create another one so you can control the roll out or you have
a big if/then or switch case to do that for you.

trying to persist with the 2 branches production and testing for now ..
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/puppet-users/4YL6D4wwJww/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/272fba59-9684-44b1-8350-417893cbfb9e%40googlegroups.com.

Christopher Wood

unread,
Aug 22, 2016, 9:45:13 AM8/22/16
to puppet...@googlegroups.com
basemodulepath helps here:

https://docs.puppet.com/puppet/latest/reference/configuration.html#basemodulepath

In puppet.conf:

basemodulepath = /etc/puppetlabs/code/environments/common/modules

If there's something that environments don't need to track specially (ntp and mcollective modules come to mind), they can leave those out of their Puppetfile and the common versions will be used.

That common branch has a puppetfile and the hiera hierarchy has 'common/hieradata/common' at the bottom. It hasn't eliminated the issue of people's environments falling behind in specific areas. Whether that's an issue depends greatly how you handle updates without a pressing platform need. There are several positions on this matter held within this company so the hybrid approach seems to be the least-worst solution.

(You should probably pick a better solution.)

Editorially: As to whether it's more or less painful to have 50 individual operating environments I think we're in a bit of the same position. Organizationally entrenched silos and decades-old practices mean we're handling it in puppet instead of getting better answers to impertinent questions ("how do these silos benefit the company's bottom line?"). There's only so much curation we can automate away with that sort of headwind.

On Sat, Aug 20, 2016 at 01:52:31PM -0400, Chadwick Banning wrote:
> Can you explain more? When you say "Having a common environment for the
> common modules" that sounds like you would need to apply multiple puppet
> environments to a node to get the full config...one "common" environment
> and one with "non-common" configuration...and I don't think this is
> currently possible?
>
> On Aug 20, 2016 12:20 PM, "Christopher Wood"
> >    email to [1][2]puppet-users...@googlegroups.com.
> >    To view this discussion on the web visit
> >   
> [2][3]https://groups.google.com/d/msgid/puppet-users/8abde89d-dfec-486e-b0ba-7ced6b6e07d8%40googlegroups.com.
> >    For more options, visit [3][4]https://groups.google.com/d/optout.
> >
> > References
> >
> >    Visible links
> >    1. mailto:[5]puppet-users...@googlegroups.com
> >    2.
> [6]https://groups.google.com/d/msgid/puppet-users/8abde89d-dfec-486e-b0ba-7ced6b6e07d8%40googlegroups.com?utm_medium=email&utm_source=footer
> >    3. [7]https://groups.google.com/d/optout
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit
> [8]https://groups.google.com/d/topic/puppet-users/4YL6D4wwJww/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> [9]puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> [10]https://groups.google.com/d/msgid/puppet-users/20160820161935.GB18127%40iniquitous.heresiarch.ca.
> For more options, visit [11]https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [12]puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> [13]https://groups.google.com/d/msgid/puppet-users/CAPwwW9G9o58-U0EBzJZhrZm6gp1tX8_oX0C-VBANF%3DWZw7jJnQ%40mail.gmail.com.
> For more options, visit [14]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. mailto:christop...@pobox.com
> 2. mailto:puppet-users%2Bunsu...@googlegroups.com
> 3. https://groups.google.com/d/msgid/puppet-users/8abde89d-dfec-486e-b0ba-7ced6b6e07d8%40googlegroups.com
> 4. https://groups.google.com/d/optout
> 5. mailto:puppet-users%2Bunsu...@googlegroups.com
> 6. https://groups.google.com/d/msgid/puppet-users/8abde89d-dfec-486e-b0ba-7ced6b6e07d8%40googlegroups.com?utm_medium=email&utm_source=footer
> 7. https://groups.google.com/d/optout
> 8. https://groups.google.com/d/topic/puppet-users/4YL6D4wwJww/unsubscribe
> 9. mailto:puppet-users%2Bunsu...@googlegroups.com
> 10. https://groups.google.com/d/msgid/puppet-users/20160820161935.GB18127%40iniquitous.heresiarch.ca
> 11. https://groups.google.com/d/optout
> 12. mailto:puppet-users...@googlegroups.com
> 13. https://groups.google.com/d/msgid/puppet-users/CAPwwW9G9o58-U0EBzJZhrZm6gp1tX8_oX0C-VBANF%3DWZw7jJnQ%40mail.gmail.com?utm_medium=email&utm_source=footer
> 14. https://groups.google.com/d/optout

Rob Nelson

unread,
Aug 22, 2016, 10:06:55 AM8/22/16
to puppet...@googlegroups.com
I think there are so many ways to do A/B deployments that it's probably not wise to codify that into your Puppet environments layout. Here's a few examples of how:

* Use DNS/Load Balancer rules/etc to control which puppet masters different agents connect to. Some of the masters have a different default environment, to force some percentage of nodes to run against your feature branch and receive the correct environment. After the feature is merged into production, those masters default back to the `production` environment.
* Your ENC provides the puppet environment to check into, forcing some percentage of nodes to receive the `feature` environment, and later to return to the `production` environment.
* Use any orchestration solution (mcollective, ansible, vRO, etc) to configure agents to check in against the `feature` environment, and later to return to the `production` environment.
* Manually check in canary nodes against the `feature` environment, and later to return to the `production` environment.

That's just a sampling of methods in no particular order (although I think we can all agree that the manual option is the least appealing) but is by no means comprehensive. However, all of these methods let you do A/B testing between production and a specific feature branch, rather than against a long lived and divergent branch than production which may have numerous changes that cause interaction in unexpected ways, which certainly tests a lot more but makes it more difficult for diagnosing where failures come from. Having your feature branches diverge from `production`, your single source of truth, greatly limits the number of variables in play at any given time and streamlines your pipeline.

I would recommend a book on Continuous Integration and/or Continuous Delivery as these are subjects far too deep and wide to properly cover here in the detail they require.


> puppet-users+unsubscribe@googlegroups.com.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/272fba59-9684-44b1-8350-417893cbfb9e%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAJ%2BQ1PWS3LHBWE6Z-h8Newbo_R7qdh4qn-yADxqWMbJLv8Ga4Q%40mail.gmail.com.

Chadwick Banning

unread,
Aug 22, 2016, 10:16:18 AM8/22/16
to puppet...@googlegroups.com
It depends on what you mean by slowly rolling out the change. Do you control the openssh version globally (all machines get same version)? Or is the version set per *operational* environment (dev machines have newer version than prod machines, etc)?

If you set the version globally, and you want to control the rollout in some way, you'd do this via Puppet environments. If you control the version in individual operational environments, and you have facts that distinguish these, you'd use these facts with Hiera changes to control the rollout across different operational environments.

Again, this complexity only comes into play when you need to be able to stage rollouts across Puppet environments AND operational environments *independently*.

This is just *one* way to possibly handle this complexity. I certainly always have my eyes and ears open for better/simpler solutions.

On Sun, Aug 21, 2016 at 11:20 PM, Alex Samad <al...@samad.com.au> wrote:
> puppet-users+unsubscribe@googlegroups.com.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/272fba59-9684-44b1-8350-417893cbfb9e%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/4YL6D4wwJww/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAJ%2BQ1PWS3LHBWE6Z-h8Newbo_R7qdh4qn-yADxqWMbJLv8Ga4Q%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Chadwick Banning

Rich Burroughs

unread,
Aug 22, 2016, 12:30:39 PM8/22/16
to puppet...@googlegroups.com
I've managed a lot of configs for apps our developers build. At times when config changes were not backwards compatible I built in the equivalent of feature flags. I added a Boolean param that controlled if it was on or off, and built logic into the ERB/EPP template that used it. Then I could control the rollout by setting the value for that param in Hiera.

Not my favorite thing to do but it worked :) And of course the bad part about feature flags is remembering to clean them up when the rollout is done.

I'm really enjoying this discussion. I haven't done A/B or canary nodes with Puppet and I think some of this stuff is hard to figure out if you haven't seen it work.


Rich

> puppet-users...@googlegroups.com.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/272fba59-9684-44b1-8350-417893cbfb9e%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAJ%2BQ1PWS3LHBWE6Z-h8Newbo_R7qdh4qn-yADxqWMbJLv8Ga4Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAC76iT_1QCeuEMx0hdBf7QVfjYkBLctfsGGR8zbHAcV8%2B3ppMg%40mail.gmail.com.

Chadwick Banning

unread,
Aug 22, 2016, 12:49:34 PM8/22/16
to puppet...@googlegroups.com
Another potential management approach I'm exploring...

Because of the complexity of my current situation, where we have multiple independent internal apps and each one of these has a more-or-less unique set of operational environments, AND we have common Puppet code that applies across everything, I'm looking into spinning off each app team into their own module and using the Puppet 4 data-in-modules to allow them to manage their own data/hierarchy. I'd then use the controlrepo as the integration point for these internal modules as well as the container for the common Puppet code.

I'm just now exploring this idea so I'm not ready to say whether it's a good idea or not!

I like the idea of doing A/B testing on feature branches off of production. In my situation, I'd have to have a pretty robust and visible way to test and report success/fail before the security team buys into it. Especially if this was automated such that canarys are moved to feature environment, test is performed, and on success feature is merged and canarys are moved back to production environment. 


> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/272fba59-9684-44b1-8350-417893cbfb9e%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAC76iT_1QCeuEMx0hdBf7QVfjYkBLctfsGGR8zbHAcV8%2B3ppMg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/4YL6D4wwJww/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users+unsubscribe@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAPGcbtB%3Dj-nKJgEj5VQkt2a%2BJd-Uf1ePcXdse%2BV%3DdCAyh8YM3g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Chadwick Banning

Rob Nelson

unread,
Aug 22, 2016, 1:06:15 PM8/22/16
to puppet...@googlegroups.com
For canaries, we have some 'dev' versions of services, i.e. 'wiki' and 'wikidev'. Because of that, we can both leave the 'dev' versions on a feature branch if we need to (or disable puppet entirely when we're down in the nitty gritty and don't want puppet to undo development efforts) AND have a tier in hiera where we can provide different values to the regular and dev versions. We mostly rely on the hiera tier for data because then there's less manual or even orchestration effort to test on those nodes, except when we want a run immediately.


To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAPwwW9GJ-aF5SWB5_WCc-ess35RtnHqKWOsGUBMxehcFJs7j8Q%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages