Puppet environments and "avalability zones"

[Iakov Gan]

Hi, 

I wonder are there any best practices for deployment of changes in a large puppet environment?

Once we change puppet code all changes are applied within 30 minutes on an environment of several thousand hosts across multiple geographical zones. An automation power in its scaring beauty.

Surely it is tested in preproduction and dev envs before, but still, there is a risk when applying it on all zones almost the same time. 

What I look for is to deploy changes progressively per equivalent of Amazon's Availability Zone via a pipeline, in order to control the propagation of system changes.  Surely it will make a change slower but it is a price to pay. 

Any best practice about it? 

Best regards,

Iakov 

[Christopher Wood]

Not sure if you'd call it a "best" practice, but with a fairly standard
control repo(1) and r10k'ish environments you can:

* branch
* make your changes in the new branch/environment
* move a few canary hosts into the new environment using your ENC(2),
see how that goes
* move a few more, see if you are comfortable with the change
* repeat until you're comfortable
* merge the control repo branch and move hosts into the live
environment with your ENC

Another alternative, that I have never used, is to temporarily
configure your environment for slower agent run cycling, say hours
instead of 30 minutes, that would slow things right down.

Similarly to option one, you could do the change in two phases:

* one change pushes up the code to make the change, but turned off
behind a feature flag
* another set of actions activates the change in different parts of
your infrastructure (probably using some hiera key)

It's late and Friday, I'm probably out of ideas by now.


(1) https://github.com/puppetlabs/control-repo

(2) https://puppet.com/docs/puppet/5.5/nodes_external.html

> -- 
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/puppet-users/18f24142-226b-4f10-98a5-
> 6db992f4a005%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.