puppet 4 environments git feedback?

[dkoleary]

Hey;

To put this in perspective, I'm a sysadmin, not a developer.  While I've used git for a couple of years, until today, I could easily count the number of times I issued a 'git branch' command.

I'm practicing setting up a new puppet 4 server and, after some research, I've got various environments under git management and have successfully 'promoted code' from test through production.  It's a wee bit tedious but I'm sure I could get used to it.

What I'm hoping is to have someone more familiar the process verify I'm doing it somewhat close to right and/or make suggestions on an improvements.  I have heard of r10k; however, I'm one of those that has to know what's going on under the covers.  Up until now, r10k has been of of those 'developer' things.  Once I run through this a few times, *then* I'll start playing with r10k.

So, bit of a build up.  Here's what I have:

one git repo covering all puppet environments thusly::

# git branch -r

  origin/HEAD -> origin/master

  origin/dev

  origin/master

  origin/prod

  origin/qa

  origin/test

  origin/uat

On a different system, one pulls the test system, develops code, commits, pushes, etc.  In the test environment, the admin pulls the updated work and tests

# pwd

/etc/puppetlabs/code/environments/test

# git branch

* test

once the tests are complete, a responsible admin accesses the dev environment and executes:

git checkout test 

git pull # if necessary

git checkout dev

git merge test

Process iterates through the environments to prod.  

# git log --oneline

e298de7 prod.rst: mved from uat

07f3ab1 uat: merged from qa

a20a85c qa: mved from dev

2f644f2 dev: renamed from test

c8c067b test: added

a432124 puppet production environment initial check in

Thanks for any hints/tips/suggestion.  

Doug O'Leary

[Rob Nelson]

Doug,

If all your git repositories are local filestores, that's probably a pretty reasonable workflow. However, most people use some form of dedicated service as their git origins, that reside external to the local systems - GitHub, Bit Bucket, Git Lab, etc. If you are using one of those systems, or you can migrate to it, you can then improve your code flow by using pull requests to review changes and merge them into the branch at the upstream. These services would also be able to fire a web hook, an event sent to your puppet master as an http/https payload, that can trigger r10k to deploy the updated code.

That might sound complicated, but I promise you, it's not. It's just using some peculiar terminology you're not familiar with. For learning git and services like GitHub, there are countless tutorials out there; I recommend https://github.com/commitmas/12-days-of-commitmas. That will introduce you to git, GitHub, Pull Requests (PRs), and code review processes. I wrote an article on using r10k with a webhook at https://rnelson0.com/2015/05/03/configuring-an-r10k-webhook-on-your-puppet-master/, and I and others have tons of articles about using r10k. With this in place, your process would be a little simpler:

  git commit -am 'Something I want to push to test'

  git push origin test_change

  <open a PR from test_change to test, merge>

The webhook fires, r10k starts deploying code, and in a few seconds to minutes, your test environment has been updated to incorporate those changes. You'd then use PRs to promote code from test -> dev -> qa -> uat -> production. This has other impacts to your workflow, of course, and you may actually be able to remove a level of environments (feature -> qa -> uat -> production). You could later add some continuous integration tests to your code, that are automatically run by GitHub/GitLab/Jenkins/etc, which could lead to removing the qa level as well. But that's down the road a bit.

There is a lot of room for improvement here as you have time to focus on your pipeline.

Rob Nelson
rnel...@gmail.com

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/729cd1f2-2828-4d46-b008-2e0033fed34d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Christopher Wood]

On Thu, May 19, 2016 at 11:39:30AM -0400, Rob Nelson wrote:
> Doug,
>
> If all your git repositories are local filestores, that's probably a
> pretty reasonable workflow. However, most people use some form of
> dedicated service as their git origins, that reside external to the local
> systems - GitHub, Bit Bucket, Git Lab, etc. If you are using one of those
> systems, or you can migrate to it, you can then improve your code flow by
> using pull requests to review changes and merge them into the branch at
> the upstream. These services would also be able to fire a web hook, an
> event sent to your puppet master as an http/https payload, that can
> trigger r10k to deploy the updated code.

Our inter-datacenter connectivity tends to skip a beat whenever there's a DDOS. It's old skule but a cron job gives us some easy resilience here.

[root@puppetmaster6 ~]# crontab -l | grep r10k
# Puppet Name: r10k deploy
* * * * * /usr/bin/lockrun --lockfile=/var/run/r10k-deploy -- /opt/puppetlabs/puppet/bin/r10k deploy environment --puppetfile >/dev/null 2>&1

Also when I set this up there were issues setting up plain git hooks with the gitlab instance we were originally using, using a cron job saved weeks of inter-team ticket discussion judging by other hook tickets.

> That might sound complicated, but I promise you, it's not. It's just using
> some peculiar terminology you're not familiar with. For learning git and
> services like GitHub, there are countless tutorials out there; I recommend

> [1]https://github.com/commitmas/12-days-of-commitmas. That will introduce

> you to git, GitHub, Pull Requests (PRs), and code review processes. I
> wrote an article on using r10k with a webhook at

> [2]https://rnelson0.com/2015/05/03/configuring-an-r10k-webhook-on-your-puppet-master/,

> and I and others have tons of articles about using r10k. With this in
> place, your process would be a little simpler:
>
>   git commit -am 'Something I want to push to test'
>   git push origin test_change
>   <open a PR from test_change to test, merge>
>
> The webhook fires, r10k starts deploying code, and in a few seconds to
> minutes, your test environment has been updated to incorporate those
> changes. You'd then use PRs to promote code from test -> dev -> qa -> uat
> -> production. This has other impacts to your workflow, of course, and you
> may actually be able to remove a level of environments (feature -> qa ->
> uat -> production). You could later add some continuous integration tests
> to your code, that are automatically run by GitHub/GitLab/Jenkins/etc,
> which could lead to removing the qa level as well. But that's down the
> road a bit.
>
> There is a lot of room for improvement here as you have time to focus on
> your pipeline.
> Rob Nelson

> [3]rnel...@gmail.com

> an email to [5]puppet-users...@googlegroups.com.

> To view this discussion on the web visit

> [6]https://groups.google.com/d/msgid/puppet-users/729cd1f2-2828-4d46-b008-2e0033fed34d%40googlegroups.com.
> For more options, visit [7]https://groups.google.com/d/optout.

>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to [8]puppet-users...@googlegroups.com.

> To view this discussion on the web visit

> [9]https://groups.google.com/d/msgid/puppet-users/CAC76iT_LczDuGm1kdJ-bmm11sM2ib%2BNbaHSvqqUOc34jj5C%2BTQ%40mail.gmail.com.
> For more options, visit [10]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. https://github.com/commitmas/12-days-of-commitmas
> 2. https://rnelson0.com/2015/05/03/configuring-an-r10k-webhook-on-your-puppet-master/
> 3. mailto:rnel...@gmail.com
> 4. mailto:dkol...@olearycomputers.com
> 5. mailto:puppet-users...@googlegroups.com
> 6. https://groups.google.com/d/msgid/puppet-users/729cd1f2-2828-4d46-b008-2e0033fed34d%40googlegroups.com?utm_medium=email&utm_source=footer
> 7. https://groups.google.com/d/optout
> 8. mailto:puppet-users...@googlegroups.com
> 9. https://groups.google.com/d/msgid/puppet-users/CAC76iT_LczDuGm1kdJ-bmm11sM2ib%2BNbaHSvqqUOc34jj5C%2BTQ%40mail.gmail.com?utm_medium=email&utm_source=footer
> 10. https://groups.google.com/d/optout

[Rob Nelson]

We use a cron job in addition to git hooks. The hooks work 99% of the time, and the cron job handles the occasional miss because the network burped. They're very complimentary. 

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/20160519172802.GA8993%40iniquitous.heresiarch.ca.

For more options, visit https://groups.google.com/d/optout.

--

Rob Nelson
rnel...@gmail.com

[dkoleary]

Thanks, all, for the information.  I appreciate it.  

Doug O'Leary