Bootstrapping Puppet, both Agent and Master

[Daniel Schaefer]

So, I've revisited my entire Puppet bootstrapping routine recently and I have come across some minor and major flaws. I have essentially created a Bash bootstrapping script for the Agent. During which they check for hostname and domain name integrity, the ability to ping the Master, install the Puppetlabs repo, install the Puppet agent, stop the service, run the agent for the first time, prompt to sign the cert on the master, and eventually run the agent again to commit the changes to the node.

So far this has been working great. I've had to modify the script a few times to produce the correct result (for multiple environments), especially if the hostname is misconfigured.

The issue I'm running into now is that I'm now using the example42/puppet-puppet module to configure the nodes to use Puppet. This seems to work okay with the Agents. But I'm having some trouble with the Puppet Master.

So let's assume that you have a completely functional Puppet configuration stored in a GitHub repo, for example. Your Puppet Master node crashes and you need to rebuild it (or rebuild it for fun). My Puppet Master configuration is the entire /etc/puppet directory, including the puppet.conf, auth.conf, fileserver.conf, nodes.pp, site.pp, etc for the master. I'm running into the issue where I need the correct conf files in place before running the agent on the master. I can't do that without first having the conf files generated from the puppet module and committed in my Puppet Master repo and cloned/pulled into /etc/puppet. I'm running into a "chicken before the egg" scenario.

My question to you all is, assuming the Master config repo, how do you, personally, configure the Master's Puppet config with the said repo, before actually using the repo as the Master's config? If not in /etc/puppet, where do you store the Master repo and what initial changes do you make to the config for this to work?

Let me know if you have any questions or if anything needs cleared up. Thanks in advance.

-Dan

[David Vestal]

Hi Dan,

There is not a lot that you can do about that situation, that I'm aware of. You do have a "chicken before the egg" situation in that scenario.

The primary way that I have approached this is to go ahead and go down the road of scaling the puppet master. Once you have multiple machines behind the actual "puppet master" the likelihood of having to bootstrap the puppet master decreases significantly. That obviously incurs the additional cost of the additional machine(s).

If you, for whatever reason, choose to not go that route yet my suggestion would be to still use that bootstrap script. By having the rest of the machines bootstrap from the puppet master you can narrow the bootstrap script usage to the puppet master itself. That would then eliminate the majority of its complications.

Even still in my case, of having multiple machines as the puppet master, you still have to account for having the ENTIRE environment dying. To cover that aspect of a disaster recovery situation you're back at the bootstrap script anyway. At that point though you only have to bootstrap the one/two/three machines and then you can bring your entire environment back up normally.

--
David Vestal

> --
> You received this message because you are subscribed to the Google Groups "Example42 Puppet Modules" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to example42-puppet-m...@googlegroups.com.
> To post to this group, send email to example42-pu...@googlegroups.com.
> Visit this group at http://groups.google.com/group/example42-puppet-modules.
> For more options, visit https://groups.google.com/groups/opt_out.

[Guillaume Lederrey]

Hello !

It is actually not that hard. If you use r10k (https://github.com/adrienthebo/r10k), you can write a simple script that will download all you puppet modules and then use "puppet apply" to configure your puppetmaster the first time. You can have a look at my solution (https://github.com/gehel/vagrant-vms/blob/master/vagrant-puppetmaster/init.sh) for some inspiration.

My solution depends also on eyaml. So the private eyaml keys are not in the script above, nor is the SSH private key used to connect to Github. You will need to replace them by your own and modify my hiera data to re-encrypt whatever is encrypted.

Regards,

   Guillaume

--
mobile : +41 76 573 32 40
skype : Guillaume.Lederrey

projects :
* http://rwanda.ledcom.ch/
* http://trock.ch/
others :
* http://kiva.org/

[David Vestal]

Hi Guilllaume,

What do you do about the necessary bootstrapping to the point of being able to use r10k?

My recovery bootstrap script uses Librarian-puppet to install the modules and things so from that perspective it looks to be fairly similar to using r10k. I sat in Adrien's talk on r10k at PuppetConf last year and opted to continue using my own script and Librarian-puppet since I didn't see much more of a benefit.

Is there something else with r10k that I'm missing?

--
David Vestal

[Guillaume Lederrey]

I dont have any experience with puppet-librarian, but from my understanding, it is fairly similar to r10k. I'm not sure I understand what is your problem with bootstrapping your puppet master. Basically, here is what my bootstrapping script does:

* deploy r10k config file

* deploy eyaml private key

* deploy SSH private key (to download modules from GitHub)

* install puppet, r10k and eyaml (with apt and gem)

* execute puppet apply

At this point, my puppet master is configured.

The limitations at this point are :

* the puppet CA keys are not deployed by puppet, so they are re-generated. All clients must submit certificate requests again

* the state of the puppet DB is not externalized, so it is lost (not that I care all that much, and it would be fairly easy to use an external database)

[David Vestal]

Ok. Your process looks to be identical to mine then. Just a different tool, r10k instead of librarian-puppet.

--
David Vestal

[Guillaume Lederrey]

I'm still curious, what exactly is the problem you have with your bootstrapping of your puppet master ? I feel like I am probably missing something in my own process...

[David Vestal]

I don't have a problem with it. I misunderstood a part of your description. I thought that you meant that you were using r10k for the entire bootstrap process. However, as you noted, you still have to deploy the config file, private keys, Puppet, r10k, etc. and then execute Puppet. You noted that's what you're bootstrapping script does.

My bootstrapping script does the same thing. I just thought that you meant that you weren't using a script to bootstrap it to that point.

--
David Vestal

[Guillaume Lederrey]

Ok, thanks for the explanation ! 

[David Schmitt]

On 2014-02-23 02:27, Daniel Schaefer wrote:
> So let's assume that you have a completely functional Puppet
> configuration stored in a GitHub repo, for example. Your Puppet Master
> node crashes and you need to rebuild it (or rebuild it for fun). My
> Puppet Master configuration is the entire /etc/puppet directory,
> including the puppet.conf, auth.conf, fileserver.conf, nodes.pp,
> site.pp, etc for the master. I'm running into the issue where I need the
> correct conf files in place before running the agent on the master. I
> can't do that without first having the conf files generated from the
> puppet module and committed in my Puppet Master repo and cloned/pulled
> into /etc/puppet. I'm running into a "chicken before the egg" scenario.
>
> My question to you all is, assuming the Master config repo, how do you,
> personally, configure the Master's Puppet config with the said repo,
> before actually using the repo as the Master's config? If not in
> /etc/puppet, where do you store the Master repo and what initial changes
> do you make to the config for this to work?

To say it bluntly, puppet is not a disaster reovery tool. You'll also
need the private keys for the CA and I well hope that you do not have
THEM in a git repo somewhere.

For the rest of bootstrapping, take a look at my vagrant config on
github, where I regularily pull up new puppet masters in vboxes:

https://github.com/DavidS/dasz-configuration/blob/master/Vagrantfile



Regards, David