How to manage a lab with puppet? (how does puppet scale?)

[Nic Scott]

I'm evaluating puppet to see if it can work in our environment and I have to admit the the learning cure with the puppet "terms" are giving me issues. I keep reading documentation into circles. I'm familiar with python, bash scripting, and use munki in my labs, but I'm stilling trying to understand manifest, modules, classes, etc.


What I have is a Master and a 1 node setup. They are talking and the Puppet Master is pushing configurations to the node. That's perfect. I handle this by having two manifest on the Master.  A puppet_client_1.pp and my site.pp. My site.pp looks like this:

import "puppet_client_1"


Next step ... manage two nodes. I have this working by creating a new .pp file called puppet_client_2.pp. I then updated my site.pp to include the second nodes manifest.

import "puppet_client_1"
import "puppet_client_2"



My question is ... is this the best practice to manage multiple nodes? What if I have a lab of 20 machines and I want the same configuration on all 20? Can I do a nested manifest somehow, or do I have to create a separate manifest for each node and then copy and paste my configuration into each manifest?

That seems like a lot of work to manage hundreds of nodes. I have to believe puppet scales better then that, but I've having a hard time finding examples.

Can anyone share an example of how they are managing multiple nodes?  Perhaps point me to an online resource or documentation?

[Doug Forster]

The idea is that you setup a class for each node function. If you have a bunch of nodes that need this one program create a class for it and add it to their profile. Some people actually use an External Node Classifier like Hiera, Puppet enterprise console, or forman. Some people find it helpful to setup a custom fact to determine what "roles and profiles" it gets.

Puppet does indeed scale if you listen to one of the recent puppet podcasts you will know Cern manages many computers in a lab environment with puppet.

Here is a puppet vid on the reasons and how to get your applications into a role and profile pattern. This will probably be one of the better places to start.
https://puppetlabs.com/presentations/designing-puppet-rolesprofiles-pattern

Forman Info

Hope this helps.

Doug Forster

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/5f21984a-dc7e-45d2-880c-adb24400361f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[jcbollinger]

On Monday, October 27, 2014 10:46:39 AM UTC-5, Nic Scott wrote:

I'm evaluating puppet to see if it can work in our environment and I have to admit the the learning cure with the puppet "terms" are giving me issues. I keep reading documentation into circles. I'm familiar with python, bash scripting, and use munki in my labs, but I'm stilling trying to understand manifest, modules, classes, etc.

Puppet jargon primer

Manifest = Puppet source file (with .pp extension).

Catalog = the master's collective description of all the machine state that should be enforced on one target machine (node), as evaluated at one point in time, based on the target machine's state as evaluated prior to catalog generation.

Resource = a physical resource is more or less any one 'thing' on the target machine that is subject to management as a unit -- a file, a system service, etc..  More often, though, discussion of "resources" is directed at the Puppet representation of configuration specifications for physical resources -- declarations in one manifest or another such as

  file { '/tmp/foo': ensure => present }

and the internal representation of the same.  Resources have 'types' (File, Service, etc), 'properties' representing persistent characteristics of the corresponding physical resource that Puppet can manage, and non-persistent 'parameters' influencing how Puppet manages the resource.

Class = abstractly, a categorization of some target nodes, synonymous in that sense with "type" or "kind".  Not to be interpreted in the more specific sense the term is used in some object oriented programming languages.  Concretely, classes' principal purpose is to organize resources into named groups.  In reality, it is sometimes hard to see how the concrete use of classes fits the abstract definition.

Defined Type = a resource type analogous to File, etc., defined in a manifest via the Puppet language.  Their representation is similar to classes', but the two constructs have important differences.

Module = a (more or less) self-contained collection of manifests, often various kinds of data, and sometimes custom plugins.  These normally focus on providing for management of a specific subsystem (e.g. firewall, web server), but sometimes they provide a lower-level general facility such as general-purpose extension functions (puppet::stdlib) or custom resource types (puppet::concat).  It is intended and hoped that modules be reusable in different sites and environments, which is more successful for some modules than for others.

 

What I have is a Master and a 1 node setup. They are talking and the Puppet Master is pushing configurations to the node. That's perfect. I handle this by having two manifest on the Master.  A puppet_client_1.pp and my site.pp. My site.pp looks like this:

import "puppet_client_1"


Next step ... manage two nodes. I have this working by creating a new .pp file called puppet_client_2.pp. I then updated my site.pp to include the second nodes manifest.

import "puppet_client_1"
import "puppet_client_2"



My question is ... is this the best practice to manage multiple nodes? What if I have a lab of 20 machines and I want the same configuration on all 20? Can I do a nested manifest somehow, or do I have to create a separate manifest for each node and then copy and paste my configuration into each manifest?

You have multiple options here, some of which you can mix and match.  Here are many of the more popular:

1. If most or all of your machines are intended to be configured the same way then you can describe that configuration in a node block labeled "default".  Any machine for which no better-matching node block is available will use the default node block.
2. You can declare the same configuration for multiple nodes by using a single node block with a comma-delimited list of node names.
3. You can use regular expressions to match node identifiers to node blocks.
4. You can use an "external node classifier" (commonly referred to via its initialism, "ENC"), an external program provided by you(*) that specifies global variables and classes for a node based on its identifier.
5. You can use external data, typically accessed via Puppet's external data subsystem "Hiera", to designate classes for nodes or groups of nodes, to assign class parameter values, and generally to provide data that can be customized based on an hierarchical scheme that you choose.
   

(*) Although I say the ENC approach requires a program provided by "you," that doesn't necessarily mean you have to write it from scratch.  The ENC in fact serves as a flexible extension point; it is leveraged by several different GUI configuration interfaces that might (or might not) appeal to you.  Among those is the rather comprehensive GUI integrated into PL's for-fee Enterprise product, but there are open source alternatives as well.

Also, recent version of Puppet support a site manifest directory, whose purpose is to relieve you of having to write all those "import" statements.  Instead, Puppet will automatically load all manifests from a specified directory for use as its starting point.  With older versions of Puppet, it is common to record node blocks in separate manifests in a directory such as manifests/nodes/, and to import them all via a single

import "nodes/*.pp"

line in manifests/site.pp.

 

That seems like a lot of work to manage hundreds of nodes. I have to believe puppet scales better then that, but I've having a hard time finding examples.

Puppet does scale better than that, but details of how to achieve it vary somewhat with your specific requirements.

 

Can anyone share an example of how they are managing multiple nodes?  Perhaps point me to an online resource or documentation?

The Puppet Reference Manual is quite good.  In particular, it contains everything I said about node blocks, in more detail.  It also describes the external node classifier form and function.  Of course, everything I put in my little primer can also be found there.

One thing that's hard to find in the reference manual is any documentation for Hiera.  Those docs are in a separate manual, and the main reference manual seems to have only one or two links to it.


HTH,

John