Puppet directory environments mysteriously breaking -- help?

[Victor Danilchenko]

Hi all,

I am running a puppetmaster setup on CentOS. We have out production environment working fine. However, I am trying to migrate from puppet 3.4, and from config environments to directory environments. I got my environments tree all set up in the puppet master, but when I activate my directory environments, puppetmaster suddently can't see any modules or manifests -- it runs, but returns empty catalog for the agent, and 'puppet modules list' says I have no modules installed:

# puppet module list
/usr/share/puppet/modules (no modules installed)

None of the other global config files contain anything that references environments. My environment directory is set up correctly (it works on our production puppetmaster), and it works when I point the puppetmaster at it explicitly:

# puppet module --confdir=/etc/puppet/environments/production/modules:/usr/share/puppet/modules list
/etc/puppet/environments/production/modules

├── apache (???)
├── awstats (???)
etc.

Here's my main config file, stripped of comments:

[main]
    certname=kmadmin102
    dns_alt_names=kmadmin102.vistaprint.net,test-kmpuppet,test-kmpuppet.vistaprint.net,kmadmin102.vistaprint.net
    logdir = /var/log/puppet/
    rundir = /var/run/puppet
    ssldir = $vardir/ssl
    pluginsync = true
    certificate_revocation = false
    report = false
    configtimeout = 4m
    environmentpath = $confdir/environments
    default_manifest = $confdir/manifests
    basemodulepath = $confdir/modules:/usr/share/puppet/modules
[agent]
    classfile = $vardir/classes.txt
    localconfig = $vardir/localconfig
[master]

Can someone see what I am doing wrong?

I also have PuppetDB installed, but that shouldn't affect 'puppet modules list' command. Puppet fails the same way with or without PuppetDB.

[Thomas Müller]

have you configured the environment.conf file? https://docs.puppetlabs.com/puppet/latest/reference/config_file_environment.html

- Thomas

 

[Victor Danilchenko]

I was trying to avoid it, because we are also migrating to Git-centric workflow, and trying to set up per-branch environments -- it seems to me that the environment config files will be hard to keep straight under such setup.

Unless I add environment.conf to the .gitignore, and instantiate it automatically for each branch as it's created?.. Is that what I am supposed to do?

I am confused, because I was under impression that the per-environment modules and manifests were supposed to be processed automatically, just by the fact the myEnvironment/modules and myEnvironment/manifests directories existed, and per-directory environments were enabled. Am I wrong? Do I have to explicitly set those in environment.conf?

[jcbollinger]

On Monday, October 27, 2014 4:43:07 PM UTC-5, Victor Danilchenko wrote:

Hi all,

I am running a puppetmaster setup on CentOS. We have out production environment working fine. However, I am trying to migrate from puppet 3.4, and from config environments to directory environments. I got my environments tree all set up in the puppet master, but when I activate my directory environments, puppetmaster suddently can't see any modules or manifests -- it runs, but returns empty catalog for the agent, and 'puppet modules list' says I have no modules installed:

# puppet module list
/usr/share/puppet/modules (no modules installed)

This is with which version of Puppet?  Running as what user?

 

None of the other global config files contain anything that references environments. My environment directory is set up correctly (it works on our production puppetmaster), and it works when I point the puppetmaster at it explicitly:

# puppet module --confdir=/etc/puppet/environments/production/modules:/usr/share/puppet/modules list
/etc/puppet/environments/production/modules

├── apache (???)
├── awstats (???)
etc.

That command looks wrong.  Did you mean to type "--modulepath=[...]"?  Because it looks like a module path is what you have given, and having given an explicit modulepath I would certainly expect the tool to report the modules it finds in that path.  It appears that the correct confdir for you is /etc/puppet.  If you run puppet as a normal user, however, then (by default) it consults different configuration files than if you run it as root.  As a normal user, you probably do need to specify the correct config dir:

puppet module --confdir=/etc/puppet list

[...]

Can someone see what I am doing wrong?

Your config file looks fine to me, so probably you've run into an issue with the contents or layout of your environment directories.  In particular, if you want puppet to consult /etc/puppet/manifests as the main manifest directory for nodes in a given environment, then that environment's directory must not have its own 'manifests' subdirectory.  The behavior you describe could be explained by your environment directories having empty 'manifests' subdirectories.


John

[Tony Thayer]

What happens when you run

puppet module list --environment=production

?

Your configuration doesn't define which environment the puppet agent lives in, so it _should_default to production but I've been running into this kind of thing myself. This directory structure is working for me:

/etc/puppet
- /modules (root modules, stuff downloaded from puppet forge)
- /manifests (Empty on purpose. Directory environments define my nodes now.)
- /environments
  - /test01
  - /test02
    - /manifests
      - site.pp
    - /modules
      - /allmyenvironment-specific-modules
    - /hieradata
  - /production
    - /manifests
      - site.pp
    - /modules
      - /allmyenvironment-specific-modules
    - /hieradata

Specifying your default_manifest will override your directory environments as per the documentation. My puppet.conf has some lines like this in it to remind me:

[main]
    # The Puppet log directory.
    # The default value is '$vardir/log'.
    logdir = /var/log/puppet


    # Where Puppet PID files are kept.
    # The default value is '$vardir/run'.
    rundir = /var/run/puppet


    # Where SSL certificates are kept.
    # The default value is '$confdir/ssl'.
    ssldir = $vardir/ssl
    environmentpath = $confdir/environments
    basemodulepath = $confdir/modules
    # default manifest prevents the directory environment from taking
    # default_manifest = /etc/puppet/manifests
    # Disabling to enable environment-specific site.pp 10/20/2014 tthayer
    # disable_per_environment_manifest = true

- Tony

[Victor Danilchenko]

John,

Sorry, I should have been more thorough -- this is Puppet 3.7 master, running as root (so it has full access to everything).

Yes, 'puppet module --modulepath=/etc/puppet/environments/production/modules list' also works -- but I was under impression that supplying the environmentpath value in puppet.conf, in the [main] section, was sufficient. I can set the module.path explicitly in each environment.conf file, but I am hoping to avoid that.

Note BTW that calling 'puppet module --confdir=/etc/puppet list' does NOT work: is results in the same error as in the OP -- Puppet simply finds no modules.

My manifest directory is populated with a working manifest: this is the environment I simply copied from our working Puppet 3.4 master installation using explicitly configured environments. Both environments/production/manifests and environments/production/modules directories are properly populated (and yes, I need to use per-environment manifests, not the global one).

I know Puppet reads my environments directory, because supplying a non-existent environment causes it to throw an error. It simply doesn't read any modules or manifests from the environments unless I specify their paths explicitly.

Thanks for your help!

[Victor Danilchenko]

Tony,

When I explicitly specify --environment=production, Puppet breaks the same way as in the OP -- i.e. it simply fails to find the modules (and manifests). If I specify a non-existent environment, it throws the normal 'could not find directory environment' error.

So clearly Puppet reads my environments -- it just doesn't read anything within them, modules or manifests. It also doesn't read my environments/production/environment.conf file, which I just created:

[root@kmadmin102 puppet]# ls -l /etc/puppet/environments/production/
total 12
-rw-r--r--  1 root root   57 Oct 28 13:39 environment.conf
drwxr-xr-x  3 root root 4096 Oct 24 15:20 manifests
drwxr-xr-x 32 root root 4096 Oct 22 14:51 modules

[root@kmadmin102 puppet]# cat /etc/puppet/environments/production/environment.conf
modulepath = /etc/puppet/environments/production/modules

[root@kmadmin102 puppet]# ls -m -w 80 /etc/puppet/environments/production/modules

apache, apt, awstats, concat, elasticsearch, firewall, gcc, git, inifile, java,

km, logrotate, memcached, module_data, mysql, nodejs, perl, php, postgresql,

puppet-puppetdbquery, rclocal, redis, repoforge, rsync, src, stdlib, sudo,

sysctl, wget, xinetd

[root@kmadmin102 puppet]# puppet module list --environment=production
/usr/share/puppet/modules (no modules installed)

Thanks for the pointer about the default manifest, I didn't realize that it would actually override the environment ones, I assumed it worked additively, like modulepath.

I have disabled the default manifest, but my puppet is still not finding the per-environment modules...

[Tony Thayer]

What happens if you take the production/environment.conf file out of the equation? It will override any default values specified by puppet.conf. I have left it out of my configuration entirely.

[Victor Danilchenko]

I started out with no environment.conf file, I just added it today as a desperate bid for a work-around. Without environment.conf, it acts exactly as with environment.conf -- no modules found.

[jcbollinger]

On Tuesday, October 28, 2014 12:08:11 PM UTC-5, Tony Thayer wrote:

Specifying your default_manifest will override your directory environments as per the documentation.

That is NOT what the documentation says.  In fact, it is exactly contrary to the documentation:

The default_manifest setting allows you to omit the manifests directory in the environments you create [...].

(https://docs.puppetlabs.com/puppet/latest/reference/environments_configuring.html, emphasis added.)

Any environment that doesn’t set the manifest setting in its environment.conf file will use [the manifest specified by default_manifest].

(https://docs.puppetlabs.com/references/3.7.latest/configuration.html#defaultmanifest, emphasis added.)


John

[Tony Thayer]

You are correct. I'll update my comments then. 

By omitting both default_manifest and an environment.conf file, my puppet install is defaulting to the ./manifests folder in the environment being called: https://docs.puppetlabs.com/references/3.7.latest/configuration.html#defaultmanifest

[jcbollinger]

On Tuesday, October 28, 2014 12:32:48 PM UTC-5, Victor Danilchenko wrote:

John,

Sorry, I should have been more thorough -- this is Puppet 3.7 master, running as root (so it has full access to everything).

Actually, the master normally shouldn't run as root.  Whether it does depends on how you start it, but Puppet packages I've seen typically set up the master to run under a for-purpose service account, typically named "puppet".  The master has no need for privilege to perform its work, so good security practices dictate that it not have any.  (The agent is a different matter.)

Regardless of the user id of the puppetmaster process, there might be an access control issue involved.  Even root is affected by some forms of access control (SELinux policy, server-enforced policy on remote network directories).

Also, since you said you were upgrading, can you be sure that you don't have any bits lying around from an older Puppet version (or even a whole copy of another Puppet version)?  That might happen if you performed multiple installs, such as a source install and an RPM / DEB install (and maybe even a gem install, too).  It can happen in such cases that Puppet loads a mixture of bits from different versions, and then all manner of strange behavior can occur.
 

Yes, 'puppet module --modulepath=/etc/puppet/environments/production/modules list' also works -- but I was under impression that supplying the environmentpath value in puppet.conf, in the [main] section, was sufficient.

I think you have the right expectation, provided that Puppet is in fact reading the config file you think it's reading.  My point was that your code excerpt specified a --confdir that appeared to in fact be a modulepath.

 

I can set the module.path explicitly in each environment.conf file, but I am hoping to avoid that.

As indeed you should be able to do.

 

Note BTW that calling 'puppet module --confdir=/etc/puppet list' does NOT work: is results in the same error as in the OP -- Puppet simply finds no modules.

Fair enough.  I'm still having difficulty spotting any problem in your config file, so I still think the problem is likely related to your directory layout, contents, or access controls.  Do satisfy yourself about the integrity of your Puppet install, but that's a bit of a long shot.

 

My manifest directory is populated with a working manifest: this is the environment I simply copied from our working Puppet 3.4 master installation using explicitly configured environments. Both environments/production/manifests and environments/production/modules directories are properly populated (and yes, I need to use per-environment manifests, not the global one).

That's more or less the way I would have proceeded myself, but if you can "[copy] from [your] working Puppet 3.4 master installation" then does that mean that you have Puppet 3.4 and Puppet 3.7 installed at the same time on the same machine?  If so, that's an uncertain proposition.  I would recommend first upgrading from 3.4 to 3.7 (keeping your existing config file environments), then, separately, moving to directory environments.


John

[Victor Danilchenko]

So, I just started randomly twiddling knobs, and I realized the 'puppet module list' starts returning proper values once I comment out the 'certname' setting in /etc/puppet/puppet.conf:

certname=kmadmin102.vistaprint.net

(I had unqualified name there before, but whether unqualified or FQDN, it makes no difference). Note that my certname is set correctly, and matches the certificate Puppetmaster is using:

# openssl s_client -connect test-kmpuppet:8140 </dev/null 2>&1 | openssl x509 -noout -issuer -subject -dates

issuer= /CN=Puppet CA: kmadmin102

subject= /CN=kmadmin102.vistaprint.net

notBefore=Oct 27 20:20:30 2014 GMT

notAfter=Oct 27 20:20:30 2019 GMT

Plus, the certname value should only affect communications with agent -- it should have absolutely nothing to do with the local operation like 'puppet module list', right?

[Victor Danilchenko]

My upgrade is actually a clean install on a new system -- I am, of course, not touching the production puppetmaster while I am putzing around with the upgrade. In the process of installing Puppet 3.7, I blow away all the RPMs and directories, and install Puppet from scratch (I have a script that's supposed to bootstrap it).

Anyway, see my post above -- apparently it was the 'certname' value that was causing 'puppet module list' to fail, which is exceedingly weird.

[jcbollinger]

This does smell fishy.  I could imagine that absent an explicit modulepath, 'puppet module list' uses the certname to determine which environment's modulepath to use, but it does not then make sense that removing the setting altogether changes the results, especially if the specified certname is the one that Puppet would use by default anyway.

Without the certname specified in your config file, does Puppet successfully serve correct manifests?  Including to the master itself?

Also does it make a difference in which section you put the 'certname' setting?  It makes sense for the master to put it in the '[main]' section, but does it work differently if instead you put it in both the '[master]' and the '[agent]' section?

As you probably recognize, I'm now for the most part probing the shape of what appears to me to be a bug.  I do hope, however, that some of these questions will help you find an acceptable workaround for the overall problem (inasmuch as I assume you had a 'certname' setting in your configuration for good reason).  I also hope that once the nature of the issue is a bit more clearly defined, you will consider filing a bug report about it.


John

[jcbollinger]

On Wednesday, October 29, 2014 8:31:12 AM UTC-5, jcbollinger wrote:

Without the certname specified in your config file, does Puppet successfully serve correct manifests?

I meant "serve correct catalogs," of course.


John

[Johan De Wit]

For the record,

Got the same error,

I solved it by adding the following to my puppet.conf :

server = <fqdn puppet server>

De default (puppet) was not resolvable in my environment

Grts