Re: [EXTERNAL] - [Puppet Users] Puppet agent is not applying changes

[Dirk Heinrichs]

Am Donnerstag, den 26.09.2019, 06:20 -0700 schrieb Dan Crisp:

/etc/puppetlabs/code/environments/production:

total 20

-rw-r--r--. 1 root root  808 Sep 25 20:47 environment.conf

What does this one contain? I don't have these in my environments.

Bye...

Dirk

-- 

Dirk Heinrichs

Senior Systems Engineer, Delivery Pipeline

OpenText ™ Discovery | Recommind

Phone: +49 2226 15966 18

Email: dhei...@opentext.com

Website: www.recommind.de

Recommind GmbH, Von-Liebig-Straße 1, 53359 Rheinbach

Vertretungsberechtigte Geschäftsführer Gordon Davies, Madhu Ranganathan, Christian Waida, Registergericht Amtsgericht Bonn, Registernummer HRB 10646

This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.

[Dan Crisp]

This file does nothing.  All the lines therein are commented out.  In fact, I have in the past moved this file out of the way then put it back just to rule out it was doing anything weird.

Dan.

[Dirk Heinrichs]

Am Freitag, den 27.09.2019, 03:43 -0700 schrieb Dan Crisp:

This file does nothing.  All the lines therein are commented out.  In fact, I have in the past moved this file out of the way then put it back just to rule out it was doing anything weird.

OK, just wanted to make sure it doesn't contain any strange configuration for that environment. Not sure what else could be the problem. Could you run the agent with --debug and post the relevant lines involving your user resource (if any)?

[Dirk Heinrichs]

Am Freitag, den 27.09.2019, 11:50 +0000 schrieb Dirk Heinrichs:

Am Freitag, den 27.09.2019, 03:43 -0700 schrieb Dan Crisp:

This file does nothing.  All the lines therein are commented out.  In fact, I have in the past moved this file out of the way then put it back just to rule out it was doing anything weird.

OK, just wanted to make sure it doesn't contain any strange configuration for that environment. Not sure what else could be the problem. Could you run the agent with --debug and post the relevant lines involving your user resource (if any)?

Oh, and do you, by chance, have a file /etc/puppetlabs/code/manifests/site.pp? Mine has just the following content (might as well be empty):

# site.pp must exist (puppet #15106, foreman #1708)

Bye...

[Dan Crisp]

Please see below.  Apologies, there is a lot of detail here:

Debug: Using settings: adding file resource 'confdir': 'File[/etc/puppetlabs/puppet]{:path=>"/etc/puppetlabs/puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'codedir': 'File[/etc/puppetlabs/code]{:path=>"/etc/puppetlabs/code", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'vardir': 'File[/opt/puppetlabs/puppet/cache]{:path=>"/opt/puppetlabs/puppet/cache", :owner=>"root", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'logdir': 'File[/var/log/puppetlabs/puppet]{:path=>"/var/log/puppetlabs/puppet", :mode=>"750", :owner=>"root", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'statedir': 'File[/opt/puppetlabs/puppet/cache/state]{:path=>"/opt/puppetlabs/puppet/cache/state", :mode=>"1755", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'rundir': 'File[/var/run/puppetlabs]{:path=>"/var/run/puppetlabs", :mode=>"755", :owner=>"root", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'libdir': 'File[/opt/puppetlabs/puppet/cache/lib]{:path=>"/opt/puppetlabs/puppet/cache/lib", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'hiera_config': 'File[/etc/puppetlabs/puppet/hiera.yaml]{:path=>"/etc/puppetlabs/puppet/hiera.yaml", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'preview_outputdir': 'File[/opt/puppetlabs/puppet/cache/preview]{:path=>"/opt/puppetlabs/puppet/cache/preview", :mode=>"750", :owner=>"root", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'certdir': 'File[/etc/puppetlabs/puppet/ssl/certs]{:path=>"/etc/puppetlabs/puppet/ssl/certs", :mode=>"755", :owner=>"root", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'ssldir': 'File[/etc/puppetlabs/puppet/ssl]{:path=>"/etc/puppetlabs/puppet/ssl", :mode=>"771", :owner=>"root", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'publickeydir': 'File[/etc/puppetlabs/puppet/ssl/public_keys]{:path=>"/etc/puppetlabs/puppet/ssl/public_keys", :mode=>"755", :owner=>"root", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'requestdir': 'File[/etc/puppetlabs/puppet/ssl/certificate_requests]{:path=>"/etc/puppetlabs/puppet/ssl/certificate_requests", :mode=>"755", :owner=>"root", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'privatekeydir': 'File[/etc/puppetlabs/puppet/ssl/private_keys]{:path=>"/etc/puppetlabs/puppet/ssl/private_keys", :mode=>"750", :owner=>"root", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'privatedir': 'File[/etc/puppetlabs/puppet/ssl/private]{:path=>"/etc/puppetlabs/puppet/ssl/private", :mode=>"750", :owner=>"root", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'hostcert': 'File[/etc/puppetlabs/puppet/ssl/certs/lhcadvdeveye05.fixnetix.com.pem]{:path=>"/etc/puppetlabs/puppet/ssl/certs/lhcadvdeveye05.fixnetix.com.pem", :mode=>"644", :owner=>"root", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'hostprivkey': 'File[/etc/puppetlabs/puppet/ssl/private_keys/lhcadvdeveye05.fixnetix.com.pem]{:path=>"/etc/puppetlabs/puppet/ssl/private_keys/lhcadvdeveye05.fixnetix.com.pem", :mode=>"640", :owner=>"root", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'localcacert': 'File[/etc/puppetlabs/puppet/ssl/certs/ca.pem]{:path=>"/etc/puppetlabs/puppet/ssl/certs/ca.pem", :mode=>"644", :owner=>"root", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'hostcrl': 'File[/etc/puppetlabs/puppet/ssl/crl.pem]{:path=>"/etc/puppetlabs/puppet/ssl/crl.pem", :mode=>"644", :owner=>"root", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'statefile': 'File[/opt/puppetlabs/puppet/cache/state/state.yaml]{:path=>"/opt/puppetlabs/puppet/cache/state/state.yaml", :mode=>"660", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'transactionstorefile': 'File[/opt/puppetlabs/puppet/cache/state/transactionstore.yaml]{:path=>"/opt/puppetlabs/puppet/cache/state/transactionstore.yaml", :mode=>"660", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'clientyamldir': 'File[/opt/puppetlabs/puppet/cache/client_yaml]{:path=>"/opt/puppetlabs/puppet/cache/client_yaml", :mode=>"750", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'client_datadir': 'File[/opt/puppetlabs/puppet/cache/client_data]{:path=>"/opt/puppetlabs/puppet/cache/client_data", :mode=>"750", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'classfile': 'File[/opt/puppetlabs/puppet/cache/state/classes.txt]{:path=>"/opt/puppetlabs/puppet/cache/state/classes.txt", :mode=>"640", :owner=>"root", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'resourcefile': 'File[/opt/puppetlabs/puppet/cache/state/resources.txt]{:path=>"/opt/puppetlabs/puppet/cache/state/resources.txt", :mode=>"640", :owner=>"root", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'deviceconfdir': 'File[/etc/puppetlabs/puppet/devices]{:path=>"/etc/puppetlabs/puppet/devices", :mode=>"750", :owner=>"root", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'clientbucketdir': 'File[/opt/puppetlabs/puppet/cache/clientbucket]{:path=>"/opt/puppetlabs/puppet/cache/clientbucket", :mode=>"750", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'lastrunfile': 'File[/opt/puppetlabs/puppet/cache/state/last_run_summary.yaml]{:path=>"/opt/puppetlabs/puppet/cache/state/last_run_summary.yaml", :mode=>"644", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'lastrunreport': 'File[/opt/puppetlabs/puppet/cache/state/last_run_report.yaml]{:path=>"/opt/puppetlabs/puppet/cache/state/last_run_report.yaml", :mode=>"640", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'graphdir': 'File[/opt/puppetlabs/puppet/cache/state/graphs]{:path=>"/opt/puppetlabs/puppet/cache/state/graphs", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'pluginfactdest': 'File[/opt/puppetlabs/puppet/cache/facts.d]{:path=>"/opt/puppetlabs/puppet/cache/facts.d", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: Using settings: adding file resource 'localedest': 'File[/opt/puppetlabs/puppet/cache/locales]{:path=>"/opt/puppetlabs/puppet/cache/locales", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'

Debug: /File[/opt/puppetlabs/puppet/cache/state/resources.txt]/seluser: Found seluser default 'system_u' for /opt/puppetlabs/puppet/cache/state/resources.txt

Debug: /File[/opt/puppetlabs/puppet/cache/state/resources.txt]/selrole: Found selrole default 'object_r' for /opt/puppetlabs/puppet/cache/state/resources.txt

Debug: /File[/opt/puppetlabs/puppet/cache/state/resources.txt]/seltype: Found seltype default 'usr_t' for /opt/puppetlabs/puppet/cache/state/resources.txt

Debug: /File[/opt/puppetlabs/puppet/cache/state/resources.txt]/selrange: Found selrange default 's0' for /opt/puppetlabs/puppet/cache/state/resources.txt

Debug: /File[/opt/puppetlabs/puppet/cache/state/resources.txt]: Adding autorequire relationship with File[/opt/puppetlabs/puppet/cache/state]

On Friday, September 27, 2019 at 10:58:49 AM UTC+1, Dirk Heinrichs wrote:

[Dan Crisp]

No.  I only have /etc/puppetlabs/code/environments/production/manifests/site.pp

[jcbollinger]

On Friday, September 27, 2019 at 7:20:51 AM UTC-5, Dan Crisp wrote:

Please see below.  Apologies, there is a lot of detail here:

Debug: Using settings: adding file resource 'confdir': 'File[/etc/puppetlabs/puppet]{:path=>"/etc/puppetlabs/puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'

[...]

If the (elided) log messages presented were all the log messages emitted, then they depict the agent applying an empty catalog, which is of course consistent with not changing anything.  All the resources shown are generated locally by the agent.  You should be able to confirm that by looking at the catalog itself, which you will find, by default, in a file in /opt/puppetlabs/puppet/cache/client_data/catalog.

If you're making changes to your manifest set but not seeing any effect at the agent then there are several possibilities, but the most likely issue is server-side caching.  Before tweaking the cache configuration, however, the easiest way to test this hypothesis is to flush the cache by restarting the puppetserver service on the master.  (That's not the only way, but it's quick and easy, and you don't need to learn anything new to do it.)

If that indeed solves the problem then you'll want to adjust the environment_timeout configuration setting on the master.  For the time being, I would suggest setting it to 0 to disable caching altogether.  This is also supposed to be the default if the setting is not explicitly specified, however.

---

If that doesn't turn out to be the issue, then do have a look at the master's logs.  You should confirm that it is logging catalog requests from the agent in question (else they must be going to a different master), and you should look for any messages providing a clue about the issue.  It may be helpful to turn up puppetserver's log level to get more detailed information.

If that's also unavailing then my last suggestion would be to confirm that the puppetserver process can successfully access everything in the environment directory.  Check file ownership, mode, ACLs, SELinux context, and anything else that affects whether the puppetserver can read the files and traverse (all) the directories.  I would pay special attention to your one manifest file, because that's the most likely one to be messed up in this regard.

John

[Dan Crisp]

Thanks for the reply John,

The issue still persists unfortunately.

I've ensured that Selinux isn't enforcing on both the server side and client and then restarted the Puppet service on the master server.  The server logs whilst running the agent read as follows:

10.20.25.83 - - - 27/Sep/2019:16:06:43 +0000 "GET /puppet/v3/node/lhcadvdeveye05.fixnetix.com?environment=production&transaction_uuid=4d2c88b1-2aec-45a5-bc7e-407c2ad8229e&fail_on_404=true HTTP/1.1" 200 13535 10.20.25.83 10.20.25.83 8140 109

10.20.25.83 - - - 27/Sep/2019:16:06:43 +0000 "GET /puppet/v3/file_metadatas/pluginfacts?environment=production&links=follow&recurse=true&source_permissions=use&ignore=.svn&ignore=CVS&ignore=.git&ignore=.hg&checksum_type=md5 HTTP/1.1" 200 220 10.20.25.83 10.20.25.83 8140 25

10.20.25.83 - - - 27/Sep/2019:16:06:43 +0000 "GET /puppet/v3/file_metadatas/plugins?environment=production&links=follow&recurse=true&source_permissions=ignore&ignore=.svn&ignore=CVS&ignore=.git&ignore=.hg&checksum_type=md5 HTTP/1.1" 200 224 10.20.25.83 10.20.25.83 8140 16

10.20.25.83 - - - 27/Sep/2019:16:06:44 +0000 "GET /puppet/v3/file_metadatas/locales?environment=production&links=follow&recurse=true&source_permissions=ignore&ignore=.svn&ignore=CVS&ignore=.git&ignore=.hg&ignore=%2A.pot&ignore=config.yaml&checksum_type=md5 HTTP/1.1" 200 224 10.20.25.83 10.20.25.83 8140 20

2019-09-27 16:06:44,620 INFO  [puppetserver] Puppet Compiled catalog for lhcadvdeveye05.fixnetix.com in environment production in 0.10 seconds

10.20.25.83 - - - 27/Sep/2019:16:06:44 +0000 "POST /puppet/v3/catalog/lhcadvdeveye05.fixnetix.com?environment=production HTTP/1.1" 200 612 10.20.25.83 10.20.25.83 8140 249

10.20.25.83 - - - 27/Sep/2019:16:06:45 +0000 "PUT /puppet/v3/report/lhcadvdeveye05.fixnetix.com?environment=production& HTTP/1.1" 200 9 10.20.25.83 10.20.25.83 8140 92

Unfortunately, I don't see anything untoward here nor anything helpful that contributes to resolving the issue.

Thanks,

Dan.

[Andreas Ntaflos]

On 27.09.19 18:11, Dan Crisp wrote:
> Thanks for the reply John,
>
> The issue still persists unfortunately.
>
> I've ensured that Selinux isn't enforcing on both the server side and
> client and then restarted the Puppet service on the master server.  The
> server logs whilst running the agent read as follows:

Are you positive the user and group really haven't been created?

And have you changed

node 'default' { ... }

to

node default { ... }

i.e. without the single quotes, as suggested? And afterwards restarted
the Puppetserver process by means of, e.g. systemctl restart puppetserver?

If so, and this hasn't helped, you may want to try to narrow the problem
down by simplifying the default node manifest even more, by making
site.pp look like this (verbatim):

node default {
fail('Failing deliberately on default node manifest')
}

Don't forget to restart the Puppetserver after that to make sure the
change is picked up.

When you then run the Puppet agent on the troublesome node it should
fail hard with the message defined above. If it does then you know at
least that site.pp is read and a catalog is created and applied for that
node. If it does not then there must be something else amiss that is not
obvious from the information and details you posted.

HTH

Andreas

[Martin Alfke]

Hi Dirk,

you are including class accounts within node default classification.
The accounts module does not do anything unless you add data to it.

Please look at https://github.com/puppetlabs/puppetlabs-accounts and check if the following example is working:

node default {
accounts::user { 'dan': }
accounts::user { 'morgan': }
}

This should create two accounts on the nodes:
User 'dan' and user 'morgen'

hth,
Martin

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/48f7aa63-ed60-d5c8-d36c-d302c01d4130%40ptmx.org.

[Andreas Ntaflos]

On 28.09.19 12:15, Martin Alfke wrote:
> Hi Dirk,
>
> you are including class accounts within node default classification.
> The accounts module does not do anything unless you add data to it.
>
> Please look at https://github.com/puppetlabs/puppetlabs-accounts and check if the following example is working:

It doesn't look like Dan is using the puppetlabs-accounts module. His
accounts module just creates a user and a group (copy/pasting the code
from the initial post):

# more
/etc/puppetlabs/code/environments/production/modules/accounts/manifests/init.pp
class accounts {

include accounts::groups

user { 'djc72uk':
ensure => present,
home => '/home/djc72uk',
shell => '/bin/bash',
managehome => true,
gid => 'djc72uk',
}

}

# more
/etc/puppetlabs/code/environments/production/modules/accounts/manifests/groups.pp
class accounts::groups {

group { 'djc72uk':
ensure => present,
}
}

Andreas

[Dan Crisp]

Thanks to all that contributed.  I managed to solve the issue.  Transpires that the permissions  on the files and directories (0640 for the most part) was not sufficient.  I had to ensure that all files in question had 0644 and directories at 0755 permissions set to get this to work.  A simple permission denied error somewhere among the logs would of been helpful!!

Thanks again,

Dan.

[Josh Cooper]

Yes agreed, there are two issues on that: https://tickets.puppetlabs.com/browse/SERVER-1717 and https://tickets.puppetlabs.com/browse/PUP-7102. We tried to fix it earlier, but had to revert due to acceptance test failures. The need for world readable (or changing the owner or group to puppet) is because puppetserver runs as the puppet user.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/4a709aee-0057-4498-92f8-7b24ec4a31b4%40googlegroups.com.

--

Josh Cooper | Software Engineer

jo...@puppet.com | @coopjn

Join us for Puppetize PDX 9-10 October.