Unreported Windows nodes (Puppet 3.7.3, Exchange 2010)

[cko]

Hi,

I'm currently having an issue with the Windows Server 2008R2 nodes in our Exchange 2010 environment.

Since I upgraded the four nodes to Puppet 3.7.3 , they do not check in to the Puppetmaster via the Windows Service every 30 minutes like all the other nodes do. 

Here is a gist of the output from "puppet agent -t --debug" that might help https://gist.github.com/anonymous/e972393ea742631ccaef

Is there a problem in my environment or are there any known issues regarding this behavior? 

[Felix Frank]

Hi,

I actually don't know how Puppet operates on Windows exactly. But
apparently, there is supposed to be a registered service.

Is the Puppet service running? Is it writing into the event log?

[Rob Reynolds]

We are not aware of a known issue for this behavior. If you could check the eventlog and determine what issues it may be reporting?  

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/4da95e8f-9c8f-4dca-a91d-dfae1ce1dd1d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Rob Reynolds

Developer, Puppet Labs

Join us at PuppetConf 2015, October 5-9 in Portland, OR - http://2015.puppetconf.com/

Register early to save 40%!

[cko]

The Puppet Service is actually running.

The eventlog shows records about successfully finished catalog runs. All files in the C:\ProgramData\PuppetLabs\puppet\var\state directory indicate that the puppet run finishes without problems.

I suspect there is a problem with the submission of the report to the puppet master unless the puppet run is triggered manually. 

That might be the reason they are marked "out of sync" in any dashboard application (puppetdb, puppetexplorer, foreman).

[Rob Reynolds]

On Sun, Dec 14, 2014 at 9:17 AM, cko <der...@gmail.com> wrote:

The Puppet Service is actually running.

The eventlog shows records about successfully finished catalog runs. All files in the C:\ProgramData\PuppetLabs\puppet\var\state directory indicate that the puppet run finishes without problems.

I suspect there is a problem with the submission of the report to the puppet master unless the puppet run is triggered manually. 

What user does the puppet agent service run under? 

 

That might be the reason they are marked "out of sync" in any dashboard application (puppetdb, puppetexplorer, foreman).

On Sunday, December 14, 2014 4:48:59 AM UTC+1, Rob Reynolds wrote:

On Tue, Dec 9, 2014 at 7:28 AM, cko <der...@gmail.com> wrote:

Hi,

I'm currently having an issue with the Windows Server 2008R2 nodes in our Exchange 2010 environment.

Since I upgraded the four nodes to Puppet 3.7.3 , they do not check in to the Puppetmaster via the Windows Service every 30 minutes like all the other nodes do. 

Here is a gist of the output from "puppet agent -t --debug" that might help https://gist.github.com/anonymous/e972393ea742631ccaef

Is there a problem in my environment or are there any known issues regarding this behavior? 

We are not aware of a known issue for this behavior. If you could check the eventlog and determine what issues it may be reporting?  

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/4da95e8f-9c8f-4dca-a91d-dfae1ce1dd1d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Rob Reynolds

Developer, Puppet Labs

Join us at PuppetConf 2015, October 5-9 in Portland, OR - http://2015.puppetconf.com/

Register early to save 40%!

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/311eff85-ba40-42bf-a983-a363fe5a61d1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[cko]

The service runs under the Local System account.

[cko]

I just noticed that the affected agents do not update the files located in "C:\ProgramData\PuppetLabs\puppet\var\state".

Also interesting: The "agent_catalog_run.lock" file is not properly removed after a puppet run.

See the timestamps on the files ( http://i.imgur.com/qE87OB5.png )

[Rob Reynolds]

On Mon, Feb 2, 2015 at 5:05 AM, cko <der...@gmail.com> wrote:

I just noticed that the affected agents do not update the files located in "C:\ProgramData\PuppetLabs\puppet\var\state".

Also interesting: The "agent_catalog_run.lock" file is not properly removed after a puppet run.

See the timestamps on the files ( http://i.imgur.com/qE87OB5.png )

Can you give me an icacls on that folder? icacls C:\ProgramData\PuppetLabs\puppet\var\state. Also I wonder if you have a connection issue in sending the report to the master? From the gist I didn't see an issue jump out at me though.

Also, 3.7.4 just came out. I don't know for sure if it will resolve the issues or not.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/74fa3e65-2974-46b5-95c4-a30bf6374a60%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[cko]

Icacls: http://i.imgur.com/s1xQy65.png

I will upgrade to Puppet 3.7.3 this week an see if that fixed the issue.

[Rob Reynolds]

On Tue, Feb 3, 2015 at 8:25 AM, cko <der...@gmail.com> wrote:

Icacls: http://i.imgur.com/s1xQy65.png

Can you run one against the last_run_report.yaml and the lock file (if still present)? 

I'm seeing a similar set of permissions for var/state but want to verify the permissions on the files (as they can be different).

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/6bddf57d-01b4-4ea6-bb8a-8c7988d0c1c5%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Christian Koep]

This is what it looks like: http://i.imgur.com/Woxi68C.png

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAMJiBK4WYMparSBj0_nF4nwVUiQ-%2BQDJoCeZcc3o%2BXN9YZo5Bg%40mail.gmail.com.

[cko]

I just updated to 3.7.4

This is the output when I manually run Puppet: http://i.imgur.com/pdrklFm.png (Still, only happening on exchange 2010 nodes)

The Agent basically stops working after loading all facts, without removing the .lock file.

Let me know if you need more information on this.

[Rob Reynolds]

On Wed, Feb 11, 2015 at 11:31 PM, cko <der...@gmail.com> wrote:

I just updated to 3.7.4

This is the output when I manually run Puppet: http://i.imgur.com/pdrklFm.png (Still, only happening on exchange 2010 nodes)

The Agent basically stops working after loading all facts, without removing the .lock file.

Let me know if you need more information on this.

Apologies on not getting back sooner. This definitely seems like something is wrong. However it would be hard to understand specifically what it is on your system that is causing the weird behavior without some low level debugging.

There are a few possibilities into why the pidlock file is not removed, including possibly something else taking a lock on the file and Puppet is not able to delete it on exit. There are tools out there that can help you see what, if anything has a lock on a specific file/directory (I use LockHunter). Timing it right though may be necessary, so a combination of that and procmon.

Other debugging tools I would recommend:

• Wireshark
• SysInternals' Process Explorer (aka procexp)
• SysInternal's Process Monitor (aka procmon)

One suggestion I would have before going to the low level debugging would be to completely remove Puppet from that machine (including manually removing the c:\programdata\puppetlabs directory), Reboot, and then try installing it again fresh. Note that you will need to remove the cert from the master as it will try to request a new cert.[1][2] I would back up the programdata\puppetlabs directory prior to removing it.

[1] https://hurricanelabs.com/blog/managing-puppet-certificates/

[2] http://www.masterzen.fr/2010/11/14/puppet-ssl-explained/

 

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/b5b4414a-fe71-4171-be8b-a4a0d90b7771%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Rob Reynolds

Developer, Puppet Labs

PuppetConf 2015 is coming to Portland, Oregon! Join us October 5-9.

Register now to take advantage of the Early Adopter discount —save $349!