Puppet Agent starting too soon

[Laverne Schrock]

I have a box on which puppet-agent does not start correctly on reboot. Well, to be more precise, the puppet-agent starts, but never contacts the server.

$ sudo journalctl -b 0 -u puppet -f
-- Logs begin at Wed 2017-02-01 18:27:11 CST. --
Mar 06 12:42:03 localhost.localdomain systemd[1]: Started Puppet agent.
Mar 06 12:42:16 localhost.localdomain puppet-agent[927]: Could not request certificate: getaddrinfo: Temporary failure in name resolution
Mar 06 12:44:16 a.real.hostname.tld puppet-agent[927]: Could not request certificate: getaddrinfo: Temporary failure in name resolution
Mar 06 12:46:16 a.real.hostname.tld puppet-agent[927]: Could not request certificate: getaddrinfo: Temporary failure in name resolution
Mar 06 12:48:16 a.real.hostname.tld puppet-agent[927]: Could not request certificate: getaddrinfo: Temporary failure in name resolution

Note how when the puppet-agent starts, the box doesn't yet know its hostname because the network stack is (apparently) not fully up. Running `systemctl restart puppet` resolves the issue until the next reboot.

I was able to find a work-around. In the systemd unit file for puppet, I changed

After=basic.target network.target

to

 After=basic.target network-online.target

See: https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/

The box is a fresh install of Fedora 25 and is using the following packages:

puppet-agent-1.9.3-1.fedoraf25.x86_64
puppetlabs-release-pc1-1.1.0-5.fedoraf25.noarch

I have another box with the same setup (but a little more RAM) and the issue does not occur there.

I have two thoughts on this.
1) This is a subtle timing issue which is why I see it on one box, but not the other.
2) puppet-agent is misbehaving and ought to  properly detect when the networking stack comes up.
3) If I want to resolve this, I should just use my workaround.

Does #3 seem like the best plan? I'd appreciate any insight into why the issue is occurring.

Cheers,
-Laverne Schrock

 

[Trevor Vaughan]

This does seem to be the fix that is required both for Fedora and EL7.

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/36ca1e68-fa09-4654-ba62-9c13c2561c76%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Trevor Vaughan
Vice President, Onyx Point, Inc

(410) 541-6699 x788

-- This account not approved for unencrypted proprietary information --

[Rob Nelson]

That wiki page says that all you should need to do is have NetworkManager or systemd-networkd services enabled. Do you by any chance have them both disabled on the affected node? I know many of us don't like NetworkManager, but disabling it entirely can cause some problems. Just a guess, as I haven't seen this issue on any of my nodes, including the low-RAM ones.

Rob Nelson
rnel...@gmail.com

On Mon, Mar 27, 2017 at 11:33 PM, Laverne Schrock <lverns...@gmail.com> wrote:

[Trevor Vaughan]

Disabling NetworkManager hasn't caused any issues for me so far. That said, EL7.3 might break that, so Fedora may already be broken.

Trevor

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAC76iT_vimf_MmN--mBLA2fnqSAxQUJ%3DBAzNRMOQOgM1F0TevA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Rob Nelson]

Did you replace it with systemd-networkd, or just ditch it? Regardless, I've updated us to EL7.3 (CentOS though) and not observed this issue, so I'm not sure it's that generic of a problem, there must be *something* triggering it. Even a simple race condition should result in some successes if Laverne has rebooted it a few dozen times.

Rob Nelson
rnel...@gmail.com

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CANs%2BFoWZ9Vfa%3DLY_MYgavrsBZaN6imUyWfZKT1776EsN5yyEXw%40mail.gmail.com.

[Trevor Vaughan]

I just ditched it.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAC76iT9o4et2Va%3DDQYCK%2BgDbqFCX1sL_YdndA4CKYx2wS%2B2hqw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Rob Nelson]

I'm guessing those race conditions are related to not replacing it with something else. Seems they're "magic" components to not requiring reliance on network-online.target. Hooray :/

Rob Nelson
rnel...@gmail.com

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CANs%2BFoVwHnJZ3WSaBHyyF_Y4XncFD3X%2BOXYuZ7nBLsQN1yNOiA%40mail.gmail.com.

[Игорь Тиунов]

HI, I have the same issue for my Redhat 7 servers (immutable hypervisors). For some reasons ppuppet.servise unit is not network depended. I don't know what is it bug or feature. I override dependency for puppet unit to depend on network.service (I use lsb network service)