possible bug: slow server connection causes clients to hang

[JonY]

Over the weekend there were some intermittent outages at our data center. The puppet server would let clients connect but they would eventually time out. 

Unf this timeout wasn't handled gracefully and the clients would hang. Doing a 'ps -ax | grep puppet' would show the service and the 'puppet agent: applying configuration'. No activity available in the message log for many hours though which indicated an unhappiness.

If I killed the process and did 'puppet agent --test --debug' I could see it get to where it tried to download the configuration. Doing this interactively it would exit properly. If I restarted the service it would stop and hang there.

Server: CentOS 6.4 (x64). 

puppet 3.4.0

apache 2 / Passenger

Clients: Debian 6.x 

puppet 3.4.0

[JonY]

Turns out to be a denial of service on a machine in the same subnet. Machine in question has been sequestered. 

All of my puppet clients are hung. Fortunately there are only a few dozen as I'm having to touch each one, stop the puppet service, kill the hung process and rerun manually.