Jira (PUP-11324) puppet-agent-6.25.0 breaks puppetserver-6.17.0

Issue Type:	Bug
Affects Versions:	PUP 6.25.0
Assignee:	Unassigned
Created:	2021/10/20 6:23 AM
Priority:	Normal
Reporter:	Chris Fox

Puppet Version: 6.25.0
Puppet Server Version: 6.17.0
OS Name/Version: SLES 12SP5

After upgrading puppetserver to 6.17.0 and puppet agent to 6.25.0 (on puppetserver) (running on SLES12 SP5) the agent freeze after some puppet runs complety and shows the following error

Request to https://<puppetserver>:8140/puppet/v3/file_metadatas/plugins?recurse=false&links=manage&checksum_type=md5&source_permissions=ignore&environment=produktion interrupted after 300.002 seconds Wrapped exception: end of file reached

The interesting thing about this error is the timing when the error occurs. Furthermore no errors logged in puppetserver.log, there seems no bottleneck (cpu/memory - each node has 4 cpus / 10G Memory, xms/xmx set to 4G), the error just occurs after some time or load?!

But let me describe observed behavior

(re)starting the puppetserver, cpu load rises for some time, but i think this is quite normal
starting 2 puppet agents on other hosts in an endless loop to simulate load ( while :; do puppet agent -t; done)
after some more time / puppet runs the error from above accurs

If we downgrade the puppet-agent to 6.24.0 on the puppetserver everything works fine again.

Desired Behavior:

We can update puppet agent and puppetserver to the latest 6.X Version and everything works as expected.

Actual Behavior:

We need to downgrade puppet-agent on the puppetserver hosts to 6.24.0.

This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)

Josh Cooper (Jira)

unread,

Oct 20, 2021, 2:54:02 PM10/20/21

to puppe...@googlegroups.com

Josh Cooper commented on

The end of file reached error message and 300 seconds indicates something killed the connection, possibly a firewall or puppetserver itself. Can you include redacted puppetserver.log around the time the error occurs?

Also it would be helpful to see the output of puppet agent -t when the server is under heavy load. Both when using 6.24.0 and 6.25.0

Chris Fox (Jira)

unread,

Oct 22, 2021, 3:58:02 AM10/22/21

to puppe...@googlegroups.com

Chris Fox updated an issue

Puppet /

Change By:	Chris Fox
Attachment:	test1_puppetrun.png
Attachment:	test1_metrics.png

Chris Fox (Jira)

unread,

Oct 22, 2021, 4:17:03 AM10/22/21

to puppe...@googlegroups.com

Chris Fox updated an issue

Puppet /

Change By:	Chris Fox
Attachment:	test2_matrics.png
Attachment:	test2_puppetrun.png

Chris Fox (Jira)

unread,

Oct 22, 2021, 4:38:03 AM10/22/21

to puppe...@googlegroups.com

Chris Fox commented on

Hi Josh Cooper,

there are no error in puppetserver.log around the time and i dont think its caused by a firewall (i tested also on localhost, deactivated iptables).

I spent some time to figure out, when the error occurs:

Test 1

setup: 1x puppetserver (with puppetserver-6.17.0-1.sles12 and puppet-agent-6.24.0), 2x clients (puppet-agent-6.25.0)

All puppet-agents run fine, without any error as you can see in test1_puppetrun.png. The load of the puppetserver you can find in test1_metrics.png

Test 2

setup: 1x puppetserver (with puppetserver-6.17.0-1.sles12 and puppet-agent-6.25.0), 2x clients (puppet-agent-6.25.0)

In the first step i only started the puppet-agents on the two clients - the puppet-agent on puppetserver was disabled and stopped. They run fine for some time (load on puppetserver marked with "1" in test2_matrics.png. In the second step i also started the puppet-agent on puppetserver which seems to cause the error after some (1,2,3,??) puppetruns. As you can see, the puppetagent on one client throws an internal error marked with "1" in test2_puppetrun.png (happens at the same time as i started the puppetrun on puppetserver). This error only appears in 1/3 of my tests. After this all clients get the error "end of file reached" and the load on puppetserver reduce to nearly zero, marked with "2" in test2_matrics.png. The only way to get everything running again is to stop the puppet-agent on the puppetserver and restart the puppetserver.

Hopefully these tests help to figure out where the error comes from. If you need further data please let me know.

Josh Cooper (Jira)

unread,

Oct 22, 2021, 1:58:02 PM10/22/21

to puppe...@googlegroups.com

Josh Cooper commented on

Thanks Chris Fox. When the agent reports "Error on SERVER: ... Attempt to redefine loader named..." there should be a corresponding stack trace in /var/log/puppetlabs/puppetserver/puppetserver.log. Could you include that in the ticket (redacting any module/class names)? Also are you running with puppetserver multithreading enabled?

Chris Fox (Jira)

unread,

Oct 25, 2021, 1:21:02 AM10/25/21

to puppe...@googlegroups.com

Chris Fox updated an issue

Puppet /

Change By:	Chris Fox
Attachment:	puppetserver_redefine_loader.log

Chris Fox (Jira)

unread,

Oct 25, 2021, 1:28:02 AM10/25/21

to puppe...@googlegroups.com

Chris Fox commented on

HI Josh Cooper,

you can find the stack strace in the attachment puppetserver_redefine_loader.log. Its the first stack trace i found, so do not wonder that the time differs. The stack trace is always the same, only the puppet modules differs.

Yes, we're using multithreading. If i have some time today, i can do the test again with multithreading disabled.

Chris Fox (Jira)

unread,

Oct 25, 2021, 2:51:03 AM10/25/21

to puppe...@googlegroups.com

Chris Fox commented on

I've done some quick tests with multithreading disabled: i was not able to reproduce the error "...Attempt to redefine loader named...". Furthermore all agents and the puppetserver running fine without freezing.

Josh Cooper (Jira)

unread,

Oct 25, 2021, 2:17:03 PM10/25/21

to puppe...@googlegroups.com

Josh Cooper commented on

Ah thanks Chris Fox that's very helpful! If possible, could you try using puppet-agent 6.25.1 with multithreading enabled and capture a thread dump when puppetserver hangs?