odd log entry warning: com.puppetlabs.http.client.HttpClientException: Request cancelled

[Mike Jeski]

Hi group,

running PE 3.8.7 (while cleaning up code for upgrade to puppet 4) seeing  HttpClientException constantly in the puppetserver.log.  

WARN  [o.e.j.s.HttpChannel] /production/certificate/ca?fail_on_404=true
com.puppetlabs.http.client.HttpClientException: Request cancelled

Happens every few minutes,  reports continue to process and then eventually I see this:

WARN  [o.e.j.s.HttpChannel] /production/certificate/ca?fail_on_404=true
java.lang.OutOfMemoryError: unable to create new native thread
   
after which (most of the time) agent runs & reports are processed, however not always.  Sometimes reports stop processing altogether and the console shows all 120 nodes as unresponsive. 

Where to start debugging this?

What I've done so far is in /etc/sysconfig/pe-puppetserver I've modified the JAVA ARGS
MaxPermSize is now 512m up from 256m. And I've added  -XX:+UseG1GC

The Machine is RHEL6 (VMWARE) 16GB RAM  & 8Cores.  r

[Thomas]

Maxpermsize isnt available in openjdk8. Look for xmx/xms. And I think UseGC1G is only available for openjdk8.

https://docs.puppet.com/puppetserver/latest/tuning_guide.html#jvm-heap-size

- Thomas

[Mike Jeski]

Cheers,
 the -XX:MaxPermSize= was there as part of the pe-puppetserver package.  I merely raised the value to 512m from 256m.

Are you saying it is syntactically incorrect?

[Ryan Whitehurst]

On Thu, Jul 27, 2017 at 1:37 PM, Mike Jeski <jesk...@gmail.com> wrote:

Cheers,
 the -XX:MaxPermSize= was there as part of the pe-puppetserver package.  I merely raised the value to 512m from 256m.

Are you saying it is syntactically incorrect?

It's not syntactically incorrect, it just has no effect on Java 8. What you actually need is to increase the Java heap size. This page has more information on tuning Puppet Server and would be valuable to read:

https://docs.puppet.com/puppetserver/1.2/tuning_guide.html

 

On Thursday, July 27, 2017 at 1:19:59 PM UTC-7, Mike Jeski wrote:

Hi group,

running PE 3.8.7 (while cleaning up code for upgrade to puppet 4) seeing  HttpClientException constantly in the puppetserver.log.  

WARN  [o.e.j.s.HttpChannel] /production/certificate/ca?fail_on_404=true
com.puppetlabs.http.client.HttpClientException: Request cancelled

Happens every few minutes,  reports continue to process and then eventually I see this:

WARN  [o.e.j.s.HttpChannel] /production/certificate/ca?fail_on_404=true
java.lang.OutOfMemoryError: unable to create new native thread
   
after which (most of the time) agent runs & reports are processed, however not always.  Sometimes reports stop processing altogether and the console shows all 120 nodes as unresponsive. 

Where to start debugging this?

What I've done so far is in /etc/sysconfig/pe-puppetserver I've modified the JAVA ARGS
MaxPermSize is now 512m up from 256m. And I've added  -XX:+UseG1GC

The Machine is RHEL6 (VMWARE) 16GB RAM  & 8Cores.  r

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/b0ea1830-6ad4-4886-9539-db700edefd0d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Mike Jeski]

Cheers mate.  This version of PE is actually running java 7.

On Thursday, July 27, 2017 at 1:19:59 PM UTC-7, Mike Jeski wrote:

[Ryan Whitehurst]

On Fri, Jul 28, 2017 at 9:01 AM, Mike Jeski <jesk...@gmail.com> wrote:

Cheers mate.  This version of PE is actually running java 7.

Ah, well in that case bumping up the MaxPermSize will have an effect, but the problem is still almost certainly caused by too low of a heap size, so you should still read the information here about tuning the JVM heap size and set the -Xmx value higher:

https://docs.puppet.com/puppetserver/1.2/tuning_guide.html

 

--

You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/81eea74d-a217-4cf7-ac97-f0cab1061ddb%40googlegroups.com.

[Mike Jeski]

cheers Ryan, I have read the guide and made changes, we'll see how it goes.

On Friday, July 28, 2017 at 11:27:29 AM UTC-7, Ryan Whitehurst wrote:

On Fri, Jul 28, 2017 at 9:01 AM, Mike Jeski <jesk...@gmail.com> wrote:

Cheers mate.  This version of PE is actually running java 7.

Ah, well in that case bumping up the MaxPermSize will have an effect, but the problem is still almost certainly caused by too low of a heap size, so you should still read the information here about tuning the JVM heap size and set the -Xmx value higher:

https://docs.puppet.com/puppetserver/1.2/tuning_guide.html

 

On Thursday, July 27, 2017 at 1:19:59 PM UTC-7, Mike Jeski wrote:

Hi group,

running PE 3.8.7 (while cleaning up code for upgrade to puppet 4) seeing  HttpClientException constantly in the puppetserver.log.  

WARN  [o.e.j.s.HttpChannel] /production/certificate/ca?fail_on_404=true
com.puppetlabs.http.client.HttpClientException: Request cancelled

Happens every few minutes,  reports continue to process and then eventually I see this:

WARN  [o.e.j.s.HttpChannel] /production/certificate/ca?fail_on_404=true
java.lang.OutOfMemoryError: unable to create new native thread
   
after which (most of the time) agent runs & reports are processed, however not always.  Sometimes reports stop processing altogether and the console shows all 120 nodes as unresponsive. 

Where to start debugging this?

What I've done so far is in /etc/sysconfig/pe-puppetserver I've modified the JAVA ARGS
MaxPermSize is now 512m up from 256m. And I've added  -XX:+UseG1GC

The Machine is RHEL6 (VMWARE) 16GB RAM  & 8Cores.  r

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

[Mike Jeski]

In case it helps anyone else,  there is likely no need to fiddle with the Heap etc. The root cause in my case was somehow during an upgrade the master was unpinned from the "Pe Certificate Authority" nodegroup.  I've seen it on a few boxes that were upgraded using nc_migrate tool FWIW.