puppet server connection limit / throttling ?

[Andy Hall]

Hi there we have a puppetserver 6.4.0 which is currently handling about 450 clients. In terms of performance over our legacy 3.8 instance it is great - clearly clojure is faster than a rails app behind a web proxy :-) But we notice that when we hit approx. 90 established connection to TCP port 8140 any new client puppet-agent connections "hang" at this stage as seen from debug output:

Debug: Creating new connection for https://server.company.com:8140

Debug: Starting connection for https://server.company.com:8140

This isn't just waiting for a catalog to compile - it often sits there for minutes with no response. In fact if we reboot the puppet server thus clearing the connections the client then runs normally as expected - with just a small delay waiting for its catalog.

I'm pretty sure this is not some OS / TCP stack limitation so wonder if there is some config which might be throttling this at the application level ? I hope approx. 450 clients is not the stage we need to start scaling to multiple catalog servers ??

Thanks for your help, Andy.

[Ramin K]

I'd look at general tuning as a first step. From puppetserver.conf

450 servers every 30 minutes is 4s. If catalog compile is taking longer
than 4s a single core instance is likely falling behind. I believe
Puppet recommends 4 core minimum to handle OS, JVM, garbage collection,
etc. Distributing file resources through Puppet will take CPU time away
from catalog compiles. I'd only be concerned about 200-300 or so.

From puppetserver.conf
https://puppet.com/docs/puppetserver/latest/config_file_puppetserver.html
max-active-instances: Optional. The maximum number of JRuby instances
allowed. The default is 'num-cpus - 1', with a minimum value of 1 and a
maximum value of 4.

Assuming a 4 core instance that's probably fine. Larger and you'll need
to tune it.

General guidelines have been 250-500MB per instance. Default tuning in
6.7 is 2g. Again on a 4 core / 3GB+ instance that should be fine.

Lastly you said reboot, but are you just restarting the Puppetserver
process? If it truly requires a reboot, let us know.

Ramin

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com
> <mailto:puppet-users...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/52b4d184-6ecd-409a-ad26-d623762539d2%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/52b4d184-6ecd-409a-ad26-d623762539d2%40googlegroups.com?utm_medium=email&utm_source=footer>.