Puppet server is slow and needs tuning

[Harish Kothuri]

Hi All,

Following is my puppet configuration details. Problem is that puppet server and agent runs are going very slow at times and restarting the server helps in production.

Puppet 3.8.7 (open source)

Facter 2.4.6

CentOS 7

Number of nodes: 500

Number of cores: 16

RAM: 32GB

I'm going through this puppet website https://docs.puppet.com/pe/latest/config_monolithic.html#tuning-monolithic-installations-(reference) to fine tune my puppet.

Can someone help me in understand location of the files exactly as i could not figure out few (marked in Red). Please verify my changes and let me know if there's anything wrong.

Section from Link: 16 cores, 32 GB of RAM (Monolithic)

Puppet Server:

1. Increased JRuby instances in /etc/hiera.yaml file as below.

---

:backends:

  - yaml

:hierarchy:

  - defaults

  - "%{clientcert}"

  - "%{environment}"

  - global

:yaml:

# datadir is empty here, so hiera uses its defaults:

# - /var/lib/hiera on *nix

# - %CommonAppData%\PuppetLabs\hiera\var on Windows

# When specifying a datadir, make sure the directory exists.

  :datadir:

  jruby-puppet: {

    max-active-instances: 10

    max-requests-per-instance: 0

  }

1.1 Heap size on puppet server - Not sure where and how to change this.

PuppetDB:

2. PuppetDB command processing threads - Where and how to change this?

    changed this in /etc/puppetdb/conf.d/config.ini

2. 1 Changed heap size of puppetdb in /etc/sysconfig/puppetdb file as below

JAVA_ARGS="-Xmx1g -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -Djava.security.egd=file:/dev/urandom"

2.2 Broker Memory - Not sure where and how to change this.

3. Node classifier- Not sure where and how to change this.

4. ActiveMQ - Not sure where and how to change this.

5. Orchestration services - Not sure where and how to change this.

6. PostgreSQL - Not sure where and how to change this.

Thanks & Regards,

Harish Kothuri

[Rob Nelson]

Harish,

You are running puppet 3 open source and looking at directions for puppet 4 enterprise edition. While some of the PE information will surely apply to the open source edition, you should update to puppet 4 as puppet 3 is End of Support/Life. That upgrade alone should improve performance significantly, at which time you can re-evaluate performance issues.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/b0fb70eb-3224-4099-b53d-e1c370e2a0f9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Rob Nelson

[Harish Kothuri]

Thanks for your reply. I understand that i need to upgrade to puppet 4. 

I'm just looking for similar options to optimize in 3 open source since i have these in production and not easy to upgrade

Thanks & Regards,

Harish Kothuri

[Ramin K]

With Puppet 3.8.7 you're hopefully running Puppet with Apache/Passenger.
If not, you have a puppetmaster (puppet master?) process that is single
threaded. You'll have to migrate to Passenger based setup. My guess is
that that you're running the standalone process since Passenger defaults
to 6 processes which should be able to serve 500 agents.

Tuning Apache/Passenger
https://ask.puppet.com/question/13433/how-should-i-tune-passenger-to-run-puppet/

Ramin

> as i could not figure out *few (marked in Red)*. Please verify

> my changes and let me know if there's anything wrong.
>
> Section from Link: 16 cores, 32 GB of RAM (Monolithic)
>

> _*Puppet Server:*_
> *1. Increased JRuby instances in /etc/hiera.yaml file as below.*

>
> ---
> :backends:
> - yaml
> :hierarchy:
> - defaults
> - "%{clientcert}"
> - "%{environment}"
> - global
>
> :yaml:
> # datadir is empty here, so hiera uses its defaults:
> # - /var/lib/hiera on *nix
> # - %CommonAppData%\PuppetLabs\hiera\var on Windows
> # When specifying a datadir, make sure the directory exists.
> :datadir:
> jruby-puppet: {
> max-active-instances: 10
> max-requests-per-instance: 0
> }
>

> *1.1 Heap size on puppet server - Not sure where and how to
> change this.*
> *
> *
> *_PuppetDB:_*
> *2. PuppetDB command processing threads - **Where and how to
> change this?*
> * changed this in /etc/puppetdb/conf.d/config.ini*
> *
> *
> *
> *
> *2. 1 Changed heap size of puppetdb in /etc/sysconfig/puppetdb
> file as below*
> *
> *
> *JAVA_ARGS="-Xmx1g -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof
> -Djava.security.egd=file:/dev/urandom"
> *
> *
> *
> *2.2 Broker Memory - Not sure where and how to change this.*
> *
> *
> *3. Node classifier-** Not sure where and how to change this.*
> *4. ActiveMQ - **Not sure where and how to change this.*
> *5. Orchestration services - **Not sure where and how to change
> this.*
> *6. PostgreSQL - **Not sure where and how to change this.*
> *
> *

> Thanks & Regards,
> Harish Kothuri
>
> --
> You received this message because you are subscribed to the
> Google Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from

> it, send an email to puppet-users...@googlegroups.com <javascript:>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/b0fb70eb-3224-4099-b53d-e1c370e2a0f9%40googlegroups.com

> <https://groups.google.com/d/msgid/puppet-users/b0fb70eb-3224-4099-b53d-e1c370e2a0f9%40googlegroups.com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.
>
> --
> Rob Nelson

>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com

> <mailto:puppet-users...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/ce7ef49d-0fdd-470f-ae89-ef5e39e4e906%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/ce7ef49d-0fdd-470f-ae89-ef5e39e4e906%40googlegroups.com?utm_medium=email&utm_source=footer>.