PuppetDB Sizing

[Tom Tucker]

Assuming 2,500 Linux clients running Puppet community edition 3.8.  Any sizing recommendation for a PuppetDB system in regards to disk size for the DB, CPU, Memory, etc.  

Thank you for your time and feedback.

[Christopher Wood]

Somewhere past 700 nodes (still puppetizing) our 1-core, 2GB-RAM puppetmasters and 2-core, 2GB-RAM puppetdb host started showing signs of overload (ssl/connect errors in agent logs, catalog/report mismatches in puppetdb). I augmented the VMs with "hardware" to stop the complaints and later on went off tuning. I moved the puppetmasters up to 4-core 8GB-RAM and the puppetdb host is now 4-core 16GB-RAM. Definitely rattling around in them now but there's lots of room for growth.

For scaling/tuning, among others:

https://ask.puppetlabs.com/question/13433/how-should-i-tune-passenger-to-run-puppet/

https://docs.puppetlabs.com/puppetdb/latest/scaling_recommendations.html

https://docs.puppetlabs.com/guides/scaling.html

http://activemq.apache.org/scaling-queues.html

http://activemq.apache.org/javalangoutofmemory.html

After all that I analyzed catalogs across the deployment and found that the datacat usage in the mcollective module (now https://github.com/puppet-community/puppet-mcollective) was an abominable percentage of the total number of resources. The firewall type (https://github.com/puppetlabs/puppetlabs-firewall) was 3% of the total resources. Since it takes less horsepower to puppet up fewer things I figure there will be a benefit in some judicious refactoring here and there. (Templates instead of file_line, iptables config instead of firewall resources, et cetera.)

However, I figure there's a benefit in a conversion effort to cram things into puppet first and sort them out later. I had a good time just throwing hardware at the problem to start and then tuning after the bulk of hosts were converted to puppet management. People at companies where incremental hardware use is expensive may want to tune early and shrink manifests more aggressively.

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [1]puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> [2]https://groups.google.com/d/msgid/puppet-users/CAGymF1CQpBBFRZ2VO5x_e5XEt2sxF6Zpe_cT%2BZUaT5NbO%2BcEYA%40mail.gmail.com.
> For more options, visit [3]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. mailto:puppet-users...@googlegroups.com
> 2. https://groups.google.com/d/msgid/puppet-users/CAGymF1CQpBBFRZ2VO5x_e5XEt2sxF6Zpe_cT%2BZUaT5NbO%2BcEYA%40mail.gmail.com?utm_medium=email&utm_source=footer
> 3. https://groups.google.com/d/optout

[Tom Tucker]

Chris,

Thanks for the detailed response.  BTW did you deploy HA, load balancing, etc on the Postgres side? 

Cheers

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/20150626031113.GA832%40iniquitous.heresiarch.ca.
For more options, visit https://groups.google.com/d/optout.

[Christopher Wood]

There's a load balancer in front of the puppetmasters (which have SAN certs) but a single puppetdb host. Nothing save puppetmasters and my own curiosity depends on puppetdb (2.2.2) right now. (My disaster recovery plan is to rebuild the host and let the data refill by itself.)

If I need another scale-up or more things rely on puppetdb I might add database replication for PostgreSQL and a few extra puppetdb instances behind the load balancer. Given current usage I have no idea when I'd hit that point. I have 2GB max heap set and it spikes to 1.4GB usage quite rarely.

On Fri, Jun 26, 2015 at 07:32:18AM -0400, Tom Tucker wrote:
> Chris,
> Thanks for the detailed response.  BTW did you deploy HA, load balancing,
> etc on the Postgres side? 
> Cheers
> On Thu, Jun 25, 2015 at 11:11 PM, Christopher Wood
> <[1]christop...@pobox.com> wrote:
>
> Somewhere past 700 nodes (still puppetizing) our 1-core, 2GB-RAM
> puppetmasters and 2-core, 2GB-RAM puppetdb host started showing signs of
> overload (ssl/connect errors in agent logs, catalog/report mismatches in
> puppetdb). I augmented the VMs with "hardware" to stop the complaints
> and later on went off tuning. I moved the puppetmasters up to 4-core
> 8GB-RAM and the puppetdb host is now 4-core 16GB-RAM. Definitely
> rattling around in them now but there's lots of room for growth.
>
> For scaling/tuning, among others:
>

> [2]https://ask.puppetlabs.com/question/13433/how-should-i-tune-passenger-to-run-puppet/
>
> [3]https://docs.puppetlabs.com/puppetdb/latest/scaling_recommendations.html
>
> [4]https://docs.puppetlabs.com/guides/scaling.html
>
> [5]http://activemq.apache.org/scaling-queues.html
>
> [6]http://activemq.apache.org/javalangoutofmemory.html

>
> After all that I analyzed catalogs across the deployment and found that
> the datacat usage in the mcollective module (now

> [7]https://github.com/puppet-community/puppet-mcollective) was an

> abominable percentage of the total number of resources. The firewall

> type ([8]https://github.com/puppetlabs/puppetlabs-firewall) was 3% of

> the total resources. Since it takes less horsepower to puppet up fewer
> things I figure there will be a benefit in some judicious refactoring
> here and there. (Templates instead of file_line, iptables config instead
> of firewall resources, et cetera.)
>
> However, I figure there's a benefit in a conversion effort to cram
> things into puppet first and sort them out later. I had a good time just
> throwing hardware at the problem to start and then tuning after the bulk
> of hosts were converted to puppet management. People at companies where
> incremental hardware use is expensive may want to tune early and shrink
> manifests more aggressively.
> On Thu, Jun 25, 2015 at 08:16:31PM -0400, Tom Tucker wrote:
> >    Assuming 2,500 Linux clients running Puppet community edition 3.8. 
> Any
> >    sizing recommendation for a PuppetDB system in regards to disk size
> for
> >    the DB, CPU, Memory, etc.  
> >    Thank you for your time and feedback.
> >
> >    --
> >    You received this message because you are subscribed to the Google
> Groups
> >    "Puppet Users" group.
> >    To unsubscribe from this group and stop receiving emails from it,
> send an

> >    email to [1][9]puppet-users...@googlegroups.com.

> >    To view this discussion on the web visit
> >   

> [2][10]https://groups.google.com/d/msgid/puppet-users/CAGymF1CQpBBFRZ2VO5x_e5XEt2sxF6Zpe_cT%2BZUaT5NbO%2BcEYA%40mail.gmail.com.
> >    For more options, visit [3][11]https://groups.google.com/d/optout.
> >
> > References
> >
> >    Visible links
> >    1. mailto:[12]puppet-users...@googlegroups.com
> >    2.
> [13]https://groups.google.com/d/msgid/puppet-users/CAGymF1CQpBBFRZ2VO5x_e5XEt2sxF6Zpe_cT%2BZUaT5NbO%2BcEYA%40mail.gmail.com?utm_medium=email&utm_source=footer
> >    3. [14]https://groups.google.com/d/optout

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to [15]puppet-users...@googlegroups.com.

> To view this discussion on the web visit

> [16]https://groups.google.com/d/msgid/puppet-users/20150626031113.GA832%40iniquitous.heresiarch.ca.
> For more options, visit [17]https://groups.google.com/d/optout.

>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to [18]puppet-users...@googlegroups.com.

> To view this discussion on the web visit

> [19]https://groups.google.com/d/msgid/puppet-users/CAGymF1CXG4cfFfFDEv4vRkko558un3Duo89LpqTTaHd1VmkQRQ%40mail.gmail.com.
> For more options, visit [20]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. mailto:christop...@pobox.com
> 2. https://ask.puppetlabs.com/question/13433/how-should-i-tune-passenger-to-run-puppet/
> 3. https://docs.puppetlabs.com/puppetdb/latest/scaling_recommendations.html
> 4. https://docs.puppetlabs.com/guides/scaling.html
> 5. http://activemq.apache.org/scaling-queues.html
> 6. http://activemq.apache.org/javalangoutofmemory.html
> 7. https://github.com/puppet-community/puppet-mcollective
> 8. https://github.com/puppetlabs/puppetlabs-firewall
> 9. mailto:puppet-users%2Bunsu...@googlegroups.com
> 10. https://groups.google.com/d/msgid/puppet-users/CAGymF1CQpBBFRZ2VO5x_e5XEt2sxF6Zpe_cT%2BZUaT5NbO%2BcEYA%40mail.gmail.com
> 11. https://groups.google.com/d/optout
> 12. mailto:puppet-users%2Bunsu...@googlegroups.com
> 13. https://groups.google.com/d/msgid/puppet-users/CAGymF1CQpBBFRZ2VO5x_e5XEt2sxF6Zpe_cT%2BZUaT5NbO%2BcEYA%40mail.gmail.com?utm_medium=email&utm_source=footer
> 14. https://groups.google.com/d/optout
> 15. mailto:puppet-users%2Bunsu...@googlegroups.com
> 16. https://groups.google.com/d/msgid/puppet-users/20150626031113.GA832%40iniquitous.heresiarch.ca
> 17. https://groups.google.com/d/optout
> 18. mailto:puppet-users...@googlegroups.com
> 19. https://groups.google.com/d/msgid/puppet-users/CAGymF1CXG4cfFfFDEv4vRkko558un3Duo89LpqTTaHd1VmkQRQ%40mail.gmail.com?utm_medium=email&utm_source=footer
> 20. https://groups.google.com/d/optout

[Erik Dalén]

We are running a PuppetDB instance with ~8000 clients using a 30 minute runinterval. We have hit some scaling issues in the past but the latest PuppetDB version (2.3.5) is running nicely for us. This is with stringify_facts=false, which does create some extra load compared with stringify_facts=true.

We have separated the PuppetDB service hosts and the postgres databases to different hosts. For postgres we have a write master with a hot standby and four read-only slaves (one for each PuppetDB host). This was scaled up when we actually had some performance issues with earlier PuppetDB versions and can likely be scaled down to just two service hosts and read-only slaves without issues now.

The hardware is pretty beefy, the service hosts are Dell PowerEdge R620 with 24 CPU cores @ 2.00GHz and 32GB RAM.

The databases are all Dell PowerEdge R620 with 32 CPU cores @ 1.80GHz and 64GB RAM. For disks they have a 3TB Fusion-IO ioDrive2 each, they have pretty impressive performance that you can read about here: http://www.fusionio.com/data-sheets/iodrive2

So, this is probably on the higher end of scaling up a PuppetDB cluster and I'm fairly confident it could handle a lot more than 8000 clients, but haven't done any load test on it so far to see what the limit is.

We have a smaller PuppetDB installation as well which is around 2000 clients, and there we can manage with just a single of these database hosts and PuppetDB service hosts.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/20150626132535.GA4476%40iniquitous.heresiarch.ca.

[Miguel Di Ciurcio Filho]

On Thu, Jul 2, 2015 at 6:39 AM, Erik Dalén <erik.gus...@gmail.com> wrote:
>
> We have separated the PuppetDB service hosts and the postgres databases to
> different hosts. For postgres we have a write master with a hot standby and
> four read-only slaves (one for each PuppetDB host). This was scaled up when
> we actually had some performance issues with earlier PuppetDB versions and
> can likely be scaled down to just two service hosts and read-only slaves
> without issues now.
>

Could you share how did you configure the write-master and the read-only slaves?

Did you use the write/read separation of PuppetDB configuration?

[Byron Miller]

I use the write/read configuration built into PuppetDB and point the interfaces to VIPs behind a netscaler..  Netscaler load balances to the puppetDB instances and they're all setup so that writes happen to the primary DB node and reads can happen on the VIP for the slave nodes.  The slave nodes just use log shipping in postgres 9.4.

I'm not running 8000 nodes, but we do have about 2k and i get away with running puppetdb in vmware on fairly small sized VMs.  We use the crap out of read access for internal tooling and 30 minute checkins don't seem to cause much load for writes.

-byron