Re: [Puppet Users] Terrible exported resources performance

[David Schmitt]

On 21.01.2013 14:05, Daniel wrote:
> Hi,
>
> Since the below is a little long, I put my question at the top: how do I
> troubleshoot awful exported resources performance in puppet and is there
> anything I can tweak to get it to run under 10 minutes in larger
> environments?

There is a "simple" answer: puppetdb.

There'll still be bottlenecks to wring, but collection performance won't
be one of them.


Best Regards, David

[GRANIER Bernard (MORPHO)]

Hi,

Is there a way to define a variable in puppet.conf file ?

Something like :
My_var = my_value in puppet.conf

Then to be able to use it in manifest with something like :
file {custo.txt':
path=> $my_value,
ensure=> present,
mode=> 0640,
content=> "this content",
}

Sincerly,

Bernard Granier

#
" This e-mail and any attached documents may contain confidential or proprietary information. If you are not the intended recipient, you are notified that any dissemination, copying of this e-mail and any attachments thereto or use of their contents by any means whatsoever is strictly prohibited. If you have received this e-mail in error, please advise the sender immediately and delete this e-mail and all attached documents from your computer system."
#

[Brian Mathis]

Something like this would typically go in your manifest files, not the
puppet.conf file. I think you will find the Puppet language guide
instructive, specifically the page on variables:
http://docs.puppetlabs.com/puppet/2.7/reference/lang_variables.html

You should think of Puppet as more of a development language than a
tool that you simply configure using a .conf file. The site.pp is
simply the entry point into the structure you build with the
manifests, and you can do quite a lot with the language.


❧ Brian Mathis

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>

[GRANIER Bernard (MORPHO)]

Thanks for the advice, I will use variables to define specific values,

Sincerely,


Bernard Granier

[Luke Bigum]

Hi Daniel,

On Monday, January 21, 2013 1:05:26 PM UTC, Daniel wrote:

In the larger env it takes about 70 minutes, if it manages to finish at all. Initially, as a "quick" test, I was running puppetdb without postgres and had to give it 5GB to get it to finish at all (70 mins). With postgres 8.4, load on the puppetmaster is significantly reduced, but with 512MB for puppetdb (128 + 1MB per node, and then double it for good measure) puppetdb still runs out of memory. I set it to 1GB and puppedb just crashed again (I've got dumps). Trying with 2GB now. I haven't fiddled with thread settings, but my puppet agents aren't deamonized or 'croned', I run them using mcollective or manually. So there's only a single puppet agent running during this test, on the core nagios server. It seems that there's a ruby process taking 100% of one core during this run and nothing else "dramatic" seems to be happening (except for puppetdb dying of course).

Given enough RAM it doesn't sound like PuppetDB is the problem any more, is that correct?

The Ruby process is most likely a Puppet Master thread doing the catalog construction. I think your suffering from a similar problem that we had recently, where it's not specifically resource collection that's taking up all the time, it's the Puppet Master turning the exported resources information into one enormous catalog that takes too long.

We got around this by bypassing exported resources and querying the information from PuppetDB directly and using that information in a template. I suggested the following to another user a few days ago in this thread:

https://groups.google.com/forum/#!topic/puppet-users/X6Lm-0_etbA

-Luke

[Daniel]

Hi Luke,

This sounds like a sensible workaround, I will definitely have a look. I haven't yet had enough time to look at the issue properly, but it seems that this very long time is indeed consumed by catalog construction. Puppetdb fails after this is finished, so it seems that it dies when nagios host tries to report its catalog back.

To be honest it's very disappointing that puppet and puppetdb can't handle few thousand exported resources and that workarounds like that are needed. I might need to look at alternatives.


Regards,
Daniel

[Ken Barber]

> This sounds like a sensible workaround, I will definitely have a look. I
> haven't yet had enough time to look at the issue properly, but it seems that
> this very long time is indeed consumed by catalog construction. Puppetdb
> fails after this is finished, so it seems that it dies when nagios host
> tries to report its catalog back.

Do you mean it dies from an OOM when it tries to report the catalogue back?

ken.

[Daniel Siechniewicz]

Yes, that's what it looks like. Of course I can prevent it by giving
it more memory (which I did), but I already have postgres backed
puppetdb and had to give puppetdb 3GB, or a puppet agent run on a
single host (OK, with thousands of exported resources to collect and
process) that takes about 70 minutes can still kill it. This waiting
70 minutes for it to die is an insult to injury... Overall not great.
I'm happy to redo this setup if I'm doing something wrong, but it just
seems like this is exponential (30-odd nodes - 2 minutes, 100-odd
nodes, 70 minutes).

Regards,
Daniel

[Ken Barber]

How many resources are we talking for the 100 nodes?

If you're running puppetdb one way to do this is to get a report on
the nagios server that does the collection:

curl -G -H 'Accept: application/json' 'http://puppetdb:8080/resources'
--data-urlencode 'query=["=", ["node", "name"], "nagiosserver"]' | p
resource | wc -l

(this at least works on Puppetdb 1.0.5 - as JSON output is pretty-printed)

It would be interesting to see if this is an artifact of collection or
the nagios resource specifically, although the only good way I know to
do this is to perhaps replace the nagios exported resource cases with
something simple - such as a noop style resource like 'whit' or
'anchor'.

When you run puppet agent -t ... if you do a summarize what is the
result? You should get something like this:

# puppet agent -t --summarize
info: Caching catalog for puppetdb1.vm
info: Applying configuration version '1358813527'
notice: exported
notice: /Stage[main]//Node[puppetdb1.vm]/Notify[exported]/message:
defined 'message' as 'exported'
notice: asdfasdfsasdf
notice: /Stage[main]//Node[puppetdb1.vm]/Notify[foo]/message: defined
'message' as 'asdfasdfsasdf'
notice: Finished catalog run in 0.03 seconds
Changes:
Total: 2
Events:
Total: 2
Success: 2
Resources:
Changed: 2
Out of sync: 2
Skipped: 6
Total: 9
Time:
Filebucket: 0.00
Notify: 0.00
Config retrieval: 0.90
Total: 0.90
Last run: 1358813994
Version:
Config: 1358813527
Puppet: 2.7.18

I think Luke's suggestion about tapping the information in puppetdb
using functions is not a bad work-around however, but its
disappointing that exported resources in this case isn't just
_working_ :-(.

ken.

[jcbollinger]

This sounds like a memory-management problem.  If you're running up against (or close to) a memory limit, then the Java-based puppetdb will start doing a lot of garbage collection as available heap memory becomes scarce.  That will greatly slow down the whole Java VM, especially so if the GC runs don't free much memory.  Also, if you give the VM more memory than available RAM then you will have slowdowns related to paging parts of the heap between physical and virtual memory; depending on memory access patterns, that could happen frequently and be very costly.

On the other hand, some versions of the Java VM have problems with large amounts of heap memory.  One would hope that such problems have been resolved by now, but you could anyway try reducing the heap memory to about 1.5G.  That's close to the maximum that you can safely use with some VM implementations.

I certainly agree that puppetdb has a serious problem if it requires such large amounts of memory to handle resource collections with a few thousand elements.  If roughly 3000 (at ~30 resources * ~100 nodes) exported resources require in excess of 3GB then that's in the vicinity of 1MB per resource -- absurd!

On the other hand, that's so absurd that I suspect something else is going on here.  It smacks of a combinatoric problem, either in your manifests or in the puppetdb implementation.  There aren't enough data points to be sure, but your run times could be scaling with the square of the number of nodes.  Is there any chance that the number of exported resources is scaling the same way?  That would explain both time and memory consumption.


John

[Daniel]

Seems that chaining exported resources might not be too efficient and produces lots of data that could be the reason for puppetdb crashing.

The culprits being these two lines in two manifest files:

./nsca/server.pp:  #File <<| tag == $get_tag |>> -> Nagios_host <<| tag == $get_tag |>>
./nrpe/server.pp:  #File <<| tag == $get_tag |>> -> Nagios_host <<| tag == $get_tag |>> 

replacing them with unchained:

File <<| tag == $get_tag |>>
Nagios_host <<| tag == $get_tag |>>

causes it to run even with 1GB for puppetdb (still a 16GB vm) in under 10 mins, which is acceptable.

This is tracked under bug #18804

[jcbollinger]

On Thursday, January 24, 2013 5:24:58 AM UTC-6, Daniel wrote:

Seems that chaining exported resources might not be too efficient and produces lots of data that could be the reason for puppetdb crashing.

The culprits being these two lines in two manifest files:

./nsca/server.pp:  #File <<| tag == $get_tag |>> -> Nagios_host <<| tag == $get_tag |>>
./nrpe/server.pp:  #File <<| tag == $get_tag |>> -> Nagios_host <<| tag == $get_tag |>> 

And there may be your combinatorial problem.  Supposing that the tags are not specific to the exporting node, the numbers of Files and Nagios_hosts increase linearly with the number of nodes, but the number of relationships among them increases with the square of the number of nodes.
 

replacing them with unchained:

File <<| tag == $get_tag |>>
Nagios_host <<| tag == $get_tag |>>

causes it to run even with 1GB for puppetdb (still a 16GB vm) in under 10 mins, which is acceptable.

Do you in fact need the order of application constraints that the chaining declared?  All of them?

This is an area that seems to give people problems.  On one hand, Puppet provides a couple of mechanisms that allow people to compactly declare large numbers of ordering relationships.  On the other hand, people sometimes lose sight of the distinction between functional dependency and application-order dependency.  Either one of these can cause trouble, as unneeded declarations of any kind slow Puppet and make it consume more resources.  They also make you manifests more susceptible to dependency cycles.  The combination is worse, however, because the number of unneeded relationships produced can be multiplicative.

Depending on what relationships you actually need, you have various options:

1) If it doesn't actually matter whether Puppet syncs given Files before or after it syncs any particular Nagios_host, then you never needed any relationships at all, and simply removing the chaining as you did is the best solution.

2) If specific Files must be synced before specific Nagios_hosts, then you should express those relationships and not any others.  In particular, if you only need relationships among Files and Nagios_hosts exported by the same node, then you should declare the relationships on the exporting side instead of on the collecting side.  (And if they always come in such pairs then perhaps you should wrap then in a defined type and export that instead).

3) If you really need something along the lines that the chaining expressed -- all Files with a given tag synced before all Nagios_hosts bearing that tag -- then you can break the combinatorial problem by introducing an artificial barrier resource for each tag:

define mymodule::barrier() {
  # no content
}

mymodule::barrier { $get_tag: }

File<<| tag == $get_tag |>> -> Mymodule::Barrier[ $get_tag ] -> Nagios_host <<| tag == $get_tag |>>

That gives you (number(Files) + number(Nagios_host)) relationships for each tag instead of (number(Files) * number(Nagios_host)) relationships.

Of course, you can use any single resource as a barrier; it doesn't need to be of a defined type.  In some cases there might be a real resource that naturally fills the role.  If you have Puppet's stdlib module then the Anchor resource would be a good fit; it is intended for a similar purpose anyway.


John

[Ken Barber]

I think John is on to something here, Daniel - I haven't seen your
full content yet, but are you creating a file for each nagios_host, so
that you can use that as the 'target'? Thus creating a single file for
each nagios_host entry?

If this is the case, the John is spot-on ... and you're creating more
relationships than necessary. You probably want one file to one
nagios_* resource, and thus one 'edge' for each, not this many-to-many
case you seem to be creating at collection time.

Of course I'm just guessing, not having seen your content.

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/2_AvVt-Qm4oJ.

[Daniel]

On Thursday, January 24, 2013 2:41:53 PM UTC, jcbollinger wrote:

On Thursday, January 24, 2013 5:24:58 AM UTC-6, Daniel wrote:

Seems that chaining exported resources might not be too efficient and produces lots of data that could be the reason for puppetdb crashing.

The culprits being these two lines in two manifest files:

./nsca/server.pp:  #File <<| tag == $get_tag |>> -> Nagios_host <<| tag == $get_tag |>>
./nrpe/server.pp:  #File <<| tag == $get_tag |>> -> Nagios_host <<| tag == $get_tag |>> 

And there may be your combinatorial problem.  Supposing that the tags are not specific to the exporting node, the numbers of Files and Nagios_hosts increase linearly with the number of nodes, but the number of relationships among them increases with the square of the number of nodes.

Do you in fact need the order of application constraints that the chaining declared?  All of them?

I don't need all of them. The reason I added it was to fix problems with files not being created when puppet tried to realize services, etc., thus producing an incorrect nagios setup. It worked in my small environment and only gave me a pause after days of trying to figure out what the hell is going on in the larger one.

Looks like I didn't understand implications of this configuration, but to be honest I haven't found much in terms of documentation on chaining exported resources, potential implications and problems. Now I know, so "I've learnt something today" :).

One suggestion - I would have a "WFT?" moment a lot sooner if --summarize displayed a number of relationships generated by my crazy config. I don't know if it's possible/easy to implement, but may be worth having a look at.

Depending on what relationships you actually need, you have various options:

1) If it doesn't actually matter whether Puppet syncs given Files before or after it syncs any particular Nagios_host, then you never needed any relationships at all, and simply removing the chaining as you did is the best solution.

 

As stated above, I've added it after failures, so I do need some sort of relationships.

2) If specific Files must be synced before specific Nagios_hosts, then you should express those relationships and not any others.  In particular, if you only need relationships among Files and Nagios_hosts exported by the same node, then you should declare the relationships on the exporting side instead of on the collecting side.  (And if they always come in such pairs then perhaps you should wrap then in a defined type and export that instead).

That's the most likely approach I'm going to implement in the "next iteration" of this module.

3) If you really need something along the lines that the chaining expressed -- all Files with a given tag synced before all Nagios_hosts bearing that tag -- then you can break the combinatorial problem by introducing an artificial barrier resource for each tag:

I most likely don't need this, but will have it in mind for future experiments with exported resources.

Thanks for your comprehensive answer.


Regards,
Daniel