Re: [Puppet Users] hiera command line performance

[R.I.Pienaar]

hello,

----- Original Message -----
> From: "Paul Colby" <pco...@gmail.com>
> To: puppet...@googlegroups.com
> Sent: Monday, August 13, 2012 6:22:10 PM
> Subject: [Puppet Users] hiera command line performance
>
> Hi everyone,
>
>
> The hiera command line program (version 0.3.0) is currently taking
> around 2 seconds per invocation. This seems quite slow to me, but is
> probably not an issue for most users.

2 seconds seem quite slow though i guess it depends on the size of your
yamls and so forth, sounds like yours are big and complex. I'd hope the
bulk of the time is spent on starting/stopping ruby each time.

A better option might be to consider writing your script in ruby rather
than bash so you can then do your lookup that way reusing the single
hiera instance and avoid the ruby start/stop cost

So a simple test on my setup doing:

for i in {1..100}
do
hiera syslocation ::location=foo
done

this takes 30 seconds, ages. Here is the same in Ruby reusing the
class and using a single interpreter:

h = Hiera.new(:config => "/etc/hiera.yaml")

Benchmark.measure do
100.times do
puts h.lookup("syslocation", "", {"::location" => "foo"})
end
end

this takes 0.1 second for 100 lookups. If you know a bit of ruby this
would be a good approach for you even if you just write your script to
configure a single machine in ruby and run it 40 times only

>
>
> We have a number of legacy machines that for one reason or another
> have not been puppetised yet, and probably won't be anytime soon, so
> I have a somewhat clever script that, for each of these hosts, uses
> the hiera command like utility to export a few puppet-templated
> documents just as puppet+hiera does for our puppeted server. This
> works very nicely, but involves ~250 hiera lookups per host, which
> at 2 seconds per lookup, is taking more than 8 minutes per host (and
> using near 100% of a CPU core the entire time). As we have ~40 such
> hosts, that's over 5 hours with no concurrency (we do use some
> concurrency, but since each invocation uses near 100% of a CPU core
> each, there's no point running more simultaneous hiera processes
> than the number of CPUs, and even then, we need some reserved for
> other services).
>
>
> So, is there some way I can make this export run faster? The bulk of
> the time seems to be spent in initialising the hiera command line
> program, not the actual data lookup (for example, if I fail to
> provide the necessary identity / scope files, it still takes 2
> seconds to get around to returning an error, but if I fail to
> provide a config file, it returns instantaneously with error). So,
> I'd either like to make the program load / initialise much faster,
> or be able to query multiple values for a single hiera command line
> invocation.
>
>
> Also, is there a more recent version of hiera that would be faster?
> (I'm a little unclear about hiera versioning - we're running 0.3.0,
> but I don't know if that's recent or old).
>
>
> Any suggestions / tips would be greatly appreciated :)
>
>
> Thanks.
>
>
> Paul.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/YTZZ7FOshJkJ .
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>

[Paul Colby]

Thanks R.I. Pienaar, that's exactly the kind of suggestion I was hoping for! :)

I haven't written any Ruby for ~9 years, but as I regularly develop in several other languages (primarily C++), I'll enjoy getting my hands dirty with Ruby again.

And thanks for including a basic single-interpreter example - that will help me get going a lot quicker!

Thanks,

Paul.

[Paul Colby]

Thanks again R.I. Pienaar.

I just got time to get back to this, I've implemented a basic ruby script to export the hiera data as you've suggested - works extremely well!! :)

Interestingly, the performance issue seems to be with puppet specifically, not hiera.

Here's a slightly cut-down version of my export script:

#!/usr/bin/ruby

require 'rubygems'

require 'hiera'

#require 'puppet'

# Load the "facts" for the host to export settings for.

if ARGV.length != 1

  puts "Usage: #{File.basename($PROGRAM_NAME)} /path/to/host-facts.yaml"

  exit

end

scope = YAML.load_file(ARGV.first)

scope['calling_module'] = 'settings'

# Fetch all settings for this host from hiera.

hiera = Hiera.new(:config => '/etc/puppet/hiera.yaml')

settings = hiera.lookup('settings', '', scope, '', :array)

settings.each do |setting|

  puts "#{setting}=#{hiera.lookup(setting, '', scope)}"

end

Note that the "require 'puppet'" line is commented out... code examples on the net use that statement, but the scripts runs fine both with and without it.

The interesting thing is, without that require, the script is nice and quick, but with it, it is 2.5 times slower.

Example times:

With "require puppet"

real    0m2.448s

user    0m2.123s

sys     0m0.323s

Without "require puppet"

real    0m0.917s

user    0m0.808s

sys     0m0.098s

Anyway, I'm very happy with the result!  The inner "*.each" call at the end means I'm calling this script ~40 times, instead of the (slower) hiera command line program ~10,000 times.  Much faster, and lower CPU usage too :)

Thanks again.

Paul.

[R.I.Pienaar]

----- Original Message -----
> From: "Paul Colby" <pco...@gmail.com>
> To: puppet...@googlegroups.com

> Sent: Friday, August 24, 2012 11:03:51 AM
> Subject: Re: [Puppet Users] hiera command line performance
>
> Thanks again R.I. Pienaar.

awesome, glad you could sort it out :)

pretty interesting that just requiring puppet slows it down that much