Hiera Backend Installation and LOAD_PATH requirement?

[Matt Wise]

I've been writing a new Hiera backend (here) and its working fine in all of my local and unit tests. I can run Puppet Apply with an appropriate LOAD_PATH and it works just fine. However, when we actually merged it in to run on our test puppet master, we ran into load problems:

Strace from Puppet Master trying to load rstag_backend: here

Hiera Config: here

Based on my read of the docs:

When using Hiera with Puppet, you can load backends from the lib directory of a Puppet module; however, these backends won’t be loaded when you run Hiera from the command line unless you specify that directory in your RUBYLIB environment variable.

We should be able to simply import our puppet module and use the backend... but that doesn't seem to be the case. It does seem like I might be able to simply add a symlink in the root of my Puppet code path from ./hiera/backends/rstag_backend.rb -> ./modules/rightscale/lib/hiera/backends/rstag_backend.rb, but that doesn't seem like it should be necessary.

Any thoughts?

We're running Puppet 3.4.1 by the way.. 

[Matt Wise]

Just to followup .. running a simple puppet apply on the puppet master works fine -- the module loads, it gets backend data properly. I just can't get it to work from within Unicorn for our remote clients.

Here's our simple unicorn.conf:

worker_processes 4
working_directory '/etc/puppet'
listen '/var/run/puppet/unicorn.sock', :backlog => 512
timeout 180
pid '/var/run/puppet/unicorn.pid'
   
preload_app true
if GC.respond_to?(:copy_on_write_friendly=)
  GC.copy_on_write_friendly = true
end
before_fork do |server, worker|
  old_pid = "#{server.config[:pid]}.oldbin"
  if File.exists?(old_pid) && server.pid != old_pid
    begin
      Process.kill("QUIT", File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
      # someone else did our job for us
    end
  end
end

and our config.ru:

$0 = "master"
ARGV << "--confdir" << "/etc/puppet"
ARGV << "--rack"
ARGV << "--vardir"  << "/var/lib/puppet"
require 'puppet/util/command_line'
require 'unicorn/worker_killer'
run Puppet::Util::CommandLine.new.execute
use Unicorn::WorkerKiller::MaxRequests, 2048 + rand(1024)
use Unicorn::WorkerKiller::Oom, (96 + rand(32)) * 1024**2 

[Matt Wise]

After working with binford2k on #puppet, we discovered two issues...

  1. We had to explicitly add /var/lib/puppet/lib to the LOAD_PATH via the config.ru file.

  2. Since our Puppet masters use 'puppet apply' to execute their own configurations locally, they were not populating /var/lib/puppet/lib at all. It seems that 'puppet apply' dynamically figures out the LIB paths properly, but even with the '--pluginsync' option it does not actually populate /var/lib/puppet/lib at all. For now I've added a hack to our servers to execute 'puppet plugin download' locally, but thats a real nasty hack.

Based on the documentation, shouldn't the puppet master daemon be able to autoload up the hiera/backend/rstag_backend.rb just like 'puppet apply' does?