Ruby script to download files without 'puppet agent'

[zu...@puzzle.ch]

We have often the Problem that some files need to be checked for updates
faster than the cycle of the puppet agent. I try to solve this with a
script which tries to download the files directly from the fileserver of
the puppetmaster. So far i couldn't get it to work.

I don't know if i got the URL right, i did not find any examples on the
REST API documentation for the fileserver.

for example if the file is /etc/puppet/manifests/files/sudo/sudoers

fileserver.conf:
[files]
path /etc/puppet/manifests/files
allow 127.0.0.1
allow *.bar

to what path would this translate?
https://foo.bar:8140/file_content/sudo/sudoers
https://bli.bla:8140/file_content/files/sudo/sudoers

Greetings
Andy

[Nigel Kersten]

On Mon, Jun 27, 2011 at 7:27 AM, <zu...@puzzle.ch> wrote:

We have often the Problem that some files need to be checked for updates
faster than the cycle of the puppet agent. I try to solve this with a
script which tries to download the files directly from the fileserver of
the puppetmaster. So far i couldn't get it to work.

Have you considered using tags instead to filter for a subset of your resources?

 

I don't know if i got the URL right, i did not find any examples on the
REST API documentation for the fileserver.

This should cover it.

http://docs.puppetlabs.com/guides/rest_api.html#file-server

[zu...@puzzle.ch]

> On Mon, Jun 27, 2011 at 7:27 AM, <zu...@puzzle.ch> wrote:
>
>> We have often the Problem that some files need to be checked for updates
>> faster than the cycle of the puppet agent. I try to solve this with a
>> script which tries to download the files directly from the fileserver of
>> the puppetmaster. So far i couldn't get it to work.
>>
>
> Have you considered using tags instead to filter for a subset of your
> resources?

I need to check 1 file and 1 directory every 5min from >600 puppet nodes.
As far as i know the puppetmaster still needs to compile the whole
catalog. I did not test this, but i think this would not scale well.

[Nigel Kersten]

Possibly.

You could also do something like this. Put the thing you need to check frequently into a module, we'll call it "frequent" for now, and structure your setup with environments like the following:

# normal environment

[normal]

modulepath=/var/lib/puppet/env/normal/modules:/var/lib/puppet/env/frequent/modules

[frequent]

modulepath=/var/lib/puppet/env/frequent/modules

Then you could choose to run every 5 minutes against the "frequent" environment, and you'd have a very minimal catalog for the rapid runs, and a more comprehensive one for your "normal" runs.

[zu...@puzzle.ch]

This isn't still clear to me. Would appreciate if someone could give a
concrete example how a path in a module or in the filepath would tanslate.

Here is what i try:

fileserver.conf
[files]
path /etc/puppet/manifests/files
allow 127.0.0.1

allow *.smobi.mobicorp.test

The file is located at /etc/puppet/manifests/files/sudo/sudoers

[root@foo me]# ./puppet-wget file_content/sudo/sudoers
https://foo.bar:8140/file_content/sudo/sudoers
warning: peer certificate won't be verified in this SSL session
/usr/lib/ruby/1.8/net/http.rb:2099:in `error!': 403 "Forbidden request:
foo.bar(10.2.3.4) access to /sudo/sudoers [find] authenticated at line 93
" (Net::HTTPServerException)
from ./puppet-wget:56
from ./puppet-wget:43:in `each'
from ./puppet-wget:43

AFAIK if something is wrong with the certs the server would answer with a
401 and not 403. It really looks to me as if i get that path wrong.

------------------
#!/bin/env ruby
#
# Puppet Wget
#
# Downloads files from the puppetmaster without the puppet agent

require 'optparse'
require 'puppet/rails'
require 'uri'
require 'net/https'

# Parse all Options
options = {}
OptionParser.new do |opts|
opts.banner = "Usage: puppet-wget [options] module/path/to/file"

opts.on("-r", "--recursive", "Download directory recursively") do |r|
options[:recursive] = r
end
end.parse!

# Get some settings for the puppet config
Puppet[:config] = "/etc/puppet/puppet.conf"
Puppet.parse_config
puppet_conf = Puppet.settings.instance_variable_get(:@values)[:main]

# default values
puppet_conf[:server] ||= 'puppet'
puppet_conf[:masterport] ||= '8140'
puppet_conf[:ssldir] ||= '/etc/puppet/ssl'

server = puppet_conf[:server]
port = puppet_conf[:masterport]
certpath = puppet_conf[:ssldir] + '/certs/' + ENV['HOSTNAME'] +
'.pem'
pkey_path = puppet_conf[:ssldir] + '/private_keys/' + ENV['HOSTNAME'] +
'.pem'

ARGV.each do |filepath|
url = URI.parse("https://#{server}:#{port}/#{filepath}")
req = Net::HTTP::Get.new("#{url.path}?#{url.query}", "Accept" => 's')

puts url

# make ssl request
connection = Net::HTTP.new(url.host, url.port)
connection.use_ssl = true
connection.cert = OpenSSL::X509::Certificate.new(File.read(certpath))
connection.key = OpenSSL::PKey::RSA.new(File.read(pkey_path))
res = connection.start { |http| http.request(req) }

res.error! unless res.code_type == Net::HTTPOK
puts res.body

end

[Ohad Levy]

you are missing the CA file... something like
connection.ca_file = Puppet[:localcacert]

>  res = connection.start { |http| http.request(req) }
>
>  res.error! unless res.code_type == Net::HTTPOK
>  puts res.body
>
> end
>
>

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>
>

[Andreas Zuber]

> > # make ssl request
> > connection = Net::HTTP.new(url.host, url.port)
> > connection.use_ssl = true
> > connection.cert = OpenSSL::X509::Certificate.new(File.read(certpath))
> > connection.key = OpenSSL::PKey::RSA.new(File.read(pkey_path))
>
> you are missing the CA file... something like
> connection.ca_file = Puppet[:localcacert]

Tested that, but it seams to make no difference. From what i unerstand the ca
is not really needed for this on the client.

[Peter Meier]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> [root@foo me]# ./puppet-wget file_content/sudo/sudoers
> https://foo.bar:8140/file_content/sudo/sudoers
> warning: peer certificate won't be verified in this SSL session
> /usr/lib/ruby/1.8/net/http.rb:2099:in `error!': 403 "Forbidden request:
> foo.bar(10.2.3.4) access to /sudo/sudoers [find] authenticated at line 93
> " (Net::HTTPServerException)
> from ./puppet-wget:56
> from ./puppet-wget:43:in `each'
> from ./puppet-wget:43

Using your script works to get a file from a module files, like:

./puppet_wget
/development/file_content/modules/site-bind/etc/rz.foo.ch/named.conf

However your error looks like the client is not authorized to find that
content. Maybe some tweaks in the auth.conf are missing to fetch files
from the fileserver instead from modules?

~pete
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk4LCqYACgkQbwltcAfKi38O9wCgnUKnyugLiukznBcjjLQ2wXbd
sK0AnA8UAYkrIiFPAkIbCELvFLu7F6q3
=iiOQ
-----END PGP SIGNATURE-----

[zu...@puzzle.ch]

The solution to this was easier than expected.
I simply write the folowing file and execute it with puppet apply

-----
$source = 'puppet.example.com'

file{'/etc/sudoers':
source => [ "puppet://$source/files/sudo/sudoers/${fqdn}/sudoers",
"puppet://$source/files/sudo/sudoers/sudoers",
"puppet://$source/sudo/sudoers/${operatingsystem}/sudoers",
"puppet://$source/sudo/sudoers/sudoers" ],
owner => root, group => 0, mode => 0440;
}
-----

It then just deploys this one file without the need of compiling a
catalog. I can call this from cron then as often as i wish.

I saw that some use the shebang "#!/usr/bin/puppet apply" to start such
manifests directly as a script. This does not seam to work for me. All i
get is:

./puppet_sync_sudo2.pp: line 3: =: command not found
./puppet_sync_sudo2.pp: line 5: file{/tmp/sudoers:: No such file or directory
./puppet_sync_sudo2.pp: line 6: =: No such file or directory
./puppet_sync_sudo2.pp: line 7: puppet:///files/sudo/sudoers/sudoers,: No
such file or directory
./puppet_sync_sudo2.pp: line 8: puppet:///sudo/sudoers//sudoers,: No such
file or directory
./puppet_sync_sudo2.pp: line 9: puppet:///sudo/sudoers/sudoers: No such
file or directory
./puppet_sync_sudo2.pp: line 10: owner: command not found
./puppet_sync_sudo2.pp: line 11: syntax error near unexpected token `}'
./puppet_sync_sudo2.pp: line 11: `}'

Any ideas?

Greetings
Andy

[vagn scott]

On 07/18/2011 11:27 AM, zu...@puzzle.ch wrote:
> I saw that some use the shebang "#!/usr/bin/puppet apply" to start such
> manifests directly as a script. This does not seam to work for me. All i
> get is:

If your script name is foo, and you have

"#!/usr/bin/puppet apply"

as the first line, then you can run the script with

chmod +x foo
./foo

what will not work is:

sh foo

it fails because it is not a shell script.

--
vagn

[Nick Fagerlund]

You're not the only one -- I've never gotten a puppet apply shebang
line to work. I investigated around the office, and I think what I
found was that everyone remembered it having worked at some point in
the unspecified past, but no one could specify a version where it
definitely worked, and it certainly doesn't work now.

[Nigel Kersten]

On Mon, Jul 18, 2011 at 3:45 PM, Nick Fagerlund
<nick.fa...@puppetlabs.com> wrote:
> You're not the only one -- I've never gotten a puppet apply shebang
> line to work. I investigated around the office, and I think what I
> found was that everyone remembered it having worked at some point in
> the unspecified past, but no one could specify a version where it
> definitely worked, and it certainly doesn't work now.

Hey you didn't ask me :)

It's worked with env for me for quite a while.

~ nbk $ cat /tmp/test.pp
#!/usr/bin/env puppet apply
notify { "woot!": }

~ nbk $ /tmp/test.pp
notice: woot!
notice: /Stage[main]//Notify[woot!]/message: defined 'message' as 'woot!'
notice: Finished catalog run in 0.01 seconds

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>
>

--
Nigel Kersten
Product Manager, Puppet Labs

Join us for PuppetConf
<http://www.bit.ly/puppetconfsig>
Sept 22/23 Portland, Oregon, USA.

[vagn scott]

On 07/18/2011 06:45 PM, Nick Fagerlund wrote:
> You're not the only one -- I've never gotten a puppet apply shebang
> line to work. I investigated around the office, and I think what I
> found was that everyone remembered it having worked at some point in
> the unspecified past, but no one could specify a version where it
> definitely worked, and it certainly doesn't work now.

vagn@nika:~$ chmod +x foo
vagn@nika:~$ ./foo
notice: Scope(Class[main]): hello world!
vagn@nika:~$ cat foo
#! /usr/bin/puppet apply

notice("hello world!")

vagn@nika:~$ puppet --version
2.6.2

vagn@nika:~$ lsb_release -d
Description: Debian GNU/Linux 6.0.2 (squeeze)
vagn@nika:~$