how to resolve hostnames to IP addresses in templates

[Tim Mooney]

Environment: puppet 2.7.14 on both master and all clients. We're also
using puppetlabs-stdlib and hiera, if that matters.

I know this is really more of a ruby/erb question, but I've been searching
for a couple hours and haven't turned up anything relatively close to what
I'm trying to do, and I'm hoping someone here has had to do this and can
provide a suggestion for how to proceed.

I'm generating a configuration file from a template. The configuration
file will need to have IP addresses in it, but I would like to be able to
use either hostnames or IP adresses in the puppet config. This means
that I need to be able to resolve the hostnames and turn them into IP
addresses, probably in the template itself.

Basically, given something like this in puppet:

class foo::data {

$webfarm = {
http_servers => hiera('webfarm_http_servers', [
{
host => 'foo1.example.com',
port => '80',
},
{
host => 'foo2.example.com',
port => '80',
},
{
host => 'foo3.example.com',
port => '80',
},
]),
https_servers => hiera('webfarm_https_servers', [
{
host => 'foo1.example.com',
port => '443',
},
{
host => 'foo22.example.com',
port => '443',
},
{
host => 'foo99.example.com',
port => '443',
},
]),
}
}

I need my template to iterate over the http_servers and https_servers
arrays and resolve the values for the host key for each element.

Anyone have an example of how to do this in a template?

Thanks,

Tim
--
Tim Mooney Tim.M...@ndsu.edu
Enterprise Computing & Infrastructure 701-231-1076 (Voice)
Room 242-J6, IACC Building 701-231-8541 (Fax)
North Dakota State University, Fargo, ND 58105-5164

[Wolf Noble]

if you're using hiera, why not something like:


foo_data_webfarm_http_servers:
foo1.example.com: {
ip: '1.2.3.4',
port: '80',
}
foo2.example.com: {
ip: '2.3.4.5',
port: '80',
}
foo_data_webfarm_https_servers:
foo1.example.com: {
ip: '1.2.3.4',
port: '443',
}
foo2.example.com: {
ip: '2.3.4.5',
port: '443',
}

class foo::data {
include foo::params
file { 'foo.conf':
path => '/etc/foo.conf',
ensure => 'file',
content => template(foo_conf.erb),
mode => '0555',
}

class foo::params {
$webfarm_http_servers = hiera('foo_data_webfarm_http_servers', '')
$webfarm_https_servers = hiera('foo_data_webfarm_https_servers', '')
}


foo_conf.erb
<% http_servers = scope.lookupvar('foo::params::webfarm_http_servers')
https_servers = scope.lookupvar('foo::params::webfarm_https_servers')
-%>
#
# http servers
<% http_servers.each_pair do |key, hash| -%>
#<%=key%>
IPADDR= <%=hash['ip'] %>
PORT= <%=hash['port'] %>
<% end%>
#
# https servers
<% https_servers.each_pair do |key, hash| -%>
#<%=key%>
IPADDR= <%=hash['ip'] %>
PORT= <%=hash['port'] %>
<% end%>

<% end%>



On Aug 9, 2012, at 3:38 PM, Tim Mooney <Tim.M...@ndsu.edu>
wrote:

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>


________________________________

This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you.

[jcbollinger]

On Thursday, August 9, 2012 4:18:17 PM UTC-5, Wolf Noble wrote:

if you're using hiera, why not something like:


foo_data_webfarm_http_servers:
  foo1.example.com: {
    ip: '1.2.3.4',
    port: '80',
  }

[...]

Because that duplicates the data, which you then need to keep synchronized.  Yuck.

It looks like you might be able to get some use out of Ruby's "Resolv" class, but that's a bit of a hack because it does not use the system's native resolver library (hence it might provide different answers in some cases).

I think a more reliable approach would be via Socket.getaddrinfo() or Socket.gethostbyname().  As far as I can tell, those are direct interfaces to appropriate system libraries, but they are probably somewhat trickier to use.


John

[Tim Mooney]

In regard to: Re: [Puppet Users] how to resolve hostnames to IP addresses...:

Thanks for the fully-formed example Wolf. John's subsequent response
was spot-on as to why I would like to avoid duplicating the IPs in the
data structure. They won't change frequently, but at the same time,
I really don't want to have to manually duplicate and sync information
that's stored in our DNS.

I do have one follow-on question regarding your example, though.

Did you choose foo_data_webfarm_http_servers as the "top level" hiera
name for any particular reason, such as how hiera is going to work in
puppet 3 with parameterized classes?

I must admit that while I find hiera fantastic, I've really struggled
with how our complex data should be organized in hiera. I'm looking
forward to the hiera examples that Kelsey Hightower mentioned a couple
weeks ago on the list, but seeing examples like this in the interim is
really helpful and appreciated.

Tim

[Tim Mooney]

In regard to: [Puppet Users] Re: how to resolve hostnames to IP addresses...:

> There's no way to do DNS lookups in a template with stock Puppet,

Thanks for confirming what my futile research seemed to be implying. :-)

> but you can pretty easily write a custom function to do that for you. By
> default, Resolv will use the settings in /etc/resolv.conf, so as long as
> your nameservers are set up correctly on the puppetmaster, you shouldn't
> run into any problems.
>
> Try plopping something like this into lib/puppet/parser/functions/get_ip_addr.rb in your module's directory:
> <a href="https://gist.github.com/3308273">https://gist.github.com/3308273</a>
>
> require 'resolv'
>
> module Puppet::Parser::Functions
> newfunction(:get_ip_addr, :type => :rvalue) do |args|
> # Super sexy regex to match valid IPs
> ip_addr_re = /\b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b/
> hostname = args[0].strip
> if hostname =~ ip_addr_re then return hostname end
> begin
> Resolv::DNS.open { |dns| return dns.getaddress hostname }
> rescue Resolv::ResolvError
> return ''
> end
> end
> end
>
>
> Then you can call it in your template file as ```scope.function_get_ip_addr``` and it will either return the IP address for a hostname, the unchanged IP address for a valid IP address, and an empty string otherwise.
>
> I'm not sure what your hiera() calls are supposed to return, but assuming $webfarm ends up as a hash with keys http_servers and https_servers containing an array of hashes with keys host and port, you could do something like this:
>
> <% webfarm = scope.lookupvar 'foo::data::webfarm' %>
> <% webfarm['http_servers'].each do |server| %>
> <%= scope.function_get_ip_addr server['host'] %>:<%= server['port'] %>
> <% end %>

Thank you for the excellent example! There are several things here that
I'm going to have to ponder for a while. I've seen scope.lookupvar before
but never personally had to use it, but the scope.function_<functionname>
is new to me.

Time to do some more reading and research, but what you've provided really
helps point me in the right direction.

Tim

[Wolf Noble]

On Aug 9, 2012, at 5:42 PM, Tim Mooney <Tim.M...@ndsu.edu>
wrote:

> In regard to: Re: [Puppet Users] how to resolve hostnames to IP addresses...:
>
>> if you're using hiera, why not something like:

[…]
heh yeah,

okay, so, well, uh... I didn't say it was a GOOD idea ;)

> Thanks for the fully-formed example Wolf. John's subsequent response
> was spot-on as to why I would like to avoid duplicating the IPs in the
> data structure. They won't change frequently, but at the same time,
> I really don't want to have to manually duplicate and sync information
> that's stored in our DNS.

I didn't know if they were already being defined somewhere else similarly, and thought if there was already a hash defining them in some way, you could just tack the ip in and call it a day.

>
> I do have one follow-on question regarding your example, though.
>
> Did you choose foo_data_webfarm_http_servers as the "top level" hiera
> name for any particular reason, such as how hiera is going to work in
> puppet 3 with parameterized classes?

I have no real knowledge of how that's going to work; I have been namespacing things based on what I felt made things very clear for people looking at the hiera values, and the templates.

afaik there's no significant penalty for longish names, but it makes it a bit easier for me to ingest when looking through a gaggle of hiera parameters.

>
> I must admit that while I find hiera fantastic, I've really struggled
> with how our complex data should be organized in hiera. I'm looking
> forward to the hiera examples that Kelsey Hightower mentioned a couple
> weeks ago on the list, but seeing examples like this in the interim is
> really helpful and appreciated.

for what it's worth, happy to help…

this gave me another idea, however which I'll create a new thread on rather than hijack this one

[Tim Mooney]

In regard to: Re: [Puppet Users] how to resolve hostnames to IP addresses...:

>> Did you choose foo_data_webfarm_http_servers as the "top level" hiera
>> name for any particular reason, such as how hiera is going to work in
>> puppet 3 with parameterized classes?
>
> I have no real knowledge of how that's going to work; I have been
> namespacing things based on what I felt made things very clear for
> people looking at the hiera values, and the templates.
>
> afaik there's no significant penalty for longish names, but it makes it
> a bit easier for me to ingest when looking through a gaggle of hiera
> parameters.

Understood about the longish names and honestly that's what we're mostly
doing with our hiera usage so far, but every time I look at my site's
hiera setup, it strikes me as wrong, or at least not the best way.

I've posted about this before, but when we converted from extlookup() to
hiera(), we didn't really do any kind of data reorganization. As an
example, we have things like

nameserver_type: client
primary_nameserver: 1.2.3.4
secondary_nameserver: 2.3.4.5
tertiary_nameserver: 3.4.5.6

All of that's fine, but since hiera supports structure, it strikes me as
more elegant to do something like:

nameserver:
type: client
addresses:
- 1.2.3.4
- 2.3.4.5
- 3.4.5.6

etc.

However, there are precious few examples of how to actually look up and
reference nested data in hiera and the ones that do exist make it appear
that it's much more complicated than just using the "namespace via _"
method that we're currently using.

My hope is that some best practices and more examples with be part of some
forthcoming documentation.

[Tomas]

It is not so complicated as others wrote here.

Here is an example how I use it:

allowed_hosts=<% for nagios_server in nagios_servers %><% ipInt = Socket.gethostbyname(nagios_server)[3] %><%= "%d.%d.%d.%d" % [ipInt[0], ipInt[1], ipInt[2], ipInt[3]] %>,<% end %>



Dne čtvrtek, 9. srpna 2012 22:38:14 UTC+2 Tim Mooney napsal(a):

[Stephen Hawes]

Thanks to Tomas!  His answer got me onto a useful track, shared here.

In my case Gethostbyname was returning a bad ip, but I could use Socket.getaddrinfo, like this:

<% ipStr = Socket.getaddrinfo('INST-STAGE-HAPROXY-A',nil)[0][3] %><%= ipStr %>

or with the environment parameterised;

<% ipStr = Socket.getaddrinfo('INST-'+scope.lookupvar('::config::uc_env')+'-HAPROXY-A',nil)[0][3] %><%= ipStr %>

Other useful pointers;

You can test the ruby functions

(yum install irb)

$ irb

2.1.2 :001 > require 'socket'

 => true 

2.1.2 :002 > Socket.getaddrinfo('localhost',nil)

 => [["AF_INET6", 0, "::1", "::1", 30, 2, 17], ["AF_INET6", 0, "::1", "::1", 30, 1, 6], ["AF_INET6", 0, "fe80::1%lo0", "fe80::1%lo0", 30, 2, 17], ["AF_INET6", 0, "fe80::1%lo0", "fe80::1%lo0", 30, 1, 6], ["AF_INET", 0, "127.0.0.1", "127.0.0.1", 2, 2, 17], ["AF_INET", 0, "127.0.0.1", "127.0.0.1", 2, 1, 6]] 

2.1.2 :004 > exit

(yum remove irb)

More ruby functions here: http://ruby-doc.org/stdlib-1.9.3/libdoc/socket/rdoc/Socket.html#method-c-gethostbyname