Extlookup() again.

[Douglas Garstang]

Can't get the extlookup() that supports yaml to work.

I did this...

On server:

mv /usr/lib/ruby/site_ruby/1.8/puppet/parser/functions/extlookup.rb /usr/lib/ruby/site_ruby/1.8/puppet/parser/functions/extlookup.rb.orig

Replaced /usr/lib/ruby/site_ruby/1.8/puppet/parser/functions/extlookup.rb with puppet-extlookup/lib/puppet/parser/functions/extlookup.rb

mkdir /usr/lib/ruby/site_ruby/1.8/puppet/util/extlookup/

Added puppet-extlookup/lib/puppet/util/extlookup.rb to /usr/lib/ruby/site_ruby/1.8/puppet/util

Added puppet-extlookup/lib/puppet/util/extlookup/csv_parser.rb to /usr/lib/ruby/site_ruby/1.8/puppet/util/extlookup

Added puppet-extlookup/lib/puppet/util/extlookup/yaml_parser.rb to /usr/lib/ruby/site_ruby/1.8/puppet/util/extlookup

Added puppet-extlookup/lib/puppet/util/extlookup/puppet_parser.rb to /usr/lib/ruby/site_ruby/1.8/puppet/util/extlookup

Created /etc/puppet/extlookup.yaml:

---

:parser: YAML

:precedence:

- environment_%{environment}

- common

:yaml:

  :datadir: /etc/puppet/extdata

Created /etc/puppet/extdata/settings.yaml:

---

testkey: testval

Calling with:

        $test_var = extlookup('testkey', '---', 'settings.yaml')

        notice("TEST VAR=$test_var")

And $test_var is always '---'. Replacing settings.yaml with a bogus file name also returns '---', which tells me that the file isn't being found. What am I missing here? Quite possibly a lot given the really bad docs.

What is 'precedence' for? Not documented. 

Doug.

[Douglas Garstang]

Actually, when I run the puppetmaster in debug mode on the server, I see:

debug: Automatically imported testmodule::test01 from testmodule/test01 into prod2

debug: extlookup/yaml: looking for key=testkey with default=---

debug: extlookup/yaml: Looking for data in /etc/puppet/extdata

debug: extlookup/yaml: Looking for data in /etc/puppet/extdata/settings.yaml.yaml

debug: extlookup/yaml: Looking for data in /etc/puppet/extdata/environment_prod2.yaml

debug: extlookup/yaml: Looking for data in /etc/puppet/extdata/common.yaml

notice: Scope(Class[Testmodule::Test01]): TEST VAR=---

debug: File[/tmp/y]: Adding default for backup

.... so, it's looking in the right place for the right key.... it just doesn't get it...

Doug.

[Aaron Grewell]

Precedence is for having it look in multiple places.  I've got node-level, site-level, and default YAML files for example.

On Wed, Aug 3, 2011 at 3:50 PM, Douglas Garstang <doug.g...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[Douglas Garstang]

Thanks Aaron. Seems you've had some experience with this, so..

I'm a little confused about how it loads complex data. With a very simple YAML file like this:

---

testkey:

- Doug

- Bob

- Bill

 

calling $test_var = extlookup('testkey', '---', 'settings') gives me a string 'DougBobBill', which isn't what I expected to see. Actually, now that I think about it, I'm not sure what I'm supposed to see.

Also, if I had this:

---

testkey1:

  testkey2:

    - Doug

how do I access the key testkey2?

Doug.

[Scott Smith]

Lists (arrays) in YAML are represented with the `-' prefix. So you created a list called `testkey'.

When you echo a list in Puppet, it concatenates the elements. That's why you got `DougBobBill'

--

[Aaron Grewell]

Example Config:
---
:parser: YAML
:precedence:
- %{environment}/nodes/%{fqdn}
- %{environment}/nodes/cluster_%{cluster}
- %{environment}/nodes/site_%{site}
- %{environment}/nodes/default
:yaml:
   :datadir: /usr/share/puppet/environments

Example call:
$default_packages = extlookup('linux_default_packages')

Given an environment called 'testing', cluster called 'cluster1', site called 'site1' and a node called localhost.localdomain it will look for variable 'linux_default_packages' in:

/usr/share/puppet/environments/testing/localhost.localdomain.yaml
/usr/share/puppet/environments/testing/cluster_cluster1.yaml
/usr/share/puppet/environments/testing/site_site1.yaml
/usr/share/puppet/environments/testing/default.yaml

In that order, which means you can set this at the default level, then override at any lower level you like.  Make sense?

On Wed, Aug 3, 2011 at 4:15 PM, Aaron Grewell <aaron....@gmail.com> wrote:

[Aaron Grewell]

Argh, just realized I left something out of those paths, should be:

/usr/share/puppet/environments/testing/nodes/localhost.localdomain.yaml
/usr/share/puppet/environments/testing/nodes/cluster_cluster1.yaml
/usr/share/puppet/environments/testing/nodes/site_site1.yaml
/usr/share/puppet/environments/testing/nodes/default.yaml

[R.I.Pienaar]

----- Original Message -----
> Argh, just realized I left something out of those paths, should be:

This all works exactly as extlookup also worked, nothing new there.

The general behavior is well documented even in the Puppet core docs

[Douglas Garstang]

On Wed, Aug 3, 2011 at 4:25 PM, Aaron Grewell <aaron....@gmail.com> wrote:

Example Config:
---
:parser: YAML
:precedence:
- %{environment}/nodes/%{fqdn}
- %{environment}/nodes/cluster_%{cluster}
- %{environment}/nodes/site_%{site}
- %{environment}/nodes/default
:yaml:
   :datadir: /usr/share/puppet/environments

Example call:
$default_packages = extlookup('linux_default_packages')

Given an environment called 'testing', cluster called 'cluster1', site called 'site1' and a node called localhost.localdomain it will look for variable 'linux_default_packages' in:

/usr/share/puppet/environments/testing/localhost.localdomain.yaml
/usr/share/puppet/environments/testing/cluster_cluster1.yaml
/usr/share/puppet/environments/testing/site_site1.yaml
/usr/share/puppet/environments/testing/default.yaml

In that order, which means you can set this at the default level, then override at any lower level you like.  Make sense?

Aaron, makes sense. I wasn't so interested in the defaulting behaviour, because you can do the same thing with extlookup itself with something like this:

$ssh_idle_timeout = extlookup("SshIdleTimeout", extlookup("SshIdleTimeout", extlookup("SshIdleTimeout", "" ,"settings/nodes/_global"), "settings/nodes/${domain}"), "settings/nodes/${fqdn}")

In fact, I haven't had a chance to think fully though it yet, but I think this implementation of extlookup actually removes functionality. By chaining extlookup() calls together like I have above you can determine what lookup order you want to use whenver you want, rather than having to rely on a centrally configured file.

But... I was much more interested in what I thought was the possibility of storing complex YAML data and retrieving it with puppet. What does $linux_default_packages end up looking like? Is it a string? If it's a string, how do you parse it?

I dunno... seems like all you can store in the yaml files is key/value pairs, which means it's no better than using a CSV file. No?

Doug

[Aaron Grewell]

It's an array, which can then be exploded into a whole set of package resources with a single entry:
package {$default_packages:}

[Douglas Garstang]

Ok, so am I getting 'DougBobBill' when I use $test_var = extlookup('testkey', '---', 'settings') on this file?

---

testkey:

- Doug

- Bob

- Bill

Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.g...@gmail.com
Cell: +1-805-340-5627

[R.I.Pienaar]

----- Original Message -----
> Ok, so am I getting 'DougBobBill' when I use $test_var =
> extlookup('testkey', '---', 'settings') on this file?

how are you examining this value?

[Scott Smith]

http://groups.google.com/group/puppet-users/msg/6432a6dd539b542f

[Douglas Garstang]

My own post....? y?