hiera consul backend Puppet 4.10
160 views
Skip to first unread message
Justin DynamicD
Aug 29, 2017, 7:17:46 PM8/29/17
to Puppet Users
Hi,
As you can see this does very little other than perform a hiera lookup (for hostname) and then dump the output into a file so I can see what it's gathering.
The file is always empty.
I've tried a few different variations to attempt to get data out, but I'm not certain of how. What I'd like to emulate is simply:
anyone have some good concise examples on how to setup using a consul backend in hiera? I've attempted with little luck to get this backend to work.
So far these are the steps I've tried:
1. Imported lynxman/hiera-consul into in the Puppetfile
So far these are the steps I've tried:
1. Imported lynxman/hiera-consul into in the Puppetfile
2. Modified puppetserver.conf "ruby-load-path" to include the agent libdir (and presumably pickup the consul_backend.rb) via: https://tickets.puppetlabs.com/si/jira.issueviews:issue-html/SERVER-571/SERVER-571.html
3. Validated basic hiera yaml was working by creating a simple "common.yaml" and ensured variables were being picked up during puppet runs (this was mostly to ensure hiera as a subsystem was t least functioning).
4. Verified I can directly query consul with either consul kv, curl, and even dig.
With all this in place, I can't seem to get any info back. For a fun test I created a truly "simply" little class:
class hieratest (
$getdata = hiera('%{::hostname}'), ) {
$myresult = join($getdata, ",")
file { '/root/dump.txt' :
ensure => present,
content => inline_template($myresult),
}
}As you can see this does very little other than perform a hiera lookup (for hostname) and then dump the output into a file so I can see what it's gathering.
The file is always empty.
I've tried a few different variations to attempt to get data out, but I'm not certain of how. What I'd like to emulate is simply:
consul catalog services -node=<nodename>
This simple command returns all the services registered to a specified node as an array. That's basically all I'm trying to get back from consul via hiera so I can then use simple If "application" in $getdata".
Anyone have any insight on what setting I might be missing? Simple syntax issue? consul trick?
Anyone have any insight on what setting I might be missing? Simple syntax issue? consul trick?
Justin DynamicD
Aug 29, 2017, 8:38:33 PM8/29/17
to Puppet Users
Quick update:
I found out i can run "hiera <nodename>" successfuly on the server as long as I link the the file to the path approrpiate to the hiera command (designated with a 'which'). still doesn't apear to function from within a puppet manifest, however ... so I'm more supicious of a syntax issue.
Also ... is hiera something we _only_ need to function on the server? or is the server going to let the local client try to process the hiera lookup? Basically I have no idea where I have to ensure hiera is running properly.
I found out i can run "hiera <nodename>" successfuly on the server as long as I link the the file to the path approrpiate to the hiera command (designated with a 'which'). still doesn't apear to function from within a puppet manifest, however ... so I'm more supicious of a syntax issue.
Also ... is hiera something we _only_ need to function on the server? or is the server going to let the local client try to process the hiera lookup? Basically I have no idea where I have to ensure hiera is running properly.
Justin DynamicD
Aug 30, 2017, 3:32:44 AM8/30/17
to Puppet Users
I finally was able to solve this issue on my own, so posting for others who may have been lost:
the hiera () syntax is _very_ sensitive, even more-so than runnning hiera and the command prompt. So while:
hiera <server>.Services would call back all the services on a node from the commanline,
the hiera () syntax is _very_ sensitive, even more-so than runnning hiera and the command prompt. So while:
hiera <server>.Services would call back all the services on a node from the commanline,
hiera ($::hostname.Services) would fail, and instead you have to settle for the "base" match of simply hiera(<server>). once you have that base array you can then call out hashes from there.
There may be a way to abuse ruby to get the info a bit more efficiently, but that's out of scope of this question.
Henrik Lindberg
Aug 30, 2017, 7:31:11 AM8/30/17
to puppet...@googlegroups.com
On 30/08/17 09:32, Justin DynamicD wrote:
> I finally was able to solve this issue on my own, so posting for others
> who may have been lost:
>
> the hiera () syntax is _very_ sensitive, even more-so than runnning
> hiera and the command prompt. So while:
>
> hiera <server>.Services would call back all the services on a node from
> the commanline,
> hiera ($::hostname.Services) would fail, and instead you have to settle
> for the "base" match of simply hiera(<server>). once you have that base
> array you can then call out hashes from there.
>
> There may be a way to abuse ruby to get the info a bit more efficiently,
> but that's out of scope of this question.
>
From Puppet 4.10 and forward you have hiera 5 available. The earlier
> I finally was able to solve this issue on my own, so posting for others
> who may have been lost:
>
> the hiera () syntax is _very_ sensitive, even more-so than runnning
> hiera and the command prompt. So while:
>
> hiera <server>.Services would call back all the services on a node from
> the commanline,
> hiera ($::hostname.Services) would fail, and instead you have to settle
> for the "base" match of simply hiera(<server>). once you have that base
> array you can then call out hashes from there.
>
> There may be a way to abuse ruby to get the info a bit more efficiently,
> but that's out of scope of this question.
>
hiera 3 and 4 formats and APIs have been deprecated in favor of hiera 5,
and the lookup CLI is favoured over the hiera CLI, and the lookup
function favoured over the hiera, hiera_array, hiera_has (etc) functions.
With hiera 5 a "hiera backend" is simply a function implemented using
puppet 4.x function API which makes backend writing much simpler.
The lookup function is more powerful than the older hiera calls, and you
can do things like 'dot style dig' into data structures directly in the
lookup.
Lots of information in the documentation for hiera 5. Yu may want to
start reading here:
https://docs.puppet.com/puppet/5.1/hiera_intro.html#whats-the-deal-with-hiera-5
Best,
- henrik
> content =>inline_template($myresult),
> }
>
> }
> |
>
>
>
> As you can see this does very little other than perform a hiera
> lookup (for hostname) and then dump the output into a file so I
> can see what it's gathering.
>
> The file is always empty.
>
> I've tried a few different variations to attempt to get data
> out, but I'm not certain of how. What I'd like to emulate is
> simply:
>
> consul catalog services -node=<nodename>
>
> This simple command returns all the services registered to a
> specified node as an array. That's basically all I'm trying to
> get back from consul via hiera so I can then use simple If
> "application" in $getdata".
>
> Anyone have any insight on what setting I might be missing?
> Simple syntax issue? consul trick?
>
>
> --
> }
>
> }
> |
>
>
>
> As you can see this does very little other than perform a hiera
> lookup (for hostname) and then dump the output into a file so I
> can see what it's gathering.
>
> The file is always empty.
>
> I've tried a few different variations to attempt to get data
> out, but I'm not certain of how. What I'd like to emulate is
> simply:
>
> consul catalog services -node=<nodename>
>
> This simple command returns all the services registered to a
> specified node as an array. That's basically all I'm trying to
> get back from consul via hiera so I can then use simple If
> "application" in $getdata".
>
> Anyone have any insight on what setting I might be missing?
> Simple syntax issue? consul trick?
>
>
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com
> <mailto:puppet-users...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/eb1a7ae9-a75e-4acc-82a4-41eb05e103bf%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/eb1a7ae9-a75e-4acc-82a4-41eb05e103bf%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.
--
Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/
Justin DynamicD
Aug 30, 2017, 12:23:54 PM8/30/17
to Puppet Users
Thanks for this info.
My biggest problem right now is the backend is Hiera 3, and I'm not aware of any written/usable consul backends written in Hiera_5. It took me long enough to get this syntax working, writing a backend is well outside my wheelhouse at this time, so writing a new custom hiera_5 consul backend isn't going to happen.
Can I use Hiera_5 syntax against a Hiera 3 backend?
My biggest problem right now is the backend is Hiera 3, and I'm not aware of any written/usable consul backends written in Hiera_5. It took me long enough to get this syntax working, writing a backend is well outside my wheelhouse at this time, so writing a new custom hiera_5 consul backend isn't going to happen.
Can I use Hiera_5 syntax against a Hiera 3 backend?
> <mailto:puppet-users+unsub...@googlegroups.com>.
Henrik Lindberg
Aug 30, 2017, 5:16:25 PM8/30/17
to puppet...@googlegroups.com
On 30/08/17 18:23, Justin DynamicD wrote:
> Thanks for this info.
>
> My biggest problem right now is the backend is Hiera 3, and I'm not
> aware of any written/usable consul backends written in Hiera_5. It took
> me long enough to get this syntax working, writing a backend is well
> outside my wheelhouse at this time, so writing a new custom hiera_5
> consul backend isn't going to happen.
>
> Can I use Hiera_5 syntax against a Hiera 3 backend?
>
Yes you can the only constraint is that hiera 3 backends are only
> Thanks for this info.
>
> My biggest problem right now is the backend is Hiera 3, and I'm not
> aware of any written/usable consul backends written in Hiera_5. It took
> me long enough to get this syntax working, writing a backend is well
> outside my wheelhouse at this time, so writing a new custom hiera_5
> consul backend isn't going to happen.
>
> Can I use Hiera_5 syntax against a Hiera 3 backend?
>
allowed in the global layer's hiera.yaml. You also get less --explain
output from a hiera 3 backend than what you get from hiera 5 backends.
- henrik
> > <mailto:puppet-users...@googlegroups.com <javascript:>>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/puppet-users/eb1a7ae9-a75e-4acc-82a4-41eb05e103bf%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/eb1a7ae9-a75e-4acc-82a4-41eb05e103bf%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/puppet-users/eb1a7ae9-a75e-4acc-82a4-41eb05e103bf%40googlegroups.com?utm_medium=email&utm_source=footer
> >
> https://groups.google.com/d/msgid/puppet-users/eb1a7ae9-a75e-4acc-82a4-41eb05e103bf%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/eb1a7ae9-a75e-4acc-82a4-41eb05e103bf%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/puppet-users/eb1a7ae9-a75e-4acc-82a4-41eb05e103bf%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> > For more options, visit https://groups.google.com/d/optout
> <http://puppet-on-the-edge.blogspot.se/>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com
> <mailto:puppet-users...@googlegroups.com>.
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/ce602930-a49a-4038-a032-be4d96fec8e8%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/ce602930-a49a-4038-a032-be4d96fec8e8%40googlegroups.com?utm_medium=email&utm_source=footer>.
Justin DynamicD
Aug 31, 2017, 12:29:02 PM8/31/17
to Puppet Users
Appreciate your feedback. After a bit of "mucking about" I've added a v5 yaml to a branch and will probably merge it in soon, but have left the global yaml inplace as v3 until I can do more testing ...
That said the lookup() feature doesn't feel any easier than the old hiera statement ... arguably harder actually. hiera('var' , []) was easier to me than lookup('var', data, 'first', []) but meh, overtime maybe I'll appreciate it or at least accept it now that I can make per-environment hiera changes.
Yes, the lookup seems to work like you said it would, but it seems I've only opened a can of worms:
1. I can't run lookup from the cli, only hiera. So guess I have to figure out why (maybe because the global hiera is still v3?)
Yes, the lookup seems to work like you said it would, but it seems I've only opened a can of worms:
1. I can't run lookup from the cli, only hiera. So guess I have to figure out why (maybe because the global hiera is still v3?)
2. the consul backend dumps an array of hashes ... eww ... looks like Ill have to get really clever about how to manipulate things to get the exact key/value I want ... out of scope of this question though.
Thank you for your help.
Thank you for your help.
> > <mailto:puppet-users+unsub...@googlegroups.com <javascript:>>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/puppet-users/eb1a7ae9-a75e-4acc-82a4-41eb05e103bf%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/eb1a7ae9-a75e-4acc-82a4-41eb05e103bf%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/puppet-users/eb1a7ae9-a75e-4acc-82a4-41eb05e103bf%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/puppet-users/eb1a7ae9-a75e-4acc-82a4-41eb05e103bf%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
> --
>
> Visit my Blog "Puppet on the Edge"
> http://puppet-on-the-edge.blogspot.se/
> <http://puppet-on-the-edge.blogspot.se/>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com
> <mailto:puppet-users+unsub...@googlegroups.com>.
John Baird
Aug 31, 2017, 8:47:09 PM8/31/17
to Puppet Users
Justin,
Funny, i'm dealing with a somewhat similar scenario, but I should let you know that you CAN run lookup from CLI...
"puppet lookup" should invoke exactly what "lookup" does from within puppet, that's actually something they did amazingly well...
Justin DynamicD
Aug 31, 2017, 11:10:22 PM8/31/17
to Puppet Users
Unfortunately puppet lookup doesn't work at all for me, instead it's complaining about being unable to load some ruby gems that are related to a windows chocolaty module that doesn't even get called in the current environment. Mind you simple puppet agent -t runs and everything else is flawless ... but anything Hiera related has been an exercise in patience. Only way I've been able to test it to make a custom role::hieratest and assign it to my server and just invoke puppet runs over and over to see how the lookup works. There isn't even a module on the forge for formatting the global hiera.yaml on the puppet master (they all stop at v4). Custom template for that ... Once I get this consul integration doing what I want I'll do everything in my power to avoid re-configuring hiera at all costs.
Integrating Consul has been a very frustrating journey, much more so than I anticipated when I started.
Does anyone even have a list of example code I can borrow from? I can find videos of peopel claiming it's awesome at puppet conf but they are all horrifically devoid of any real detail or usable example. I can't imagine I'm the first person who's tried to lookup what services are registered to a node yet ... can't find a single working example.
Does anyone even have a list of example code I can borrow from? I can find videos of peopel claiming it's awesome at puppet conf but they are all horrifically devoid of any real detail or usable example. I can't imagine I'm the first person who's tried to lookup what services are registered to a node yet ... can't find a single working example.
John Baird
Sep 14, 2017, 2:55:01 PM9/14/17
to Puppet Users
Justin,
I just wanted to say that I am currently working on implementing the hiera_http to interact with consul. It's not nearly as clean/concise, but it works nonetheless.
Still implementing it at the moment and testing to make sure things work, but for now, assuming your scope is correct, works like a champ. Maybe that would help?
I just wanted to say that I am currently working on implementing the hiera_http to interact with consul. It's not nearly as clean/concise, but it works nonetheless.
Still implementing it at the moment and testing to make sure things work, but for now, assuming your scope is correct, works like a champ. Maybe that would help?
Justin DynamicD
Sep 18, 2017, 1:08:19 AM9/18/17
to Puppet Users
Thank you very much for your reply.
I did eventually get things working (nearly) the way I wanted. In the end I'm using Hiera 5 but still have a v3 global in order to get hiera-consol working. I then forked that and wrote my own custom function to get it do return data in a format I expected ... eww? But hey ... it works the way I need it to so I'm good there.
Essentially I now have a custom function (Written in ruby) that simply returns all registered services when you feed it a node, or all nodes when you feed it a service. Took awhile ot get there ... I think I cried a little ... but it's rather effective at the end of the day.
I did eventually get things working (nearly) the way I wanted. In the end I'm using Hiera 5 but still have a v3 global in order to get hiera-consol working. I then forked that and wrote my own custom function to get it do return data in a format I expected ... eww? But hey ... it works the way I need it to so I'm good there.
Essentially I now have a custom function (Written in ruby) that simply returns all registered services when you feed it a node, or all nodes when you feed it a service. Took awhile ot get there ... I think I cried a little ... but it's rather effective at the end of the day.
Reply all
Reply to author
Forward
0 new messages