Hiera 5 + Puppet 5 File Backend

[John Baird]

There was an old hiera 3.X hiera-file backend now owned on github by voxpupuli... does anyone have a working example of how to reproduce this in Puppet 5 with the next data_hash contexts?  Puppet's documentation seems woefully lacking in this area.

Basically, I would like to be able to serve up `/etc/hosts` file from hiera backend, for example.

[Henrik Lindberg]

That should be incredibly simple to achieve with hiera 5.
What is it that you feel is lacking in terms of documentation?

- henrik


--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

[John Baird]

I see the "yaml_data" and "json_data" backends, but unfortunately, I'm not a Ruby developer and trying to mock those into a file_data backend is proving frustratingly obnoxious.  Honestly, I wish a "file_data" backend was included by default as I feel there a TON of value in it's existence, but that's another topic.

Basically, from the documentation at https://docs.puppet.com/puppet/5.0/hiera_custom_data_hash.html, I am able to make a custom backend, but unable to get it to parse the directory structure properly and retrieve the entire contents of the file.

If you have something that would work, I would love to get it working on my end.

[Henrik Lindberg]

On 21/08/17 22:18, John Baird wrote:
> I see the "yaml_data" and "json_data" backends, but unfortunately, I'm
> not a Ruby developer and trying to mock those into a file_data backend
> is proving frustratingly obnoxious.  Honestly, I wish a "file_data"
> backend was included by default as I feel there a TON of value in it's
> existence, but that's another topic.
>
> Basically, from the documentation
> at https://docs.puppet.com/puppet/5.0/hiera_custom_data_hash.html, I am
> able to make a custom backend, but unable to get it to parse the
> directory structure properly and retrieve the entire contents of the file.
>
> If you have something that would work, I would love to get it working on
> my end.
>

You can write functions in the puppet language as well as in Ruby if
that is more convenient for you. Then depending on what the format of
the data file you want read is, you could either use an existing
function that reads JSON/YAML etc. If there is no reader you would be
best off to write a separate function that can read a particular format,
and then use that to parse the file and serve data from it using a hiera
5 backend compliant function.

The functions that just read a file and returns it a a hash are very
simple
(https://github.com/puppetlabs/puppet/blob/master/lib/puppet/functions/json_data.rb)
which is basically JSON.parse and some error handling around that.

I would expect a custom backend function that you write to be given an
argument that is the absolute path to the file you want read. That is
given in the hiera.yaml.

Is it that you want a structure that defines the keys and that the files
themselves are the resulting value? (Have not looked at hiera-file
backend). If so, then it sounds like just translating the key to lookup
to a path (splitting on '::' and and joining with '/') plus appending
that to a base path where the files are to be found (given as an option
when adding the function to the hiera.yaml hierarchy). Then reading that
file or producing a "not found" if the file does not exists.

If you have the start of this in Ruby somewhere I am happy to help you
review and point out what to change.

Best,
- henrik

> On Monday, August 21, 2017 at 3:14:42 PM UTC-5, Henrik Lindberg wrote:
>
> On 21/08/17 22:03, John Baird wrote:
> > There was an old hiera 3.X hiera-file backend now owned on github by
> > voxpupuli... does anyone have a working example of how to
> reproduce this
> > in Puppet 5 with the next data_hash contexts?  Puppet's
> documentation
> > seems woefully lacking in this area.
> >
> > Basically, I would like to be able to serve up `/etc/hosts` file
> from
> > hiera backend, for example.
> >
> That should be incredibly simple to achieve with hiera 5.
> What is it that you feel is lacking in terms of documentation?
>
> - henrik
>
>
> --
>
> Visit my Blog "Puppet on the Edge"
> http://puppet-on-the-edge.blogspot.se/

> <http://puppet-on-the-edge.blogspot.se/>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com
> <mailto:puppet-users...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/9154ad28-5f52-40b9-87a6-62ba6ac89e85%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/9154ad28-5f52-40b9-87a6-62ba6ac89e85%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[John Baird]

Henrik,

Thanks for helping... I tried originally using the hiera-file backend but, there's too much code to update since the repo hasn't been touched in nearly 2+ years.  So I went down the path of trying to modify the json_data and/or yaml_data backends to suit my needs, since I noticed the biggest delta between the two files was the loading of the respective JSON or YAML.  

I am unsure as to what the exact expectations are when returning the data within the *_data.rb backend.

If you can assist in starting with either the json_data or yaml_data ruby files and modifying them to work with what I need, I would be forever grateful.

Here are my requirements and expectations...

1. Using lookup() should return/query the same way as it currently does for variables.

2. I want to be able to specify a "filepath" like "/etc/hosts" which would then be appended at the end of the search path, so "environments/%{environment}", for example.

3. The contents of "environment/%{environment}/etc/hosts" would then be returned.

3. The backend should only return the contents of the first match, as I won't want duplicative entries within the files.

4. Anything else that may make sense to modify would be good.

I appreciate any assistance you can provide.  Thanks!

> <mailto:puppet-users+unsub...@googlegroups.com>.

[John Baird]

Henrik,

I would also like to add that having the ability to serve up binary files, not just ASCII files would also be a great bonus.  Thanks!

[John Baird]

Henrik,

Whether the existing YAML/JSON/HOCON backend is referenced, everything is using the context of `path` to do the lookups.  The issues with this in regards to a file backend, is I am unsure as to how to access the value or file being looked up.  The `path` for the other backends is statically defined and they load the content accordingly.  Do you know how to access the value being looked up or if that is even possible in this context ?

context.cached_file_data(path) do |content|

Path in this case is the full URI to the YAML file being ingested, for instance. But I need to append the variable being looked up to that in order to find the file I am seeking.

I believe that is the missing piece I need to make this all work. Any insight would be appreciated. Thanks!

[Henrik Lindberg]

On 22/08/17 16:57, John Baird wrote:
> Henrik,
>
> Whether the existing YAML/JSON/HOCON backend is referenced, everything
> is using the context of `path` to do the lookups.  The issues with this
> in regards to a file backend, is I am unsure as to how to access the
> value or file being looked up.  The `path` for the other backends is
> statically defined and they load the content accordingly.  Do you know
> how to access the value being looked up or if that is even possible in
> this context ?
>

> *context.cached_file_data(path) do |content|
> *

>
> Path in this case is the full URI to the YAML file being ingested, for
> instance. But I need to append the variable being looked up to that in
> order to find the file I am seeking.
> I believe that is the missing piece I need to make this all work. Any
> insight would be appreciated. Thanks!
>

What you want to do is to use the kind of function that handles lookup
key by key. You would otherwise have to load all the data files and
return all of them as a hash. That would mean that it is very slow the
first time, and that you may waste lots of memory if only looking up
some of the values.

The cache that is available can be used if you want to keep contents in
memory. That is only of value if the content is used many times as is
the case if a backend function reads a yaml or json file containing a
hash. I suggest skipping trying to cache in your first implementation
(guessing that the same key is typically not looked up over and over again).

Simply resolve the name of the key into a path and read that file and
return the value. (Alternatibely, if there is no such file call the
not_found on the context).

There is support for reading binary files. The result is an instance of
Binary. That data type is not fully supported everywhere, but I think it
is fine to use as content of a File data type. (There is a file()
function that reads a text file and a binary_file that reads it as a
binary).

Hope this helps you get a bit further. Keep the questions coming...

Best,
- henrik

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com

> <mailto:puppet-users...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/28c0c2d2-b350-47d6-a775-e282abd4f158%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/28c0c2d2-b350-47d6-a775-e282abd4f158%40googlegroups.com?utm_medium=email&utm_source=footer>.

[John Baird]

Henrik,

I have something that appears to be at least "attempting" to find the proper files.  The issue I am currently running into is that the "key" or file that I am looking for is being "chopped" at the first ".", which in this case is actually in a folder name, not the file extension.  Do you know how to avoid that?

# File Backend for Hiera

Puppet::Functions.create_function(:'file_backend') do

  dispatch :file_backend do

    param "String", :key

    param "Hash", :options

    param "Puppet::LookupContext", :context

  end

  argument_mismatch :missing_path do

    param 'Hash', :options

    param 'Puppet::LookupContext', :context

  end

  def file_backend(key, options, context)

    hieradir   = context.interpolate(options['hieradir'])

    searchroot = context.interpolate(options['searchroot'])

    searchpath = hieradir + searchroot

    if (File.exists?("#{searchpath}/#{key}"))

      data = File.read("#{searchpath}/#{key}")

    else

      context.explain() { "SearchPath: #{searchpath}, Key: #{key}"}

      context.not_found()

    end

  end

  def missing_path(options, context)

    "one of 'path', 'paths', 'glob', 'globs', or 'mapped_paths' must be declared in hiera.yaml when using this file_backend function"

  end

end

When looking up '/etc/dd-agent/checks.d/filename.py' I receive a `Function lookup() did not find a value for the name '/etc/dd-agent/checks.d/filename.py`.

The debugging from the puppet lookup outputs KEY: /etc/dd-agent/checks which does not contain the rest of the filename or path.

Thoughts ?

[Henrik Lindberg]

On 22/08/17 18:30, John Baird wrote:
> Henrik,
>
>
> I have something that appears to be at least "attempting" to find the
> proper files.  The issue I am currently running into is that the "key"
> or file that I am looking for is being "chopped" at the first ".", which
> in this case is actually in a folder name, not the file extension.  Do
> you know how to avoid that?
>

> *# File Backend for Hiera*
> *Puppet::Functions.create_function(:'file_backend') do*
> *
> *
> *  dispatch :file_backend do*
> *    param "String", :key*
> *    param "Hash", :options*
> *    param "Puppet::LookupContext", :context*
> *  end*
> *
> *
> *  argument_mismatch :missing_path do*
> *    param 'Hash', :options*
> *    param 'Puppet::LookupContext', :context*
> *  end*
> *
> *
> *  def file_backend(key, options, context)*
> *    hieradir   = context.interpolate(options['hieradir'])*
> *    searchroot = context.interpolate(options['searchroot'])*
> *    searchpath = hieradir + searchroot*
> *    if (File.exists?("#{searchpath}/#{key}"))*
> *      data = File.read("#{searchpath}/#{key}")*
> *    else*
> *      context.explain() { "SearchPath: #{searchpath}, Key: #{key}"}*
> *      context.not_found()*
> *    end*
> *  end*
> *
> *
> *  def missing_path(options, context)*
> *    "one of 'path', 'paths', 'glob', 'globs', or 'mapped_paths' must be
> declared in hiera.yaml when using this file_backend function"*
> *  end*
> *end*
> *
> *
> When looking up '*/etc/dd-agent/checks.d/filename.py'* I receive a
> `*Function lookup() did not find a value for the name
> '/etc/dd-agent/checks.d/filename.py*`.
>
> The debugging from the puppet lookup outputs *KEY: /etc/dd-agent/checks
> *which does not contain the rest of the filename or path.

Only had a quick look at the code. I think the issue you are seeing is
that a '.' has special meaning - it navigates into a structure, you need
to escape dots in the key - IIRC by quoting them. Try with a key that
does not have a '.', and then try with quoting the dot with "" or ''

The dot-syntax is typically used to navigate into arrays and hashes.

Best,
- henrik
>
> Thoughts ?

>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com
> <mailto:puppet-users...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/7a86b718-434c-42ff-ad50-649f276c75ba%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/7a86b718-434c-42ff-ad50-649f276c75ba%40googlegroups.com?utm_medium=email&utm_source=footer>.

[John Baird]

Henrik,

Thanks for the feedback.  My concern stems, really, from being able to use this with Puppet.  I would think that I should be able to specify a filename and/or filepath that contain a "." without having to escape those myself.  The backend should handle that, in my opinion.  You can image if a user/developer wanted to have a domain-oriented folder structure with multiple domains, how that could get extremely hard to read quickly... "/path/to/file/domain1\.example\.com" and then the variable would have to be unescaped for all other references to that variable.  That seems silly.  I appreciate your time on this, but I feel like there has to be a better solution, I just haven't found it yet.  I am still working on making this code work...

[Henrik Lindberg]

The escaping is not done with '\' - the dot should be quoted with single
or double quotes. In your case "/path/to/file/domain1'.'example'.'com"

Since you will not ever be able to use file paths as keys that are
handled by APL (the paths are not valid names in the language) you can
simply write a small wrapper function that users call instead of
lookup() to perform this kind of lookup - this function would quote the
given path and then delegate to the existing lookup.

i.e. something like

$content = lookup_path_data("/path/to/file/domain1.example.com")

- henrik

[John Baird]

Henrik,

Thanks for the feedback, while I wish there was a "better/native" way, I have accepted that I need to write my own function.  I would like to be able to use a similar syntax as "lookup()" and simply specify a filepath and have the contents returned as the content of the file being created.  I am fairly certain I can accomplish this, but I am not sure how I gain access to the hiera scope or hierarchy that is defined by either the module or the global/environment scope.  I want my custom module to be able to search the hiera path(s) for the necessary file structure.  How would I go about getting that scope into a custom function?  Thanks!

[Henrik Lindberg]

Unless you are using the bad practice of using 'calling_module' or even
worse: variables in the calling scope (if you managed to get that
working) - you do not need to relay the *calling* scope. If all you are
accessing in your hiera.yaml is top scope variables, then you can just
use 'call_function' (or in puppet language, just call it with normal
puppet syntax).

If you however must delegate *calling* scope, then this is an example of
a Ruby function that does that:
https://github.com/puppetlabs/puppet/blob/ebd96213cab43bb2a8071b7ac0206c3ed0be8e58/lib/puppet/functions/include.rb

You cannot do the same with a function written in the puppet language.
(FWIW: In general it is a really bad idea to design functions that do
different things depending on from where they are called).

When you write such a delegating function, you can call another function
and pass in the scope to use by using 'internal_call_function' instead
of 'call_function' - the signature is the same but it takes a scope as
its first argument.

Best,

[John Baird]

Henrik,  I feel like this is becoming overly complicated and perhaps that is my fault for lack of explanation.

 I am simply trying to return the contents of a file that is stored in hiera... Perhaps my gist would make more sense ?

https://gist.github.com/soudaburger/e253bdce191c731491581df0b0f99234

[Henrik Lindberg]

I commented on your gist
https://gist.github.com/soudaburger/e253bdce191c731491581df0b0f99234

(for others) I wrote this on the gist:

You need an additional function to transform a pathname with dots to one
where they are escaped. But skip that problem first and only test it
with a path that does not contain any dots

Your backend function does not have the correct kind, and signature
it should not be a 'data_hash' kind - use a 'lookup_key' kind of backend
function (you must otherwise return the content of all files as one hash)

read about how it should be written here:
https://docs.puppet.com/puppet/5.1/hiera_custom_lookup_key.html

Your backend function should not have a 'scope_param' in the dispatcher
Use Puppet::Filesystem.read(path) to read the content of the file
referenced by 'path'

The backend function receives a path in the options hash - that path is
guaranteed to exist - all non existing paths are simply skipped by hiera
(see the linked docs how it works). Thus, you do not need to use any of
the find file, etc. You can use relative paths in the hiera.yaml config
and hiera will figure it out (see the docs).

[John Baird]

Thank you.  I will take that and run with it.  I appreciate the comments.

[John Baird]

The backend function receives a path in the options hash - that path is 
guaranteed to exist - all non existing paths are simply skipped by hiera 
(see the linked docs how it works). Thus, you do not need to use any of 
the find file, etc. You can use relative paths in the hiera.yaml config 
and hiera will figure it out (see the docs). 

Henrik,

I believe I fully understand the context here, however, is the "options" hash not fully referenced within the "hiera.yaml" ?  Basically, how do I access the lookup of the key itself from within a puppet manifest as an appendage of the options['path'].  I really believe that is the piece I am just not able to connect.  Can you help me understand how to do this?  Thanks! 

[Henrik Lindberg]

Not quite sure exactly where there is a piece of information missing, so
pardon if I explain too much here.

When hiera 5 processes a lookup it will visit each entry in the
hiera.yaml. Such an entry may use one out of several ways to specify a
set of file paths to check for existence (path, paths, glob). If the
path corresponds to something existing, a call is made to the function
associated with that entry. It receives an options hash where path is
set to the path that was found to be existing. This is repeated for all
the existing paths found in that entry.

A user may have given additional options in the options hash.

If the mode of the lookup is "priority" (first found) the search for a
value stops when a function produces a value for the key being searched
for. If the search is a kind of "merge" then the search continues until
all values for the key have been found.

You can use the fact that the hiera framework performs the existence
check. If you do a glob to match all of the files your function will be
called with each - then simply check if the leaf file name matches the
key being looked up. Call "not_found" for all others, and read and
return the content for the one that matches.

By doing that you get a small advantage in that you will not be
incurring checks for lots of file existence for all of the keys that are
irrelevant and never will have a matching file.

Does this help with what you were wondering about?

[John Baird]

Henrik,

I think the disconnect is coming from the fact that the documentation only goes so far.  In that, I mean, what should my lookup be from my manifest?

Should my manifest be:

$somefile = lookup('/etc/example.txt')

and then specify the file_backend in my hiera.yaml?

Or should my manifest be:

$somefile = file_backend('/etc/example.txt')

and then specify the file_backed in my hiera.yaml?

If I do the former, is specifying the ":key" just implied?

If I do the latter, I end up in a situation where the file_backend function is looking for more arguments than just the simple filename.

I feel like that's what's not really clear, since the YAML and JSON backends simply append the key to the options path without ever specifying the actual key existing in the backend.  So it's a little confusing how I should actually be interacting with the custom backend.

[Henrik Lindberg]

On 01/09/17 01:32, John Baird wrote:
> Henrik,
>
> I think the disconnect is coming from the fact that the documentation
> only goes so far.  In that, I mean, what should my lookup be from my
> manifest?
>
> Should my manifest be:
> $somefile = lookup('/etc/example.txt')
> and then specify the file_backend in my hiera.yaml?
>
> Or should my manifest be:
> $somefile = file_backend('/etc/example.txt')
> and then specify the file_backed in my hiera.yaml?
>
> If I do the former, is specifying the ":key" just implied?
> If I do the latter, I end up in a situation where the file_backend
> function is looking for more arguments than just the simple filename.
>
> I feel like that's what's not really clear, since the YAML and JSON
> backends simply append the key to the options path without ever
> specifying the actual key existing in the backend.  So it's a little
> confusing how I should actually be interacting with the custom backend.
>

The big difference between the YAML and JSON backends is that they load
files with keys in them. In your case the key is the "name" of the file!

The JSON and YAML backends are of the data_hash kind. They are called
once and must produce a hash of all the keys in the file pointed to by
the path the backend is given. You do not want that since it would mean
that you would have to slurp in all of the data files on the initial
call. That could be a problem if you have lots of large files and they
are not all needed for every compilation.

Instead you need to deliver the value for a key per lookup of that key.
The simple implementation takes the key appends it to the root path,
reads that file and returns it as the value. If the file does not exist
it should instead call $context.not_found().

- henrik

> <http://puppet-on-the-edge.blogspot.se/>

>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com
> <mailto:puppet-users...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/9f084cd5-73b7-4b17-9c81-8a386af77dc4%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/9f084cd5-73b7-4b17-9c81-8a386af77dc4%40googlegroups.com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.

[John Baird]

Henrik,

I really appreciate your patience here, but I feel like there is something obvious that I am either not grasping or isn't spelled out in the documentation.

Let's say I make a function called `lookup_file` which will do the File.Read in whatever manner I need.  I can call that from my manifests to lookup the file, but how, then do I lookup the context from hiera?  It's the combination of these two things that seems abstract to me and is preventing me from moving forward.  I understand how to do both individually, but I'm not sure how to retrieve the hiera context for the path from within this "other" function....

I can see that any file_backend function I create chomps the key on the first "." as we previously discussed, which means that I need to call a function BEFORE that in order to call my file_backend from within my file_lookup function, for instance.

How do I get the options and context to be used from the function my manifest will be using?  Let's assume in my manifest:

content => lookup_file('/tmp/somefile.txt')

does this work in my backend?  

Puppet::Functions.create_function(:file_backend) do

  dispatch :file_backend do

    param 'String', :key

    param 'Hash', :options

    param 'Puppet::LookupContext', :context

  end

  def file_backend(key, options, context)

    searchroot = options['path']

    filename = "#{searchroot}/#{filename}"

    # If file is found, return contents of file

    # If file is not found, context.not_found()

  end

  def lookup_file(key)

    options = :options

    context = :context

    file_backend(key, :options, :context)

  end

end

I think that's the part I am struggling with... hiera knows about the context and options, but my manifest knows (or should know) the proper key to look up.  How do I combine them so I retrieve the contents of the file?

You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/TNDd6f8K6H8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/oodq2v%241su%241%40blaine.gmane.org.

[Henrik Lindberg]

On 02/09/17 13:02, John Baird wrote:
> Henrik,
>

You probably wanted to interpolate #{key} here, not #{filename} since it
will always be nil here.

>     # If file is found, return contents of file
>     # If file is not found, context.not_found()
>   end
>
>   def lookup_file(key)
>     options = :options
>     context = :context
>     file_backend(key, :options, :context)
>   end
> end
>

That looks like a decent start of that backend. You want to give it a
name that is in your module's namespace.

> I think that's the part I am struggling with... hiera knows about the
> context and options, but my manifest knows (or should know) the proper
> key to look up.  How do I combine them so I retrieve the contents of the
> file?
>

You (and the users of your module) interact with the backend via lookup.
The backend function should not be called directly from a manifest -
that is difficult since you have to context in which to do the lookup.
(There are ways, for testing purposes, or when delegating - but that is
not the way it is supposed to be done normally).

Your backend is called from hiera because it is configured to be called
in a hiera.yaml where you also give the path to the root of where the
files are - and you get that in the options parameter (as you already
have figured out).

There are some alternatives how you can support users - which I am
showing below:

(I use a module name of "my" here.)

Alternative 1
---
A user needs to add escapes to all '.' in the filename manually when
looking up. (Try this first to make sure your backend works).

lookup("/tmp/foo'.'bar")

Not so great.

Alternative 2
---
A user calls a function that escapes all '.' in the filename.

lookup(my::escape_dot("/tmp/foo.bar"))

You provide that escape function (see further down)

Alternative 3
---
Make your own lookup wrapping function - tell users to use that as a
convenience.

function my::lookup_file(String $file_name) {
lookup(my::escape_dot($file_name))
}

Users call that instead of lookup()

my::lookup_file('/tmp/foo.bar')

They can also call lookup() directly if they want to use the options
lookup() provides, but then they need to call the my::escape_dot() function.


my::escape_dot function
---
The my::escape_dot is something like:

Puppet::Functions.create_function(:'my::escape_dot') do
dispatch :escape_dot do
param 'String', :key
end

def escape_dot(str)
str.split('.').join("'.'")
end
end


With that escaping in place, your backend function will get the key with
the '.' without the quotes.

There are a couple of corner cases with the quoting function if file
names used as keys can contain single quotes, or if it can contain both
single and double quotes. I hope you do not have to support those cases.
(I suggest you simply document that it is illegal to use file names
containing quotes, and possible check for that and give an error in the
quoting function).

- henrik

> <mailto:puppet-users%2Bunsu...@googlegroups.com>
> > <mailto:puppet-users...@googlegroups.com
> <mailto:puppet-users%2Bunsu...@googlegroups.com>>.

> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/puppet-users/9f084cd5-73b7-4b17-9c81-8a386af77dc4%40googlegroups.com
> >
> <https://groups.google.com/d/msgid/puppet-users/9f084cd5-73b7-4b17-9c81-8a386af77dc4%40googlegroups.com?utm_medium=email&utm_source=footer>.
> > For more options, visit https://groups.google.com/d/optout.
>
>
> --
>
> Visit my Blog "Puppet on the Edge"
> http://puppet-on-the-edge.blogspot.se/
>
> --
> You received this message because you are subscribed to a topic in
> the Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/puppet-users/TNDd6f8K6H8/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> puppet-users...@googlegroups.com

> <mailto:puppet-users%2Bunsu...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/oodq2v%241su%241%40blaine.gmane.org.
> For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com
> <mailto:puppet-users...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/CACSjNA4Uk7hHP5zbODY4YT9BTboEVuu5WqKSJeAR83sO7ScD6A%40mail.gmail.com
> <https://groups.google.com/d/msgid/puppet-users/CACSjNA4Uk7hHP5zbODY4YT9BTboEVuu5WqKSJeAR83sO7ScD6A%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[John Baird]

I just realized, while doing some other hiera work with a custom HTTP backend, that this won't work.  It's the same problem I have with the way scoping works for the HTTP backend.  If I don't specify the lookup() key in the manifest with a scope, then puppet ONLY uses GLOBAL and ENVIRONMENT scope and ignores module scope.  This seems undesired.  In that manner, I would have to have my filenames (in the case of file backend) or URIs (as is the case with HTTP backend) contain module scoping in the lookup, so my filename would become lookup('somemodule::somefile.txt") which would THEN have to actually exist in that lookup...

I would expect Puppet to use module scope when a lookup is performed within the module if scope is not specified for this lookup.

https://groups.google.com/forum/#!topic/puppet-users/9q__saaa26w

[Henrik Lindberg]

On 15/09/17 16:46, John Baird wrote:
> I just realized, while doing some other hiera work with a custom HTTP
> backend, that this won't work.  It's the same problem I have with the
> way scoping works for the HTTP backend.  If I don't specify the lookup()
> key in the manifest with a scope, then puppet ONLY uses GLOBAL and
> ENVIRONMENT scope and ignores module scope.  This seems undesired.  In
> that manner, I would have to have my filenames (in the case of file
> backend) or URIs (as is the case with HTTP backend) contain module
> scoping in the lookup, so my filename would become
> lookup('somemodule::somefile.txt") which would THEN have to actually
> exist in that lookup...
>

yes, modules cannot bind to names in the global (top) name space.
If they could they would have the potential to step on each other and it
will become impossible to resolve a lookup without visiting every module
on the modulepath for every lookup.

hiera is not ignoring module scope - it is simply illegal to try to bind
to anything that is not in the module's namespace.

> I would expect Puppet to use module scope when a lookup is performed
> within the module if scope is not specified for this lookup.

It simply does not work that way. There are no "relative keys".

[John Baird]

Right, and that's exactly my concern with NOT being able to do exactly that.

Scenarios

1.  I wish to lookup a URI that is to be configured for a module.  The module name here is obviously not important, but let's call it "mymodule".  In the manifest, I wish to lookup ssh keys being served up by Consul (as I am currently doing).  I would like to simply lookup('username') where 'username' is the user whose public key I would like to authorize onto my system.

1a. This means, lookup('username') will only work for top scope.

1b. Option 1, change scope of lookup.  my lookup now needs to be "sshkeys::username", but now my URI lookup path is going to be "/path/to/some/uri/sshkeys::username" in consul.  Surely there's a better way than having a bunch of k/v pairs with namespaced keys in them ?

1c. Option 2, everything is top scope when I care about doing URI lookups.

1d. This would be fine except if now I want to have a list of all active users on the system as a simple list in another module.  Now whichever URI path is first will return the data, and not necessarily the appropriate data for the lookup.

I understand that it's not supposed to work the way it used to, but come one, this functionality almost seems common place.  Using an HTTP or file backend is CRITICAL in many ecosystems.

I suppose I can write some regsub on the lookup to parse out the namespace prior to appending it to the URI, but that seems insanely hackish...  Personally, if I do a lookup from within a module, it should automatically append the current namespace to it for lookup, then this issue goes away.

[Henrik Lindberg]

On 15/09/17 19:18, John Baird wrote:
> Right, and that's exactly my concern with NOT being able to do exactly that.
>
> Scenarios
> 1.  I wish to lookup a URI that is to be configured for a module.  The
> module name here is obviously not important, but let's call it
> "mymodule".  In the manifest, I wish to lookup ssh keys being served up
> by Consul (as I am currently doing).  I would like to simply
> lookup('username') where 'username' is the user whose public key I would
> like to authorize onto my system.
> 1a. This means, lookup('username') will only work for top scope.
> 1b. Option 1, change scope of lookup.  my lookup now needs to be
> "sshkeys::username", but now my URI lookup path is going to be
> "/path/to/some/uri/sshkeys::username" in consul.  Surely there's a
> better way than having a bunch of k/v pairs with namespaced keys in them ?
> 1c. Option 2, everything is top scope when I care about doing URI lookups.
> 1d. This would be fine except if now I want to have a list of all active
> users on the system as a simple list in another module.  Now whichever
> URI path is first will return the data, and not necessarily the
> appropriate data for the lookup.
>
> I understand that it's not supposed to work the way it used to, but come
> one, this functionality almost seems common place.  Using an HTTP or
> file backend is CRITICAL in many ecosystems.
>

What do you mean by "to work the way it used to" - you have never been
able to bind to anything but module namespaced keys in modules.
Earlier everything was in top scope.

There is absolutely nothing stopping you from using the old practice of
having the global layer refer to data files inside of modules. This is a
choice you make - it is bad in general but works if you are in control
of where a module gets used (you are creating a dependency that the
module must be present). I.e. a conscious decision made at the env or
global level.

You could also write a simple aggregation function that takes a list of
modules as input (from hiera.yaml) - this is a list of modules that for
example contribute "users". You function when being asked to supply the
key for a user "fred_flintstone", it looks up in each module and you are
then in charge of dealing with that merge).

There are many other options here - I am not sure I understand the
concerns about URIs HTTP/DB queries well enough to advice in more detail.

Hope you got something to work with from the above.

Best,
- henrik

> <http://puppet-on-the-edge.blogspot.se/>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com

> <mailto:puppet-users...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/80759536-03b7-4f9b-80ea-39e13dcf0ef8%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/80759536-03b7-4f9b-80ea-39e13dcf0ef8%40googlegroups.com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.

[Justin DynamicD]

I had a similar but "not quite the same" challenge.  Basically I just added a custom ruby function to let you parse the data you're looking for.  So in addition to lookup (), I now have things like consul_servicenames() which dumps all services presented by a host in an array format (handy for if $a in $b scenarios).  It basically means building a few custom parsers in ruby into your backend module to cover common lookup "Styles" that match your CM layout.  By storing them all in the backend module they are available essentially globally and simply called differently.

[John Baird]

Justin,

I would love to see what you have done.  Would you be willing to send me what you've written.  It's possible I could extend what you've done to resolve my huge lingering issue, which is being able to some sort of file-based lookup WITH file extensions (as default lookup behavior chops at first '.').  Feel free to message directly if you wish.  Thanks for anything you can send my way!