Issues with Data in Modules and nested classes and scope...

[Gavin Williams]

Morning all... 

I've hit a bit of an issue with our usage of Data-In-Modules (DIM) provided by ripienaar/module_data with Puppet 3.7.x and was wondering what the best option might be... 

TL;DR: How can I change the DIM Hiera lookup scope to use the correct module? 

Currently, we have the following module architecture:

1. Profile module service class, which wraps:
2. Specific service module, which wraps:
3. A common 'service-framework' module. 

The 'service module' uses DIM to provide sane defaults for lots of param values, which can then be over-ridden in the profiles class. 

It also provides a config template snippet that's specific to that service, which then gets evaluated in the 'service-framework' module. 

The issue I'm hitting is that the scope used for the DIM Hiera lookups is preventing Hiera from resolving the specific service modules data files... 

In order to illustrate this better, I've done a simple example available here [1].

The README shows an example failure when attempting to run the RSpec unit tests within the 'profile_module', which correlates with the error I'm getting with trying to actually use this same functionality... 

It also shows a working test from within the 'calling_module', which is effectively the service specific module... 

I did manage to come up with a slightly ugly hack[2] to the module_data hiera backend, which makes the tests pass... 

However my understanding is that the same logic has been adopted by Puppet core in the 4.x series, which obviously means we'll hit exactly the same issue if/when we come to upgrade to 4.x... 

So yeh, any pointers on how best to 'resolve' this issue? 

Cheers

Gavin 

[1] https://github.com/fatmcgav/puppet_dim_example

[2] https://gist.github.com/fatmcgav/e72d8f5bd8d3a5307d3225a15bccd096

[Henrik Lindberg]

On 16/05/16 13:00, Gavin Williams wrote:
> Morning all...
>
> I've hit a bit of an issue with our usage of Data-In-Modules (DIM)
> provided by ripienaar/module_data with Puppet 3.7.x and was wondering
> what the best option might be...
>
> TL;DR: How can I change the DIM Hiera lookup scope to use the correct
> module?
>
> Currently, we have the following module architecture:
>

> 1. Profile module service class, which wraps:
> 2. Specific service module, which wraps:
> 3. A common 'service-framework' module.

>
> The 'service module' uses DIM to provide sane defaults for lots of param
> values, which can then be over-ridden in the profiles class.
> It also provides a config template snippet that's specific to that
> service, which then gets evaluated in the 'service-framework' module.
>
> The issue I'm hitting is that the scope used for the DIM Hiera lookups
> is preventing Hiera from resolving the specific service modules data
> files...
>
> In order to illustrate this better, I've done a simple example available
> here [1].
>
> The README shows an example failure when attempting to run the RSpec
> unit tests within the 'profile_module', which correlates with the error
> I'm getting with trying to actually use this same functionality...
> It also shows a working test from within the 'calling_module', which is
> effectively the service specific module...
>
> I did manage to come up with a slightly ugly hack[2] to the module_data
> hiera backend, which makes the tests pass...
> However my understanding is that the same logic has been adopted by
> Puppet core in the 4.x series, which obviously means we'll hit exactly
> the same issue if/when we come to upgrade to 4.x...
>

The logic from R.I's DIM module is not in puppet 4.x. It is a quite
different implementation. If you are having problems with the 4.x
implementation please log an issue.

There was too much to look at in your example + R.I's implementation and
I could not quite figure out if what you patched fixes a bug or if it
changed something conceptually.

The 4.x implementation allows a module to bind to keys in its own
namespace (and not in others). When looking something up, if the key is
in a module namespace, it will be looked up there if not overridden by
something with higher precedence. There is never an issue with "lookup
from" as the 4.x implementation looks at keys/namespaces being looked up.

> So yeh, any pointers on how best to 'resolve' this issue?
>

Unless you are trying to bind to non module namespace keys from within
a module, the 4.x implementation should work for you.

- henrik

--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

[Gavin Williams]

Henrik

Cheers for the quick response. I haven't tried this scenario on 4.x, and I doubt I will for quite a while... 

OK, so to elaborate on the namespace thing, and possibly try and explain what I'm trying to achieve... 

I've got module_a, which uses DIM, and defines a key of 'module_a::key_a'. Module_a also contains a template, which has a simple 'scope.function_hiera(['module_a::key_a'])'.

If I evaluate that template in a resource within module_a, then it should look up the value without issue?

What happens if module_b uses the template defined by module_a? Is that now namespaced to 'module_b'? Ergo Puppet would never be able to get the key value from module_a DIM? 

I think that explains the issue I'm seeing a bit better... 

Cheers

Gavin 

[Gavin Williams]

I also went back and looked at my tweaks to R.I's code, and it looks like I was/am actually using a value from 'scope' that gets defined by my code :( 

So I've got a defined type that looks like [1], which sets the value of 'module' to either the value provided, or 'caller_module_name' if not provided... 

So where it works in my tests is because I'm specifying the value of 'module' on the input [2]. 

So possibly not the most sustainable work-around... :( 

Cheers
Gav

[1] https://gist.github.com/fatmcgav/48d30a8a9b29d342035120528fec1767#file-microservice-config_snippet-pp

[2] https://gist.github.com/fatmcgav/48d30a8a9b29d342035120528fec1767#file-snippet_creation-pp