Jira (PUP-7332) add ability to lookup "below" in hierarchy

3 views
Skip to first unread message

Henrik Lindberg (JIRA)

unread,
Mar 10, 2017, 8:16:02 AM3/10/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
 
Puppet / New Feature PUP-7332
add ability to lookup "below" in hierarchy
Change By: Henrik Lindberg
Fix Version/s: PUP 4.10.0
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
Atlassian logo

Henrik Lindberg (JIRA)

unread,
Mar 10, 2017, 8:16:02 AM3/10/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Team: Puppet Developer Experience

Henrik Lindberg (JIRA)

unread,
Mar 10, 2017, 8:16:03 AM3/10/17
to puppe...@googlegroups.com
Henrik Lindberg created an issue
Issue Type: New Feature New Feature
Assignee: Unassigned
Created: 2017/03/10 5:15 AM
Priority: Normal Normal
Reporter: Henrik Lindberg

In order to be able to filter or modify a value bound at a lower precedence in ways that the built in hiera interpolation or aliasing features can handle it is of great value to be able to lookup the value that would be the result of lower precedence entries and return that.

This mechanism should be available via the $context object via a lookup_below(key). This method takes a no args lambda that is called if the lookup results in not_found. If no lambda is given and the lookup_below(key) results in not_found that also becomes the result of the function calling lookup_below (it does not return to it).

The rest of the arguments to the lookup_below are the same as for a lookup.

As an example a user may do something like this for the lookup of key

return "some prefix: #{context.lookup_below(key)"

Without the lookup_below feature this is impossible since a lookup of the same key would lead to an endless recursion and hiera would error out.

This mechanism can thus be used to implement filter/map like behavior, change the data type etc. As an example, a function could filter all "password" keys to wrap them as Sensitive values, a lookup of a hash 'x', could add entries to that hash that comes from other keys, and other similar advanced cases.

Henrik Lindberg (JIRA)

unread,
Mar 16, 2017, 8:32:01 AM3/16/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Sprint: PDE 2017-03-22

Henrik Lindberg (JIRA)

unread,
Mar 20, 2017, 5:40:02 PM3/20/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Fix Version/s: PUP 4.10.0
Fix Version/s: PUP 5.0.0

Henrik Lindberg (JIRA)

unread,
Mar 22, 2017, 7:14:11 PM3/22/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Sprint: PDE 2017-03-22 , PDE 2017-04-05

Thomas Hallgren (JIRA)

unread,
Mar 23, 2017, 4:15:02 AM3/23/17
to puppe...@googlegroups.com
Thomas Hallgren commented on New Feature PUP-7332
 
Re: add ability to lookup "below" in hierarchy

Wouldn't it be of interest for a function in the global layer to bypass the environment layer? Use of "below" will block that. Why not have a #lookup_in_layer(key, layer) instead, with the same constraints as in PUP-7333, i.e. a module would not be allowed to use that function, an environment would only be allowed to use 'module', but global could use both 'environment' and 'module'. I think that would vouch for some consistency between this ticket and PUP-7333.< /p>

Alternatively, change the semantics in PUP-7333 to only allow the layer below, which would prevent the global from bypassing environment.

Henrik Lindberg (JIRA)

unread,
Mar 23, 2017, 5:10:02 AM3/23/17
to puppe...@googlegroups.com

I don't understand the use case for that. The purpose of "lookup_below" is to be able to transform or filter what would otherwise have been returned for a given key.
I also don't understand how a lookup_in_layer would be an alternative, it would need "lookup in levels below in same layer" as well. Also given PUP-7333, you don't really know the ordering of layers and levels. So something like "below" is needed as a filtering/transforming function would otherwise have constraints on where it can be placed.

The name can naturally be something other than "lookup_below", but it is important that it returns the result that would have been returned if the function calling lookup_below did not exist.

The suggestion to change PUP-7333 to say "layer below" would be fine, but "below" is not the same thing in both cases - in this ticket "below" means all levels in all layers that are below the level making the call. In PUP-7333 it would need to mean the layer below this layer. That change in 7333 would work for me.

Thomas Hallgren (JIRA)

unread,
Mar 23, 2017, 5:48:02 AM3/23/17
to puppe...@googlegroups.com

Strike my proposal regarding change to lookup_below. It's fine as it is. I've come to realize that it's PUP-7333 I'm worried about. Let's move the discussion to that ticket.

Henrik Lindberg (JIRA)

unread,
Mar 24, 2017, 6:02:03 AM3/24/17
to puppe...@googlegroups.com

When discussing this - we found a design flaw. A function cannot simply filter without the framework knowing it is filtering as that would produce strange results when a merge lookup is performed (it would merge the unfiltered and filtered results).

New proposal:

A new kind of function: filter_key is added. It has a signature that is the same as the lookup_key function, but with an added parameter value_below.
The framework will call this function "on the way up the hierarchy" after having produced the partial result that is produced by all definitions/levels/layers below the filter_key entry. The call is then made to the given function and the partial result is given as the value_below. If the function wishes to do no filtering it simply returns the found value. If it wants to replace it with something else, it returns that value. If it wants to, it can call context.not_found to filter out the value completely.

The value_below is not just the value since the lookup below can result in both undef and a not_found exception. To enable the function to react to all situations the value_below is therefore a Struct[found => Boolean, value => Any].

Thomas Hallgren (JIRA)

unread,
Mar 24, 2017, 6:30:02 AM3/24/17
to puppe...@googlegroups.com

It's unclear to me how the result from this filter would participate in a merge. Let's assume that we have the participants A, F, B, and C where F is the filter. Without the filter, A, B, and C would be merged. What should be merged instead? Is it A, F, and C or just A and F? I.e. does F "take over" all subsequent entries or just the next one? If it's the latter, then I assume that F should be fed the already merged result of B and C?

Thomas Hallgren (JIRA)

unread,
Mar 24, 2017, 6:34:02 AM3/24/17
to puppe...@googlegroups.com

A follow-up question is of course if it wouldn't be better if the F took over everything so that the framework stops processing at F. Instead it's up to F to do the rest using a lookup_below. That will give F a chance to override merge strategies and even change the key.

Thomas Hallgren (JIRA)

unread,
Mar 24, 2017, 8:56:02 AM3/24/17
to puppe...@googlegroups.com

Alternative proposal.

Combine the ability to reorder layers with the ability to add a filter. Make it clear what it is that a filter acts on by declaring a sub-hierarchy:

hierarchy:
 
  - name: Common
 
    path: common.yaml
 
 
 
  - sub_hierarchy:
 
      filter: <function name> # optional lookup_key function acting on sub-hierarchy using Context methods.
 
      hierarchy:
 
         # normal hierarchy
 
 
 
  - name: Roles
 
    path: roles.yaml

The presence of a sub_hierarchy element will prevent the search in the next layer from the hierarchy containing it. This is true for nested sub-hierarchies as well which has three consequences:

  1. A sub-hierarchy can be used to reorder the search (i.e. as a replacement for a next_layer keyword) if it has an empty hierarchy and no function. That would simply mean, "search next layer here".
  2. In a nested sub-hierarchy, only the leafs will search the next layer.
  3. Multiple parallel sub-hierarchies will result in multiple searches in the next layer.

Pros:

  • A "filter" is simply an entry in the hierarchy. Yet with power to have it's own hierarchy that cannot be reached from the outside.
  • The result produced by sub-hierarchies can be merged with "normal" data that comes after the sub-hierarchy without having that data passing through a special function.
  • Solves both PUP-7332 and PUP-7333
  • The declaration makes the search order more visible through the Hiera indentation.

Caveats:
In a scenario with multiple parallel sub-hierarchies, it might be desirable to prevent some of them (perhaps all but one) from accessing the next layer.

Henrik Lindberg (JIRA)

unread,
Mar 24, 2017, 12:55:03 PM3/24/17
to puppe...@googlegroups.com

I don't see how that solves "default hierarchy" inside of a module. Including the underlying layer more than once is too weird.
For every filter I want I must introduce one sub hierarchy. I must be explicit about the lower layer or the filter will not filter data from lower layer(s).

I also struggle to come up with meaningful use cases where I would need an actual nested hierarchy (except for the default hierarchy in a module, but that cannot be intermixed into the regular hierarchy).

Thomas Hallgren (JIRA)

unread,
Mar 24, 2017, 6:31:02 PM3/24/17
to puppe...@googlegroups.com

The default hierarchy of a module will still be consulted when everything else fails, and only then.

And you don't need to introduce any more hierarchy elements than before. Just organize them differently (which also highlights what the filter is using). I'd consider that a very useful way to declare what it is you are doing. The filters are in fact mechanism that introduces nesting.

Thomas Hallgren (JIRA)

unread,
Mar 27, 2017, 5:50:02 AM3/27/17
to puppe...@googlegroups.com

Revised version of my alternative proposal.
Instead of using the term sub_hierarchy, we should simply use hierarchy for the nested hierarchies as well and allow that a hierarchy has a filter functions. Filter functions can be inserted anywhere in a hierarchy as a single string element (as opposed to the hash elements used for other hierarchy entries). A filter then acts on everything "below" its declaration.

In essence, this means that a filter doesn't need to act on a sub hierarchy. I may (if declared as the first element in a hierarchy), but it can also be declared in arbitrary places in a flat hierarchy, e.g.:

hierarchy:
 
  - name: Common
 
    path: common.yaml
 
  - my_filter_func
 
  - name: Filtered data
 
    path: filtered.yaml

Logically, this would be exactly the same as declaring:

hierarchy:
 
  - name: Common
 
    path: common.yaml
 
  - hierarchy:
 
    - my_filter_func
 
    - name: Filtered data
 
      path: filtered.yaml

The difference between the two declarations is only relevant when something is declared after the sub hierarchy, e.g.

hierarchy:
 
  - name: Common
 
    path: common.yaml
 
  - hierarchy:
 
    - my_filter_func
 
    - name: Filtered data
 
      path: filtered.yaml
 
  - name: Roles
 
    path: roles.yaml

Here, the "my_filter_func" would not have see to the "Roles" entry when doing a lookup_below because it resides in the parent hierarchy.

Henrik Lindberg (JIRA)

unread,
Mar 27, 2017, 7:26:02 AM3/27/17
to puppe...@googlegroups.com

So, are you saying that if the filter is in the top level hierarchy, then all entries below it in the same hierarchy and all entries in all lower layers are filtered? (assuming that the next layer down is not specifically referenced and therefore ends up last in the top hierarchy in the hiera.yaml in question).

If that is true, then this proposal is workable. I still think the extra ability to have a sub hierarchy may be overkill. (I understand that this is made possible because it becomes a composition of uniform parts which is nice because of its generality).

One thing to note - it must be possible to pass options to a filter function (e.g. you want to replace "blue" with "green" and don't want that hardcoded in the function itself). You may also want to call the same filter function with different arguments in different places in the hierarchy, and you therefore need a name for the entry to understand explain output.

Thomas Hallgren (JIRA)

unread,
Mar 27, 2017, 8:08:02 AM3/27/17
to puppe...@googlegroups.com

Yes, that's what I'm saying.

Having sub hierarchies might be overkill, but please consider that their presence solves three problems.

  1. The ability to reorder (PUP-7333), since any hierarchy containing a sub-hierarchy is prevented from continuing with the next layer.
  2. The ability to have more than one filter that acts on diverse parts of the hierarchy.
  3. The ability to add unfiltered parts after the filter.

If options are needed, then the filter function cannot be an array. It must be a hash, and as such, must be uniquely identified from normal hierarchy entries. Perhaps by using the unique key 'filter' for the function. Locations (urls, paths, globs, mapped paths) must be forbidden. While using 'options' as a key might be appealing at first, we might want to consider some other key to avoid that it is assigned the 'options' from the 'defaults' entry.

Henrik Lindberg (JIRA)

unread,
Mar 27, 2017, 8:33:02 AM3/27/17
to puppe...@googlegroups.com

I don't understand your comment in #1 - It must naturally be possible to filter what comes from a lower layer - the feature is more or less useless otherwise. Are you saying that it is illegal to reference a layer in a subhierarchy? Thus, if you want to filter a layer you must have your filter in the hiera.yaml's top hierarchy?

The #2 is the feature I think is overkill and that would require allowing lower layer multiple times in different sub hierarchies in order to be of real and general value. I can however accept that you can do this as there may be cases where you are combining some backends in the same layer and you need to filter something locally. The filter acts like a wrapper where you can introduce your own post processing.

The #3 seems useful. If answer to my question about #1 is 'yes', then the combination of filtered sublayer with an unfiltered thing after the filter does not seem possible... If #1 is really "only the top hierarchy, or one of the sub-hierarchies may include the lower layer(s), then it works.

Some details:

  • Options are required
  • path(s)/glob(s)/mapped-paths not supported (error)
  • uri support is useful - for example, may filter by talking to a service and need a URL. Seems strange if not the same param ("uri") for this.
  • I think using a different name for "options" is more confusing than applying the general rule. Also think filter functions are more likely to take parameters than being completely hardcoded, having to pass an empty options hash to override defaults is therefore a very rare operation.

Thomas Hallgren (JIRA)

unread,
Mar 27, 2017, 8:35:02 AM3/27/17
to puppe...@googlegroups.com

Of course you should filter what comes from a lower layer. That's why I propose that it works the way it does. A sub-hierarchy takes over the responsibility for the lower layers from it's parent.

Henrik Lindberg (JIRA)

unread,
Mar 27, 2017, 8:44:02 AM3/27/17
to puppe...@googlegroups.com

So if I have a sub hierarchy then it automatically filters the lower layer(s)? That was surprising.
What if I have two sub hierarchies ?
Confused.

Thomas Hallgren (JIRA)

unread,
Mar 27, 2017, 9:36:02 AM3/27/17
to puppe...@googlegroups.com

This is covered in my original proposal (starts with "Alternative proposal."). See #3 above "pros" and also what I mention as a caveat.

Henrik Lindberg (JIRA)

unread,
Mar 27, 2017, 11:48:03 AM3/27/17
to puppe...@googlegroups.com

How I think it would work is that you can include the lower layer(s) anywhere - either in the top hierarchy in the hiera.yaml, or in a sub hierarchy. If it is not included anywhere, it is placed last in the top hierarchy. You are only allowed to include the lower level(s) once in a hiera.yaml.

Is this correct? Then it works for me.

Thomas Hallgren (JIRA)

unread,
Mar 27, 2017, 5:35:02 PM3/27/17
to puppe...@googlegroups.com

That's not how I intended it, but what you propose would work too.

I though of a hierarchy as something that always includes the lower layers except when it contains a sub-hierarchy. When it does, the sub-hierarchy "takes over" that responsibility. This behavior is consistent with how the top layer works today (it traverses lower layers without any explicit instruction to do so).

In a nested sub-hierarchy, that "take over" would still result in just one hierarchy (the leaf) actually performing the traversal. The exception would be when there are multiple parallel sub-hierarchies, in which case, each leaf would traverse the lower layers unless that could somehow be blocked. That would be solved by adding a keyword for blocking such traversal.

Henrik Lindberg (JIRA)

unread,
Mar 29, 2017, 7:04:02 AM3/29/17
to puppe...@googlegroups.com

I prefer it the way I described.

Thomas Hallgren (JIRA)

unread,
Mar 29, 2017, 7:14:02 AM3/29/17
to puppe...@googlegroups.com

How do you propose that "include the lower layers here" is declared?

Henrik Lindberg (JIRA)

unread,
Apr 5, 2017, 4:28:05 PM4/5/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Sprint: PDE 2017-03-22, PDE 2017-04-05 , PDE 2017-04-19

Eric Sorenson (JIRA)

unread,
Apr 13, 2017, 6:57:02 PM4/13/17
to puppe...@googlegroups.com
Eric Sorenson commented on New Feature PUP-7332
 
Re: add ability to lookup "below" in hierarchy

Is this truly necessary? Has anyone asked for it? It seems like in practice it could lead to a great deal of complexity.

Henrik Lindberg (JIRA)

unread,
Apr 14, 2017, 5:48:02 AM4/14/17
to puppe...@googlegroups.com

Eric Sorenson the intention of this feature is to be an open way of handling various requests for additions/changes/custom "deep merge" behaviors and requests for interpolation functions. The typical thing asked for is the need to exclude values from a collection and where "knockout prefix" cannot do the job.

There has been no requests for general filtering, but I suspect that is because users have not yet encountered the situations where it becomes important (as they are just starting out using hiera 5 features). The alternative when not being able to filter is to change manifests to use manual calls to lookup rather than using APL as that is the only way a user can modify the returned values.

At the moment as we are discussing how to implement this feature, it has snowballed into a proposal for "filtered-sub-hierarchy" which I think is conceptually overly complex. Even if it is how the implementation would work I don't think that is something users really need to be able to do.

Eric Sorenson (JIRA)

unread,
Apr 20, 2017, 8:18:02 PM4/20/17
to puppe...@googlegroups.com
Eric Sorenson updated an issue
 
Change By: Eric Sorenson
Fix Version/s: PUP 5.0.0

Henrik Lindberg (JIRA)

unread,
Apr 25, 2017, 5:14:04 AM4/25/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Sprint: PDE 2017-03-22, PDE 2017-04-05, PDE 2017-04-19 , PDE 2017-05-03

Henrik Lindberg (JIRA)

unread,
Apr 25, 2017, 5:16:02 AM4/25/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Sprint: PDE 2017-03-22, PDE 2017-04-05, PDE 2017-04-19, PDE 2017-05- 03 17

Henrik Lindberg (JIRA)

unread,
Apr 25, 2017, 5:20:02 AM4/25/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Fix Version/s: PUP 5.y

Henrik Lindberg (JIRA)

unread,
May 2, 2017, 10:28:17 AM5/2/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Sprint: PDE 2017-03-22, PDE 2017-04-05, PDE 2017-04-19, PDE 2017-05- 17 31

Ethan Brown (JIRA)

unread,
May 17, 2017, 1:38:04 PM5/17/17
to puppe...@googlegroups.com
Ethan Brown updated an issue
Change By: Ethan Brown
Team: Puppet Developer Experience Agent

Ethan Brown (JIRA)

unread,
May 17, 2017, 1:39:03 PM5/17/17
to puppe...@googlegroups.com
Ethan Brown updated an issue
Change By: Ethan Brown
Sprint: PDE 2017-03-22, PDE 2017-04-05, PDE 2017-04-19,  PDE 2017-06-07  Agent N+1

Ethan Brown (JIRA)

unread,
May 17, 2017, 1:39:04 PM5/17/17
to puppe...@googlegroups.com
Ethan Brown updated an issue
Change By: Ethan Brown
Labels: triaged

Henrik Lindberg (JIRA)

unread,
May 31, 2017, 9:10:04 AM5/31/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Sprint: PDE 2017-03-22, PDE 2017-04-05, PDE 2017-04-19,  Agent N+1  Language Triage

Henrik Lindberg (JIRA)

unread,
Nov 24, 2017, 4:24:02 AM11/24/17
to puppe...@googlegroups.com
 
Re: add ability to lookup "below" in hierarchy

I read through this again - and I find the subhierarchy idea too complex to understand. If PUP-7333 ends up being that you can define where the lower precedence layer(s) are inserted with insert_next, then you have the ability to define the order of all levels in all layers. You should then be able to insert a filter anywhere in there to filter a particular key or set of keys.

When resolving simply skip over the filter, when the value from lower precedence levels is known at the point where the filter is, the filter function is called with the key and the resolved value. What it returns replaces what was resolved. It should have a regexp that keys must match in order for the filter to kick in (without such a regexp it would be called for every key). Different merge strategies does not affect the filtering as it applies to the normal resolution, then there is a replacement (or not), that value is then merged with other higher precedence resolutions.

With such an implementation there is no need for the earlier proposed "lookup_below" feature.
This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db)
Atlassian logo

David McTavish (Jira)

unread,
Dec 6, 2021, 12:23:01 PM12/6/21
to puppe...@googlegroups.com
David McTavish updated an issue
 
Change By: David McTavish
Labels: final_triage
This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages