Jira (PUP-7333) make it possible to control partial ordering of layers

1 view
Skip to first unread message

Henrik Lindberg (JIRA)

unread,
Mar 10, 2017, 8:31:02 AM3/10/17
to puppe...@googlegroups.com
Henrik Lindberg created an issue
 
Puppet / New Feature PUP-7333
make it possible to control partial ordering of layers
Issue Type: New Feature New Feature
Assignee: Unassigned
Created: 2017/03/10 5:30 AM
Priority: Normal Normal
Reporter: Henrik Lindberg

When transitioning to data in environments and when dealing with different versions of modules (with various degree of quality of their data). It is of importance to be able to define that entries in a global or environment hiera.yaml has lower precedence than those normally having lower precedence.

While these can be achieved with PUP-7332 as a kind of filter it makes it difficult to do the same with an existing data function. The typical example here would be to have a common layer in the global hiera.yaml that kicks in after the environment, or an entry in an environment that kicks in after modules. It is meaningless to have this in a module's hiera.yaml.

An easy way to configure this is to add a kind of function called layer that accepts environment or modules. If this is present in a hierarchy the lower level is visited at that point in the higher layer's hierarchy. If not present, the lower precedence layer is automatically visited after the higher layer's precedence (as before this ticket).

The entry layer: environment is only allowed in a global hiera.yaml, and the entry layer: modules is only allowed in an environment's hiera.yaml. The module layer accepts none of those.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
Atlassian logo

Geoff Nichols (JIRA)

unread,
Mar 10, 2017, 12:52:02 PM3/10/17
to puppe...@googlegroups.com
Geoff Nichols updated an issue
Change By: Geoff Nichols
Team: Puppet Developer Experience

Henrik Lindberg (JIRA)

unread,
Mar 16, 2017, 8:26:01 AM3/16/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Story Points: 2
Sprint: PDE 2017-03-22

Henrik Lindberg (JIRA)

unread,
Mar 16, 2017, 8:34:01 AM3/16/17
to puppe...@googlegroups.com

Henrik Lindberg (JIRA)

unread,
Mar 16, 2017, 8:34:02 AM3/16/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Sprint: PDE 2017- 03 04 - 22 05

Henrik Lindberg (JIRA)

unread,
Mar 16, 2017, 8:35:02 AM3/16/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Fix Version/s: PUP 4.10.0
Fix Version/s: PUP 5.0.0

Henrik Lindberg (JIRA)

unread,
Mar 23, 2017, 5:29:02 AM3/23/17
to puppe...@googlegroups.com
Henrik Lindberg commented on New Feature PUP-7333
 
Re: make it possible to control partial ordering of layers

In PUP-7332 it is suggested that we should instead use lookup: below here. The "below" then means "env" for a global hierarchy and "module" for a "env" hierarchy, and it is illegal in a module's hiera.yaml. Since "below" has a different meaning in PUP-7332 and the only real benefit of using "below" instead of "environment"/"modules" is that it is somewhat easier to copy paste between a global and an env based hiera.yaml. Implementation wise, validation is needed in any case. Use of "environment" and "modules" is somewhat more descriptive and it becomes a reminder of where you are when you read it.

Thomas Hallgren (JIRA)

unread,
Mar 23, 2017, 5:52:02 AM3/23/17
to puppe...@googlegroups.com

What I'm after in PUP-7332 is consistency. The term "below" must have the same meaning in all cases. As you point out, it cannot when used together with the term "layer" and still give the same meaning as in lookup_below, so introducing that here would actually make things worse. Some other terminology must be used altogether, like "yield: below" instead of "layer: <some layer>".

What I wonder is, when is there ever a need to state 'layer: module' in the global layer? The only time I can think of, is when you actually want to accomplish a "below" in the sense that it has in lookup_below. The global layer must then have two entries, one using "layer: environment" and another using "layer: module".

I'm not so sure it's a good thing to introduce a mechanism that will allow you to bypass the environment or reverse the priority of the environment and module layer. I mean, it's OK for a layer to state, "I yield to the things below me" but it's not OK to start messing with how those things are organized internally. Not to mention the mess it will create in the current implementation.

Thomas Hallgren (JIRA)

unread,
Mar 23, 2017, 6:20:02 AM3/23/17
to puppe...@googlegroups.com

Thinking one step ahead. The hierarchy is an array of hashes. Each hash as a name, a function, etc. Here, we could replace the whole hash with a simple keyword such as "yield" or "yield_below". There's no point in having name.

Henrik Lindberg (JIRA)

unread,
Mar 23, 2017, 6:21:02 AM3/23/17
to puppe...@googlegroups.com

There is no need (it should be an error) to specify layer: modules in a global hiera.yaml.

If we want to make the concepts related and to also reinforce the terminology, we could use next: layer (i.e. env in global, and modules in an env). And change the name of context.lookup_below in PUP-7332 to context.next_level_lookup - i.e. they are both about "next-ness", one about "layer" and one about "level".

Henrik Lindberg (JIRA)

unread,
Mar 23, 2017, 6:27:03 AM3/23/17
to puppe...@googlegroups.com

ok, the only reason I used two words was to have that entry be like the others and that you could name your "yield" so to say. That is not required at all and you would probably struggle to say something meaningful in its name anyway. So, I like your proposal to just have a keyword instead of name, etc.

Suggest we use next_layer as the keyword. I don't think the concept of "yield" works for the people that work with data and hierarchies.

Thomas Hallgren (JIRA)

unread,
Mar 23, 2017, 6:30:02 AM3/23/17
to puppe...@googlegroups.com

The difference is that next_layer just suggests a layer, not what's below that layer in turn (which I think it should). I would like this to have the functional meaning "perform lookup below this layer here instead of at the end" so that the logic could trigger on this and prevent further lookups once the current layer is done.

Henrik Lindberg (JIRA)

unread,
Mar 23, 2017, 7:03:02 AM3/23/17
to puppe...@googlegroups.com

"perform lookup below this layer here instead of at the end"

yes, agree.

Consider that "lookup below" is changed to "next_level_lookup", which makes the concept of "below" free to mean "layer below". I do however associate the word "below" with the actual text in the file in question as you often read "see below", so I don't know if "below" is the best term for either concept.

Trying to come up with a single household word, but end up with names like go_down, include_next, process_next, search_next, include_lower_layers, search_lower_layers, drill_down. Of those search_lower_layers is quite descriptive.

Thomas Hallgren (JIRA)

unread,
Mar 23, 2017, 7:23:02 AM3/23/17
to puppe...@googlegroups.com

If "lookup_below" is changed to "next_level_lookup", then I would suggest the keyword here to be "next_level". It's part of a definition so I don't thing it needs to be prefixed with a verb.

Henrik Lindberg (JIRA)

unread,
Mar 23, 2017, 8:02:03 AM3/23/17
to puppe...@googlegroups.com

did you mean "next_layer"? (next_level would be a noop, it is the level below this in the same hierarchy).

Thomas Hallgren (JIRA)

unread,
Mar 23, 2017, 8:54:02 AM3/23/17
to puppe...@googlegroups.com

So, the terminology for "level" is different if you call next_level_lookup and use "next_level" here? That doesn't make sense to me. The temr "next_layer", as I wrote before, suggests a layer, not the layers below that layer.

I'm trying to find a term that describes all layers below this layer, both in the hierarchy and in the proposed context function. It should be the same term. I think the best so-far is "below". I don't quite buy that it would be perceived as pointing to the next declaration in the same file as that makes no sense at all. Why then add it in the first place? The same is true for the term "next_level". I think users will understand that it's not a noop.

Henrik Lindberg (JIRA)

unread,
Apr 5, 2017, 4:28:06 PM4/5/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Sprint: PDE 2017-04-05 , PDE 2017-04-19

Eric Sorenson (JIRA)

unread,
Apr 18, 2017, 2:51:02 PM4/18/17
to puppe...@googlegroups.com
Eric Sorenson commented on New Feature PUP-7333
 
Re: make it possible to control partial ordering of layers

Is this truly necessary? It seems like it adds a great deal of complexity and I can't tell whether anyone has actually asked for it.

Eric Sorenson (JIRA)

unread,
Apr 20, 2017, 8:21:02 PM4/20/17
to puppe...@googlegroups.com
Eric Sorenson updated an issue
 
Change By: Eric Sorenson
Fix Version/s: PUP 5.0.0

Henrik Lindberg (JIRA)

unread,
Apr 25, 2017, 5:14:05 AM4/25/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Sprint: PDE 2017-04-05, PDE 2017-04-19 , PDE 2017-05-03

Henrik Lindberg (JIRA)

unread,
Apr 25, 2017, 5:16:03 AM4/25/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Sprint: PDE 2017-04-05, PDE 2017-04-19, PDE 2017-05- 03 17

Henrik Lindberg (JIRA)

unread,
Apr 25, 2017, 5:21:02 AM4/25/17
to puppe...@googlegroups.com

Henrik Lindberg (JIRA)

unread,
May 2, 2017, 10:28:07 AM5/2/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Sprint: PDE 2017-04-05, PDE 2017-04-19, PDE 2017-05- 17 31

Moses Mendoza (JIRA)

unread,
May 17, 2017, 3:10:02 PM5/17/17
to puppe...@googlegroups.com

Moses Mendoza (JIRA)

unread,
May 17, 2017, 3:11:02 PM5/17/17
to puppe...@googlegroups.com
Moses Mendoza commented on New Feature PUP-7333
 
Re: make it possible to control partial ordering of layers

Henrik Lindberg is this still an issue planned for completion?

Henrik Lindberg (JIRA)

unread,
May 18, 2017, 10:01:02 AM5/18/17
to puppe...@googlegroups.com

We know that there are problems for some to migrate to hiera 5 with data in environments because this is missing. It occurs when there are many environments (some users have several thousand envs) and you need to have some common layer that must be below environments since not all environments will have that common level, and the common in a global will override an environment. That effectively blocks a gradual migration.

I think this is an important features. It is not needed with great urgency since those that have a very large set of environments are usually not on the bleeding edge in terms of upgrading to the latest.

Henrik Lindberg (JIRA)

unread,
May 31, 2017, 8:12:05 AM5/31/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
 
Change By: Henrik Lindberg
Sprint: PDE 2017-04-05, PDE 2017-04-19 , PDE 2017-06-07

Henrik Lindberg (JIRA)

unread,
May 31, 2017, 8:12:07 AM5/31/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Sprint: PDE  Triage, PDE  2017-04-05, PDE 2017-04-19

Henrik Lindberg (JIRA)

unread,
May 31, 2017, 9:01:14 AM5/31/17
to puppe...@googlegroups.com

Henrik Lindberg (JIRA)

unread,
May 31, 2017, 9:01:22 AM5/31/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Sprint: PDE 2017-04-05,  Forge - To Accept,  PDE 2017-04-19

Henrik Lindberg (JIRA)

unread,
May 31, 2017, 9:09:04 AM5/31/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Sprint: PDE 2017-04-05,  Forge - To Accept,  PDE 2017-04-19 , Language Triage

David McTavish (Jira)

unread,
Dec 6, 2021, 1:39:01 PM12/6/21
to puppe...@googlegroups.com
David McTavish updated an issue
Change By: David McTavish
Priority: Normal Low
This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages