Jira (PUP-7479) ENC-enforced environment makes `puppet lookup` environment simulation difficult

[Gary Larizza (JIRA)]

Gary Larizza created an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult Issue Type: Improvement Affects Versions: PUP 4.9.4 Assignee: Henrik Lindberg Components: CLI Created: 2017/04/27 11:28 AM Priority: Normal Reporter: Gary Larizza The use case here is that we're trying to move people away from using the `hiera` binary to simulate/debug lookups to using `puppet lookup`. With `hiera` you could pass environment variables that matched whatever "variables" were in hiera.yaml to simulate what Puppet would provide, so doing: `hiera testkey environment=test` would allow us to simulate this lookup in the test environment. When we tried doing that with: `puppet lookup testkey - environment test` we found that we were still getting values from the production environment. This is because `puppet lookup` (even without using ` -compile`) will pull down a node object for whatever node is being targeted, and that means it's going through paths of indirection and ultimately the ENC. If the ENC is enforcing an environment, then `puppet lookup` is bound to that environment. In our case above (using the Console), setting the node to a nodegroup enforcing "Agent-specified environment" allowed us to simulate a lookup in another environment properly. I'm marking this as an "Improvement" instead of a bug, but the reality is that it's a pretty major use-ability issue because being able to simulate lookups decoupled from the ENC is a pretty common exercise for people debugging Hiera data. Add Comment

This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)

[Gary Larizza (JIRA)]

Gary Larizza updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Gary Larizza

The use case here is that we're trying to move people away from using the  `  {{ hiera ` }}  binary to simulate/debug lookups to using  `  {{ puppet lookup ` }}

.  With `hiera` you could pass environment variables that matched whatever "variables" were in hiera.yaml to simulate what Puppet would provide, so doing:  `hiera testkey environment=test` would allow us to simulate this lookup in the test environment. When we tried doing that with:  ` puppet lookup testkey --environment test `  we found that we were still getting values from the production environment. This is because `puppet lookup` (even without using `--compile`) will pull down a node object for whatever node is being targeted, and that means it's going through paths of indirection and ultimately the ENC.  If the ENC is enforcing an environment, then `puppet lookup` is bound to that environment.  In our case above (using the Console), setting the node to a nodegroup enforcing "Agent-specified environment" allowed us to simulate a lookup in another environment properly.

I'm marking this as an "Improvement" instead of a bug, but the reality is that it's a pretty major use-ability issue because being able to simulate lookups decoupled from the ENC is a pretty common exercise for people debugging Hiera data.

Add Comment

[Gary Larizza (JIRA)]

[Gary Larizza (JIRA)]

Gary Larizza updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult Change By: Gary Larizza

The use case here is that we're trying to move people away from using the {{hiera}} binary to simulate/debug lookups to using {{puppet lookup}}.  With {{hiera}} you could pass environment variables that matched whatever "variables" were in hiera.yaml to simulate what Puppet would provide, so doing:  {{hiera testkey environment=test}} would allow us to simulate this lookup in the test environment.

When we tried doing that with:  {{  puppet lookup testkey - - environment test }}  we found that we were still getting values from the production environment. This is because {{puppet lookup}} (even without using {{- -compile}}) will pull down a node object for whatever node is being targeted, and that means it's going through paths of indirection and ultimately the ENC.  If the ENC is enforcing an environment, then {{puppet lookup}} is bound to that environment.  In our case above (using the Console), setting the node to a nodegroup enforcing "Agent-specified environment" allowed us to simulate a lookup in another environment properly.

I'm marking this as an "Improvement" instead of a bug, but the reality is that it's a pretty major use-ability issue because being able to simulate lookups decoupled from the ENC is a pretty common exercise for people debugging Hiera data.

Add Comment

[Gary Larizza (JIRA)]

Gary Larizza updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult Change By: Gary Larizza

The use case here is that we're trying to move people away from using the {{hiera}} binary to simulate/debug lookups to using {{puppet lookup}}.  With {{hiera}} you could pass environment variables that matched whatever "variables" were in hiera.yaml to simulate what Puppet would provide, so doing:  {{hiera testkey environment=test}} would allow us to simulate this lookup in the test environment.

When we tried doing that with: puppet lookup testkey --environment test we found that we were still getting values from the production environment. This is because {{puppet lookup}} (even without using {{--compile}}) will pull down a node object for whatever node is being targeted, and that means it's going through paths of indirection and ultimately the ENC.  If the ENC is enforcing an environment, then {{puppet lookup}} is bound to that environment.  In our case above (using the Console), setting the node to a nodegroup enforcing "Agent-specified environment" allowed us to simulate a lookup in another environment properly.

I'm marking this as an "Improvement" instead of a bug, but the reality is that it's a pretty major use-ability issue because being able to simulate lookups decoupled from the ENC is a pretty common exercise for people debugging Hiera data.

Add Comment

[Gary Larizza (JIRA)]

[Gary Larizza (JIRA)]

Gary Larizza updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult Change By: Gary Larizza

The use case here is that we're trying to move people away from using the {{hiera}} binary to simulate/debug lookups to using {{puppet lookup}}.  With {{hiera}} you could pass environment variables that matched whatever "variables" were in hiera.yaml to simulate what Puppet would provide, so doing:  {{hiera testkey environment=test}} would allow us to simulate this lookup in the test environment.

When we tried doing that with: {{puppet lookup testkey - -environment test}} we found that we were still getting values from the production environment. This is because {{puppet lookup}} (even without using {{--compile}}) will pull down a node object for whatever node is being targeted, and that means it's going through paths of indirection and ultimately the ENC.  If the ENC is enforcing an environment, then {{puppet lookup}} is bound to that environment.  In our case above (using the Console), setting the node to a nodegroup enforcing "Agent-specified environment" allowed us to simulate a lookup in another environment properly.

I'm marking this as an "Improvement" instead of a bug, but the reality is that it's a pretty major use-ability issue because being able to simulate lookups decoupled from the ENC is a pretty common exercise for people debugging Hiera data.

Add Comment

[Lindsey Smith (JIRA)]

Lindsey Smith updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Lindsey Smith Priority: Normal Major

Add Comment

[Thomas Kishel (JIRA)]

Thomas Kishel commented on PUP-7479

Re: ENC-enforced environment makes `puppet lookup` environment simulation difficult I was about to submit a similar ticket, and I'd hate to waste formatted text ... The puppet lookup environment parameter does not override the server-specified node environment unless the node is in the 'Agent-specified environment' node group. This matches the behavior of puppet agent on a node, but is not documented (in the command help, and in the explain output} and is possibly unexpected when using puppet lookup. [root@pe-201711-master ~]# puppet lookup --node pe-201711-agent test --environment development --explain Searching for "lookup_options" Searching for "test" Global Data Provider (hiera configuration version 5) Using configuration "/etc/puppetlabs/puppet/hiera.yaml" Hierarchy entry "v5 global common heiradata" Path "/etc/puppetlabs/puppet/hieradata/common.yaml" Original path: "common.yaml" Path not found Environment Data Provider (hiera configuration version 5) Using configuration "/etc/puppetlabs/code/environments/production/hiera.yaml" Hierarchy entry "v5 env prod common data" Path "/etc/puppetlabs/code/environments/production/data/common.yaml" Original path: "common.yaml" Found key: "test" value: "PROD" Given: [root@pe-201711-master ~]# cat /etc/puppetlabs/puppet/hiera.yaml --- version: 5 defaults: datadir: hieradata data_hash: yaml_data hierarchy: - name: "v5 global common heiradata" path: "common.yaml"     [root@pe-201711-master ~]# cat /etc/puppetlabs/code/environments/development/data/common.yaml --- test: 'DEV'     [root@pe-201711-master ~]# cat /etc/puppetlabs/code/environments/production/data/common.yaml --- test: 'PROD' Results: # node in production environment via console/puppetdb [root@pe-201711-master ~]# puppet lookup --node pe-201711-agent test --environment development --- PROD ...     # node in development environment via console/puppetdb [root@pe-201711-master ~]# puppet lookup --node pe-201711-agent test --environment production --- DEV ...     # node in 'agent-specified environment' node group via console/puppetdb [root@pe-201711-master ~]# puppet lookup --node pe-201711-agent test --environment development --- DEV ...

Add Comment

[Eric Thompson (JIRA)]

Eric Thompson updated an issue

Puppet / PUP-7479

ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Eric Thompson QA Risk Assessment: Needs Assessment Automate

Add Comment

[Eric Thompson (JIRA)]

Eric Thompson updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Eric Thompson QA Risk Assessment Reason: could be automated in an integration test in rspec

Add Comment

[Thomas Hallgren (JIRA)]

Thomas Hallgren commented on PUP-7479

Re: ENC-enforced environment makes `puppet lookup` environment simulation difficult This is one of those cases where it would be beneficial to write the acceptance test first. With an acceptance test that provokes the problem, it's easy to debug and assert that the problem is fixed and such a test will be needed anyway. I'm therefore assigning this to QA.

Add Comment

[Thomas Hallgren (JIRA)]

Thomas Hallgren assigned an issue to Thomas Hallgren

Puppet / PUP-7479

ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Thomas Hallgren Assignee: Henrik Lindberg Thomas Hallgren

Add Comment

[Thomas Hallgren (JIRA)]

Thomas Hallgren assigned an issue to qa

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Thomas Hallgren Assignee: Thomas Hallgren qa

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-7479

Re: ENC-enforced environment makes `puppet lookup` environment simulation difficult Lindsey Smith the solution is to override the override for "agent may not specify environment", but it may be a bit tricky due to how those interfaces work together and that --environment is a setting that is automatically handled. Not sure an acceptance test is needed to run the CLI with a configuration that denies the agent from setting the environment - but I could be wrong.

Add Comment

[Nick Walker (JIRA)]

Nick Walker commented on PUP-7479

Re: ENC-enforced environment makes `puppet lookup` environment simulation difficult

Could we ship an additional classifier terminus that reads the output of the existing terminus and strips out the environment key? Then when --environment is passed to lookup we could seamlessly switch to using the terminus that strips the environment key.

Add Comment

[Gary Larizza (JIRA)]

Gary Larizza commented on PUP-7479

Re: ENC-enforced environment makes `puppet lookup` environment simulation difficult

I think visibility is going to be the most important thing here - if you specify --environment test and the classifier hands you back the results for the production environment, then there needs to be some data to even let you know that there's a problem (with a recommendation as to how to correct it). In this case the coverup is worse than the crime. Nick's idea seems more of a hack than a long-term solution, but I suspect once customers understand what's going on that they'll have a way to work around it.

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-7479

Re: ENC-enforced environment makes `puppet lookup` environment simulation difficult

Wrapping the configured terminus in one that does what Nick Walker suggests would solve the problem. Gary Larizza Are you saying that it is a bad idea in the first place to allow a user to specify the environment when doing a lookup, and that they must configure the ENC to allow it? Any implementation that "pokes" the CLI given environment would behave the same way. An implementation could: Add the following option to the lookup application option('--environment name') do |arg| options[:environment] = arg end And then get the node returned from the indirection: node = Puppet::Node.indirection.find(node) And then do the following override (i.e. always) node.environment = options[:environment] unless options[:environment].nil? if we need to output that the node switched env then it is difficult to get that into the actual explain output, but it could be logged to console.

Add Comment

[Gary Larizza (JIRA)]

Gary Larizza commented on PUP-7479

Re: ENC-enforced environment makes `puppet lookup` environment simulation difficult

My comment about it being a hack is based adding another terminus that "seamlessly switches" - that feels like more magic. And it's definitely not a bad thing that a user ask for a different environment when performing a lookup (quite the opposite, it's the reason I filed the ticket). I like Henrik's solution of handling it inside puppet lookup. I don't think there's ever been the expectation that users couldn't perform a lookup from an environment OTHER than the one the ENC is enforcing, so I feel like that's the best place for this.

Add Comment

[Nick Walker (JIRA)]

Nick Walker commented on PUP-7479

Re: ENC-enforced environment makes `puppet lookup` environment simulation difficult

Gary Larizza to clarify, I was trying to propose that puppet lookup simply strip the environment key returned from any classifier if the --environment flag is passed. I muddied the waters by suggesting implementation details. I definitely don't want a terminus to be doing anything magical. I want puppet lookup to do the magic and not have it in the puppet master code.

Add Comment

[Gary Larizza (JIRA)]

Gary Larizza commented on PUP-7479

Re: ENC-enforced environment makes `puppet lookup` environment simulation difficult

Nick Walker Ahhh, that makes more sense - sounds like we're all looking at the same place then

Add Comment

[Ethan Brown (JIRA)]

Ethan Brown updated an issue

Puppet / PUP-7479

ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Ethan Brown Labels: triaged

Add Comment

[Ethan Brown (JIRA)]

Ethan Brown updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Ethan Brown Team: Puppet Developer Experience Agent

Add Comment

[Jarret Lavallee (JIRA)]

Jarret Lavallee updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Jarret Lavallee CS Priority: Needs Priority

Add Comment

This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db)

[Owen Rodabaugh (JIRA)]

Owen Rodabaugh updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Owen Rodabaugh CS Priority: Needs Priority Major CS Impact: `puppet lookup` is a great tool for helping understand what hiera is doing. Improving it by adding the ability for the user to specify the environment would make it even better and reduce this one point of friction. Usability/intuitiveness is an area we need to improve on as we've gotten a lot of user feedback. Improving this would help reduce user surprise. CS Severity: 3 - Serious CS Business Value: 5 - $$$$$$ CS Frequency: 3 - 25-50% of Customers

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-7479

Re: ENC-enforced environment makes `puppet lookup` environment simulation difficult This issue is somewhat related to PUP-8094, in which Eric Sorenson said it (lookup) should treat the ENC as authoritative. The difference here is whether the --environment flag means lookup should "start in that environment and possibly change like the agent" or it should "always use" that environment. I think the latter is fine, since puppet lookup runs in a compiler context puppet parser, where the --environment option is taken to mean "always compile in that environment". It should be easy to modify puppet lookup to always enforce the specified environment.

Add Comment

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-7479

ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Josh Cooper Acceptance Criteria: If the --environment flag is used then puppet lookup should always use that environment regardless of what the ENC says. If the environment doesn't exist locally then it should error since we don't have the necessary per-environment hiera configuration and data. When the " \ - \ -compile" flag is omitted, then this "accidentally works" due to the change made in PUP-8502. Once lookup defaults to the ENC environment (in PUP-8094) then it will cause this issue to "regress" when the " \ - \ -compile" flag is specified. So this ticket is to make sure " \ - \ -environment" works consistent with or without " \ - \ -compile" and is blocked on PUP-8502

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Josh Cooper Acceptance Criteria:

If the --environment flag is used then puppet lookup should always use that environment regardless of what the ENC says. If the environment doesn't exist locally then it should error since we don't have the necessary per-environment hiera configuration and data.

When the "--compile" flag is omitted, then this "accidentally works" due to the change made in PUP-8502. Once lookup defaults to the ENC environment (in PUP-8094) then it will cause this issue to "regress" when the "--compile" flag is specified. So this ticket is to make sure "--environment" works consistent with or without "--compile" and is blocked on PUP-8502 Sub-team: Language Team: Froyo Coremunity

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Josh Cooper Acceptance Criteria:

If the --environment flag is used then puppet lookup should always use that environment regardless of what the ENC says. If the environment doesn't exist locally then it should error since we don't have the necessary per-environment hiera configuration and data.

When the "\-\-compile" flag is omitted, then this "accidentally works" due to the change made in PUP-8502. Once lookup defaults to the ENC environment (in PUP-8094) then it will cause this issue to "regress" when the "\-\-compile" flag is specified. So this ticket is to make sure "\-\-environment" works consistent with or without "\-\-compile" and is blocked on PUP- 8502 8094

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Josh Cooper Sprint: Coremunity Grooming

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Josh Cooper Sprint: Coremunity Grooming Hopper

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Josh Cooper Epic Link: PUP- 6870 11052

Add Comment

This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Josh Cooper Sprint: Coremunity Hopper , Community PRs 2

Add Comment

[Ciprian Badescu (Jira)]

Ciprian Badescu updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Ciprian Badescu Sprint: NW - 2021-10-20

Add Comment

[Ciprian Badescu (Jira)]

Ciprian Badescu updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Ciprian Badescu Sprint: NW - 2021-11-03

Add Comment

[Ciprian Badescu (Jira)]

Ciprian Badescu commented on PUP-7479

Re: ENC-enforced environment makes `puppet lookup` environment simulation difficult To be reproduced after environment convergence changes and assess if it is still possible to overwrite environment from cmd line

Add Comment

[Ciprian Badescu (Jira)]

Ciprian Badescu updated an issue

Puppet / PUP-7479

ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Ciprian Badescu Story Points: 5

Add Comment

[Ciprian Badescu (Jira)]

Ciprian Badescu updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Ciprian Badescu Sprint: NW - 2021-11-03

Add Comment

[Ciprian Badescu (Jira)]

Ciprian Badescu updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Ciprian Badescu Sprint: ready for triage

Add Comment

[Ciprian Badescu (Jira)]

Ciprian Badescu updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Ciprian Badescu Sprint: ready for triage NW - 2021-11-17

Add Comment

[Victor Bobosila (Jira)]

Victor Bobosila assigned an issue to Victor Bobosila

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Victor Bobosila Assignee: Victor Bobosila

Add Comment

[Ciprian Badescu (Jira)]

Ciprian Badescu updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Ciprian Badescu Sprint: NW - 2021-11-17 , NW - 2021-12-06

Add Comment

[Victor Bobosila (Jira)]

Victor Bobosila updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Victor Bobosila Release Notes: Enhancement

Add Comment

[Victor Bobosila (Jira)]

Victor Bobosila updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Victor Bobosila Release Notes Summary: Changed the priority of the `--environment` option by skipping the classification whenever the flag is specified. This allows the user to bypass the ENC enforced environment with the CLI option.

Add Comment

[Michael Hashizume (Jira)]

Michael Hashizume updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Michael Hashizume Fix Version/s: PUP 6.26.0 Fix Version/s: PUP 7.13.0

Add Comment

[Parker Leach (Jira)]

Parker Leach updated an issue

Puppet / PUP-7479 ENC-enforced environment makes `puppet lookup` environment simulation difficult

Change By: Parker Leach Labels: docs_reviewed

Add Comment

This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)