Jira (PUP-6739) Error running puppet with non-existent and non-default environment

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-6739 Error running puppet with non-existent and non-default environment Change By: Josh Cooper Summary: If no matching local $environment folder on agent, error when Error  running  any ' puppet  config' command  with non-existent and non-default environment Add Comment

This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)

[Maggie Dreyer (JIRA)]

Maggie Dreyer updated an issue

Puppet / PUP-6739 Error running puppet with non-existent and non-default environment

Change By: Maggie Dreyer Labels: triaged

Add Comment

[Geoff Nichols (JIRA)]

Geoff Nichols assigned an issue to Unassigned

Puppet / PUP-6739 Error running puppet with non-existent and non-default environment

Change By: Geoff Nichols Assignee: Henrik Lindberg

Add Comment

[Kenn Hussey (JIRA)]

Kenn Hussey updated an issue

Puppet / PUP-6739 Error running puppet with non-existent and non-default environment

Change By: Kenn Hussey Fix Version/s: PUP 5.0.0 Fix Version/s: PUP 5.y

Add Comment

[Kenn Hussey (JIRA)]

Kenn Hussey updated an issue

Puppet / PUP-6739 Error running puppet with non-existent and non-default environment

Change By: Kenn Hussey Sprint: Puppet 5 Race to the Finish! Agent N+1

Add Comment

[Jesse Reynolds (JIRA)]

Jesse Reynolds commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment The same error is reported if you run {puppet resource service puppet} after setting an agent's environment to something other than production. Eg: # puppet --version 4.10.1 # grep 'environment =' /etc/puppetlabs/puppet/puppet.conf environment = foobar # puppet resource service puppet /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/environments.rb:38:in `get!': Could not find a directory environment named 'foobar' anywhere in the path: /etc/puppetlabs/code/environments. Does the directory exist? (Puppet::Environments::EnvironmentNotFound) from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application_support.rb:29:in `push_application_context' from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:346:in `run' from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:132:in `run' from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:72:in `execute' from /usr/local/bin/puppet:5:in `<main>'

Add Comment

[Paul Anderson (JIRA)]

Paul Anderson commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment

Jesse Reynolds When this bug is rearing its ugly head, everything is broken including puppet help.... The workaround is to slap --environment production as a closing argument to the puppet command.

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-6739

Error running puppet with non-existent and non-default environment

Change By: Josh Cooper Fix Version/s: PUP 5.y Fix Version/s: PUP 5.1.0

Add Comment

[Dylan Ratcliffe (JIRA)]

Dylan Ratcliffe commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment It's worth noting here that this will also break puppetserver if you try to start puppetserver on a machine that doesn't have it's environment on disk. A good example of this is if you try to promote a non-production agent to a compile master, puppetserver fails to start because of this bug. Which is extra annoying because you need puppetserver running to get code manager/file-sync to actually create the directory and fix the issue.

Add Comment

[Kenn Hussey (JIRA)]

Kenn Hussey updated an issue

Puppet / PUP-6739

Error running puppet with non-existent and non-default environment

Change By: Kenn Hussey Flagged: Impediment

Add Comment

[Karen Van der Veer (JIRA)]

Karen Van der Veer updated an issue

Puppet / PUP-6739 Error running puppet with non-existent and non-default environment

Change By: Karen Van der Veer Sprint: Platform Core 2017-09- 05 19

Add Comment

[Karen Van der Veer (JIRA)]

Karen Van der Veer updated an issue

Puppet / PUP-6739 Error running puppet with non-existent and non-default environment

Change By: Karen Van der Veer

Sprint: Platform Core  2017-09-19  Grooming

Add Comment

[Karen Van der Veer (JIRA)]

Karen Van der Veer updated an issue

Puppet / PUP-6739 Error running puppet with non-existent and non-default environment

Change By: Karen Van der Veer

Sprint: Platform Core Grooming

Add Comment

[Karen Van der Veer (JIRA)]

Karen Van der Veer updated an issue

Puppet / PUP-6739 Error running puppet with non-existent and non-default environment

Change By: Karen Van der Veer

Fix Version/s: PUP 5.2.0 Fix Version/s: PUP 5.y

Add Comment

[Sam McLeod (JIRA)]

Sam McLeod commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment This bug is very annoying. Our Puppet 3 (Open Source) workflow was: 1. Create branch (r10k environment) 2. Set nodes to run off branch by setting `environment = <branchname>` in the nodes puppet.conf 3. Develop on and test branch 4. Merge branch in to production Now the only way we can reliably test branches is to enable puppet on the node, do interactive puppet runs passing `--environment=<branchname>` and disable puppet again so it doesn't default back to running off production. I tried creating the branch directory in `/etc/puppetlabs/code/environments/<branchname>` as suggested above, but that just results in no-change puppet runs. I'm really surprised this has gone unfixed for so long.

Add Comment

[Sam McLeod (JIRA)]

Sam McLeod commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment

I've just found a repeatable workaround for this bug. Simply creating the environment directory does not work (`/etc/puppetlabs/code/environment/<branchname>`) however if you copy the `production` environment's directory and it's substructure - it does seem to work: ``` cp -a /etc/puppetlabs/code/environment/production /etc/puppetlabs/code/environment/<branchname> puppet agent -t ... ```` So, while it's not a fix, this hopefully will serve as a temporary workaround until this bug is squashed!

Add Comment

[Adam Bottchen (JIRA)]

Adam Bottchen updated an issue

Puppet / PUP-6739

Error running puppet with non-existent and non-default environment

Change By: Adam Bottchen CS Priority: Needs Priority

Add Comment

[Owen Rodabaugh (JIRA)]

Owen Rodabaugh updated an issue

Puppet / PUP-6739 Error running puppet with non-existent and non-default environment

Change By: Owen Rodabaugh CS Priority: Needs Priority Major CS Impact: While this issue isn't difficult for an experienced user to work around. For many new or non-english reading users the error may not be easily understood and breaks CLI usage of puppet on the node. CS Severity: 3 - Serious CS Business Value: 4 - $$$$$ CS Frequency: 4 - 50-90% of Customers

Add Comment

[Martijn Pepping (JIRA)]

Martijn Pepping commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment We resolved (eg. worked around) this issue by moving the environment key from the [main] section of puppet.conf to the [agent] section. Using puppet config set this should be something like puppet config set --section agent environment dev. With this in place commands like puppet config print don't bork with the Could not find a directory environment named 'dev' anywhere in the path: /etc/puppetlabs/code/environments error.

Add Comment

[Sam McLeod (JIRA)]

Sam McLeod commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment

Good to know Martijn Pepping thanks for the update! Perhaps then - the Puppetlabs documentation needs to be updated to reflect this?

Add Comment

[Karen Van der Veer (JIRA)]

Karen Van der Veer updated an issue

Puppet / PUP-6739

Error running puppet with non-existent and non-default environment

Change By: Karen Van der Veer

Labels: the-goods

Add Comment

[Maggie Dreyer (JIRA)]

Maggie Dreyer updated an issue

Puppet / PUP-6739 Error running puppet with non-existent and non-default environment

Change By: Maggie Dreyer Sprint: Platform Core Hopper

Add Comment

[Melissa Stone (JIRA)]

Melissa Stone assigned an issue to Melissa Stone

Puppet / PUP-6739 Error running puppet with non-existent and non-default environment

Change By: Melissa Stone Assignee: Melissa Stone

Add Comment

This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db)

[Melissa Stone (JIRA)]

Melissa Stone updated an issue

Puppet / PUP-6739 Error running puppet with non-existent and non-default environment

Change By: Melissa Stone Sprint: Platform Core  Hopper  KANBAN

Add Comment

[Melissa Stone (JIRA)]

Melissa Stone commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment Alright, so moving this ticket forward, I see a two part solution being the best course of action. The first is to improve documentation around the definition of environment and the implications of setting environment in different contexts. I believe a lot of people running into this issue are using environment to refer to the environment that is present on the master filesystem, but setting that globally in puppet.conf. Ideally this should be set in the [master] section. This way, when running commands that require an environment to exist locally, puppet will not expect that remote environment to exist locally. However, running puppet agent will still request the correct environment from the master. The second is to allow certain applications to not care about loading a local environment. They will if they find it, but they will not error out if they do not find it. This will allow users to continue to run these applications even when, for example, they have a typo in the environment name defined in puppet.conf. I have a pull request up for the second part of this solution which can be found at https://github.com/puppetlabs/puppet/pull/6343

Add Comment

[Paul Anderson (JIRA)]

Paul Anderson commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment

Melissa Stone, I think the drop dead simple most people encounter this issue is that they, like all good sysadmins, want to remove the warning they see on non-production managed puppet nodes (not the server): Notice: Local environment: 'production' doesn't match server specified node environment 'development', switching agent to 'development'. Some people see that and get concerned and do something like this: https://groups.google.com/forum/#!topic/puppet-users/RjdQ7jXLx5w The unfortunate issue is that that advice USED to work, but now causes fireworks. I suppose it could still work if we added an additional resource to the base profile: {{class profile::base { ... ini_setting { "puppet_environment": ensure => present, path => '/etc/puppet/puppet.conf', section => 'agent', setting => 'environment', value => $::environment, } file { 'create empty local environment on agent nodes so puppet commands don't explode' : ensure => present, path => "/etc/puppetlabs/code/environments/$ {environment} , } ... }}} (This is related to your second scenario).

Add Comment

[Thomas Mueller (JIRA)]

Thomas Mueller commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment

+1 for Paul Anderson the Notice used to be a Warning in <4.x days which was a bit confusing - thus I know many who have added a [agent] environment = xyz puppet.conf setting by puppet to avoid it. this used to work and now fails.

Add Comment

[Melissa Stone (JIRA)]

Melissa Stone commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment

Thomas Mueller are you saying that setting [agent] environment = xyz is causing the following failure? # puppet config print environment /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/environments.rb:38:in `get!': Could not find a directory environment named 'xyz' anywhere in the path: /etc/puppetlabs/code/environments. Does the directory exist? (Puppet::Environments::EnvironmentNotFound)

from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application_support.rb:29:in `push_application_context' from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:346:in `run' from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:132:in `run' from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:72:in `execute'

from /opt/puppetlabs/puppet/bin/puppet:5:in `<main>' In my exploration of this issue, that failure only occurred with # cat /etc/puppetlabs/puppet/puppet.conf environment = xyz or # cat /etc/puppetlabs/puppet/puppet.conf [main] environment = xyz Setting environment under the [agent] section allowed puppet config and puppet help to function as expected. Paul Anderson Are you asking me to address the warning that folks are seeing when the local environment doesn't match the server specified node environment, rather than the error users are reporting here when running commands like puppet config?

Add Comment

[Thomas Mueller (JIRA)]

Thomas Mueller commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment

Melissa Stone The error is indeed {{ Could not find a directory environment named }}. Its originating not from puppet-agent but from puppet server. I think my situation is similiar but not the same: on the puppetserver the ENC enforces "environment=env_A" for node hostA puppet-agent adds [agent] environment = env_A to puppet.conf on hostA Admin X changes environment to env_B in ENC for hostA puppet-agent on hostA requests catalog with env_A (because defined in puppet.conf), fails with {{ Could not find a directory environment named }} and (silently) applies the cached catalog.

Add Comment

[Eric Delaney (JIRA)]

Eric Delaney updated an issue

Puppet / PUP-6739

Error running puppet with non-existent and non-default environment

Change By: Eric Delaney QA Risk Assessment: Manual

Add Comment

[Michael Smith (JIRA)]

Michael Smith commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment Thomas Mueller is it that env_A is renamed to env_B, so env_A no longer exists on the master? If so, I believe that should be handled as a separate ticket, PUP-6802 in fact.

Add Comment

[Paul Anderson (JIRA)]

Paul Anderson commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment

Melissa Stone I see how I was unclear in my communication (I'm not talking about the notice/warning; it's the crash!). The issue does not occur when someone sets their environment in the agent section of the configuration. However, if one sets the environment in the [main] section on a Puppet-managed system running only the agent, they will encounter this error. This is quickly reproducible via puppet config set agent other_than_production and then puppet help. Also, to your documentation point, [the current documentation]( https://puppet.com/docs/puppet/5.3/config_file_main.html#config-sections) would seem to indicate that setting the environment in [main] should be fine, since one would assume no [server] functions would occur on an agent-only node: main is the global section used by all commands and services. It can be overridden by the other sections.

Add Comment

[Melissa Stone (JIRA)]

Melissa Stone commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment

Paul Anderson The issue with the documentation is that we are overloading environment. It is used for different things in different contexts. You can find a good summary at https://tickets.puppetlabs.com/browse/PUP-6048?focusedCommentId=275369&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-275369. In the situation where users want to set the environment in puppet.conf, but want to refer to an environment that only exists on the master, that should always be set in the [agent] section. If set either at the top level or in [main], commands like puppet apply and puppet config assume environment is referring to an environment that is defined locally (i.e., exists on the local filesystem). My argument is that the current documentation is misleading, as it's apparent that setting the environment in [main] isn't doing what users expect it to do. Do you agree?

Add Comment

[Paul Anderson (JIRA)]

Paul Anderson commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment

Melissa Stone I agree that if I put my end user had on, and I set the environment in [main], and Puppet explodes, and double checking the documentation doesn't give me any reason to prefer the [agent] environment, that the current documentation is misleading.

Add Comment

[Garrett Guillotte (JIRA)]

Garrett Guillotte commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment

Docs PR reviewed by Melissa Stone and ready to merge.

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment

Merged to 4.10.x in https://github.com/puppetlabs/puppet/commit/767c454bb6b8efa11bc3c1eb8252740050173749

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-6739

Error running puppet with non-existent and non-default environment

Change By: Josh Cooper Fix Version/s: PUP 5.y Fix Version/s: PUP 4.10.10

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg updated an issue

Puppet / PUP-6739 Error running puppet with non-existent and non-default environment

Change By: Henrik Lindberg Fix Version/s: PUP 5.4.0 Fix Version/s: PUP 5.3.4

Add Comment

[Craig Gomes (JIRA)]

Craig Gomes updated an issue

Puppet / PUP-6739 Error running puppet with non-existent and non-default environment

Change By: Craig Gomes Sub-team: Coremunity

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper assigned an issue to Eric Delaney

Puppet / PUP-6739 Error running puppet with non-existent and non-default environment

Change By: Josh Cooper Assignee: Eric Delaney

Add Comment

[Eric Delaney (JIRA)]

Eric Delaney commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment Tested on SHA=85eed087b7c38e34d6d6465d94d016ab129268df SUITE_VERSION=5.3.3.140.g85eed08 PASSED [root@wmflkyw0qx7ablp ~]# puppet config print environment production [root@wmflkyw0qx7ablp ~]# puppet config set environment 'foo' Warning: The environment should be set in either the `[user]`, `[agent]`, or `[master]` section. Variables set in the `[agent]` section are used when running `puppet agent`. Variables set in the `[user]` section are used when running various other puppet subcommands, like `puppet apply` and `puppet module`; these require the defined environment directory to exist locally. Set the config section by using the `--section` flag. For example, `puppet config --section user set environment foo`. For more information, see https://puppet.com/docs/puppet/latest/configuration.html#environment [root@wmflkyw0qx7ablp ~]# puppet config --section user set environment 'foo' [root@wmflkyw0qx7ablp ~]# puppet config print environment foo [root@wmflkyw0qx7ablp ~]# egrep 'environment|^[[]' /etc/puppetlabs/puppet/puppet.conf environment = foo [user] environment = foo

Add Comment

[Kenn Hussey (JIRA)]

Kenn Hussey commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment

Melissa Stone please provide release notes for this issue, if needed. Thanks!

Add Comment

[Melissa Stone (JIRA)]

Melissa Stone updated an issue

Puppet / PUP-6739

Error running puppet with non-existent and non-default environment

Change By: Melissa Stone Release Notes Summary: Certain puppet subcommands like `puppet help` and `puppet config` no longer require a local environment to exist in order to function. They will check for the local environment, but if it does not exist they will fall back to assuming the defined environment exists on the master filesystem. Release Notes: Bug Fix

Add Comment

[Thomas Kishel (JIRA)]

Thomas Kishel commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment This appears to be an issue in 2017.3.4 / Puppet 5.3.4: [root@pe-201734-agent ~]# puppet agent -t Warning: Unable to fetch my node definition, but the agent run will continue: Warning: Could not intern from application/json: Could not find a directory environment named 'development' anywhere in the path: /etc/puppetlabs/code/environments. Does the directory exist? Info: Retrieving pluginfacts Info: Retrieving plugin Info: Retrieving locales Info: Loading facts Info: Caching catalog for pe-201734-agent.puppetdebug.vlan Notice: Local environment: 'production' doesn't match server specified environment 'development', restarting agent run with environment 'development' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Retrieving locales Info: Loading facts Info: Caching catalog for pe-201734-agent.puppetdebug.vlan Info: Applying configuration version '1518126453' Notice: Applied catalog in 0.36 seconds   [root@pe-201734-agent ~]# puppet -V 5.3.4

Add Comment

This message was sent by Atlassian JIRA (v7.5.1#75006-sha1:7df2574)

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment

Thomas Kishel this was a regression in 5.3.4, see PUP-8419. It will be fixed in 5.3.5.

Add Comment

[Nick (Jira)]

Nick commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment

This is happening with Puppet 5.5.21. $ sudo puppet filebucket -l list --bucket /opt/puppetlabs/puppet/cache/clientbucket/ /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/environments.rb:38:in `get!': Could not find a directory environment named 'testenv' anywhere in the path: /etc/puppetlabs/code/environments. Does the directory exist? (Puppet::Environments::EnvironmentNotFound) from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application_support.rb:34:in `push_application_context' from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:363:in `run' from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:139:in `run' from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'

from /opt/puppetlabs/puppet/bin/puppet:5:in `<main>'

$ puppet --version 5.5.21 $ ls -lhatr /etc/puppetlabs/code/environments/ total 0 drwxr-xr-x. 4 root root 41 Jul 3 07:34 .. drwxr-xr-x. 3 root root 24 Jul 3 07:34 . drwxr-xr-x. 5 root root 92 Aug 21 01:30 production ${noformat} Josh Cooper can I provide any more information to help get this resolved?

Add Comment

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment

Nick this issue was resolved for config, help and a few other built-in applications. Can you file a new ticket for filebucket? The fix is relative simple, just need to add environment_mode :not_required to https://github.com/puppetlabs/puppet/blob/8bf64bb72eb48cca24d7e6e4c99e6b21e497fa86/lib/puppet/application/filebucket.rb#L4

Add Comment

[Nick (Jira)]

Nick commented on PUP-6739

Re: Error running puppet with non-existent and non-default environment

Josh Cooper sure thing, I created  PUP-10796 for filebucket command.

Add Comment