Jira (PUP-6802) Agent cannot compile catalog if it specifies an non-existent environment in puppet.conf even when the classifier is controlling environment

29 views
Skip to first unread message

Lindsey Smith (JIRA)

unread,
Oct 7, 2016, 12:05:03 PM10/7/16
to puppe...@googlegroups.com
Lindsey Smith updated an issue
 
Puppet / Bug PUP-6802
Agent cannot compile catalog if it specifies an non-existent environment in puppet.conf even when the classifier is controlling environment
Change By: Lindsey Smith
Summary: Agent cannot compile catalog if it specifies an non- existant existent  environment in puppet.conf even when the classifier is controlling environment
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
Atlassian logo

Nick Walker (JIRA)

unread,
Oct 7, 2016, 12:16:03 PM10/7/16
to puppe...@googlegroups.com
Nick Walker commented on Bug PUP-6802
 
Re: Agent cannot compile catalog if it specifies an non-existent environment in puppet.conf even when the classifier is controlling environment

I think PUP-6048 is related but different. That has to do with `puppet facts` looking for an environment locally which is unexpected. In the case of this ticket I can successfully run `puppet facts` .

If I set environment in [main] on the box I was testing with we see that I reproduce PUP-6048.

[root@agent201621-centos ~]# puppet facts
{
  "name": "agent201621-centos",
...
}
[root@agent201621-centos ~]# puppet config set environment fake --section main
[root@agent201621-centos ~]# puppet facts
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/environments.rb:38:in `get!': Could not find a directory environment named 'fake' anywhere in the path: /etc/puppetlabs/code/environments. Does the directory exist? (Puppet::Environments::EnvironmentNotFound)
        from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application_support.rb:29:in `push_application_context'
        from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:337:in `run'
        from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:128:in `run'
        from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:72:in `execute'
        from /usr/local/bin/puppet:5:in `<main>'

Nick Walker (JIRA)

unread,
Oct 7, 2016, 12:17:03 PM10/7/16
to puppe...@googlegroups.com

Branan Riley (JIRA)

unread,
Nov 2, 2016, 4:30:07 PM11/2/16
to puppe...@googlegroups.com
Branan Riley commented on Bug PUP-6802
 
Re: Agent cannot compile catalog if it specifies an non-existent environment in puppet.conf even when the classifier is controlling environment

This is "working as intended". The agent needs a valid environment in its puppet.conf in order to pluginsync custom fact implementations. If it does not do that sync, it may not have all the facts that the classifier expects in order to select an environment appropriately.

It might be valuable to allow users to specify an "initial pluginsync" environment master-side (or console-side, in the PE case), but for now, that environment comes from the agent config.

Nick Walker (JIRA)

unread,
Nov 2, 2016, 5:15:03 PM11/2/16
to puppe...@googlegroups.com
Nick Walker commented on Bug PUP-6802

If it does not do that sync, it may not have all the facts that the classifier expects in order to select an environment appropriately.

I think if the environment from an ENC is going to be authoritative on environment then that should apply to pluginsync as well. If your ENC requires facts that you need pluginsynced then you should make sure your ENC returns a default environment when it does not match on those facts.

In the case of PE, if you're using facts to determine the environment then I expect that you would make child groups from a default environment group that use the facts to determine an environment. This means that if you don't have the facts yet then you just fall back into the default environment group.

This is actually what we document for users. https://docs.puppet.com/pe/latest/console_classes_groups_environment_override.html#workflow

Reid Vandewiele (JIRA)

unread,
Nov 2, 2016, 6:31:04 PM11/2/16
to puppe...@googlegroups.com

This ticket raises that in the event a non-valid environment is specified, Puppet vomits up a screenful of unhelpful, low-level error text. That's a problem. I don't think it's our desired user experience, and I'm sure we can do better.

If we need to abort because we're afraid of allowing the master to select an environment for the agent if the agent doesn't submit facts, we need to clean up this error so it's clear what's going on and why the run aborted.

Alternatively, don't abort. If we can't pluginsync, purge plugins, submit what we have and use whatever environment we get back. In a master-chooses-the-environment situation it's against best practice to use anything user-mutable to select an environment anyway.

Eric Sorenson (JIRA)

unread,
Jan 30, 2017, 5:11:03 PM1/30/17
to puppe...@googlegroups.com

Eric Sorenson (JIRA)

unread,
Jan 30, 2017, 5:12:15 PM1/30/17
to puppe...@googlegroups.com

Geoff Nichols (JIRA)

unread,
Apr 5, 2017, 1:07:06 AM4/5/17
to puppe...@googlegroups.com

Owen Rodabaugh (JIRA)

unread,
Sep 14, 2017, 6:44:09 PM9/14/17
to puppe...@googlegroups.com
Owen Rodabaugh updated an issue
Change By: Owen Rodabaugh
CS Priority: Major
CS Impact: Similar impact as PUP-6739
CS Severity: 3 - Serious
CS Business Value: 4 - $$$$$
CS Frequency: 4 - 50-90% of Customers

Karen Van der Veer (JIRA)

unread,
Sep 19, 2017, 2:52:04 PM9/19/17
to puppe...@googlegroups.com

Josh Cooper (JIRA)

unread,
Mar 16, 2018, 4:25:04 PM3/16/18
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Sub-team: Coremunity
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Josh Cooper (Jira)

unread,
Apr 1, 2020, 2:11:03 AM4/1/20
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-6802
 
Re: Agent cannot compile catalog if it specifies an non-existent environment in puppet.conf even when the classifier is controlling environment

The problem is the agent makes a node request to ask the server which environment it is supposed to be in. This is routed to puppet's indirected_routes logic, which handles all indirected REST requests. That code raises if the requested environment doesn't exist on the master. Generally that's a good thing, because we want a valid environment from which to serve plugins, catalogs, etc.

But it doesn't make sense to force the agent to specify an environment that must exist on the master in order to ask the question "which environment should I be in". I think the ideal thing is to have a different REST API for asking the environment question. It should probably take the agent's current environment and facts (so the ENC doesn't need to get cached values from puppetdb, which may not exist yet), and return the server-specified environment, instead of a node object and all of its facts, which the agent already knows. The returned environment might be the same as what the agent is already in, if the agent is allowed to choose.

Another option is to special case the node request, so if the requested environment doesn't exist on the server, have it fallback to a known good environment that does exist on the server, like:

    configured_environment = Puppet.lookup(:environments).get(environment)
    if configured_environment.nil?
      configured_environment = Puppet.lookup(:current_environment) if indirection_name == 'node'
    else
      configured_environment = configured_environment.override_from_commandline(Puppet.settings)
      params[:environment] = configured_environment
    end

However that's a bit of a hack and we have to be careful about now allowing environments to be enumerated by an unauthenticated user.

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Justin Stoller (Jira)

unread,
Apr 1, 2020, 3:17:04 PM4/1/20
to puppe...@googlegroups.com

I was also thinking about this, though I think my two ideas are probably too large to have a good ROI:

  1. I think in general the indirector is doing too much here and pushing the error handling for if there's a valid environment into the individual indirections/terminii would be good. In this scenario, we could remove the requirement that requests contain an environment parameter and indirections/termini that need it could err if they require it and it isn't provided by an ENC.
  2. If the setting "environment" is set to "production" (its default) and it doesn't exist in an environment path, we create it from the modulepath - which allows a bare "/etc/puppetlabs/code" to be the "production" env. In doing so, the "production" environment almost always has a Ruby representation w/in the server, and consequently has been passed when the user means, "An environment isn't applicable to me", or "It's an error for the Server/ENC not to classify me". Something else we could do is to tease apart those use cases and how the effectively use "production" as a magic string. Maybe give them their own magic string? So we'd have calls that provide "environment=:not-applicable:" or "environment=:server-specified:"?

Josh Cooper (Jira)

unread,
Jun 5, 2020, 5:47:04 PM6/5/20
to puppe...@googlegroups.com

Ciprian Badescu (Jira)

unread,
Jun 16, 2021, 4:45:02 AM6/16/21
to puppe...@googlegroups.com

Ciprian Badescu (Jira)

unread,
Jun 30, 2021, 5:07:02 AM6/30/21
to puppe...@googlegroups.com

Dorin Pleava (Jira)

unread,
Jul 5, 2021, 5:07:03 AM7/5/21
to puppe...@googlegroups.com

Dorin Pleava (Jira)

unread,
Jul 14, 2021, 3:16:05 AM7/14/21
to puppe...@googlegroups.com
Dorin Pleava updated an issue
Change By: Dorin Pleava
Release Notes: Bug Fix
Release Notes Summary: When an agent runs with a environment that does not exist on the server,
puppet will fail.
Now in case it cannot get the facts or catalog for the specified
environment, it will fallback to the environment configured in the
new default_agent_environment puppet setting.

Ciprian Badescu (Jira)

unread,
Jul 14, 2021, 4:49:03 AM7/14/21
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Jul 14, 2021, 5:44:03 PM7/14/21
to puppe...@googlegroups.com

Ciprian Badescu (Jira)

unread,
Jul 28, 2021, 5:23:02 AM7/28/21
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Aug 2, 2021, 12:35:04 PM8/2/21
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Aug 2, 2021, 12:42:01 PM8/2/21
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-6802

This change requires that the puppet-agent package is updated on the agent and the server. The agent works as expected if configured to use a server-specified environment web:

[root@vice-disarray ~]# mkdir -p /etc/puppetlabs/code/environments/web/manifests
[root@vice-disarray ~]# chown -R puppet:puppet /etc/puppetlabs/code/environments/
[root@vice-disarray ~]# cat <<END > /etc/puppetlabs/code/environments/web/manifests/site.pp
notify \{ 'in web': }
END
[root@vice-disarray ~]# cat /etc/puppetlabs/puppet/enc.sh
#!/bin/sh
 
cat <<END
---
classes:
environment: web
END
[root@vice-disarray ~]# puppet agent -t
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Notice: Local environment: 'production' doesn't match server specified environment 'web', restarting agent run with environment 'web'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Caching catalog for vice-disarray.delivery.puppetlabs.net
Info: Applying configuration version '1627922221'
Notice: in web
Notice: /Stage[main]/Main/Notify[in web]/message: defined 'message' as 'in web'
Notice: Applied catalog in 0.01 seconds

In addition, due to this change, the agent can request an environment that doesn't exist, and it will be reassigned to the server-specified environment:

[root@vice-disarray ~]# puppet agent -t -E doesntexist
Notice: Environment 'doesntexist' not found on server, skipping initial pluginsync.
Notice: Local environment: 'doesntexist' doesn't match server specified environment 'web', restarting agent run with environment 'web'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Caching catalog for vice-disarray.delivery.puppetlabs.net
Info: Applying configuration version '1627922235'
Notice: in web
Notice: /Stage[main]/Main/Notify[in web]/message: defined 'message' as 'in web'
Notice: Applied catalog in 0.01 seconds

Josh Cooper (Jira)

unread,
Aug 2, 2021, 12:45:04 PM8/2/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
 
Change By: Josh Cooper
Release Notes Summary: When an agent runs with Fixes a bug that caused the agent run to fail if the agent requested an environment that does not didn't exist on the server,
puppet will fail.
Now in case it cannot get
even when the facts or catalog for classifier was controlling the specified
environment , it will fallback to the environment configured in the
new default_agent_environment puppet setting
.

Josh Cooper (Jira)

unread,
Aug 4, 2021, 11:06:03 AM8/4/21
to puppe...@googlegroups.com

Claire Cadman (Jira)

unread,
Aug 11, 2021, 9:08:02 AM8/11/21
to puppe...@googlegroups.com

Nate McCurdy (Jira)

unread,
Jan 28, 2022, 3:42:01 PM1/28/22
to puppe...@googlegroups.com
Nate McCurdy commented on Bug PUP-6802
 
Re: Agent cannot compile catalog if it specifies an non-existent environment in puppet.conf even when the classifier is controlling environment

This looks like it introduced a regression in behavior related to PUP-1763 and PUP-10582 for non-existent environments.

In cases where and ENC is not providing an environment, the "ignore_plugin_failures: false" setting should be causing a failed/skipped pluginsync to immediately fail the Puppet run.

But in Puppet 7.14.0, the pluginsync failure is ignored and the agent continues to request a catalog from the puppetserver, which results in a 500 error due to the environment not existing.

For example, on Puppet 6.19.1, using "ignore_plugin_failures: false" correctly stops the run:

[agent]> puppet --version
6.19.1
 
[agent]>  sudo puppet agent -t --environment fake --no-ignore_plugin_errors
Info: Retrieving pluginfacts
Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Could not evaluate: Could not retrieve information from environment fake source(s) puppet:///pluginfacts
Error: Failed to apply catalog: Failed to retrieve pluginfacts: Could not retrieve information from environment fake source(s) puppet:///pluginfacts

But with Puppet 7.14.0, the run does not stop and a catalog is requested. Ultimately, in my case the run fails because the code-id script can't read from a non-existent environment:

[agent]> puppet --version
7.14.0
 
[agent]>  sudo puppet agent -t --environment fake --no-ignore_plugin_errors
Info: Using environment 'fake'
Notice: Environment 'fake' not found on server, skipping initial pluginsync.
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Internal Server Error: java.lang.IllegalStateException: Non-zero exit code returned while running '/etc/puppetlabs/puppet/code-id.sh'. exit-code: '1', stdout: '', stderr: '/etc/puppetlabs/puppet/code-id.sh: line 19: cd: /etc/puppetlabs/code/environments/fake: No such file or directory
'
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)
Atlassian logo

Josh Cooper (Jira)

unread,
Feb 10, 2022, 2:59:02 AM2/10/22
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-6802

The "ignore_plugin_errors" really means "if we perform a pluginsync and there are errors, then ignore them" This ticket was "if the environment we want to use doesn't exist, then skip pluginsync and give the server a chance to redirect us to the correct environment" The setting you really want is "strict_environment_mode". I'll add more details in the other ticket about why that's not working as expected.

Reply all
Reply to author
Forward
0 new messages