Jira (PUP-10582) Puppet runs should fail fast when no environment can be found

0 views
Skip to first unread message

Nate McCurdy (Jira)

unread,
Jul 9, 2020, 10:28:04 PM7/9/20
to puppe...@googlegroups.com
Nate McCurdy created an issue
 
Puppet / Bug PUP-10582
Puppet runs should fail fast when no environment can be found
Issue Type: Bug Bug
Affects Versions: PUP 6.15.0, PUP 5.5.20
Assignee: Unassigned
Created: 2020/07/09 7:27 PM
Priority: Normal Normal
Reporter: Nate McCurdy

Puppet Version: 6.15.0
Puppet Server Version: 6.12.0 (open source)
OS Name/Version: CentOS 7

When running puppet agent -t against an environment that doesn't exist, the Puppet run doesn't fail quickly and instead makes the following requests in order, all of which respond with a 404 (except for the catalog request, which gets a 500 response):

  • GET /puppet/v3/node
  • GET /puppet/v3/file_metadatas/pluginfacts
  • GET /puppet/v3/file_metadata/pluginfacts
  • GET /puppet/v3/file_metadatas/plugins
  • GET /puppet/v3/file_metadata/plugins
  • GET /puppet/v3/file_metadatas/locales
  • GET /puppet/v3/file_metadata/locales
  • POST /puppet/v3/catalog/

From the agent's view, that looks like this:

$ sudo puppet agent -t --environment not_a_real_env
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Find /puppet/v3/node/agent.corp.net resulted in 404 with the message: {"message":"Not Found: Could not find environment 'not_a_real_env'","issue_kind":"RUNTIME_ERROR"}
Info: Retrieving pluginfacts
Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Could not evaluate: Could not retrieve information from environment not_a_real_env source(s) puppet:///pluginfacts
Info: Retrieving plugin
Warning: /File[/opt/puppetlabs/puppet/cache/lib/facter]: Skipping because of failed dependencies
Warning: /File[/opt/puppetlabs/puppet/cache/lib/facter/facter_dot_d.rb]: Skipping because of failed dependencies
Warning: /File[/opt/puppetlabs/puppet/cache/lib/facter/package_provider.rb]: Skipping because of failed dependencies
Warning: /File[/opt/puppetlabs/puppet/cache/lib/facter/pe_version.rb]: Skipping because of failed dependencies
Warning: /File[/opt/puppetlabs/puppet/cache/lib/facter/puppet_settings.rb]: Skipping because of failed dependencies
 
...(snip 100's of lines of failed pluginsync files)...
 
Info: Retrieving locales
Error: /File[/opt/puppetlabs/puppet/cache/locales]: Could not evaluate: Could not retrieve information from environment not_a_real_env source(s) puppet:///locales
Notice: /File[/opt/puppetlabs/puppet/cache/locales/ja]: Dependency File[/opt/puppetlabs/puppet/cache/locales] has failures: true
Warning: /File[/opt/puppetlabs/puppet/cache/locales/ja]: Skipping because of failed dependencies
Warning: /File[/opt/puppetlabs/puppet/cache/locales/ja/puppetlabs-stdlib.po]: Skipping because of failed dependencies
Warning: /File[/opt/puppetlabs/puppet/cache/locales/ja/puppetlabs-tomcat.po]: Skipping because of failed dependencies
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Internal Server Error: java.lang.IllegalStateException: Non-zero exit code returned while running '/etc/puppetlabs/puppet/code-id.sh'. exit-code: '1', stdout: '', stderr: '/etc/puppetlabs/puppet/code-id.sh: line 19: cd: /etc/puppetlabs/code/environments/not_a_real_env: No such file or directory
'
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

The problem here is that there is very clearly a "404 environment not found" error on the very first line yet the agent continues to request a catalog and the master tries to serve one. Even going so far as to run the static catalog code-id.sh scripts which also error out because there is no environment.

This spews a ton of errors and warning at the user and doesn't make it very clear that the problem is simply that the requested environment doesn't exist. This also causes more load than it should when the entire transaction could probably be killed early once the request for the node object returns a 404.

Desired Behavior: When an agent requests an environment that doesn't exist, the entire transaction stops after the initial request of GET /puppet/v3/node, pluginsync doesn't get attempted, a catalog doesn't get requested, and the static catalog code-id scripts don't get run.

Actual Behavior: All of the above happens even though the requested environment doesn't exist, and the user that ran Puppet has to wade through dozens of lines of errors messages to figure out what went wrong.

Additional Thoughts: Is there maybe a setting to "fail fast" that I'm just not aware of? Like a "stop if environment not found" type of setting?

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Nate McCurdy (Jira)

unread,
Jul 9, 2020, 10:33:03 PM7/9/20
to puppe...@googlegroups.com
Nate McCurdy updated an issue
Change By: Nate McCurdy
*Puppet Version: 6.15.0*
*Puppet Server Version: 6.12.0 (open source)*
*OS Name/Version: CentOS 7*


When running {{puppet agent -t}} against an environment that doesn't exist, the Puppet run doesn't fail quickly and instead makes the following requests in order, all of which respond with a {{404}} (except for the catalog request, which gets a {{500}} response):
* {{GET /puppet/v3/node}}
* {{GET /puppet/v3/file_metadatas/pluginfacts}}
* {{GET /puppet/v3/file_metadata/pluginfacts}}
* {{GET /puppet/v3/file_metadatas/plugins}}
* {{GET /puppet/v3/file_metadata/plugins}}
* {{GET /puppet/v3/file_metadatas/locales}}
* {{GET /puppet/v3/file_metadata/locales}}
* {{POST /puppet/v3/catalog/}}


From the agent's view, that looks like this:
{noformat}

$ sudo puppet agent -t --environment not_a_real_env
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Find /puppet/v3/node/agent.corp.net resulted in 404 with the message: {"message":"Not Found: Could not find environment 'not_a_real_env'","issue_kind":"RUNTIME_ERROR"}
Info: Retrieving pluginfacts
Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Could not evaluate: Could not retrieve information from environment not_a_real_env source(s) puppet:///pluginfacts
Info: Retrieving plugin
Warning: /File[/opt/puppetlabs/puppet/cache/lib/facter]: Skipping because of failed dependencies
Warning: /File[/opt/puppetlabs/puppet/cache/lib/facter/facter_dot_d.rb]: Skipping because of failed dependencies
Warning: /File[/opt/puppetlabs/puppet/cache/lib/facter/package_provider.rb]: Skipping because of failed dependencies
Warning: /File[/opt/puppetlabs/puppet/cache/lib/facter/pe_version.rb]: Skipping because of failed dependencies
Warning: /File[/opt/puppetlabs/puppet/cache/lib/facter/puppet_settings.rb]: Skipping because of failed dependencies

...(snip 100's of lines of failed pluginsync files)...

Info: Retrieving locales
Error: /File[/opt/puppetlabs/puppet/cache/locales]: Could not evaluate: Could not retrieve information from environment not_a_real_env source(s) puppet:///locales
Notice: /File[/opt/puppetlabs/puppet/cache/locales/ja]: Dependency File[/opt/puppetlabs/puppet/cache/locales] has failures: true
Warning: /File[/opt/puppetlabs/puppet/cache/locales/ja]: Skipping because of failed dependencies
Warning: /File[/opt/puppetlabs/puppet/cache/locales/ja/puppetlabs-stdlib.po]: Skipping because of failed dependencies
Warning: /File[/opt/puppetlabs/puppet/cache/locales/ja/puppetlabs-tomcat.po]: Skipping because of failed dependencies
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Internal Server Error: java.lang.IllegalStateException: Non-zero exit code returned while running '/etc/puppetlabs/puppet/code-id.sh'. exit-code: '1', stdout: '', stderr: '/etc/puppetlabs/puppet/code-id.sh: line 19: cd: /etc/puppetlabs/code/environments/not_a_real_env: No such file or directory
'
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
{noformat}


The problem here is that there is
very clearly a "404 environment not found" error on the very first line yet the agent continues to request a catalog and the master tries to serve one. Even going so far as to run the static catalog code-id.sh scripts which also error out because there is no environment.


This spews a ton of errors and warning at the user and doesn't make it
very clear that the problem is simply that the requested environment doesn't exist. This also causes more load than it should when the entire transaction could probably be killed early once the request for the {{node}} object returns a 404.

*Desired Behavior:* When an agent requests an environment that doesn't exist, the entire transaction stops after the initial request of {{GET /puppet/v3/node}}, pluginsync doesn't get attempted, a catalog doesn't get requested, and the static catalog code-id scripts don't get run.

*Actual Behavior:* All of the above happens even though the requested environment doesn't exist, and the user that ran Puppet has to wade through dozens of lines of errors messages to figure out what went wrong.

*Additional Thoughts:* Is there maybe a setting to "fail fast" that I'm just not aware of? Like a "stop if environment not found" type of setting?

Josh Cooper (Jira)

unread,
Jul 10, 2020, 12:39:03 PM7/10/20
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-10582
 
Re: Puppet runs should fail fast when no environment can be found

This is a dup of PUP-1763 due to confusion between the server doesn't have any modules (empty) versus the server modules are missing metadata. Pluginsync should only fail in the latter case.

Josh Cooper (Jira)

unread,
Jul 10, 2020, 2:10:04 PM7/10/20
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-10582

Also wanted to mention you can get the desired behavior now by setting strict_environment_mode=true. If the initial node requests fails, then the agent will fail the run, since the ENC is supposed to be authoritative.

Nate McCurdy (Jira)

unread,
Jul 10, 2020, 2:40:03 PM7/10/20
to puppe...@googlegroups.com
Nate McCurdy commented on Bug PUP-10582

Ah, thanks for the tip on strict_environment_mode Josh Cooper!

Nate McCurdy (Jira)

unread,
Jul 10, 2020, 2:53:03 PM7/10/20
to puppe...@googlegroups.com
Nate McCurdy commented on Bug PUP-10582

Josh Cooper It appears that adding `--strict_environment_mode` does not cause the run to fail early.

It does hide this error about the node object not being found:

Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Find /puppet/v3/node/agent.corp.net resulted in 404 with the message: {"message":"Not Found: Could not find environment 'not_a_real_env'","issue_kind":"RUNTIME_ERROR"}

But the rest of the run continues with pluginsync and a catalog request similar the my original example in the ticket description.

Josh Cooper (Jira)

unread,
Jul 10, 2020, 5:38:04 PM7/10/20
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-10582

Doh, sorry my bad. I was running on in a branch with a fix for 1763.

Nate McCurdy (Jira)

unread,
Jan 31, 2022, 5:27:03 PM1/31/22
to puppe...@googlegroups.com
Nate McCurdy commented on Bug PUP-10582

Opened PUP-11440 to look at a regression in Puppet 7

This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages