Jira (PUP-11328) Puppet agent may download all plugins after updating

52 views
Skip to first unread message

Josh Cooper (Jira)

unread,
Oct 25, 2021, 6:36:01 PM10/25/21
to puppe...@googlegroups.com
Josh Cooper created an issue
 
Puppet / Bug PUP-11328
Puppet agent may download all plugins after updating
Issue Type: Bug Bug
Assignee: Unassigned
Created: 2021/10/25 3:35 PM
Priority: Normal Normal
Reporter: Josh Cooper

When using a server-specified environment, the agent will re-download all of its plugins after updating to 6.25.0 or 7.12.0. This occurs, because the agent cannot determine what the last server-specified environment was. So it will:

  • fallback to "production"
  • download plugins from "production"
  • request a catalog from "production"
  • be redirected to the original environment
  • download plugins from the original environment
  • request a catalog from the original environment

If the puppet-agent package on the server is not updated to 6.25/7.12 or later, and the server-specified environment references a fact that doesn't exist in the "production" environment, then the "hop back" to the original environment will fail as described in PUP-9570.

To prevent the agent from redownloading all plugins, the agent should fallback to making a node request if its last run summary does not contain the last server-specified environment.

This was originally reported in PUP-11323 and PUP-11327.
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Atlassian logo

Josh Cooper (Jira)

unread,
Oct 25, 2021, 6:36:02 PM10/25/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Team: Night's Watch

Josh Cooper (Jira)

unread,
Oct 25, 2021, 6:36:02 PM10/25/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Fix Version/s: PUP 6.25.1
Fix Version/s: PUP 7.12.1

Josh Cooper (Jira)

unread,
Oct 25, 2021, 6:36:02 PM10/25/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Affects Version/s: PUP 6.25.0
Affects Version/s: PUP 7.12.0

Josh Cooper (Jira)

unread,
Oct 25, 2021, 6:43:02 PM10/25/21
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Oct 26, 2021, 9:05:02 PM10/26/21
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-11328

Josh Cooper (Jira)

unread,
Oct 26, 2021, 9:08:01 PM10/26/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Release Notes: Bug Fix
Release Notes Summary: 6.25.0 and 7.12.0 introduced a regression which caused a newly upgraded agent to download all of its plugins. Now the agent will perform a onetime node request to resynchronize its environment with the server.

Josh Cooper (Jira)

unread,
Oct 26, 2021, 9:17:02 PM10/26/21
to puppe...@googlegroups.com
Josh Cooper assigned an issue to Gabriel Nagy
Change By: Josh Cooper
Assignee: Gabriel Nagy

Claire Cadman (Jira)

unread,
Oct 27, 2021, 9:42:02 AM10/27/21
to puppe...@googlegroups.com
Claire Cadman updated an issue
Change By: Claire Cadman
Labels: doc-reviewed

Breno Fernandes (Jira)

unread,
Dec 1, 2021, 2:18:02 PM12/1/21
to puppe...@googlegroups.com
Breno Fernandes commented on Bug PUP-11328
 
Re: Puppet agent may download all plugins after updating

Josh Cooper would you mind clarifying an issue I'm facing?

I use node_terminus to specify dynamic environments (on the agent).

After this ticket, now if last_run_summary.yaml has a converged_environment specified, puppet agent will not use the environment that node_terminus is telling the agent to use. It will use instead what's in last_run_summary.yaml which is very bad.

If before running puppet I remove the last_run_summary.yaml file I have my expected behavior.

If I remove the converged_environment line, puppet will try to apply the "production" (default) environment.

 Is that expected? And if so, why is that?

 

The expected behavior IMO should be: if we have node_terminus configured, that will always have precedence over what is last_run_summary.yaml.

 

Josh Cooper (Jira)

unread,
Dec 1, 2021, 2:49:02 PM12/1/21
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-11328

The agent will start off in the converged environment in the last run summary, but the node terminus will always have the final say about what environment the agent should switch to. You should see the agent report that its environment doesn't match the server specified environment and it should switch to the server specified one.

Breno Fernandes (Jira)

unread,
Dec 1, 2021, 3:26:01 PM12/1/21
to puppe...@googlegroups.com

So that's what I am not having here.

Can you confirm what you're saying is actually what is happening?

In my tests with puppetserver 6.10 + puppet agent 6.25.1, and a client with puppet agent 6.25.1. Puppet agent will not use the env specified by node_terminus if the file $statedir/last_run_summary.yaml exists.

If we just remove the converged environment line, puppet will try to apply the default environment.

 

That's the behavior I have here with multiple different boxes.

Breno Fernandes (Jira)

unread,
Dec 1, 2021, 3:31:02 PM12/1/21
to puppe...@googlegroups.com

Btw, if you are going to test it, switch the environment via node_terminus more than once. Ensure it works every single time.

Because it works on the first time, but if you switch to another environment, it doesn't work anymore.

Ciprian Badescu (Jira)

unread,
Dec 2, 2021, 8:12:01 AM12/2/21
to puppe...@googlegroups.com
Ciprian Badescu updated an issue
 
Change By: Ciprian Badescu
Labels: doc-reviewed needs_repro

Ciprian Badescu (Jira)

unread,
Dec 6, 2021, 3:35:01 AM12/6/21
to puppe...@googlegroups.com
Ciprian Badescu updated an issue
Change By: Ciprian Badescu
Sprint: NW - 2021-12-17

Ciprian Badescu (Jira)

unread,
Dec 6, 2021, 4:03:02 AM12/6/21
to puppe...@googlegroups.com

Gabriel Nagy (Jira)

unread,
Dec 7, 2021, 7:34:02 AM12/7/21
to puppe...@googlegroups.com
Gabriel Nagy commented on Bug PUP-11328
 
Re: Puppet agent may download all plugins after updating

Hi brandfbb,

I wasn't able to reproduce your issue. Indeed, the logs are a bit misleading since Puppet always prints the environment it starts in (not necessarily the converged one).

I set up a fresh environment where I first configured the ENC to return env1, then env2, then env3. The behavior looks correct to me, I'm sharing my logs below:

 

[root@jutish-crushing ~]# grep environment /etc/puppetlabs/puppet/enc.sh
 
environment: env1
 
 
 
[root@jutish-crushing ~]# puppet agent -t 
Notice: Local environment: 'production' doesn't match server specified node environment 'env1', switching agent to 'env1'.
 
Info: Using environment 'env1'
 
Info: Retrieving pluginfacts
 
Info: Retrieving plugin
 
Info: Caching catalog for jutish-crushing.delivery.puppetlabs.net
 
Info: Applying configuration version '1638880188'
 
Info: Creating state file /opt/puppetlabs/puppet/cache/state/state.yaml
 
Notice: Applied catalog in 0.01 seconds
 
 
 
[root@jutish-crushing ~]# sed -i 's/env1/env2/' /etc/puppetlabs/puppet/enc.sh
 
 
 
[root@jutish-crushing ~]# puppet agent -t 
Info: Using environment 'env1'
 
Info: Retrieving pluginfacts
 
Info: Retrieving plugin
 
Notice: Local environment: 'env1' doesn't match server specified environment 'env2', restarting agent run with environment 'env2'
 
Info: Retrieving pluginfacts
 
Info: Retrieving plugin
 
Info: Caching catalog for jutish-crushing.delivery.puppetlabs.net
 
Info: Applying configuration version '1638880206'
 
Notice: Applied catalog in 0.01 seconds
 
 
 
[root@jutish-crushing ~]# puppet agent -t 
Info: Using environment 'env2'
 
Info: Retrieving pluginfacts
 
Info: Retrieving plugin
 
Info: Caching catalog for jutish-crushing.delivery.puppetlabs.net
 
Info: Applying configuration version '1638880213'
 
Notice: Applied catalog in 0.02 seconds
 
 
 
[root@jutish-crushing ~]# sed -i 's/env2/env3/' /etc/puppetlabs/puppet/enc.sh
 
 
 
[root@jutish-crushing ~]# puppet agent -t 
Info: Using environment 'env2'
 
Info: Retrieving pluginfacts
 
Info: Retrieving plugin
 
Notice: Local environment: 'env2' doesn't match server specified environment 'env3', restarting agent run with environment 'env3'
 
Info: Retrieving pluginfacts
 
Info: Retrieving plugin
 
Info: Caching catalog for jutish-crushing.delivery.puppetlabs.net
 
Info: Applying configuration version '1638880224'
 
Notice: Applied catalog in 0.01 seconds

 
This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)
Atlassian logo

Breno Fernandes (Jira)

unread,
Dec 7, 2021, 8:46:01 AM12/7/21
to puppe...@googlegroups.com

Do you mind sharing the output of your enc.sh? I'll do the exact same test on my end.

Also, is this with puppet 7 or puppet 6? What about puppetserver?

It doesn't seem like you have the issue. I am testing with puppet 6 from EPEL 8 [1].

I'm the packager, but from the previous version, not much has changed. I'm sharing the bootstrap file (spec) maybe you can spot an issue there?

 

1 https://src.fedoraproject.org/rpms/puppet/blob/rawhide/f/puppet.spec

Breno Fernandes (Jira)

unread,
Dec 7, 2021, 8:54:03 AM12/7/21
to puppe...@googlegroups.com

Oh please, also share your puppet.conf

Gabriel Nagy (Jira)

unread,
Dec 7, 2021, 8:59:02 AM12/7/21
to puppe...@googlegroups.com
Gabriel Nagy commented on Bug PUP-11328

I'm doing this on a single VM with the latest versions of puppet and puppetserver. I'll also try to reproduce using your specific setup:

 

[root@jutish-crushing ~]# cat /etc/puppetlabs/puppet/enc.sh
 
cat <<ENV
 
---
 
environment: env3
 
ENV
 
 
 
[root@jutish-crushing ~]# puppet --version
 
7.12.1
 
 
 
[root@jutish-crushing ~]# puppetserver --version
 
puppetserver version: 7.4.2
 
 
 
[root@jutish-crushing ~]# cat /etc/puppetlabs/puppet/puppet.conf
 
[main]
 
server = ...
 
 
 
[server]
 
vardir = /opt/puppetlabs/server/data/puppetserver
 
logdir = /var/log/puppetlabs/puppetserver
 
rundir = /var/run/puppetlabs/puppetserver
 
pidfile = /var/run/puppetlabs/puppetserver/puppetserver.pid
 
codedir = /etc/puppetlabs/code
 
external_nodes = /etc/puppetlabs/puppet/enc.sh
 
node_terminus = exec

Breno Fernandes (Jira)

unread,
Dec 7, 2021, 9:29:01 AM12/7/21
to puppe...@googlegroups.com

One relevant change is that I am using puppetserver and puppet agent in two different boxes.

Don't you think that would make a difference?

Specially because the actual environments (the files themselves) won't be present on puppet agent.

Gabriel Nagy (Jira)

unread,
Dec 7, 2021, 9:38:03 AM12/7/21
to puppe...@googlegroups.com
Gabriel Nagy commented on Bug PUP-11328

I don't believe that changes anything; I updated my comment above to mention that I also tested using your setup (with separate server and agent boxes) and I get the same behavior.

Breno Fernandes (Jira)

unread,
Dec 7, 2021, 10:09:03 AM12/7/21
to puppe...@googlegroups.com

Well, I'm clearly seeing a bug on my end, like I shared. The only difference now is that I'm using the package we build in fedora.

What's the version of puppetserver you're running and do you believe that would make a difference if I'm using an older version?

Breno Fernandes (Jira)

unread,
Dec 7, 2021, 10:29:03 AM12/7/21
to puppe...@googlegroups.com

Can you share your puppet.conf when you use the server separete from the agent?

Because the external_nodes and node_terminus would be located in the [main] section, not in the [server].

Did you test it like so?

Gabriel Nagy (Jira)

unread,
Dec 8, 2021, 5:00:02 AM12/8/21
to puppe...@googlegroups.com
Gabriel Nagy commented on Bug PUP-11328

I've spoken with brandfbb over Slack and the issue stems from the fact that the ENC is configured on the agent, not the server. This is a workflow not currently supported/documented by Puppet, as the documentation explicitly states the ENC is called by Puppet Server.

The way this ticket impacts the workflow is the following, assuming Puppet Server has no ENC configured (so it doesn't decide the agent's environment):

  • with last_run_summary.yaml absent, the agent run will make a node request; because the indirection class is exec, the local ENC script will be executed
  • the ENC returns the environment the agent should run in, the agent switches to that environment and continues the run
  • because the server is not authoritative, it will compile a catalog for the environment requested by the agent
  • the agent applies the catalog and writes the initial environment (production or whatever's in puppet.conf) and the converged environment (ENC-specified environment) to last_run_summary.yaml
  • because the initial and converged environment are different, subsequent agent runs will assume the last used environment was authoritative and will skip the node request
  • the agent will no longer honor the environment specified by the ENC or the environment set in puppet.conf, and will always use the converged environment specified in last_run_summary.yaml

A viable workaround for this case could be to specify the environment on the CLI: puppet agent -t --environment=$(/path/to/enc)

Josh Cooper (Jira)

unread,
Dec 10, 2021, 1:26:02 AM12/10/21
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-11328

I'm going to close this issue, because the change was already released.

TBH, I've never heard of an agent executing a local enc script before. Usually you'd want to pass the environment to use for the onetime run as gabriel.nagy mentioned.

Breno Fernandes (Jira)

unread,
Dec 13, 2021, 12:05:01 AM12/13/21
to puppe...@googlegroups.com

Hi josh , like I was telling gabriel.nagy , we've been using puppet since puppet 3.X.

And we've been using this feature like I described to Gabriel forever. I believe that probably at some point, someone changed the docs, but didn't change the behaviour of the feature.

 

I'd like to gently ask you to consider putting back this feature, since it's something that was working before and was removed suddenly, and catch us and probably others out there, off the guard.

I'm the official packager of puppet for Fedora and RedHat (EPEL). I'd like to also volunteer to submit a PR with the documents update.

 

Would you consider putting the feature back? I ask some consideration in the name of the community.

 

 

Breno Fernandes (Jira)

unread,
Dec 13, 2021, 12:21:03 AM12/13/21
to puppe...@googlegroups.com

Oh, and by the way, this change seems to put the behaviour back in place working:

[root@server /usr/share/ruby/vendor_ruby/puppet]# git diff
 
diff --git a/configurer.rb b/configurer.rb
 
index b55ad5d..211b1d3 100644
 
--- a/configurer.rb
 
+++ b/configurer.rb
 
@@ -490,7 +490,7 @@ class Puppet::Configurer
 
       converged_environment = summary['application']['converged_environment']
 
       @last_server_specified_environment = converged_environment if initial_environment != converged_environment
 
       Puppet.debug(_("Successfully loaded last environment from the lastrunfile"))
 
-      @loaded_last_environment = true
 
+      @loaded_last_environment = false
 
     end

Josh Cooper (Jira)

unread,
Dec 14, 2021, 1:03:01 PM12/14/21
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-11328

brandfbb thanks for bringing this to our attention. I filed PUP-11379 for the regression. Please watch/comment on that ticket instead of this one.

Breno Fernandes (Jira)

unread,
Jan 5, 2022, 7:47:03 AM1/5/22
to puppe...@googlegroups.com

Thank you, josh and gabriel.nagy for putting this together.

 

 

 
Reply all
Reply to author
Forward
0 new messages