Jira (PUP-10856) Slower puppet agent run after agent upgrade to version 6

33 views
Skip to first unread message

Josh Cooper (Jira)

unread,
Jan 15, 2021, 1:34:03 PM1/15/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
 
Puppet / Bug PUP-10856
Slower puppet agent run after agent upgrade to version 6
Change By: Josh Cooper
Affects Version/s: PUP 6.19.0
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Josh Cooper (Jira)

unread,
Jan 15, 2021, 1:34:03 PM1/15/21
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-10856
 
Re: Slower puppet agent run after agent upgrade to version 6

Thanks Ashok Mahajan. I moved this to the PUP project. We'll take a look.

Josh Cooper (Jira)

unread,
Jan 15, 2021, 1:34:04 PM1/15/21
to puppe...@googlegroups.com
Josh Cooper moved an issue
 
Change By: Josh Cooper
Component/s: puppet-runtime
Key: PA PUP - 3542 10856
Affects Version/s: puppet-agent 6.19.0
Project: Puppet Agent

Josh Cooper (Jira)

unread,
Jan 15, 2021, 2:49:03 PM1/15/21
to puppe...@googlegroups.com

Mihai Buzgau (Jira)

unread,
Jan 19, 2021, 10:52:03 AM1/19/21
to puppe...@googlegroups.com

Mihai Buzgau (Jira)

unread,
Jan 20, 2021, 4:37:04 AM1/20/21
to puppe...@googlegroups.com

Gheorghe Popescu (Jira)

unread,
Jan 20, 2021, 9:21:03 AM1/20/21
to puppe...@googlegroups.com

Gheorghe Popescu (Jira)

unread,
Jan 28, 2021, 4:25:04 AM1/28/21
to puppe...@googlegroups.com
Gheorghe Popescu commented on Bug PUP-10856
 
Re: Slower puppet agent run after agent upgrade to version 6

Ashok Mahajan is this happening for every run or it happened only for the first run after upgrade?
Can you provide a manifest that would help us to reproduce the slowness?

Ashok Mahajan (Jira)

unread,
Jan 28, 2021, 11:49:05 AM1/28/21
to puppe...@googlegroups.com

This is happening for every run and for every manifest consistently.

Here is the puppet run output for the same two hosts for which I  added the run summary two weeks back when I created the ticket. Schedule puppet run is happening on these nodes every 2 hours.

These are same node types and apply exact same manifests (Configuration version fac0d39 in both output)

64.24 seconds seconds (v4) vs 146.40 seconds (v6)

Puppet 4 agent:

 

# apt-cache policy puppet-agent | head -10
puppet-agent:
  Installed: 1.10.14-1stretch
  Candidate: 6.20.0-1buster
 
# puppetenv -p
>> Running puppet agent -t --environment production
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for store-app-mb2p.c.bigcommerce-production.internal
Info: Applying configuration version 'fac0d39 - Wed Jan 27 10:06:52 2021 +1100'
Notice: Applied catalog in 64.24 seconds

 

Puppet 6 agent:

 

# apt-cache policy puppet-agent | head -10
puppet-agent:
  Installed: 6.20.0-1buster
  Candidate: 6.20.0-1buster
 
# puppetenv -p
>> Running puppet agent -t --environment production
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for store-app-cn0v.c.bigcommerce-production.internal
Info: Applying configuration version 'fac0d39 - Wed Jan 27 10:06:52 2021 +1100'
Notice: Applied catalog in 146.40 seconds

 

 

Ashok Mahajan (Jira)

unread,
Jan 28, 2021, 11:56:03 AM1/28/21
to puppe...@googlegroups.com

Now I am adding he puppet run output for two database nodes (so the manifest is the very different than the nodes used previously)

(configuration version fac0d39)

31 sec vs 304 sec.

puppet 4 agent

 

# apt-cache policy puppet-agent
puppet-agent:
  Installed: 1.10.14-1stretch
  Candidate: 6.20.0-1buster
  Version table:
     6.20.0-1buster 700
        500 http://apt.bigcommerce.net/aptly/puppet6 buster/puppet6 amd64 Packages
     6.19.1-1buster 700
        500 http://apt.bigcommerce.net/aptly/puppet6 buster/puppet6 amd64 Packages
 
# puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for <db-node>
Info: Applying configuration version 'fac0d39 - Wed Jan 27 10:06:52 2021 +1100'
Notice: Applied catalog in 31.65 seconds

puppet 6 agent

 

# apt-cache policy puppet-agent | head -5
puppet-agent:
  Installed: 6.19.1-1buster
  Candidate: 6.20.0-1buster
  Version table:
     6.20.0-1buster 700
 
# puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for <db-node>
Info: Applying configuration version 'fac0d39 - Wed Jan 27 10:06:52 2021 +1100'
Notice: Applied catalog in 304.56 seconds

 

 

 

Ashok Mahajan (Jira)

unread,
Jan 28, 2021, 12:02:58 PM1/28/21
to puppe...@googlegroups.com

As per out analysis, package installation phase is causing extra time and most probably due to this specific statement, which is not there in earlier version of the agent.

```

Puppet (debug): Executing: '/usr/bin/apt-mark showmanual'

```

Ashok Mahajan (Jira)

unread,
Jan 28, 2021, 12:06:46 PM1/28/21
to puppe...@googlegroups.com

Puppet is heavily used in organization and for years. We are sure this issue is not due any specific statement(s) in manifest(s) written by us.

Josh Cooper (Jira)

unread,
Jan 28, 2021, 5:50:04 PM1/28/21
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Jan 28, 2021, 5:50:05 PM1/28/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Fix Version/s: PUP 7.4.0

Gheorghe Popescu (Jira)

unread,
Jan 29, 2021, 4:52:04 AM1/29/21
to puppe...@googlegroups.com
Gheorghe Popescu commented on Bug PUP-10856
 
Re: Slower puppet agent run after agent upgrade to version 6

Ashok Mahajan thanks for the info.
I was asking about the manifest in order to have some resources that would help me to reproduce exactly the same issues that you are seeing, in my environment. Definitely the slowness is there and was added in puppet6.
I'm working on a fix to reduce the calls to `'/usr/bin/apt-mark showmanual'`: https://github.com/puppetlabs/puppet/pull/8497

Gheorghe Popescu (Jira)

unread,
Feb 1, 2021, 2:40:03 AM2/1/21
to puppe...@googlegroups.com
Gheorghe Popescu updated an issue
 
Change By: Gheorghe Popescu
Release Notes Summary: Improve the performance of Apt package provider when removing packages by reducing the calls to `apt-mark showmanual`

Gheorghe Popescu (Jira)

unread,
Feb 1, 2021, 9:31:03 AM2/1/21
to puppe...@googlegroups.com

Ashok Mahajan (Jira)

unread,
Feb 1, 2021, 10:31:04 AM2/1/21
to puppe...@googlegroups.com

Claire Cadman (Jira)

unread,
Feb 5, 2021, 3:46:03 AM2/5/21
to puppe...@googlegroups.com

Gheorghe Popescu (Jira)

unread,
Feb 8, 2021, 3:00:04 AM2/8/21
to puppe...@googlegroups.com
Gheorghe Popescu commented on Bug PUP-10856
 
Re: Slower puppet agent run after agent upgrade to version 6

Ashok Mahajan the fix will be available in the 6.21.0 and 7.4.0 versions.

Ashok Mahajan (Jira)

unread,
Feb 8, 2021, 11:35:04 AM2/8/21
to puppe...@googlegroups.com

Gheorghe Popescu :  Any tentative/approximate date for the release of these version ? Thank you.

Gheorghe Popescu (Jira)

unread,
Feb 8, 2021, 11:49:03 AM2/8/21
to puppe...@googlegroups.com

Ashok Mahajan the releases are planned for this week. Most likely the packages will be available on 10th feb.

Reply all
Reply to author
Forward
0 new messages