Jira (PUP-10540) Performance of puppet grows worse as the size of /proc/mounts increases

9 views
Skip to first unread message

Michele Baldessari (Jira)

unread,
Jun 8, 2020, 12:41:04 PM6/8/20
to puppe...@googlegroups.com
Michele Baldessari created an issue
 
Puppet / Bug PUP-10540
Performance of puppet grows worse as the size of /proc/mounts increases
Issue Type: Bug Bug
Assignee: Unassigned
Created: 2020/06/08 9:40 AM
Priority: Normal Normal
Reporter: Michele Baldessari

Puppet Version: puppet-5.5.10-5.el8ost
Puppet Server Version: N/A
OS Name/Version: Centos 8

A system that has many containers (~1000) will make /proc/mounts rather large (700-900 kbytes). Running any trivial puppet command will take a very long time there due to the following line:
https://github.com/puppetlabs/puppet/blob/master/lib/puppet/util/selinux.rb#L210

In there we're reading /proc/mounts 1024 bytes at the time, which is clearly suboptimal in this case (but also in most default cases as /proc/mounts is rarely < 1024 bytes these days on most modern linux systems).

On such an environment we will observe:
[root@database-0 ~]# time puppet apply --noop --detailed-exitcodes --color=false -e 'file

{"/tmp/foobar": content => "baz" }

'
Notice: Compiled catalog for database-0.redhat.local in environment production in 0.16 seconds
Notice: /Stage[main]/Main/File[/tmp/foobar]/ensure: current_value absent, should be file (noop)
Notice: Class[Main]: Would have triggered 'refresh' from 1 events
Notice: Stage[main]: Would have triggered 'refresh' from 1 events
Notice: Applied catalog in 0.05 seconds

real 5m43.258s
user 5m7.929s
sys 0m5.603s

Increasing the read size to 1024 ** 2 provides immediate benefits:
[root@database-0 ~]# time puppet apply --noop --detailed-exitcodes --color=false -e 'file

{"/tmp/foobar": content => "baz" }

'
Notice: Compiled catalog for database-0.redhat.local in environment production in 0.11 seconds
Notice: /Stage[main]/Main/File[/tmp/foobar]/ensure: current_value absent, should be file (noop)
Notice: Class[Main]: Would have triggered 'refresh' from 1 events
Notice: Stage[main]: Would have triggered 'refresh' from 1 events
Notice: Applied catalog in 0.13 seconds

real 1m22.954s
user 0m50.423s
sys 0m5.115s

In such an environment strace confirms that the number of small read() calls goes from:
[root@database-0 ~]# grep -ir -e 'read(7.*1024' /tmp/strace-puppet.txt |wc -l
231381

To:
[root@database-0 ~]# grep -ir -e 'read(7.*1024' /tmp/strace-puppet-after.txt |wc -l
810

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Michele Baldessari (Jira)

unread,
Jun 8, 2020, 1:02:04 PM6/8/20
to puppe...@googlegroups.com
Michele Baldessari updated an issue
Change By: Michele Baldessari
*Puppet Version: puppet-5.5.10-5.el8ost*
*Puppet Server Version: N/A*
*OS Name/Version: Centos 8*
Reply all
Reply to author
Forward
0 new messages