Jira (PUP-10657) Resource collector overrides cause Puppet Server to retain Compiler instances

29 views
Skip to first unread message

Charlie Sharpsteen (Jira)

unread,
Sep 2, 2020, 7:16:04 PM9/2/20
to puppe...@googlegroups.com
Charlie Sharpsteen created an issue
 
Puppet / Bug PUP-10657
Resource collector overrides cause Puppet Server to retain Compiler instances
Issue Type: Bug Bug
Affects Versions: PUP 5.5.21, PUP 6.18.0
Assignee: Unassigned
Created: 2020/09/02 4:15 PM
Priority: Normal Normal
Reporter: Charlie Sharpsteen

Use of a resource collector with an override block will cause Puppet Server to retain Puppet::Parser::Catlog objects in memory after requests finish.

See SERVER-2874 for more context and PE reproduction case.

Reproduction Case

  • Install Puppet Server on CentOS 7 along with the Java development tools:

yum install -y http://yum.puppetlabs.com/puppet6-release-el-7.noarch.rpm
yum install -y puppetserver java-devel
source /etc/profile.d/puppet-agent.sh

  • Install the `pupeptlabs-inifile` module and use it to enable JRuby class reification. This allows Ruby objects, like Puppet::Parser::Compiler, to be identified easily in the Java heap:

puppet module install puppetlabs-inifile
 
puppet apply <<'EOF'
ini_subsetting { 'Enable JRuby reify.classes':
  ensure            => present,
  path              => '/etc/sysconfig/puppetserver',
  section           => '',
  key_val_separator => '=',
  setting           => 'JAVA_ARGS',
  subsetting        => '-Djruby.reify.classes',
  value             => '=true',
}
EOF

  • Bootstrap the CA, configure the agent, and start the puppetserver service:

puppetserver ca setup
puppet config set server $(hostname -f)
systemctl start puppetserver

  • Give the service a trivial catalog to compile:

node default {
  notify { 'test message':
    message => regsubst('Hello, foo!', 'foo', 'world')
  }
 
  Notify <| title == 'test message' |> {
    message => 'collector override',
  }
}

  • Run puppet a couple of times:

puppet agent -t
puppet agent -t

  • Check the number of Puppet::Parser::Compiler instances retained in Puppet Server memory:

sudo -u puppet jmap -histo:live $(systemctl show -p MainPID puppetserver|cut -d= -f2)|fgrep 'rubyobj.Puppet.Parser.Compiler'

Outcome

After running the agent twice, the puppetserver service is retaining two Compiler instances in memory:

# sudo -u puppet jmap -histo:live $(systemctl show -p MainPID puppetserver|cut -d= -f2)|fgrep 'rubyobj.Puppet.Parser.Compiler'
 
5895:             2             64  rubyobj.Puppet.Parser.Compiler

Expected Outcome

No compiler instances should be retained as the compiler is discarded at the end of the request and the -histo:live flag forces a full garbage collection.

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Charlie Sharpsteen (Jira)

unread,
Sep 2, 2020, 7:31:03 PM9/2/20
to puppe...@googlegroups.com
Charlie Sharpsteen commented on Bug PUP-10657
 
Re: Resource collector overrides cause Puppet Server to retain Compiler instances

This leak is caused by the following lines of code in the Resource Collector implementation:

      overrides[:source].meta_def(:child_of?) do |klass|
        true
      end

https://github.com/puppetlabs/puppet/blob/6.18.0/lib/puppet/pops/evaluator/collectors/abstract_collector.rb#L48-L50

That loops over resources the collector is operating on and modifies their child_of? methods to always return true. This forces the collector overrides to be merged in later by short-circuiting the logic in Puppet::Parser::Resource that only accepts overrides from a parent scope:

However, overriding a method like this results in the creation of a Ruby "eigenclass" to hold the new method definition. The body of the method, which accepts klass and always returns true, is provided by Ruby block. This block is an instance of a Proc, which means that it closes over all variables that happen to be in scope. One of those variables is @scope, which is a reference to the compiler scope.

JRuby doesn't always free eigenclass instances when the objects they were created to modify go out of scope:

https://github.com/jruby/jruby/issues/4968

So, whenever a Resource Collector is used to override parameters, we end up with unused eigenclasses that hold onto references for one or more compilers.

Josh Cooper (Jira)

unread,
Sep 3, 2020, 11:19:03 AM9/3/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
 
Change By: Josh Cooper
Fix Version/s: PUP 6.19.0
Fix Version/s: PUP 5.5.22

Josh Cooper (Jira)

unread,
Sep 3, 2020, 11:19:03 AM9/3/20
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Sep 3, 2020, 11:19:05 AM9/3/20
to puppe...@googlegroups.com

Mihai Buzgau (Jira)

unread,
Sep 3, 2020, 11:25:04 AM9/3/20
to puppe...@googlegroups.com

Charlie Sharpsteen (Jira)

unread,
Sep 3, 2020, 11:37:04 AM9/3/20
to puppe...@googlegroups.com
Charlie Sharpsteen updated an issue
Change By: Charlie Sharpsteen
Use of a resource collector with an override block will cause Puppet Server to retain {{Puppet::Parser::Catlog}} objects in memory after requests finish.


See SERVER-2874 for more context and PE reproduction case.

h2. Reproduction Case

  - Install Puppet Server on CentOS 7 along with the Java development tools:

{code:bash}

yum install -y http://yum.puppetlabs.com/puppet6-release-el-7.noarch.rpm
yum install -y puppetserver java-devel
source /etc/profile.d/puppet-agent.sh
{code}

  - Install the `pupeptlabs-inifile` module and use it to enable JRuby class reification. This allows Ruby objects, like {{Puppet::Parser::Compiler}}, to be identified easily in the Java heap:

{code:bash}

puppet module install puppetlabs-inifile

puppet apply <<'EOF'
ini_subsetting { 'Enable JRuby reify.classes':
  ensure            => present,
  path              => '/etc/sysconfig/puppetserver',
  section           => '',
  key_val_separator => '=',
  setting           => 'JAVA_ARGS',
  subsetting        => '-Djruby.reify.classes',
  value             => '=true',
}
EOF
{code}

  - Bootstrap the CA, configure the agent, and start the {{puppetserver}} service:

{code:bash}

puppetserver ca setup
puppet config set server $(hostname -f)
systemctl start puppetserver
{code}

  - Give the service a trivial catalog to compile:

{code:bash}
cat <<'EOF' > /etc/puppetlabs/code/environments/production/manifests/site.pp
node default {
  notify { 'test message':
    message => regsubst('Hello, foo!', 'foo', 'world')
  }

  Notify <| title == 'test message' |> {
    message => 'collector override',
  }
}
EOF
{code}

  - Run {{puppet}} a couple of times:

{code:bash}

puppet agent -t
puppet agent -t
{code}

  - Check the number of {{Puppet::Parser::Compiler}} instances retained in Puppet Server memory:

{code:bash}

sudo -u puppet jmap -histo:live $(systemctl show -p MainPID puppetserver|cut -d= -f2)|fgrep 'rubyobj.Puppet.Parser.Compiler'
{code}

h3. Outcome


After running the agent twice, the {{puppetserver}} service is retaining two Compiler instances in memory:

{noformat}

# sudo -u puppet jmap -histo:live $(systemctl show -p MainPID puppetserver|cut -d= -f2)|fgrep 'rubyobj.Puppet.Parser.Compiler'

5895:             2             64  rubyobj.Puppet.Parser.Compiler
{noformat}

h3. Expected Outcome


No compiler instances should be retained as the compiler is discarded at the end of the request and the {{-histo:live}} flag forces a full garbage collection.

Charlie Sharpsteen (Jira)

unread,
Sep 3, 2020, 11:40:04 AM9/3/20
to puppe...@googlegroups.com
Charlie Sharpsteen commented on Bug PUP-10657
 
Re: Resource collector overrides cause Puppet Server to retain Compiler instances

Ideally, we wouldn't override the child_of? method to short-circuit the logic. Something like a force: true option for adding overrides to resources would be a much nicer way of doing this.

That said, using instance_eval to re-define the method seems to produce the same effect as meta_def but without leaving a reference to the Proc binding on the eigenclass. So, this may be a quick solution tat requires fewer modifications:

overrides[:source].instance_eval do
  def child_of?(klass)
    true
  end
end

Charlie Sharpsteen (Jira)

unread,
Sep 4, 2020, 2:34:04 PM9/4/20
to puppe...@googlegroups.com
Charlie Sharpsteen updated an issue
Change By: Charlie Sharpsteen
    message => regsubst( 'Hello, foo world !', 'foo', 'world')
  }

  Notify <| title == 'test message' |> {
    message => 'collector override',
  }
}
EOF
{code}

  - Run {{puppet}} a couple of times:

{code:bash}
puppet agent -t
puppet agent -t
{code}

  - Check the number of {{Puppet::Parser::Compiler}} instances retained in Puppet Server memory:

{code:bash}
sudo -u puppet jmap -histo:live $(systemctl show -p MainPID puppetserver|cut -d= -f2)|fgrep 'rubyobj.Puppet.Parser.Compiler'
{code}

h3. Outcome

After running the agent twice, the {{puppetserver}} service is retaining two Compiler instances in memory:

{noformat}
# sudo -u puppet jmap -histo:live $(systemctl show -p MainPID puppetserver|cut -d= -f2)|fgrep 'rubyobj.Puppet.Parser.Compiler'

5895:             2             64  rubyobj.Puppet.Parser.Compiler
{noformat}

h3. Expected Outcome

No compiler instances should be retained as the compiler is discarded at the end of the request and the {{-histo:live}} flag forces a full garbage collection.

Austin Boyd (Jira)

unread,
Sep 8, 2020, 7:02:04 PM9/8/20
to puppe...@googlegroups.com

Austin Boyd (Jira)

unread,
Sep 8, 2020, 7:02:04 PM9/8/20
to puppe...@googlegroups.com
Austin Boyd updated an issue
Change By: Austin Boyd
Zendesk Ticket Count: 1
Zendesk Ticket IDs: 40701

Mihai Buzgau (Jira)

unread,
Sep 9, 2020, 4:23:03 AM9/9/20
to puppe...@googlegroups.com

Mihai Buzgau (Jira)

unread,
Sep 9, 2020, 4:24:04 AM9/9/20
to puppe...@googlegroups.com

Gabriel Nagy (Jira)

unread,
Sep 10, 2020, 2:54:03 AM9/10/20
to puppe...@googlegroups.com

Mihai Buzgau (Jira)

unread,
Sep 16, 2020, 5:57:03 AM9/16/20
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Sep 16, 2020, 5:24:04 PM9/16/20
to puppe...@googlegroups.com

Claire Cadman (Jira)

unread,
Oct 12, 2020, 9:17:03 AM10/12/20
to puppe...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages