Trouble after Agent upgrade -- Error: Could not autoload puppet/provider/package/gem: superclass must be a Class (Module given)

[Patrick Whitney]

Hello Everyone,

I hope you can help.

Our puppet agent was updated to version 6.5.0 on our servers last night.  Things are working fine, except for two servers are throwing the error below (SNIP #1).    A little digging found I could reproduce the error by running 'facter -p' or 'puppet facts'  (SNIP #2).  Trying to identify exactly what is going on, I got as far as running the offending ruby script (package_provider.rb) and it also returns the error, but, at least with a stack trace (SNIP #3).  

Does anyone have any idea what we've encountered?  

Any help would be greatly appreciated. 

Thanks!

-Pat

SNIP #1

# puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Error: Could not autoload puppet/provider/package/gem: superclass must be a Class (Module given)
Error: Facter: error while resolving custom facts in /opt/puppetlabs/puppet/cache/lib/facter/package_provider.rb: Could not autoload puppet/provider/package/gem: superclass must be a Class (Module given)
Info: Caching catalog for eu-assist.aws-eu.int.luminoso.com
Info: Applying configuration version '1561063233'
Notice: Applied catalog in 15.24 seconds

SNIP #2

# puppet facts > /dev/null
Error: Could not autoload puppet/provider/package/gem: superclass must be a Class (Module given)
Error: Facter: error while resolving custom facts in /opt/puppetlabs/puppet/cache/lib/facter/package_provider.rb: Could not autoload puppet/provider/package/gem: superclass must be a Class (Module given)
# facter -p > /dev/null
2019-06-20 20:42:36.539351 ERROR puppetlabs.facter - error while resolving custom facts in /opt/puppetlabs/puppet/cache/lib/facter/package_provider.rb: Could not autoload puppet/provider/package/gem: superclass must be a Class (Module given)

SNIP #3

# /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/cache/lib/facter/package_provider.rb
Traceback (most recent call last):
 15: from /opt/puppetlabs/puppet/cache/lib/facter/package_provider.rb:11:in `<main>'
 14: from /opt/puppetlabs/puppet/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
 13: from /opt/puppetlabs/puppet/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
 12: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/package.rb:9:in `<top (required)>'
 11: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/package.rb:10:in `<module:Puppet>'
 10: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/metatype/manager.rb:127:in `newtype'
 9: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:215:in `loadall'
 8: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:79:in `loadall'
 7: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:79:in `each'
 6: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:81:in `block in loadall'
 5: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:66:in `load_file'
 4: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:66:in `load'
 3: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider/package/gem.rb:1:in `<top (required)>'
 2: from /opt/puppetlabs/puppet/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
 1: from /opt/puppetlabs/puppet/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider/package_targetable.rb:23:in `<top (required)>': Could not autoload puppet/provider/package/gem: superclass must be a Class (Module given) (Puppet::Error)

[Charlie Sharpsteen]

Could you post the content of package_provider.rb? Since that is in cache/lib/facter, it is a custom fact and the specifics of its implementation could shed some light on what is happening.

-Charlie

[Patrick Whitney]

Hi Charlie,

Thanks for the help!

# Fact: package_provider
#
# Purpose: Returns the default provider Puppet will choose to manage packages
#   on this system
#
# Resolution: Instantiates a dummy package resource and return the provider
#
# Caveats:
#
require 'puppet/type'
require 'puppet/type/package'


Facter.add(:package_provider) do
  setcode do
    if defined? Gem && Gem::Version.new(Facter.value(:puppetversion).split(' ')[0]) >= Gem::Version.new('3.6')
      Puppet::Type.type(:package).newpackage(:name => 'dummy', :allow_virtual => 'true')[:provider].to_s
    else
      Puppet::Type.type(:package).newpackage(:name => 'dummy')[:provider].to_s
    end
  end
end

[Jeff Cheng]

We're seeing similar issues with the official v6.5.0 docker image when running in circleci:

circleci@default-d4c9fa03-e104-4e3d-bb5e-3510a1a9eed8:~/project$ docker run --rm puppet/puppet-agent-ubuntu:6.5.0 resource --types

Error: Could not autoload puppet/provider/package/gem: superclass must be a Class (Module given)

Error: Could not autoload puppet/type/package: Could not autoload puppet/provider/package/gem: superclass must be a Class (Module given)
Error: Could not parse application options: Could not autoload puppet/type/package: Could not autoload puppet/provider/package/gem: superclass must be a Class (Module given)

Running with v6.4.2 works fine on the same box:

circleci@default-d4c9fa03-e104-4e3d-bb5e-3510a1a9eed8:~/project$ docker run --rm puppet/puppet-agent-ubuntu:6.4.2 resource --types
augeas
cron
exec
file
filebucket
group
host
mount
notify
package
resources
schedule
scheduled_task
selboolean
selmodule
service
ssh_authorized_key
sshkey
stage
tidy
user
whit
yumrepo
zfs
zone
zpool

[Charlie Sharpsteen]

A quick update on this one:

With Patrick's help, we were able to re-produce the issue in AWS. The root cause appears to be that the Puppet::Provider class includes the Puppet::Util module, which also happens to have a Puppet::Util::Package sub-module. So, depending on the load order of puppet/util/package and puppet/provider/package files, providers using the new PackageTargatable class could trip this error as there is a window where Puppet::Provider::Package may actually be the Puppet::Util::Package module.

TL/DR: Including a module in Ruby brings in more than just methods.

A JIRA bug is open and a PR is up with changes to the package providers to ensure the right files are required before class inheritance happens:

https://tickets.puppetlabs.com/browse/PUP-9794

Thanks a bunch to everyone who reported this issue and especially to Patrick Whitney for working with us to troubleshoot it.

-Charlie