| During environment convergence, puppet loads types and providers for pluginsynced modules, potentially multiple times. Type/provider loading is triggered when writing the cached catalog and/or when converting the resource catalog to a RAL catalog. However, if the environment doesn't converge the first time, then puppet will switch environments, and pluginsync in a different environment, potentially receiving different versions of type/provider code. Puppet can reload new versions of types and providers, but can't reload helper code required by type/providers. This can result in newer providers calling methods in older helper code still in memory. This problem may occur when the classifier uses pluginsynced facts to determine environment, a node doesn't have that fact yet (or a wrong version of the fact), puppetdb doesn't have a facts for that node in the "final" environment, the node attempts to get a catalog, and there are different versions of a module in different environments. The problem may not occur, or may not be noticed if the different versions of modules don't conflict in a way that causes an exception. The problem won't occur if the agent resolves the environment during the node request (before pluginsync). However, the agent doesn't send facts with the node request (a long standing request), so this generally applies to nodes that have already resolved the environment once. Here's a stacktrace showing the inifile type/provider being loaded during the environment resolution process (using 5.5.17):
# puppet --version |
5.5.17 |
# rm -rf /opt/puppetlabs/puppet/cache/lib/ && puppet config set certname "agent55x$RANDOM" --section main && puppet agent -t --trace --logdest /tmp/out.log |
|
--> #0 #<Class:Puppet::Util::Autoload>.mark_loaded(name#String, file#String) at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:43 |
#1 #<Class:Puppet::Util::Autoload>.load_file(name#String, env#Puppet::Node::Environment::Remote) at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:70 |
#2 Puppet::Util::Autoload.load(name#Symbol, env#Puppet::Node::Environment::Remote) at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:198 |
#3 Puppet::MetaType::Manager.type(name#Symbol) at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/metatype/manager.rb:174 |
#4 #<Class:Puppet::Resource>.resource_type(type#String, title#String, environment#Puppet::Node::Environment::Remote) at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource.rb:360 |
#5 Puppet::Resource.resource_type at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource.rb:348 |
#6 Puppet::Resource.parse_title at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource.rb:656 |
#7 Puppet::Resource.to_hash at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource.rb:392 |
#8 Puppet::Resource.to_data_hash at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource.rb:93 |
#9 block in Puppet::Resource::Catalog.block in to_data_hash at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:506 |
ͱ-- #10 Array.map at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:506 |
#11 Puppet::Resource::Catalog.to_data_hash at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:506 |
#12 Puppet::Network::FormatSupport.to_json(*args#Array) at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/format_support.rb:100 |
#13 Puppet::Network::Format.render(instance#Puppet::Resource::Catalog) at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/format.rb:65 |
#14 Puppet::Network::FormatSupport.render(format#String) at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/format_support.rb:106 |
#15 Puppet::Resource::Catalog::Json.to_json(object#Puppet::Resource::Catalog) at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/json.rb:19 |
#16 block in Puppet::Indirector::JSON.block in save(request#Puppet::Indirector::Request) at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/json.rb:17 |
#17 #<Class:Puppet::Util>.replace_file(file#Pathname, default_mode#Integer, &block#Proc) at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:603 |
#18 Puppet::Indirector::JSON.save(request#Puppet::Indirector::Request) at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/json.rb:17 |
#19 Puppet::Indirector::Indirection.find(key#String, options#Hash) at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:204 |
#20 block in Puppet configuration client.block in retrieve_new_catalog(query_options#Hash) at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:455
|
To fix this problem the conversion of resource catalog to RAL catalog and writing the cached catalog need to occur after the environment converges. |
