Jira (PUP-9602) puppet 6 apply fails if puppet types have been generated

[Vadym Chepkov (JIRA)]

Vadym Chepkov created an issue

Puppet / PUP-9602 puppet 6 apply fails if puppet types have been generated Issue Type: Bug Affects Versions: PUP 6.4.0 Assignee: Unassigned Created: 2019/04/03 5:33 AM Priority: Normal Reporter: Vadym Chepkov Puppet Version: 6.4.0 Puppetserver Version 6.3.0 OS Name/Version: CentOS 7.5 Stumbled upon an unusual behavior in Puppet 6 (works as expected on Puppet 5) If command puppet generate types has been executed, puppet apply fails. Puppet agent works normally. [root@master ~]# puppet apply -e 'include bootstrap::master' Notice: Compiled catalog for master.localdomain in environment production in 0.81 seconds Notice: Applied catalog in 0.89 seconds   [root@master ~]# puppet generate types Notice: Generating Puppet resource types. Warning: The a2mod provider is deprecated, please use apache::mod instead Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/a2mod.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/concat_file.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/concat_fragment.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/git_config.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/ini_setting.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/ini_subsetting.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/postgresql_conf.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/postgresql_conn_validator.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/postgresql_psql.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/postgresql_replication_slot.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/puppetdb_conn_validator.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/anchor.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/file_line.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/vcsrepo.pp' using 'pcore' format.   [root@master ~]# puppet apply -e 'include bootstrap::master' Notice: Compiled catalog for master.localdomain in environment production in 0.75 seconds Error: Could not autoload puppet/type/anchor: Attempt to redefine entity 'http://puppet.com/2016.1/runtime/type/anchor'. Set at unknown location Error: Could not autoload puppet/type/anchor: Attempt to redefine entity 'http://puppet.com/2016.1/runtime/type/anchor'. Set at unknown location Add Comment

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)

[Vadym Chepkov (JIRA)]

Vadym Chepkov updated an issue

Puppet / PUP-9602 puppet 6 apply fails if puppet types have been generated

Change By: Vadym Chepkov Attachment: debug-trace.log

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated Does it work if you after having run generate types manually remove the generated file for the anchor type? Asking this to possibly narrow down what is wrong. Alternatively, remove all other generated files and keep only the one for anchor - does it error then?

Add Comment

[Vadym Chepkov (JIRA)]

Vadym Chepkov commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

Removed anchor.pp, it fails on the next file: [root@master ~]# rm /etc/puppetlabs/code/environments/production/.resource_types/anchor.pp  rm: remove regular file ‘/etc/puppetlabs/code/environments/production/.resource_types/anchor.pp’? y

[root@master ~]# puppet apply -e 'include bootstrap::master'

Notice: Compiled catalog for master.localdomain in environment production in 0.74 seconds Error: Could not autoload puppet/type/postgresql_conn_validator: Attempt to redefine entity 'http://puppet.com/2016.1/runtime/type/postgresql_conn_validator'. Set at unknown location Error: Could not autoload puppet/type/postgresql_conn_validator: Attempt to redefine entity 'http://puppet.com/2016.1/runtime/type/postgresql_conn_validator'. Set at unknown location

Add Comment

[Vadym Chepkov (JIRA)]

Vadym Chepkov commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

Same result with only one file present: [root@master .resource_types]# ls -lA total 4 -rw-r--r--. 1 root root 1746 Apr  3 13:26 anchor.pp [root@master .resource_types]# puppet apply -e 'include bootstrap::master'  Notice: Compiled catalog for master.localdomain in environment production in 0.82 seconds Error: Could not autoload puppet/type/anchor: Attempt to redefine entity 'http://puppet.com/2016.1/runtime/type/anchor'. Set at unknown location Error: Could not autoload puppet/type/anchor: Attempt to redefine entity 'http://puppet.com/2016.1/runtime/type/anchor'. Set at unknown location

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

After bouncing this off some other folks - this is probably related to the work done on the agent to support loading of data types (specifically the `Deferred` data type). What happens is that it probably ended up with a mix of loading both the generated types and the actual resource type implementations. The agent most certainly needs to load the resource type implementations and should ignore the generated types completely. Vadym Chepkov This means that it is broken in the apply case, but should work just fine when used when compiling catalogs with a master.

Add Comment

[Vadym Chepkov (JIRA)]

Vadym Chepkov commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

Henrik Lindberg, yes, I mentioned earlier, puppet agent works properly

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

I'm having difficulty reproducing this: [root@evnap2q7bofh6pf ~]# rpm -q puppet-agent puppet-agent-6.4.0-1.el7.x86_64 root@evnap2q7bofh6pf ~]# puppet module install puppetlabs-stdlib Notice: Preparing to install into /etc/puppetlabs/code/environments/production/modules ... Notice: Downloading from https://forgeapi.puppet.com ... Notice: Installing -- do not interrupt ... /etc/puppetlabs/code/environments/production/modules └── puppetlabs-stdlib (v5.2.0) [root@evnap2q7bofh6pf ~]# puppet generate types

Notice: Generating Puppet resource types.

Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/anchor.pp' using 'pcore' format.

Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/file_line.pp' using 'pcore' format.

[root@evnap2q7bofh6pf ~]# puppet apply -e "anchor { 'here': }" Notice: Compiled catalog for evnap2q7bofh6pf.delivery.puppetlabs.net in environment production in 0.02 seconds Notice: Applied catalog in 0.02 seconds I also tried installing stdlib in a different environment (with and without generated types in that environment) and I get the same as above.

Add Comment

[Vadym Chepkov (JIRA)]

Vadym Chepkov commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

Hmm, so, that means some other module contributes to it? You can check my bootstrap code here - https://github.com/vchepkov/puppet-bootstrap, it's easily reproducible

Add Comment

[Vadym Chepkov (JIRA)]

Vadym Chepkov commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

Interesting, so some code compiles fine. I reduced code fragment to this and it fails:   [root@master ~]# cat psql.pp    class { 'puppetdb::database::postgresql':     manage_package_repo => true,     postgres_version    => '9.6',   }     [root@master ~]# puppet apply psql.pp  Notice: Compiled catalog for master.localdomain in environment production in 0.41 seconds

Error: Could not autoload puppet/type/anchor: Attempt to redefine entity 'http://puppet.com/2016.1/runtime/type/anchor'. Set at unknown location Error: Could not autoload puppet/type/anchor: Attempt to redefine entity 'http://puppet.com/2016.1/runtime/type/anchor'. Set at unknown location

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

When we have had issues with this in the past it has usually been because of logic using old or internal APIs. Could be that one of the modules is doing something it is not supposed to do. The last one that I remember was fixed with a PR from eric sorensson - the PR is here https://github.com/puppetlabs/puppet/pull/7162. (A lot more detective work needed). While a solution is to simply skip the generated types in an apply case, that would also mean it is not possible to debug/try-out compilation with puppet apply running with generated types, which makes testing a lot more difficult.

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

Vadym Chepkov When you reduced the reproducer did you also take out all unused modules on the module path so it contained only exactly what was needed?

Add Comment

[Vadym Chepkov (JIRA)]

Vadym Chepkov commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

Henrik Lindberg, I have experimented with all of the generated puppet types and determined that if any of these are present (even just alone) catalog fails anchor.pp postgresql_conf.pp postgresql_conn_validator.pp postgresql_psql.pp

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

Vadym Chepkov That seems to contradict what Josh Cooper showed in his example using anchor - does the example he showed work for you?

Add Comment

[Vadym Chepkov (JIRA)]

Vadym Chepkov commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

Yes, his example works, it must be combination of other types that causes it

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

Hence my question regarding what you tried and what was on the module path. So, when you say "if any of these are present (even just alone)" does that mean that they were all on the modulepath, but you only declared a resource from one of them? If so, try by adding modules one by one to the module path to try to find the one that causes the problem.

Add Comment

[Vadym Chepkov (JIRA)]

Vadym Chepkov commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

If I am removing modules, I do not have catalog compiled. If you check output I provided earlier, catalog compiles and then it fails during applying it. I have tried to remove files from /etc/puppetlabs/code/environments/production/.resource_types/ one by one to see which one triggers the failure and all 4 listed type files do that.

Add Comment

[Vadym Chepkov (JIRA)]

Vadym Chepkov commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

To summarize my reproducer: [root@localhost ~]# puppet module install puppetlabs-puppetdb

Notice: Preparing to install into /etc/puppetlabs/code/environments/production/modules ... Notice: Downloading from https://forgeapi.puppet.com ... Notice: Installing -- do not interrupt ... /etc/puppetlabs/code/environments/production/modules

└─┬ puppetlabs-puppetdb (v7.1.0)   ├── puppetlabs-firewall (v1.15.2)   ├─┬ puppetlabs-inifile (v2.5.0)   │ └── puppetlabs-translate (v1.2.0)   └─┬ puppetlabs-postgresql (v5.12.1)     ├── puppetlabs-apt (v6.3.0)     ├── puppetlabs-concat (v5.3.0)     └── puppetlabs-stdlib (v5.2.0) [root@localhost ~]# puppet generate types

Notice: Generating Puppet resource types.

Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/apt_key.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/concat_file.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/concat_fragment.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/firewall.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/firewallchain.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/ini_setting.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/ini_subsetting.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/postgresql_conf.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/postgresql_conn_validator.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/postgresql_psql.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/postgresql_replication_slot.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/puppetdb_conn_validator.pp' using 'pcore' format.

Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/anchor.pp' using 'pcore' format. Notice: Generating '/etc/puppetlabs/code/environments/production/.resource_types/file_line.pp' using 'pcore' format.

[root@localhost ~]# puppet apply -e "class { 'puppetdb::database::postgresql': manage_package_repo => true, postgres_version => '9.6'}" Notice: Compiled catalog for localhost.localdomain in environment production in 0.36 seconds

Error: Could not autoload puppet/type/anchor: Attempt to redefine entity 'http://puppet.com/2016.1/runtime/type/anchor'. Set at unknown location Error: Could not autoload puppet/type/anchor: Attempt to redefine entity 'http://puppet.com/2016.1/runtime/type/anchor'. Set at unknown location

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

Thanks for the repro steps Vadym Chepkov. I was able to reproduce this in 6.4.0. The issue seems as we expected. The pops loaders load the pcore model to compile the catalog. For reasons unknown, when converting to a RAL catalog, the metatype manager is called to (re)load types that it already knows about: [root@sfofmfwbut1qvgr modules]# puppet apply -e 'include puppetdb::database::postgresql' Warning: METAMANAGER LOADED file Warning: METAMANAGER LOADED user Warning: METAMANAGER LOADED group Warning: METAMANAGER LOADED stage Warning: METAMANAGER LOADED whit Warning: METAMANAGER LOADED component Warning: METAMANAGER LOADED package Warning: METAMANAGER LOADED service Warning: METAMANAGER LOADED yumrepo Warning: METAMANAGER LOADED exec Warning: POPS LOADED resource http://puppet.com/2016.1/runtime/resource_type_pp/postgresql_psql from pcore model Warning: POPS LOADED resource http://puppet.com/2016.1/runtime/resource_type_pp/anchor from pcore model Warning: POPS LOADED resource http://puppet.com/2016.1/runtime/resource_type_pp/postgresql_conn_validator from pcore model Warning: POPS LOADED resource http://puppet.com/2016.1/runtime/resource_type_pp/concat_file from pcore model Warning: POPS LOADED resource http://puppet.com/2016.1/runtime/resource_type_pp/concat_fragment from pcore model Warning: POPS LOADED resource http://puppet.com/2016.1/runtime/resource_type_pp/postgresql_conf from pcore model Notice: Compiled catalog for sfofmfwbut1qvgr.delivery.puppetlabs.net in environment production in 0.44 seconds Warning: METAMANAGER LOADED anchor

Error: Could not autoload puppet/type/anchor: Attempt to redefine entity 'http://puppet.com/2016.1/runtime/type/anchor'. Set at unknown location

This ticket looks related: PUP-6653 I wonder if this is a regression due to the agent-side rich data added in 6.0?

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

This must be an edge case as it doesn't have the issue when using the following example: /etc/puppetlabs/code/environments/production/modules/ntp/init.pp class ntp { include ntp::service   anchor { 'ntp_first': } -> Class['ntp::service'] -> anchor { 'ntp_last': } } /etc/puppetlabs/code/environments/production/modules/ntp/service.pp class ntp::service { notify { 'service': } } [root@sfofmfwbut1qvgr modules]# puppet apply -e "include ntp"

Warning: METAMANAGER LOADED file Warning: METAMANAGER LOADED user Warning: METAMANAGER LOADED group Warning: METAMANAGER LOADED stage Warning: METAMANAGER LOADED whit Warning: METAMANAGER LOADED component Warning: METAMANAGER LOADED package Warning: METAMANAGER LOADED service

Warning: METAMANAGER LOADED notify

Warning: POPS LOADED resource http://puppet.com/2016.1/runtime/resource_type_pp/anchor from pcore model

Notice: Compiled catalog for sfofmfwbut1qvgr.delivery.puppetlabs.net in environment production in 0.02 seconds Warning: METAMANAGER LOADED anchor Warning: METAMANAGER LOADED schedule Warning: METAMANAGER LOADED filebucket Notice: service Notice: /Stage[main]/Ntp::Service/Notify[service]/message: defined 'message' as 'service'

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-9602

puppet 6 apply fails if puppet types have been generated

Change By: Josh Cooper Sub-team: Language

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-9602 puppet 6 apply fails if puppet types have been generated

Change By: Josh Cooper Team: Server

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated Josh Cooper since you got the repro case running and with logging that shows "METAMANAGER loads anchor", could you do a stack trace for that point?

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

I can reproduce the issue easily using a resource default for ini_setting and anchor resource types: $ bx puppet module install puppelabs-inifile $ bx puppet generate types $ bx puppet apply -e 'Ini_setting { section => "main" } ini_setting { "sample": ensure => present, path => "/tmp/foo.ini" }' --trace Warning: REGISTERING file from file:///Users/josh/work/puppet/lib/puppet/type/file.rb?line=15 Warning: REGISTERING user from file:///Users/josh/work/puppet/lib/puppet/type/user.rb?line=9 Warning: REGISTERING group from file:///Users/josh/work/puppet/lib/puppet/type/group.rb?line=7 Warning: REGISTERING stage from file:///Users/josh/work/puppet/lib/puppet/type/stage.rb?line=1 Warning: REGISTERING whit from file:///Users/josh/work/puppet/lib/puppet/type/whit.rb?line=1 Warning: REGISTERING component from file:///Users/josh/work/puppet/lib/puppet/type/component.rb?line=6 Warning: REGISTERING package from file:///Users/josh/work/puppet/lib/puppet/type/package.rb?line=10 Warning: REGISTERING service from file:///Users/josh/work/puppet/lib/puppet/type/service.rb?line=10 Warning: POPS CREATE http://puppet.com/2016.1/runtime/resource_type_pp/ini_setting Notice: Compiled catalog for localhost in environment production in 0.02 seconds Warning: REGISTERING ini_setting from file:///Users/josh/.puppetlabs/etc/code/environments/production/modules/inifile/lib/puppet/type/ini_setting.rb?line=4 Error: Could not autoload puppet/type/ini_setting: Attempt to redefine entity 'http://puppet.com/2016.1/runtime/type/ini_setting'. Set at unknown location /Users/josh/work/puppet/lib/puppet/pops/loader/base_loader.rb:123:in `fail_redefine' /Users/josh/work/puppet/lib/puppet/pops/loader/base_loader.rb:73:in `set_entry' /Users/josh/work/puppet/lib/puppet/pops/loaders.rb:151:in `register_runtime3_type' /Users/josh/work/puppet/lib/puppet/metatype/manager.rb:132:in `newtype' /Users/josh/.puppetlabs/etc/code/environments/production/modules/inifile/lib/puppet/type/ini_setting.rb:4:in `<top (required)>' /Users/josh/work/puppet/lib/puppet/util/autoload.rb:66:in `load' /Users/josh/work/puppet/lib/puppet/util/autoload.rb:66:in `load_file' /Users/josh/work/puppet/lib/puppet/util/autoload.rb:201:in `load' /Users/josh/work/puppet/lib/puppet/metatype/manager.rb:172:in `type' /Users/josh/work/puppet/lib/puppet/resource.rb:477:in `to_ral' /Users/josh/work/puppet/lib/puppet/resource/catalog.rb:629:in `block in to_catalog' If the resource default is removed, then it applies fine: bx puppet apply -e 'ini_setting { "sample": ensure => present, path => "/tmp/foo.ini" }' Warning: REGISTERING file from file:///Users/josh/work/puppet/lib/puppet/type/file.rb?line=15 Warning: REGISTERING user from file:///Users/josh/work/puppet/lib/puppet/type/user.rb?line=9 Warning: REGISTERING group from file:///Users/josh/work/puppet/lib/puppet/type/group.rb?line=7 Warning: REGISTERING stage from file:///Users/josh/work/puppet/lib/puppet/type/stage.rb?line=1 Warning: REGISTERING whit from file:///Users/josh/work/puppet/lib/puppet/type/whit.rb?line=1 Warning: REGISTERING component from file:///Users/josh/work/puppet/lib/puppet/type/component.rb?line=6 Warning: REGISTERING package from file:///Users/josh/work/puppet/lib/puppet/type/package.rb?line=10 Warning: REGISTERING service from file:///Users/josh/work/puppet/lib/puppet/type/service.rb?line=10 Warning: POPS CREATE http://puppet.com/2016.1/runtime/resource_type_pp/ini_setting Notice: Compiled catalog for localhost in environment production in 0.02 seconds Warning: REGISTERING ini_setting from file:///Users/josh/.puppetlabs/etc/code/environments/production/modules/inifile/lib/puppet/type/ini_setting.rb?line=4 Warning: REGISTERING schedule from file:///Users/josh/work/puppet/lib/puppet/type/schedule.rb?line=2 Warning: REGISTERING filebucket from file:///Users/josh/work/puppet/lib/puppet/type/filebucket.rb?line=4 Notice: /Stage[main]/Main/Ini_setting[sample]/ensure: created Notice: Applied catalog in 0.04 seconds It appears the issue occurs if the pops loader loads the type first during catalog compilation, and later we try to load the resource type directly using Puppet::MetaType::Manager#newtype, which will load the type and attempt to register it a second time. I pushed a branch with the debugging messages to https://github.com/joshcooper/puppet/tree/redefine_entity_9602. Henrik Lindberg can you take a look?

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper assigned an issue to Henrik Lindberg

Puppet / PUP-9602

puppet 6 apply fails if puppet types have been generated

Change By: Josh Cooper Assignee: Henrik Lindberg

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated I can reproduce this as far back as 6.0.0, and git bisect shows: 29d515761eab068381900c74bf8c4b6165853fa9 is the first bad commit commit 29d515761eab068381900c74bf8c4b6165853fa9 Author: Henrik Lindberg <henrik....@puppetlabs.com> Date: Sat Apr 7 10:41:44 2018 +0200   (PUP-8640) Add resolution of Deferred values in agent and apply

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-9602

puppet 6 apply fails if puppet types have been generated

Change By: Josh Cooper Affects Version/s: PUP 6.0.0

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg assigned an issue to Unassigned

Puppet / PUP-9602 puppet 6 apply fails if puppet types have been generated

Change By: Henrik Lindberg Assignee: Henrik Lindberg

Add Comment

[Trevor Vaughan (JIRA)]

Trevor Vaughan commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated So, after much poking around with Charlie Sharpsteen on Slack, I've determined a few things. puppet apply is crazy slow for some reason The following code snippet appears to work around the issue but I'll leave it to Henrik Lindberg to tell me if it makes any sort of sense. diff --git a/lib/puppet/pops/loader/base_loader.rb b/lib/puppet/pops/loader/base_loader.rb index d3c80b63e7..9691113c1b 100644 --- a/lib/puppet/pops/loader/base_loader.rb +++ b/lib/puppet/pops/loader/base_loader.rb @@ -70,8 +70,11 @@ class BaseLoader < Loader def set_entry(typed_name, value, origin = nil) # It is never ok to redefine in the very same loader unless redefining a 'not found' entry = @named_values[typed_name] + + user_mode_skip = (entry && !entry.value.nil? && Puppet.run_mode.name == :user) + if entry - fail_redefine(entry) unless entry.value.nil? + fail_redefine(entry) unless entry.value.nil? || (user_mode_skip && entry.value == value) end # Check if new entry shadows existing entry and fail @@ -79,7 +82,7 @@ class BaseLoader < Loader if typed_name.type == :type && !allow_shadowing? entry = loaded_entry(typed_name) if entry - fail_redefine(entry) unless entry.value.nil? #|| entry.value == value + fail_redefine(entry) unless entry.value.nil? || (user_mode_skip && entry.value == value) end end

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

Pinging type loading domain expert: Thomas Hallgren - see comment above.

Add Comment

[Thomas Hallgren (JIRA)]

Thomas Hallgren commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

Looking at Trevor's functional workaround, I can only concur with Henrik's assessment earlier: "What happens is that it probably ended up with a mix of loading both the generated types and the actual resource type implementations. The agent most certainly needs to load the resource type implementations and should ignore the generated types completely". That would also explain at least part of the slow down.

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

I looked at Trevor's patch and while it "may work", it is really a band aid on the actual problem, so I don't think it should be fixed that way. I think the correct thing to do is what I commented with earlier - i.e. do not include the loader that loads resource types from metadata (i.e. the "generated types" loader) in the configuration of the loaders on the agent side. I think it was Tony Vu who worked on this logic last. With the support for Deferred (and improved handling of Sensitive) values in the catalog there is one walk over the entire catalog at the beginning of the apply. This needs to take place even if there are no Deferred or Sensitive values since the walk is required to find them. I can imagine that a huge catalog would take time to walk over. The other piece of logic that was added is the setting up of a PAL context for the resolution of deferred values. The setting up of PAL will configure loaders for all modules on the modulepath, but won't do any loading until something is asked for. The invocation of PAL itself would be more or less constant and what it does per module is not much. Next, when resolving deferred values, it would load functions and evaluate puppet logic. The loading of a function would search for it using the loaders config. Given that the agent has pretty much all squashed everything together the search should be faster than the corresponding on the master side (fewer paths to look at) - unless of course there is something wrong with the loaders config on the agent side. To get to grip with the performance issue I think it would be easiest to instrument the deferred loading with profiling around what it is doing, and around setting up PAL. Hopefully it is something other than the catalog walk that is the cause since that is much harder to tackle. A very simple test if it is the deferred resolution at all would be to simply patch it out and compare - that would only work naturally if the catalog in question does not contain any Deferred or nested Sensitive values. I mean, it could be something else completely.

Add Comment

[Marty Ewings (JIRA)]

Marty Ewings updated an issue

Puppet / PUP-9602

puppet 6 apply fails if puppet types have been generated

Change By: Marty Ewings CS Priority: Needs Priority

Add Comment

[Trevor Vaughan (JIRA)]

Trevor Vaughan commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated While I completely agree with Henrik Lindberg's assessment that my patch is completely the wrong way to do things, if this isn't going to get much attention in the near future, could it be implemented just to prevent things from crashing?

Add Comment

[Thomas Hallgren (JIRA)]

Thomas Hallgren commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

Given that the patch makes things "crazy slow", I would avoid implementing the patch. I think there's a fair chance that the slowness hits a lot of customers and the reason for it is forgotten. Raise the priority on this ticket if needed, but the root cause must be dealt with.

Add Comment

[Trevor Vaughan (JIRA)]

Trevor Vaughan commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

Thomas Hallgren I may not have been clear. While it is crazy slow, it is no slower than if you remove the entire cache and run without an error.

Add Comment

[Jarret Lavallee (JIRA)]

Jarret Lavallee updated an issue

Puppet / PUP-9602

puppet 6 apply fails if puppet types have been generated

Change By: Jarret Lavallee CS Priority: Needs Priority Reviewed

Add Comment

[Trevor Vaughan (JIRA)]

Trevor Vaughan commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated Just checking to see if there has been any progress on this and/or if the workaround is acceptable for the short term since it doesn't actually make anything slower than the non-cached version.

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

Removing the pcore resource loader seems to do the trick. It's the loader that knows to load from the .resource_types directory: diff --git a/lib/puppet/pops/loader/runtime3_type_loader.rb b/lib/puppet/pops/loader/runtime3_type_loader.rb index f9be7ac0be..7b4f09d809 100644 --- a/lib/puppet/pops/loader/runtime3_type_loader.rb +++ b/lib/puppet/pops/loader/runtime3_type_loader.rb @@ -10,7 +10,8 @@ class Runtime3TypeLoader < BaseLoader def initialize(parent_loader, loaders, environment, env_path) super(parent_loader, environment.name) @environment = environment - @resource_3x_loader = env_path.nil? ? nil : ModuleLoaders.pcore_resource_type_loader_from(parent_loader, loaders, env_path) + #@resource_3x_loader = env_path.nil? ? nil : ModuleLoaders.pcore_resource_type_loader_from(parent_loader, loaders, env_path) + @resource_3x_loader = env_path.nil? ? nil : parent_loader end def discover(type, error_collector = nil, name_authority = Pcore::RUNTIME_NAME_AUTHORITY, &block) Using the repro above: $ bx puppet module install puppetlabs-inifile Notice: Preparing to install into /Users/josh/.puppetlabs/etc/code/environments/production/modules ...

Notice: Downloading from https://forgeapi.puppet.com ... Notice: Installing -- do not interrupt ...

/Users/josh/.puppetlabs/etc/code/environments/production/modules └─┬ puppetlabs-inifile (v4.1.0) └── puppetlabs-translate (v2.1.0)

$ bx puppet generate types

Notice: Generating Puppet resource types. Notice: Generating '/Users/josh/.puppetlabs/etc/code/environments/production/.resource_types/ini_setting.pp' using 'pcore' format. Notice: Generating '/Users/josh/.puppetlabs/etc/code/environments/production/.resource_types/ini_subsetting.pp' using 'pcore' format.

$ bx puppet apply -e 'Ini_setting { section => "main" } ini_setting { "sample": ensure => present, path => "/tmp/foo.ini" }' --trace

Notice: Compiled catalog for XX in environment production in 0.08 seconds

Notice: /Stage[main]/Main/Ini_setting[sample]/ensure: created

Notice: Applied catalog in 0.01 seconds Now to figure out how to make that conditional based on puppet apply vs puppet resource vs puppetserver vs puppet agent vs bolt.

Add Comment

[Trevor Vaughan (JIRA)]

Trevor Vaughan commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

Josh Cooper Yep, I tried that for about 10 minutes, realized that I was lost, and posted what I had here :-|

Add Comment

[Vadym Chepkov (Jira)]

Vadym Chepkov commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

The ticket is a year old, wonder if there are any updates?

Add Comment

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)

[Josh Cooper (Jira)]

Josh Cooper assigned an issue to Josh Cooper

Puppet / PUP-9602

puppet 6 apply fails if puppet types have been generated

Change By: Josh Cooper Assignee: Josh Cooper

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-9602 puppet 6 apply fails if puppet types have been generated

Change By: Josh Cooper Team: Froyo Coremunity

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-9602 puppet 6 apply fails if puppet types have been generated

Change By: Josh Cooper Sprint: Platform Core KANBAN

Add Comment

[Tim Meusel (Jira)]

Tim Meusel commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated Hey Josh Cooper, we sadly run into this bug as well. It's still present in Puppet 6.17.0.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

Tim Meusel and Vadym Chepkov I have a PR at https://github.com/puppetlabs/puppet/pull/8215. It'd be great if you could give it a try. The PR causes puppet apply to exclude the pcore resource type loader, which functionally is the same as if the .resource_types directory didn't exist. Please leave comments on the PR if you have them. Thanks!

Add Comment

[Vadym Chepkov (Jira)]

Vadym Chepkov commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated

Josh Cooper, it did fix it for me, thank you Before: Notice: Compiled catalog for master.localdomain in environment production in 1.08 seconds Error: Could not autoload puppet/type/yumrepo: Attempt to redefine entity 'http://puppet.com/2016.1/runtime/type/yumrepo'. Set at unknown location Error: Could not autoload puppet/type/yumrepo: Attempt to redefine entity 'http://puppet.com/2016.1/runtime/type/yumrepo'. Set at unknown location After: Notice: Compiled catalog for master.localdomain in environment production in 1.34 seconds Notice: Applied catalog in 2.23 seconds

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-9602

puppet 6 apply fails if puppet types have been generated

Change By: Josh Cooper Fix Version/s: PUP 6.18.0

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-9602 puppet 6 apply fails if puppet types have been generated

Change By: Josh Cooper Release Notes: Bug Fix Release Notes Summary: Running `puppet apply` could fail if the `puppet generate types` command was run previously as described in Environment Isolation: https://puppet.com/docs/puppet/latest/environment_isolation.html. With this fix, `puppet apply` will ignore resource type in the environment's {{.resource_types}} directory.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated Merged to master in https://github.com/puppetlabs/puppet/commit/dc1ce25b87700584c7a342a3644557e2e31bf4c3

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-9602

puppet 6 apply fails if puppet types have been generated

Change By: Josh Cooper

Release Notes Summary: Running `puppet apply` could fail if the `puppet generate types` command was run previously as described in Environment Isolation: https://puppet.com/docs/puppet/latest/environment_isolation.html. With this fix, `puppet apply` will ignore resource type in the environment's {{ ` .resource_types }} ` directory.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-9602

Re: puppet 6 apply fails if puppet types have been generated Passed CI in dc1ce25b87

Add Comment

[Claire Cadman (Jira)]

Claire Cadman updated an issue

Puppet / PUP-9602

puppet 6 apply fails if puppet types have been generated

Change By: Claire Cadman Labels: doc_reviewed

Add Comment