Jira (PUP-7408) Declaring a resource on a huge list causes an error

[Doug Rosser (JIRA)]

Doug Rosser created an issue

Puppet / PUP-7408 Declaring a resource on a huge list causes an error Issue Type: Bug Assignee: Unassigned Created: 2017/03/29 3:39 PM Priority: Normal Reporter: Doug Rosser Following code: https://github.com/puppetlabs/clamps/blob/iterate_test/manifests/agent.pp#L37 Works fine with about 3000 items in the list, but hits the following error at 4000: [root@glsn-agent-01 ~]# puppet agent -t --trace Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Error: Could not run: stack level too deep /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:434 Workaround in Puppet 4 is to use an iterator, like $nonroot_usernames.each |$name| { ::clamps::users { $name: servername => $master, ca_server => $ca, metrics_server => $metrics_server, metrics_port => $metrics_port, daemonize => $daemonize, splay => $splay, splaylimit => $splaylimit, } } Add Comment

This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)

[Doug Rosser (JIRA)]

Doug Rosser updated an issue

Puppet / PUP-7408 Declaring a resource on a huge list causes an error

Change By: Doug Rosser Following code: https://github.com/puppetlabs/clamps/blob/ iterate_test master /manifests/agent.pp#L37

Works fine with about 3000 items in the list, but hits the following error at 4000:

{noformat}

[root@glsn-agent-01 ~]# puppet agent -t --trace Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Error: Could not run: stack level too deep /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:434

{noformat}

Workaround in Puppet 4 is to use an iterator, like

{noformat}

$nonroot_usernames.each |$name| {       ::clamps::users { $name:         servername         => $master,         ca_server          => $ca,         metrics_server     => $metrics_server,         metrics_port       => $metrics_port,         daemonize          => $daemonize,         splay              => $splay,         splaylimit         => $splaylimit,       }   }

{noformat}

Add Comment

[Nick Lewis (JIRA)]

Nick Lewis updated an issue

Puppet / PUP-7408 Declaring a resource on a huge list causes an error

Change By: Nick Lewis Following code: https://github.com/puppetlabs/clamps/blob/ master 059076a8d278b7e4dc3014bb1e6e308d10994b64 /manifests/agent.pp#L37

[Doug Rosser (JIRA)]

Doug Rosser commented on PUP-7408

Re: Declaring a resource on a huge list causes an error And unfortunately, the workaround above does NOT work. I hit the same error, it just takes longer to hit it.

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-7408

Re: Declaring a resource on a huge list causes an error

Doug Rosser This looks like a problem on the agent side - if it crashed during compilation I think the error would be different. However, your comment that it makes a difference if you use an array as the title, or if you iterate speaks against this. Can you try to just compile the catalog and see if that works? There should really be no difference in the catalogs between an iteration and using an array title. (Except the obvious difference in file/line information)

Add Comment

[Doug Rosser (JIRA)]

Doug Rosser commented on PUP-7408

Re: Declaring a resource on a huge list causes an error

Henrik Lindberg Results below, but it looks like the catalog compiled successfully on the master, although the agent was unable to apply it. [root@glsn-mom ~]# puppet master --compile glsn-agent-01.us-west-2.compute.internal > glsn-agent-01.us-west-2.compute.internal.json Warning: Unknown variable: '::pe_server_version'. at /etc/puppetlabs/code/environments/production/manifests/site.pp:13:4 Warning: /etc/puppetlabs/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 (in /etc/puppetlabs/puppet/hiera.yaml) Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Bool. There is further documentation for validate_legacy function in the README. (at /etc/puppetlabs/code/environments/production/modules/stdlib/lib/puppet/functions/deprecation.rb:19:in `deprecation') Warning: This method is deprecated, please use match expressions with Stdlib::Compat::String instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. (at /etc/puppetlabs/code/environments/production/modules/stdlib/lib/puppet/functions/deprecation.rb:19:in `deprecation') Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Array instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. (at /etc/puppetlabs/code/environments/production/modules/stdlib/lib/puppet/functions/deprecation.rb:19:in `deprecation') Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. (at /etc/puppetlabs/code/environments/production/modules/stdlib/lib/puppet/functions/deprecation.rb:19:in `deprecation') Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Array. There is further documentation for validate_legacy function in the README. (at /etc/puppetlabs/code/environments/production/modules/stdlib/lib/puppet/functions/deprecation.rb:19:in `deprecation') Warning: Defining "data_provider": "hiera" in metadata.json is deprecated (in /etc/puppetlabs/code/environments/production/modules/ntp/metadata.json) [root@glsn-mom ~]# file glsn-agent-01.us-west-2.compute.internal.json glsn-agent-01.us-west-2.compute.internal.json: ASCII text, with very long lines, with escape sequences [root@glsn-mom ~]# ls -lh glsn-agent-01.us-west-2.compute.internal.json -rw-r--r-- 1 root root 150M Mar 31 17:50 glsn-agent-01.us-west-2.compute.internal.json   [root@glsn-agent-01 ~]# puppet apply --catalog glsn-agent-01.us-west-2.compute.internal.json Error: Could not run: Could not deserialize catalog from pson: Could not intern from pson: source 'Notice: Compiled ' not in PSON!

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-7408

Re: Declaring a resource on a huge list causes an error

Thanks Doug Rosser that helps - either the written Pson is bad, or there is something wrong on the agent side. Ping Josh Cooper.

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-7408

Re: Declaring a resource on a huge list causes an error

Doug Rosser could you attach the glsn-agent-01.us-west-2.compute.internal.json catalog?

Add Comment

[Doug Rosser (JIRA)]

Doug Rosser commented on PUP-7408

Re: Declaring a resource on a huge list causes an error

The file is 150MB, so I've put a copy at http://int-resources.ops.puppetlabs.net/drosser/glsn-agent-01.us-west-2.compute.internal.json

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-7408

Re: Declaring a resource on a huge list causes an error

Doug Rosser The command puppet master --compile glsn-agent-01.us-west-2.compute.internal > glsn-agent-01.us-west-2.compute.internal.json causes the message Notice: Compiled catalog for glsn-agent-01.us-west-2.compute.internal in environment production in 137.37 seconds to be written to the file, so parsing fails. If I remove that line, I can at least parse the catalog using both JSON and PSON: $ bundle exec irb ... irb(main):003:0> require 'json' => true irb(main):004:0> hash = JSON.parse(File.read(File.expand_path("~/Downloads/glsn-agent-01.us-west-2.compute.internal.json"))); nil => nil irb(main):005:0> require 'puppet' => true irb(main):006:0> hash = PSON.parse(File.read(File.expand_path("~/Downloads/glsn-agent-01.us-west-2.compute.internal.json"))); nil => nil

Add Comment

[Doug Rosser (JIRA)]

Doug Rosser commented on PUP-7408

Re: Declaring a resource on a huge list causes an error

Cleaned up the json file. I get the same error as from a regular puppet run. [root@glsn-agent-01 ~]# puppet apply --debug --catalog glsn-agent-01.take2.json Debug: Runtime environment: puppet_version=4.10.0, ruby_version=2.1.9, run_mode=user, default_encoding=UTF-8 Debug: Evicting cache entry for environment 'production' Debug: Caching environment 'production' (ttl = 0 sec) Debug: Failed to load library 'msgpack' for feature 'msgpack' Debug: Puppet::Network::Format[msgpack]: feature msgpack is missing Debug: catalog supports formats: pson yaml dot binary Debug: Facter: searching for custom fact "osfamily". Debug: Facter: searching for osfamily.rb in /opt/puppetlabs/puppet/cache/lib/facter. Debug: Facter: searching for osfamily.rb in /opt/puppetlabs/puppet/cache/lib/facter. Debug: Facter: searching for osfamily.rb in /opt/puppetlabs/puppet/cache/facts. Debug: Evicting cache entry for environment 'production' Debug: Caching environment 'production' (ttl = 0 sec) Debug: Evicting cache entry for environment 'production' Debug: Caching environment 'production' (ttl = 0 sec) Debug: Failed to load library 'cfpropertylist' for feature 'cfpropertylist' Debug: Evicting cache entry for environment 'production' Debug: Caching environment 'production' (ttl = 0 sec) Debug: Evicting cache entry for environment 'production' Debug: Caching environment 'production' (ttl = 0 sec) Debug: Executing: '/usr/bin/rpm --version' Debug: Executing: '/usr/bin/rpm -ql rpm' Debug: Evicting cache entry for environment 'production' Debug: Caching environment 'production' (ttl = 0 sec) Debug: Evicting cache entry for environment 'production' Debug: Caching environment 'production' (ttl = 0 sec) Debug: Executing: '/usr/bin/rpm --version' Debug: Executing: '/usr/bin/rpm --version' Debug: Evicting cache entry for environment 'production' Debug: Caching environment 'production' (ttl = 0 sec) Debug: Failed to load library 'cfpropertylist' for feature 'cfpropertylist' Debug: Executing: '/usr/bin/rpm --version' Debug: Evicting cache entry for environment 'production' Debug: Caching environment 'production' (ttl = 0 sec) Debug: Evicting cache entry for environment 'production' Debug: Caching environment 'production' (ttl = 0 sec) Debug: Facter: searching for custom fact "operatingsystemrelease". Debug: Facter: searching for operatingsystemrelease.rb in /opt/puppetlabs/puppet/cache/lib/facter. Debug: Facter: searching for operatingsystemrelease.rb in /opt/puppetlabs/puppet/cache/lib/facter. Debug: Facter: searching for operatingsystemrelease.rb in /opt/puppetlabs/puppet/cache/facts.

Error: Could not run: stack level too deep

[root@glsn-agent-01 ~]#

Add Comment

[Moses Mendoza (JIRA)]

Moses Mendoza updated an issue

Puppet / PUP-7408

Declaring a resource on a huge list causes an error

Change By: Moses Mendoza Team: Agent

Add Comment

[Moses Mendoza (JIRA)]

Moses Mendoza updated an issue

Puppet / PUP-7408 Declaring a resource on a huge list causes an error

Change By: Moses Mendoza Labels: triaged

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-7408

Re: Declaring a resource on a huge list causes an error Can't reproduce this, closing

Add Comment

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)