[Puppet - Bug #9290] (Unreviewed) Puppet master fails with 'stack level too deep' error when storeconfigs = true with rails stack 3.1.0
tic...@puppetlabs.com
Bug #9290: Puppet master fails with 'stack level too deep' error when storeconfigs = true with rails stack 3.1.0
- Author: Mark Stanislav
- Status: Unreviewed
- Priority: Normal
- Assignee:
- Category:
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord
- Branch:
Out of nowhere, a known-working Puppet stack build script started to fail. The tell-tale error is ‘stack level too deep’ which from other historical bug reports always seems related to Ruby directly. After many hours of digging around, I checked for updated gem versions on the system. There was an update to rails 3.1.0 (and activerecord, etc.) which apparently is breaking the Puppet master when working with storeconfigs = true. Disabling storeconfigs immediately works as expected again.
When reverting from rails and friends 3.1.0 to 3.0.10, Puppet again works as expected.
I don’t have any insight into where exactly this is all failing, only the version which is causing the issue and the condition to emulate it.
This was only tested again Puppet 2.7.3 using the gem install with both webrick and Apache+Passenger.
Attached is a debug output.
You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account
tic...@puppetlabs.com
- Category set to Rails
- Keywords changed from storeconfigs rails activerecord to storeconfigs rails activerecord 3.1.0
- Author: Mark Stanislav
- Status: Unreviewed
- Priority: Normal
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
- Status changed from Unreviewed to Accepted
- Priority changed from Normal to High
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
I am having the same error using the git repository as of today. Downgrading to activerecord 3.0.10 fixed the issue. It errored with both webrick and apache+passenger. The following was repeated several times with webrick:
/usr/lib/ruby/site_ruby/1.8/puppet/rails/resource.rb:177:in parameter'
/usr/lib/ruby/site_ruby/1.8/puppet/rails/resource.rb:87:in[]‘
/usr/lib/ruby/gems/1.8/gems/activerecord-3.1.0/lib/active_record/associations/association_scope.rb:81:in add_constraints'
/usr/lib/ruby/gems/1.8/gems/activemodel-3.1.0/lib/active_model/attribute_methods.rb:102:ineach_with_index’
/usr/lib/ruby/gems/1.8/gems/activerecord-3.1.0/lib/active_record/associations/association_scope.rb:55:in each'
/usr/lib/ruby/gems/1.8/gems/activerecord-3.1.0/lib/active_record/associations/association_scope.rb:55:ineach_with_index'
/usr/lib/ruby/gems/1.8/gems/activerecord-3.1.0/lib/active_record/associations/association_scope.rb:55:in add_constraints'
/usr/lib/ruby/gems/1.8/gems/activerecord-3.1.0/lib/active_record/associations/association_scope.rb:33:inscope'
/usr/lib/ruby/gems/1.8/gems/activerecord-3.1.0/lib/active_record/associations/association.rb:99:in association_scope'
/usr/lib/ruby/gems/1.8/gems/activerecord-3.1.0/lib/active_record/associations/association.rb:88:inscoped'
/usr/lib/ruby/gems/1.8/gems/activerecord-3.1.0/lib/active_record/associations/collection_proxy.rb:51:in __send__'
/usr/lib/ruby/gems/1.8/gems/activerecord-3.1.0/lib/active_record/associations/collection_proxy.rb:51:inscoped'
/usr/lib/ruby/gems/1.8/gems/activerecord-3.1.0/lib/active_record/associations/collection_proxy.rb:102:in method_missing'
/usr/lib/ruby/site_ruby/1.8/puppet/rails/resource.rb:177:inparameter'
/usr/lib/ruby/site_ruby/1.8/puppet/rails/resource.rb:87:in `[]'
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
Confirmed for me – Puppet 2.7.3, Unicorn 4.1.1
Downgrading from activerecord 3.1.0 to 3.0.10 resolved it here, as well. Will try to test on 2.7.5 this weekend.
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
Same Error with 2.7.5
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
Same error with 2.7.6
err: Could not retrieve catalog from remote server: Error 400 on SERVER: stack level too deep
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
Also ran into this one, thx for the downgrade tip.
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
I hit this also, downgrading to activerecord 3.0.10 and its related dependencies resolved it for me.
K
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
Another “it’s broken” observation here — Puppet 2.7.6, infinite loop with ARec 3.1.1, works fine with 3.0.11. Might be worth at least putting a version constraint on ARec at load time, so people don’t get bitten, until the problem can be found and fixed properly.
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
Confirmed still in ActiveRecord 3.2.0
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
If you hvae more than one Rails version installed, you can force an older activerecord version to puppet:
gem ‘activerecord’, ‘=2.2.2’ require ‘activerecord’
Add this into your config.ru file.
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
looks like the same is happening with
puppet-server-2.7.1-1
activerecord (3.2.1)
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
I’m also experiencing this after upgrading to activerecord 3.2.1. I’ve found the newest branch that works is 3.0.x (3.0.11 as of this comment). The 3.1.x and 3.2.x seem to produce this error.
gem install activerecord -v 3.0.11
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
If anyone experiencing this problem could test out the code in https://github.com/puppetlabs/puppet/pull/497 and see if it makes any difference, that would be awesome.
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
- File master-trace.txt added
I have applied the patch for lib/puppet/rails/inventory_node.rb from pull req. 497, but still, same problem on my old Debian Lenny. See the attached trace.
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
Well, it was a hope. Thanks for confirming this is still a problem.
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
the workaround proposed at note 11 made my rack crash, instead I’ve modified lib/puppet/feature/rails.rb:
Puppet.features.add(:rails) do
begin
+ # http://projects.puppetlabs.com/issues/9290
+ gem 'activerecord', '=3.0.6'
require 'active_record'
require 'active_record/version'
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
With the sql injection vuln in older versions of active record 3.x this has become an issue for us.
activerecord 3.2.5 puppet 2.7.14 puppet dashboard 1.2.8
puppetmaster¶
notice: Compiled catalog for in environment production in 4.45 seconds info: Caching catalog for debug: Searched for resources in 0.00 seconds debug: Searched for resource params and tags in 0.00 seconds debug: Resource removal in 0.00 seconds debug: Resource merger in 0.00 seconds err: stack level too deep
puppet agent¶
err: Could not retrieve catalog from remote server: Error 400 on SERVER: stack level too deep warning: Not using cache on failed catalog err: Could not retrieve catalog; skipping run
Another relevant note this doesn’t happen on all of our nodes. Only on nodes created after the active record upgrade so far.
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
Also editing /usr/lib/ruby/1.8/puppet/feature/rails.rb and making the change above except point it at active record 3.0.13 fixed the problem for me.
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
FYI, I ran into this problem today on a FreeBSD host, using puppet 2.7.16 and activerecord 3.2.5 and 3.2.6. Installing activerecord 3.0.15, and adding that 1-line patch to force the older-version gem, fixed things. Without this workaround, any use of @@syntax in a config would cause a thread on the server to consume an entire core, recurse infinitely, consume scads of memory, and eventually bomb out with a “stack too deep” message. With the workaround, things behave as you’d expect.
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
Just got bitten by this again with an accidental ActiveRecord gem update. Is anyone looking into this?
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
Using Puppet 2.7.18 and ruby 1.8.7 on the puppetmaster and 2.6.16 on the puppet client. Hit this bug immediately after enabling stored configs. Installed activerecord version 3.0.11 and now have 3 versions intalled (I have other rails apps which need 3.2.1):
$gem list | grep activerecord
activerecord (3.2.1, 3.0.11, 2.3.14)
According to the fix in msg 11 above I need to add “gem ‘activerecord’, ‘=3.0.11’ require ‘activerecord’” to my config.ru file. I have no idea which config.ru file to add this to nor where in the file to place this line. Which file exactly do I need to edit and it would be much appreciated if anyone could give me an indication of which line to replace. Msg 19 seems to indicate that I need to edit /usr/lib/ruby/1.8/puppet/feature/rails.rb however I only have /usr/lib/ruby/site_ruby/1.8/puppet/feature/rails.rb which I assume is the same. Again not sure what to replace.
Any help would be greatly appreciated as we’re stuck until we can resolve this.
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
Jeremy Cole wrote:
Using Puppet 2.7.18 and ruby 1.8.7 on the puppetmaster and 2.6.16 on the puppet client. Hit this bug immediately after enabling stored configs. Installed activerecord version 3.0.11 and now have 3 versions intalled (I have other rails apps which need 3.2.1):
$gem list | grep activerecord activerecord (3.2.1, 3.0.11, 2.3.14)According to the fix in msg 11 above I need to add “gem ‘activerecord’, ‘=3.0.11’ require ‘activerecord’” to my config.ru file. I have no idea which config.ru file to add this to nor where in the file to place this line. Which file exactly do I need to edit and it would be much appreciated if anyone could give me an indication of which line to replace. Msg 19 seems to indicate that I need to edit /usr/lib/ruby/1.8/puppet/feature/rails.rb however I only have /usr/lib/ruby/site_ruby/1.8/puppet/feature/rails.rb which I assume is the same. Again not sure what to replace.
Any help would be greatly appreciated as we’re stuck until we can resolve this.
Hey Jeremy,
There should already be everything besides these two lines,
be sure to add them between existing lines like so
Puppet.features.add(:rails) do begin + # http://projects.puppetlabs.com/issues/9290 + gem 'activerecord', '=3.0.6' require 'active_record' require 'active_record/version'
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
- File resource-fix.diff added
I had this same problem, with Puppet 2.7.6 and Rails 3.2.6. I tried downgrading Rails, but it didn’t initially work, and it’s been a long time since Rails 3.0.x, hasn’t it? So I decided to find out what the problem was.
By adding a print statement to the Puppet::Rails::Resource.parameter method, I found that the parameter causing the infinite loop was id. After staring at the backtrace for a long time, I thought it was odd that after fifteen calls inside ActiveRecord it would suddenly call back into Puppet:
... /usr/lib/ruby/site_ruby/1.8/puppet/rails/resource.rb:186:in `parameter' /usr/lib/ruby/site_ruby/1.8/puppet/rails/resource.rb:95:in `[]' /usr/lib/ruby/gems/1.8/gems/activerecord-3.2.6/lib/active_record/associations/association_scope.rb:70:in `add_constraints' /usr/lib/ruby/gems/1.8/gems/activesupport-3.2.6/lib/active_support/dependencies.rb:202:in `each_with_index' ...
So I changed the [] method as in the attached diff. Everything appeared to suddenly work, and my node began configuring itself. The node having succeeded once, it kept working even after I put the code back to the original, until I nuked the storedconfig database. Perhaps this is a clue as to the nature of the bug.
I don’t fully understand the change I’ve made. I haven’t run any of the tests. I haven’t made a pull request, because I can’t push to GitHub from the network I’m on. Maybe later.
The code I changed originated in commit:6314745c three years ago, and still exists in the master branch today, so this patch would probably apply to any recent Puppet version.
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
I can confirm the issue on FreeBSD with puppet 2.7.19 and rails 3.2.8.
I can also confirm that Jared’s workaround in the previous update fixes it. It’d be great if somebody familiar with that code could check it and commit it if appropriate (or feedback if it’s not).
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
I’d really like to see this fixed. I’m using the absolute latest stable software; puppet 3.0.0 and activerecord 3.2.8 and a perfectly reasonable configuration is completely broken…
tic...@puppetlabs.com
- Affected Puppet version changed from 2.7.3 to 3.0.1
Still seeing this with Puppet 3.0.1. Fix in comment 24 still works.
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee:
- Category: Rails
- Target version:
- Affected Puppet version: 3.0.1
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
- Assignee set to eric sorenson
- Affected Puppet version changed from 3.0.1 to 2.7.3
Hi Tim, we try to use the affected field to keep track of when the bug was first encountered, but it’s “good” to hear that this is still affecting you in 3.0.1.
- Author: Mark Stanislav
- Status: Accepted
- Priority: High
- Assignee: eric sorenson
- Category: Rails
- Target version:
- Affected Puppet version: 2.7.3
- Keywords: storeconfigs rails activerecord 3.1.0
- Branch:
tic...@puppetlabs.com
Sorry, my mistake. I assumed the opposite; that it was the most recent version. But upon reflection, it makes a lot more sense for it to be the first version :–)
tic...@puppetlabs.com
Running into this issue with 2.6.17 from EPEL. Switching to activerecord 3.0.11 from gem allowed me to work around the issue.
tic...@puppetlabs.com
We’ve received a ticket indicating that a commercial customer has run into this as well, and that the patch attached successfully resolved the issue for them. Version information they provided is Puppet 2.7.18, Fedora ~18, using mod_passenger and Rails 3.2.8.