activerecord and puppet-3.0.0

[Jonathan Gazeley]

Yesterday my puppetmaster and nodes got upgraded to puppet-3.0.0.

Since then, all puppet runs have been failing with this error:

Error: Could not retrieve catalog from remote server: Error 400 on
SERVER: Could not autoload puppet/indirector/node/active_record:
uninitialized constant ActiveRecord


My colleague and I have put a few hours into trying to work out what's
wrong. rubygem-activerecord-2.1.1-2.el6.noarch is installed from the
puppetlabs RPM repo. We've reinstalled all components but made no progress.

I found this thread which seems to describe the same behaviour, but
there are no replies:

https://groups.google.com/forum/?fromgroups=#!topic/puppet-dev/D85TsZ70LKQ

Anyone got any ideas?

Thanks,
Jonathan

[Joehillen]

Same issue here. I'd love to get some info on this.

[llowder]

Comments inline.

I can't find the sources now, but I vaguely remember seeing something related to this and they had to downgrade their activerecord install. But unfortunately, I do not remember the specifics.

 

Thanks,
Jonathan

[Joe Hillenbrand]

btw, I'm running Ubuntu 10.04.1

[Eric Sorenson]

Thanks for reporting this, we're looking into it.  I've made a ticket for your issue: https://projects.puppetlabs.com/issues/16770

If you can do so, could you please run the puppet master with the '--trace' option ( either on the command line or with a `ARGV << '--trace'` line in config.ru if you are starting from passenger, and update the ticket with the output? Thanks!!

eric0

On Wednesday, October 3, 2012 2:26:42 AM UTC-7, Jonathan Gazeley wrote:

[Jonathan Gazeley]

Hi Eric,

Thanks for your attention on this.

Right now I'm not able to run any tests on my puppet master because we
were forced to roll it back to 2.7 to restore service. I work at a
university, this week is freshers' week and we totally need our config
management to work at this busy time!

Hopefully in the next few days I'll try to set up a second puppet master
with puppet 3 and see if I can recreate the problem without breaking all
of my infrastructure ;)

Cheers,
Jonathan

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/KPQEZyWqezMJ.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.

[Jonathan Gazeley]

I managed to set up a fresh virtual machine with puppet 3 on it. Using
the same server config and certs and a demo puppet 3 client, the system
worked. So I wasn't able to run any debugging.

Therefore, I am forced to conclude that the problem lies not with puppet
3 itself, but probably with some old gems that are left lying around on
my old system.

So I have a new plan: rather than upgrade my old puppetmaster from 2.7
to 3.0, I'll create a new puppet 3 server and migrate to that. Then I'll
re-upgrade my old 2.7 puppetmaster and run the debugs... if you are
still interested in them by then.

Cheers,
Jonathan

[Andy Parker]

I encountered this issue yesterday when trying to reproduce another
issue. I discovered that you need to be using activerecord 3.0.11.
Later versions don't seem to work (error with a "stack too deep") and
earlier versions don't seem to work (error with "uninitialized
constant ActiveRecord").

[Joe Hillenbrand]

Installing 3.0.11 (and removing all the old versions manually, fucking
ruby gems!) didn't help for me:

~$ sudo gem list

*** LOCAL GEMS ***

activemodel (3.0.11)
activerecord (3.0.11)
activesupport (3.0.11)
arel (2.0.10)
bigdecimal (1.1.0)
builder (2.1.2)
bundler (1.1.5)
charlock_holmes (0.6.8)
i18n (0.5.0)
io-console (0.3)
json (1.5.4)
minitest (2.5.1)
multi_json (1.3.6)
puppet-lint (0.2.1)
rake (0.9.2.2)
rdoc (3.9.4)
tzinfo (0.3.33)


I think there used to be an ubuntu package for activerecord but it's
not there anymore.

[Jennings, Jared L CTR USAF AFMC 96 SK/CCI]

> I encountered this issue yesterday when trying to reproduce another
> issue. I discovered that you need to be using activerecord 3.0.11.
> Later versions don't seem to work (error with a "stack too deep") and
> earlier versions don't seem to work (error with "uninitialized
constant
> ActiveRecord").

About "stack too deep" with later activerecord versions:
<http://projects.puppetlabs.com/issues/9290#note-24>

-- Jared Jennings, RHCE, Network Admin, SURVICE Engineering Co.

[Jonathan Gazeley]

On 04/10/12 17:30, Andy Parker wrote:
> I encountered this issue yesterday when trying to reproduce another
> issue. I discovered that you need to be using activerecord 3.0.11.
> Later versions don't seem to work (error with a "stack too deep") and
> earlier versions don't seem to work (error with "uninitialized
> constant ActiveRecord").

Interesting. On my *old* puppetmaster (2.7 upgraded to 3.0) I'm running
activerecord-2.1.1-2 from the puppetlabs-deps repo. It doesn't work,
failing with the "uninitialized constant ActiveRecord" error.

On my *new* puppetmaster (fresh install of 3.0 on a blank box) it is
also running activerecord-2.1.1-2 from the puppetlabs-deps repo. It
*does* work.

Both puppetmasters are running identical puppet configs. The *old*
puppetmaster has these gems:

activemodel (3.2.8, 3.2.3, 3.0.9)
activerecord (2.1.1)
activesupport (3.2.8, 3.2.3, 3.0.9, 2.1.1)
arel (3.0.2, 2.0.10)
builder (3.0.0, 2.1.2)
hiera (1.0.0)
i18n (0.6.0, 0.5.0)
multi_json (1.3.2)
rake (0.8.7)
tzinfo (0.3.33, 0.3.29)


The *new* puppetmaster has these:

activerecord (2.1.1)
activesupport (2.1.1)

Possibly there's an issue with the multiple versions of activesupport? I
don't really know anything about ruby - I'm merely a user of puppet.

Cheers,
Jonathan

[Luca Gioppo]

Same problem for me:

here is the trace:
Error: Cached catalog for backup.cortile.cloudlabcsi.local failed: Could not autoload puppet/indirector/catalog/active_record: uninitialized constant ActiveRecord
/usr/lib/ruby/site_ruby/1.8/puppet/util/autoload.rb:75:in `load_file'
/usr/lib/ruby/site_ruby/1.8/puppet/util/autoload.rb:208:in `load'
/usr/lib/ruby/site_ruby/1.8/puppet/util/instance_loader.rb:68:in `loaded_instance'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/terminus.rb:108:in `terminus_class'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:313:in `make_terminus'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:126:in `terminus'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/store_configs.rb:6:in `initialize'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:316:in `new'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:316:in `make_terminus'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:126:in `terminus'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:47:in `cache'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:217:in `find_in_cache'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:186:in `find'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:108:in `do_find'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:71:in `send'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:71:in `process'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick/rest.rb:24:in `service'
/usr/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'
/usr/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:33:in `listen'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `call'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:95:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `each'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:23:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:82:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:30:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:29:in `initialize'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:29:in `new'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:29:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:26:in `synchronize'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:26:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:92:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:104:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:136:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:199:in `main'
/usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:148:in `run_command'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:346:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:438:in `plugin_hook'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:346:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/util.rb:500:in `exit_on_fail'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:346:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:76:in `execute'
/usr/bin/puppet:10

and

gem list

*** LOCAL GEMS ***

json (1.5.1)
rake (0.8.7)
stomp (1.2.2)

[Josh Cooper]

Jonathan and Luca,

How did you install puppet (from source, gem, apt-get, ...)? In the
cases that fail, is it always that you installed 2.7.x and upgraded
from there? If so, how did you update?

We are tracking this issue at http://projects.puppetlabs.com/issues/16770

Josh

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/puppet-users/-/ZaGpkrLIlRAJ.

>
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.



--

Josh Cooper
Developer, Puppet Labs

[Jonathan Gazeley]

My version of puppet 2.7 was installed on CentOS 6 from the puppetlabs
RPM repo with "yum install puppet". When puppet 3 was released, I did a
"yum update" to get the latest. The issue does not occur with a blank
CentOS box starting out with puppet 3.

Want me to add this information to the tracked issue? Don't want to add
unwanted noise to the issue unnecessarily :)

Cheers,
Jonathan

>> Il giorno mercoled� 3 ottobre 2012 11:26:42 UTC+2, Jonathan Gazeley ha

[Joe Hillenbrand]

Same story here. Started with puppet 2.7 on Ubuntu 10.04, did apt-get
upgrade, then it broke.

I have since switched from storedconfig to PuppetDB, which I highly
recommend to anyone else having this issue.

>>> Il giorno mercoledě 3 ottobre 2012 11:26:42 UTC+2, Jonathan Gazeley ha

[Josh Cooper]

We've been able to reproduce this issue in a variety of ways. As far
as we can tell, it is always caused by using incompatible versions of
activerecord gems. Removing all active* gems and installing the
platform-specific packaging containing the activerecord ruby bindings,
e.g. apt-get install libactiverecord-ruby, seems to resolve the
problem. Can users who are experiencing this problem, please give that
a try and let us know? When replying please include the OS & version,
ruby versions, gem list and whatever active* packages you are using.

Also if you are using alternatives to select ruby 1.9, then puppet 3
will automatically start using it due to its dependency on ruby in
addition to ruby1.8. This can cause problems if you were using older
activerecord gems, e.g. 2.2.2, in puppet 2.7.x and then upgraded to
puppet 3. However, deleting the old gems and installing the platform
specific package should resolve the issue.

Josh