Puppet 3.0 Puppet labs repo package problem on rhel5

[Alaric]

Hi,

I'm having a weird issue and was wondering if anyone else had run into it. I recently upgraded from puppet 2.7 -> 3.0.1 After cleaning up some gems on my puppet master everything seemed to be working ok. I had originally used the EPEL repo's to deploy puppet, but switched to the Puppet Labs repos so I could upgrade to 2.7 then 3. On RHEL5 only, I get a RSTRING_PTR error if I upgrade to the Puppet Labs version (1.4.6) if I roll back to the EPEL veriosn of rubygem-json (1.4.3) Everything works again.


Has anyone seen anything like this before?

Thanks

-a


Rolling back to rubygem-json-1.4.3-3.el5.1 and everything works

After Upgrade of rubygem-json

Info: Retrieving plugin
/usr/bin/ruby: symbol lookup error: /usr/lib64/ruby/site_ruby/1.8/x86_64-linux/json/ext/parser.so: undefined symbol: RSTRING_PTR

[root@rhel5build-10wa gems]# rpm -qi rubygem-json
Name : rubygem-json Relocations: (not relocatable)
Version : 1.4.6 Vendor: Puppet Labs
Release : 1.el5 Build Date: Wed 26 Sep 2012 06:10:33 PM EDT
Install Date: Fri 23 Nov 2012 05:11:36 PM EST Build Host: rpm-builder.puppetlabs.lan
Group : Development/Languages Source RPM: rubygem-json-1.4.6-1.el5.src.rpm
Size : 592354 License: Ruby or GPLv2
Signature : RSA/SHA1, Wed 26 Sep 2012 06:15:36 PM EDT, Key ID 1054b7a24bd6ec30
URL : http://json.rubyforge.org
Summary : A JSON implementation in Ruby
Description :
This is a implementation of the JSON specification according
to RFC 4627 in Ruby.
You can think of it as a low fat alternative to XML,
if you want to store data to disk or transmit it over
a network rather than use a verbose markup language.




Before Upgrade:

[root@rhel5build-10wa ~]# rpm -qi rubygem-json
Name : rubygem-json Relocations: (not relocatable)
Version : 1.4.3 Vendor: Fedora Project
Release : 3.el5.1 Build Date: Thu 16 Sep 2010 03:40:59 AM EDT
Install Date: Mon 19 Nov 2012 03:54:16 PM EST Build Host: x86-03.phx2.fedoraproject.org
Group : Development/Languages Source RPM: rubygem-json-1.4.3-3.el5.1.src.rpm
Size : 629927 License: Ruby or GPLv2
Signature : DSA/SHA1, Mon 20 Sep 2010 12:40:46 PM EDT, Key ID 119cc036217521f6
Packager : Fedora Project
URL : http://json.rubyforge.org
Summary : A JSON implementation in Ruby
Description :
This is a implementation of the JSON specification according
to RFC 4627 in Ruby.
You can think of it as a low fat alternative to XML,
if you want to store data to disk or transmit it over
a network rather than use a verbose markup language.

[root@rhel5build-10wa ~]# puppet agent --test
Info: Retrieving plugin
Info: Loading facts in /var/lib/puppet/lib/facter/born_on.rb
Info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
Info: Caching catalog for rhel5build-10wa
Info: Applying configuration version '1353708172'
Finished catalog run in 11.65 seconds

[Michael Stahnke]

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>

I tried to reproduce this and just couldn't. Does this happen on
other systems?

Is there any more information you could think of?

[Alaric]

On Nov 26, 2012, at 4:46 PM, Michael Stahnke <sta...@puppetlabs.com> wrote:

>
> I tried to reproduce this and just couldn't. Does this happen on
> other systems?
>
> Is there any more information you could think of?
>

Sadly, it's happening on every RHEL5 system I have! All packages are either RedHat or EPEL, or now PuppetLabs, I can't really think of anything that would cause a conflict. It happens on both VM's and Physical hardware. I did have a number of gems running to support hiera before I upgrade to Puppet 3.0.1 but I got rid of them on the master when I upgraded. It's really perplexing, especially since all the other packages from puppetlabs seem to have been installed and are working...

[Michael Stahnke]

If you disable hiera (like have a really simple catalog or something)
does it still happen?

[Alaric]

On Nov 26, 2012, at 10:29 PM, Michael Stahnke <sta...@puppetlabs.com> wrote:

> On Mon, Nov 26, 2012 at 2:16 PM, Alaric <paxind...@gmail.com> wrote:
>>
>> On Nov 26, 2012, at 4:46 PM, Michael Stahnke <sta...@puppetlabs.com> wrote:
>>
>>>
>>> I tried to reproduce this and just couldn't. Does this happen on
>>> other systems?
>>>
>>> Is there any more information you could think of?
>>>
>>
>>
>> Sadly, it's happening on every RHEL5 system I have! All packages are either RedHat or EPEL, or now PuppetLabs, I can't really think of anything that would cause a conflict. It happens on both VM's and Physical hardware. I did have a number of gems running to support hiera before I upgrade to Puppet 3.0.1 but I got rid of them on the master when I upgraded. It's really perplexing, especially since all the other packages from puppetlabs seem to have been installed and are working...
>
> If you disable hiera (like have a really simple catalog or something)
> does it still happen?
>

Yup, I get the same error:

/usr/bin/ruby: symbol lookup error: /usr/lib64/ruby/site_ruby/1.8/x86_64-linux/json/ext/parser.so: undefined symbol: RSTRING_PTR

My one thought is that maybe my version stdlib is old... I checked and it's version 2.3.1 I'll give it an upgrade and see if that helps, it's just weird that on the RHEL6 servers nothing seems off.

[Matthew Burgess]

On Tue, Nov 27, 2012 at 2:09 PM, Alaric <paxind...@gmail.com> wrote:
> Yup, I get the same error:
>
> /usr/bin/ruby: symbol lookup error: /usr/lib64/ruby/site_ruby/1.8/x86_64-linux/json/ext/parser.so: undefined symbol: RSTRING_PTR
>
>
> My one thought is that maybe my version stdlib is old... I checked and it's version 2.3.1 I'll give it an upgrade and see if that helps, it's just weird that on the RHEL6 servers nothing seems off.

This looks like your version of Ruby is too old.

RSTRING_PTR was added to Ruby-1.8.6, but RHEL5 and its clones only
provide Ruby-1.8.5. I use Ruby-1.8.7 available from
http://yum.theforeman.org/development/el5/x86_64/.

Thanks,

Matt.

[jcbollinger]

On Friday, November 23, 2012 4:47:03 PM UTC-6, alaric wrote:

After cleaning up some gems on my puppet master [...]

That you are working with gems on your RHEL system makes me pretty suspicious.  I strongly discourage use of gem on systems such as RHEL that have decent package managers with system-wide scope.  I especially urge you to avoid managing your Ruby library via a mix of yum/rpm and gem.  Each of those keeps its own, separate metadata and manages software independently, so you can easily produce conflicts, incompatibilities, and breakage that would never occur with either package manager alone.

I'm willing to cut some slack for gems that are packaged in RPM format and managed via yum / rpm, but I'm not fond even of those.

I suggest you clean out all gems not managed via yum/rpm, then try an "rpm --verify --all" (or clean out every single gem, and let package verification fail on the ones managed by RPM).  Perform a "yum reinstall" on all packages that are not successfully verified.

You might also want to look for altogether unpackaged Ruby binaries and modules.  Something like this might do the trick, albeit slowly:

  find /usr/lib{,64}/ruby -type f -exec rpm -q -f {} \; | grep "not owned"

If you're going to try that then do it after cleaning out everything you are going to remove and before reinstalling anything; that will minimize the number of files tested.


John



John

[jcbollinger]

That makes sense, somewhat.  It would constitute a pretty weird packaging issue, because rpmbuild normally does a very good job of identifying library version dependencies, and yum and rpm are very reliable about ensuring dependencies are installed (unless you start overriding them, in which case all bets are off).

Do you have more than one version of Ruby installed on the affected systems?


John

[Matthew Burgess]

I know you were probably asking this of the OP, but I'll answer from
my point of view anyway.

When I got the error it was exactly because I was doing what the OP
was doing and mixing RPMs and GEMs on the same system. The gem
installed without a hitch, because, if I remember rightly, they do
very little install-time checking other than for other gems that they
directly depend on. They simply assume that if they can be installed,
you have a recent enough Ruby environment; they leave any issues to be
discovered at runtime.

Having had my hands burned by that episode, I now only ever use RPMs.
Thankfully, the repos available from puppetlabs and theforeman coupled
with EPEL, provide for everything I've needed so far (I'd like to see
up to date RPMs for Razor-related dependencies, but that's very low
priority and for another mailing list anyway).

Kind Regards,

Matt.

[Alaric]

Only the one version that I can find! her's a list of the installed ruby packages 

libselinux-ruby-1.33.4-5.7.el5

ruby-1.8.7.370-1.el5

ruby-augeas-0.4.1-1.el5

ruby-devel-1.8.7.370-1.el5

ruby-devel-1.8.7.370-1.el5

rubygem-json-1.4.6-1.el5

rubygems-1.3.7-1.el5

rubygem-stomp-1.2.2-1.el5

rubygem-systemu-1.2.0-3.el5

ruby-irb-1.8.7.370-1.el5

ruby-libs-1.8.7.370-1.el5

ruby-libs-1.8.7.370-1.el5

ruby-mysql-2.7.3-2

ruby-rdoc-1.8.7.370-1.el5

ruby-shadow-1.4.1-7.el5

I did like the idea that gems might have been conflicting, and actually did find some hiera gems installed, after removing those and reinstalling I still get the same error, but I'm going through the package list with a fine tooth comb and verifying installs now...

All I get with find now is this:, which I think is just a cache of the json gem, would that actually have any effect?

find /usr/lib{,64}/ruby -type f -exec rpm -q -f {} \; | grep "not owned"

file /usr/lib/ruby/gems/1.8/cache/json-1.6.6.gem is not owned by any package

[Alaric]

Swing and a miss... even after verifying and manually removing any gems, and any cached gems, reinstalling effected packages and verifying, I still get the same error... 

[jcbollinger]

Things to try:

Run the affected shared object (/usr/lib64/ruby/site_ruby/1.8/x86_64-linux/json/ext/parser.so) through ldd.  Look for any library dependencies that cannot be resolved, and look carefully for the possibility that one or more lingering Ruby 1.8.5 libraries are being linked in place of the libraries from your v 1.8.7 installation.

Run ldconfig to ensure that the dynamic linker's cache is up to date (though any package that installs libraries ought to do this in its post-install script).

Check the output of "ruby --version".

Check the agent's startup script for a reference to a non-standard ruby binary.

Consider a ground-up wipe-and-rebuild of one of your affected systems, making sure to install only RPM-packaged software (no gems, especially).  Check whether the rebuilt system is still affected.  (If it is, then there is definitely a packaging problem somewhere.)


John

[Jakov Sosic]

On 11/27/2012 03:09 PM, Alaric wrote:

> Yup, I get the same error:
>
> /usr/bin/ruby: symbol lookup error: /usr/lib64/ruby/site_ruby/1.8/x86_64-linux/json/ext/parser.so: undefined symbol: RSTRING_PTR

Upgrade to the ruby provided by puppetlabs repo (RHEL 5 has older
version), and don't mix GEMs with RPMs... use only RPMs.

[jcbollinger]

On Tuesday, November 27, 2012 8:37:22 PM UTC-6, Brian Jolly wrote:

I am experiencing the same problem with a 3.0.1 install.

If I set "enable_inventory_service: false" there are no problems. 

ruby: symbol lookup error: /usr/lib/ruby/gems/1.8/gems/json-1.5.1/ext/json/ext/json/ext/parser.so: undefined symbol: RSTRING_PTR

ruby 1.8.7 (2012-06-29 patchlevel 370) [x86_64-linux]

# ldd /usr/lib/ruby/gems/1.8/gems/json-1.5.1/ext/json/ext/json/ext/parser.so

        linux-vdso.so.1 =>  (0x00007fff7f920000)

        libruby.so.1.8 => /usr/lib64/libruby.so.1.8 (0x00002aeccf1a9000)

        libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aeccf4a7000)

        libdl.so.2 => /lib64/libdl.so.2 (0x00002aeccf6c3000)

        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00002aeccf8c7000)

        libm.so.6 => /lib64/libm.so.6 (0x00002aeccfaff000)

        libc.so.6 => /lib64/libc.so.6 (0x00002aeccfd83000)

        librt.so.1 => /lib64/librt.so.1 (0x00002aecd00da000)

        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002aecd02e3000)

        /lib64/ld-linux-x86-64.so.2 (0x00002aecced7c000)

rpm -q -f /usr/lib64/libruby.so.1.8
rpm -q -f /usr/lib/ruby/gems/1.8/gems/json-1.5.1/ext/json/ext/json/ext/parser.so
rpm -q -f $(which ruby)


John

[Matthew Burgess]

On Tue, Nov 27, 2012 at 3:53 PM, Alaric <paxind...@gmail.com> wrote:
>
> Swing and a miss... even after verifying and manually removing any gems, and
> any cached gems, reinstalling effected packages and verifying, I still get
> the same error...

Are you able to reduce this down to a specific module or class that
triggers this issue? I've now got a fresh centos 5.8 VM with an out
of the box puppet-3.0.1 install on it with all dependencies met via
puppetlabs' repo. I can't trigger it with a simple module, but am
willing to try to reproduce it for you.

Thanks,

Matt.

[Jeff McCune]

Jakov,

I'm really sorry to step in on this one but our ruby should fully
support gems installed using the gem command. I understand it's not
ideal and that RPM's are definitely preferable. I encourage people to
use RPMs whenever possible in this situation. However, I'm deeply
concerned that we may be replacing the system ruby, which does support
gem install, with something that does not.

To this end, I'd like to clarify that if you run into this problem and
you're using gem install instead of yum, and we find out there's a bug
here, then this issue should also be resolved for you and your
deployment scenario. Please keep troubleshooting the issue even if
you're using gems and not RPMs.

-Jeff

[Jeff McCune]

On Fri, Nov 23, 2012 at 2:46 PM, Alaric <paxind...@gmail.com> wrote:

> Hi,
>
> I'm having a weird issue and was wondering if anyone else had run into it. I recently upgraded from puppet 2.7 -> 3.0.1 After cleaning up some gems on my puppet master everything seemed to be working ok. I had originally used the EPEL repo's to deploy puppet, but switched to the Puppet Labs repos so I could upgrade to 2.7 then 3. On RHEL5 only, I get a RSTRING_PTR error if I upgrade to the Puppet Labs version (1.4.6) if I roll back to the EPEL veriosn of rubygem-json (1.4.3) Everything works again.

Has anyone affected by this issue seen
https://bugzilla.redhat.com/show_bug.cgi?id=634380 ?

This information leads me to believe the rubygem-json package from
Puppet Labs may not be carrying the same patch that the rubygem-json
1.4.3 package from the Fedora Project is carrying. This difference
may be the cause of the error. Can anyone confirm?

-Jeff

[Matthaus Owens]

The problem was that our el5 rubygem-json package was the el6 src
rebuilt, without the patch, against ruby 1.8.5. Rebuilding the
rubygem-json package against the ruby 1.8.7 packages in our
dependencies repo resolved the parser.so linking errors. I've included
the ldd of the parser.so before and after below as well. The updated
package is now available in our dependencies repo. Please let us know
if this doesn't address your problem.

Apologies for the problems this caused you and thanks much for
bringing the issue to our attention.


Here is the ldd of parser.so with the broken rubygem-json package

ldd -r /usr/lib64/ruby/site_ruby/1.8/x86_64-linux/json/ext/parser.so
undefined symbol:
RSTRING_PTR (/usr/lib64/ruby/site_ruby/1.8/x86_64-linux/json/ext/parser.so)
undefined symbol:
RSTRING_LEN (/usr/lib64/ruby/site_ruby/1.8/x86_64-linux/json/ext/parser.so)
linux-vdso.so.1 => (0x00002ae120f0c000)
libruby.so.1.8 => /usr/lib64/libruby.so.1.8 (0x00002ae12131b000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002ae121619000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002ae121835000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00002ae121a39000)
libm.so.6 => /lib64/libm.so.6 (0x00002ae121c71000)
libc.so.6 => /lib64/libc.so.6 (0x00002ae121ef5000)
librt.so.1 => /lib64/librt.so.1 (0x00002ae12224c000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002ae122455000)
/lib64/ld-linux-x86-64.so.2 (0x00002ae120ef0000)

And the ldd of parser.so from the rebuilt package

ldd -r /usr/lib64/ruby/site_ruby/1.8/x86_64-linux/json/ext/parser.so
linux-vdso.so.1 => (0x00007fff02dfc000)
libruby.so.1.8 => /usr/lib64/libruby.so.1.8 (0x00002b58c627f000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b58c657d000)
librt.so.1 => /lib64/librt.so.1 (0x00002b58c6799000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002b58c69a2000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00002b58c6ba6000)
libm.so.6 => /lib64/libm.so.6 (0x00002b58c6ddf000)
libc.so.6 => /lib64/libc.so.6 (0x00002b58c7062000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b58c73b9000)
/lib64/ld-linux-x86-64.so.2 (0x00002b58c5e54000)

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>

--
Matthaus Owens
Release Manager, Puppet Labs

[Brian Jolly]

That worked for me. Thanks!!!

[Alaric]

For me as well! Thanks for all the help!! 

[Michael Stahnke]

Thanks Jeff for pointing out that the EPEL maintainer and the puppet labs maintainer should have been coordinating on rubygem-json.  (It's the same guy....me.  I'm an idiot).

Mike

[Jakov Sosic]

On 11/28/2012 05:58 PM, Jeff McCune wrote:
> Jakov,
>
> I'm really sorry to step in on this one but our ruby should fully
> support gems installed using the gem command. I understand it's not
> ideal and that RPM's are definitely preferable. I encourage people to
> use RPMs whenever possible in this situation. However, I'm deeply
> concerned that we may be replacing the system ruby, which does support
> gem install, with something that does not.
>
> To this end, I'd like to clarify that if you run into this problem and
> you're using gem install instead of yum, and we find out there's a bug
> here, then this issue should also be resolved for you and your
> deployment scenario. Please keep troubleshooting the issue even if
> you're using gems and not RPMs.

Let me clarify myself - I don't even try to suggest that ruby provided
by puppetlabs for el5 does not support 'gem install', I'm only strongly
advising against it - no matter if it's RedHat's ruby package in
questions or PuppetLabs one. IMHO it's a really bad practice to mix
different packaging systems and that practice will byte you, sooner or
later.




--
Jakov Sosic
www.srce.unizg.hr

[jcbollinger]

On Thursday, December 13, 2012 6:51:57 PM UTC-6, Jakov Sosic wrote:

IMHO it's a really bad practice to mix
different packaging systems and that practice will byte you, sooner or
later.

+1


John