Jira (PUP-10161) Regresssion: puppet 6.11 can't manage selinux on centos/redhat 8

17 views
Skip to first unread message

Michael Watters (JIRA)

unread,
Dec 13, 2019, 12:10:04 PM12/13/19
to puppe...@googlegroups.com
Michael Watters commented on Bug PUP-10161
 
Re: Regresssion: puppet 6.11 can't manage selinux on centos/redhat 8

This error can also cause management of resources that do not depend on selinux to fail as well.  For example, on a node running CentOS 8 custom types are not applied due to the agent failures.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

James Ralston (JIRA)

unread,
Dec 14, 2019, 3:58:05 PM12/14/19
to puppe...@googlegroups.com

People who are attempting to use the RHEL8 Puppet packages on Fedora will also be burned by this, as every version of Fedora I have examined (including Rawhide) ships a libselinux package that was built with export DISABLE_RPM="y".

(And yes, I know Puppet doesn't officially support running RHEL packages on Fedora. But I experiment with running puppetserver on Fedora, and Puppet doesn't build the puppetserver packages for the official Fedora repositories. Furthermore, because Fedora moves so rapidly, it's generally the case that by the time official Puppet packages appear for Fedora [n] are available, I've already migrated to Fedora [n+1]. For these reasons, I eschew Puppet's official Fedora repositories, and instead use the latest RHEL repository on my Fedora systems.)

That notwithstanding, it is unusual for Red Hat to roll out a change for RHEL that hasn't hit Fedora already, as Fedora is essentially RHEL next. It's also somewhat unusual (but not without precedent) for Red Hat to break backwards shared library compatibility with RHEL minor point releases. As such, I think there is a possibility that this change wasn't intentional—that it was an oversight in the libselinux 2.8 → 2.9 rebase between RHEL 8.0 and RHEL 8.1

We are a RHEL shop and have a Red Hat support contract. I filed the following support case with Red Hat, gently questioning this change:

why is libselinux now built with 'export DISABLE_RPM="n"'?

What problem/issue/behavior are you having trouble with? What do you expect to see?

Starting with libselinux-2.9-2.1.el8, the %build section of libselinux.spec file now contains:

export DISABLE_RPM="n"

This is a change from the previous (RHEL 8.0) version of libselinux, libselinux-2.8-6.el8, which contained:

export DISABLE_RPM="y"

This change means that the libselinux-2.9-2.1.el8 libselinux.so shared library has a symbol, rpm_execcon, that the libselinux-2.8-6.el8 libselinux.so shared library does not. This change means that any executable linked against the libselinux-2.9-2.1.el8 libselinux.so will die at runtime if executed on a system that has not yet updated from libselinux-2.8-6.el8.

This change also breaks cross-distro compatibility. Every Fedora version of libselinux that I have examined, including libselinux-3.0-1.fc32 from Rawhide, builds libselinux with:

export DISABLE_RPM="y"

While it is usually the case that an executable compiled on RHEL can be run on Fedora (as long as the necessary compat-* library packages are installed), that is not the case with this change, as no version of Fedora has a libselinux.so shared library that has the rpm_execcon symbol.

Furthermore, this change can break third-party vendors who are attempting to ship packages for RHEL8. Here is one instance:

https://tickets.puppetlabs.com/browse/PUP-10161

Finally, I can find no explanation for this change. There were no (public) Bugzilla tickets requesting it that I can find. And it was implemented without comment during the libselinux 2.8 → 2.9 rebase for RHEL 8.1:

https://git.centos.org/rpms/libselinux/c/29ef3f732b5b513cd8e11a5f02f5498837565688?branch=c8#_11

I am beginning to wonder if this change was in fact unintentional—that flipping DISABLE_RPM from "y" to "n" was an oversight of the libselinux 2.8 → 2.9 rebase for RHEL 8.1, and that Red Hat did not actually intend for programs linked against the RHEL 8.1 libselinux.so to break on RHEL 8.0 systems.

Was this change intentional?

If so, can you reveal the reasons for the change?

If not, is there any realistic possibility of reverting the change? (My guess is no, because if the RHEL 8.2 libselinux.so reverts to export DISABLE_RPM="y", then anything linked against the RHEL 8.1 libselinux.so will fail at runtime for RHEL 8.2 in the exact same way it fails at runtime for RHEL 8.0.)

What information can you provide around timeframes and the business impact?

Vendors and other developers who link against libselinux.so must compile on RHEL 8.0, not RHEL 8.1. This is inconvenient, but not a showstopper.

But even if this change wasn't intentional, as per above, I don't see how Red Hat would be able to revert it for RHEL 8.2 without breaking break forward shared library compatibility with RHEL 8.1. This is something Red Hat cannot do.

As such, I think the only way to safely build Puppet packages for RHEL8 that will work correctly on any version of RHEL8 (not to mention on other distros, such as Fedora) is to build using libselinux-devel-2.8 from RHEL 8.0. You're stuck with that for the lifetime of RHEL8, alas.

Michael Watters (JIRA)

unread,
Dec 16, 2019, 10:28:04 AM12/16/19
to puppe...@googlegroups.com

I was able to correct this by downgrading the puppet-agent package to 6.10.1.  

 

```dnf install puppet-agent-6.10.1```

 

This was done on a node running CentOS 8 with the puppet6-release package installed.

 

 

Mihai Buzgau (JIRA)

unread,
Dec 17, 2019, 8:30:04 AM12/17/19
to puppe...@googlegroups.com

Mihai Buzgau (JIRA)

unread,
Jan 7, 2020, 5:31:03 AM1/7/20
to puppe...@googlegroups.com

Mihai Buzgau (JIRA)

unread,
Jan 7, 2020, 5:31:04 AM1/7/20
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
Change By: Mihai Buzgau
Sprint: PR NW - Triage 2020-01-22

Mihai Buzgau (JIRA)

unread,
Jan 7, 2020, 6:52:05 AM1/7/20
to puppe...@googlegroups.com

James Ralston (JIRA)

unread,
Jan 8, 2020, 6:59:04 PM1/8/20
to puppe...@googlegroups.com
James Ralston commented on Bug PUP-10161
 
Re: Regresssion: puppet 6.11 can't manage selinux on centos/redhat 8

Red Hat responded to me.

Red Hat asserts (and I have verified) that RHEL7 did build libselinux.so.1 with the rpm_execcon() function. The specific line in the RHEL7 libselinux.spec file (for every version of the RHEL7 libselinux package) is:

# FIXME: export DISABLE_RPM="y"

This is trivial to verify on a RHEL7 system, using the stock Ruby and libselinux-ruby packages:

$ /usr/bin/irb 
irb(main):001:0> require 'selinux'
=> true
irb(main):002:0> Selinux.rpm_execcon(1, 2, 3, 4)
TypeError: Expected argument 1 of type char const *, but got Fixnum 2
        in SWIG method 'rpm_execcon'
        from (irb):2:in `rpm_execcon'
        from (irb):2
        from /usr/bin/irb:12:in `<main>'
irb(main):003:0> 

For RHEL 8.0, someone acted the FIXME comment, and removed the rpm_execcon() function. But since libselinux is an ABI Compatibility level 1 package starting with RHEL7:

https://access.redhat.com/articles/rhel-abi-compatibility

…removing rpm_execcon() broke the ABI compatibility promise, and thus had to be reverted. That is why it was brought back in RHEL 8.1.

But there is another change in play that landed in libselinux 2.7 (between RHEL7's libselinux 2.5 and RHEL8's libselinux 2.9). That change was the removal of the rpm_execcon() function from the Python and Ruby bindings. Via commit 581dde73:

commit 581dde735af139daaefb5888bff3ec7ca2470dee
Author: Nicolas Iooss <nicola...@m4x.org>
Date:   Sat Nov 5 21:55:32 2016 +0100
 
    libselinux: remove rpm_execcon from SWIG wrappers
    
    The Python wrapper of rpm_execcon() has several flaws:
    * An invalid call like selinux.rpm_execcon() triggers a segmentation
      fault.
    * The size of the buffer which is allocated to copy argv and envp is
      too small to hold all the values.
    * This allocated memory is leaked if one argument of rpm_execon() is not
      a sequence of bytes.
    
    The Ruby wrapper has no such flaws but can not be used as it is because
    it misses some glue code to convert argv and envp arguments to char
    *const [] values (even though the destructor is present!).
    
    As it is not possible to remove rpm_execcon() without changing
    libselinux soname (it would be an ABI break) like b67fefd991dd
    ("libselinux: set DISABLE_RPM default to y.") tried to do, disable this
    interface locally in the SWIG wrappers.
    
    Signed-off-by: Nicolas Iooss <nicola...@m4x.org>

So starting with RHEL8, the Ruby selinux.so bindings no longer include rpm_execcon(), even though libselinux.so.1 does.

I assert that the real problem here is how the Ruby selinux.so module (that is packaged in the puppet-agent package) is being built between 6.10 and 6.11.

Specifically, since Fedora has never supplied a libselinux.so.1 that includes rpm_execcon(), this should not work:

$ cat /etc/fedora-release 
Fedora release 31 (Thirty One)
 
$ rpm -q puppet-agent
puppet-agent-6.10.1-1.el8.x86_64
 
$ /opt/puppetlabs/puppet/bin/irb
irb(main):001:0> require 'selinux'
=> true
irb(main):002:0> result = Selinux.rpm_execcon(1,2,3,4)
Traceback (most recent call last):
        3: from /opt/puppetlabs/puppet/bin/irb:11:in `<main>'
        2: from (irb):2
        1: from (irb):2:in `rpm_execcon'
TypeError (Expected argument 1 of type char const *, but got Integer 2)
        in SWIG method 'rpm_execcon'

The rpm_execcon() function is not available in libselinux.so.1. There is no way Ruby could have resolved that symbol.

Furthermore, the fact that the RHEL8 puppet-agent selinux.so library knows Selinux.rpm_execcon() means that it was not built from the stock RHEL8 libselinux packages, as the stock RHEL8 selinux.so does not know that symbol:

$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.1 (Ootpa)
 
$ /usr/bin/irb
irb(main):001:0> require 'selinux'
=> true
irb(main):002:0> result = Selinux.rpm_execcon(1,2,3,4)
Traceback (most recent call last):
        2: from /usr/bin/irb:11:in `<main>'
        1: from (irb):2
NoMethodError (undefined method `rpm_execcon' for Selinux:Module)

The version of Ruby is identical between Puppet agent 6.10 and 6.11:

$ /opt/puppetlabs/puppet/bin/ruby --version
ruby 2.5.7p206 (2019-10-01 revision 67816) [x86_64-linux]

…so this behavior difference cannot be explained by a change in Ruby.

At this point, the only possible explanation is that up through 6.10, the selinux.so Ruby library that the puppet-agent package has shipped with (which does not seem to correspond to any selinux.so Ruby library as supplied by Red Hat) has inlined the rpm_execcon() function entirely, so it never mattered whether it was actually present in libselinux.so.1.

But starting with 6.11, selinux.so no longer inlines rpm_execcon(). So if 6.11 is built on a distribution where the system libselinux.so.1 library does not contain rpm_execcon(), then selinux support is not available.

The file sizes of the 6.10 and 6.11 selinux.so objects tend to support the hypothesis that the 6.10 objects have code that 6.11 does not:

$ ls -lsa selinux*
268 -rwxr-xr-x. 1 root root 272800 Oct 14 01:23 selinux-6.10.1.so*
224 -rwxr-xr-x. 1 root root 225432 Nov 20 08:51 selinux-6.11.1.so*
  4 lrwxrwxrwx. 1 root root     17 Jan  8 16:25 selinux.so -> selinux-6.10.1.so*

Finally, if I replace the 6.11 selinux.so with the 6.10 version (as I have done above), then 6.11's SELinux support behaves no differently than 6.10, right down to providing the Selinux.rpm_execcon() function.

My guess is that the selinux.so Ruby library that is packaged in puppet-agent is built from an out-of-tree libselinux fork, where the build process just plucks out the Ruby selinux.so library and leaves everything else. If that is indeed the case, then the only thing that should be necessary to resolve this problem is to pull commit 581dde73 into that build tree.

Reply all
Reply to author
Forward
0 new messages