RHEL9?

[t.d...@servicemusic.org.uk]

Now that RHEL9 has been available for a few months, are there any plans to support it, please?  And if so, what is the likely timescale?

Thanks.

-- David Lee

[Nick Anderson]

"t.d...@servicemusic.org.uk" <t.d...@servicemusic.org.uk> writes:

> Now that RHEL9 has been available for a few months, are there any plans to support it, please? And if so, what is the likely timescale?

We are trying to get it in with 3.21.0 which should be out before the
end of the year.

--
Nick Anderson | Doer of Things | (+1) 785-550-1767 | https://northern.tech

[t.d...@servicemusic.org.uk]

Thanks, Nick.

No particular hurry.  End of year would be good.  Good luck with it!

-- David

[Shane Ripley]

Will support for RHEL9 be back-ported to 3.15, or alternatively would 3.21 clients work with 3.15 policy hubs?

[Vratislav Podzimek]

On Wed, 2022-11-02 at 16:07 -0700, Shane Ripley wrote:
> Will support for RHEL9 be back-ported to 3.15, or alternatively would 3.21 clients work with 3.15
> policy hubs?

3.15 is approaching EOL, the support for RHEL 9 will probably only be backported to 3.18.x, like in
this PR [1]. Regarding the second question -- no, hub should always be running the same version of
CFEngine or newer than the clients.

[1] https://github.com/cfengine/core/pull/5092

--
Vratislav

>
> On Friday, October 21, 2022 at 1:44:54 PM UTC-6 t.d...@servicemusic.org.uk wrote:
> > Thanks, Nick.
> >
> > No particular hurry.  End of year would be good.  Good luck with it!
> >
> > -- David
> >
> > On Friday, 21 October 2022 at 20:37:21 UTC+1 nick.a...@northern.tech wrote:
> > >
> > > "t.d...@servicemusic.org.uk" <t.d...@servicemusic.org.uk> writes:
> > >
> > > > Now that RHEL9 has been available for a few months, are there any plans to support it,
> > > > please? And if so, what is the likely timescale?
> > >
> > > We are trying to get it in with 3.21.0 which should be out before the
> > > end of the year.
> > >
> > > --
> > > Nick Anderson | Doer of Things | (+1) 785-550-1767 | https://northern.tech
> > >

> --
> You received this message because you are subscribed to the Google Groups "help-cfengine" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> help-cfengin...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/help-cfengine/bf3f6e55-bc49-4464-a621-b824852f9527n%40googlegroups.com
> .

[t.d...@servicemusic.org.uk]

I see that 3.21 is now released (Dec 22) and that it says it supports RHEL9.  That sounds great.  Thank you!

Background to following questions:

• Until about a year ago we were way back at release 3.10 and our policy was not MPF-compatible at all
• We are now at 3.15 (software) and 3.12 (MPF masterfiles)
• Somewhat over 2000 clients; mostly RH7 and RH8 (a little residual RH6; starting RH9 development)
  

I understand 3.15 is now EOL, so we need to start moving up.  (We were going to anyway, but 3.15/EOL encourages that!)

The message above from Vratislav (Nov 3) suggests that RHEL9 ie being backported to 3.18.  Has this been done?  (The message references a PR which appears to have been merged.  But has this yet made it to the community RPMs?)  If so that means that I can start my RH9 development at 3.18 rather than needing to start at 3.21, which eases somewhat the version "stretch" with masterfiles.

Vratislav's message also says that the version of the hub software should generally be the same as, or ahead of, the clients.  Fortunately, I've seen no bad effects when the reverse is the case (client a little ahead of hub), but I take the point for general production.

But within any machine, what about software version vs. MPF/masterfiles version?  We are currently 3.15 software and 3.12 masterfiles.  It seems OK.  Nevertheless I plan to move masterfiles upwards.  Can it go ahead of the software?  Indeed, I suspect masterfiles probably should be ahead of the software.  This becomes more important as we cross the 3.18 threshold (and look further ahead towards 3.21) because a quick test with very new RPM (3.18 and 3.21) showed issues about "rxdirs", which I suspect will be eased by having masterfiles significantly later than our current 3.12.

This last point raises another thought.  Given that our software version is 3.15 (and likely to stay like that for a few months), can I take the masterfiles version way up in a giant leap from 3.12 to 3.21?  Or am I better doing masterfiles by a few smaller steps (3.12->3.15->3.18->3.21)?   I would probably do a similar series of steps for the software, although they may not exactly rise in step with each other.

Advice welcome!

-- David Lee

-- Diamond

[Nick Anderson]

"t.d…@servicemusic.org.uk" <t.d...@servicemusic.org.uk> writes:

Hi David,

The message above from Vratislav (Nov 3) suggests that RHEL9 ie being backported to 3.18. Has this been done? (The message references a PR which appears to have been merged. But has this yet made it to the community RPMs?) If so that means that I can start my RH9 development at 3.18 rather than needing to start at 3.21, which eases somewhat the version "stretch" with masterfiles.

Yes, there has been work towards bringing el9 to 3.18.x. cf-remote is a great way to keep an eye out for new packages.

+ cf-remote --version 3.18.x list el9
Available releases: master, 3.21.x, 3.21.0, 3.18.x, 3.18.3, 3.18.2, 3.18.1, 3.18.0
Using 3.18.x LTS:
http://buildcache.cfengine.com/packages/testing-pr/jenkins-3.18.x-nightly-pipeline-395/PACKAGES_x86_64_linux_redhat_9/cfengine-nova-3.18.4a.5dd596487-24850.el9.x86_64.rpm
+ cf-remote --version 3.18.x list --edition community el9
Available releases: master, 3.21.x, 3.21.0, 3.18.x, 3.18.3, 3.18.2, 3.18.1, 3.18.0
Using 3.18.x LTS:
http://buildcache.cfengine.com/packages/testing-pr/jenkins-community-nightly-3.18.x-228/PACKAGES_x86_64_linux_redhat_9/cfengine-community-3.18.4a.5dd596487-24846.el9.x86_64.rpm
+ :

But within any machine, what about software version vs. MPF/masterfiles version? We are currently 3.15 software and 3.12 masterfiles. It seems OK. Nevertheless I plan to move masterfiles upwards. Can it go ahead of the software? Indeed, I suspect masterfiles probably should be ahead of the software. This becomes more important as we cross the 3.18 threshold (and look further ahead towards 3.21) because a quick test with very new RPM (3.18 and 3.21) showed issues about "rxdirs", which I suspect will be eased by having masterfiles significantly later than our current 3.12.

As noted in the upgrade documentation, upgrading the MPF is the second step in the process (behind taking a backup) we always recommend that you run MPF version greater than or equal to your binary versions. This is because MPF from 3.12 can't possibly know about how to handle some change in behavior for 3.15.x clients. A good recent example of this is the rxdirs change. If you are running 3.18.0 MPF and you install 3.18.3 binaries you are going to get a bunch of warnings about not having rxdirs explicitly set. If you upgrade to 3.21 binaries and run 3.18.0 MPF you will have a bunch of warnings AND a behavior change.

This last point raises another thought. Given that our software version is 3.15 (and likely to stay like that for a few months), can I take the masterfiles version way up in a giant leap from 3.12 to 3.21? Or am I better doing masterfiles by a few smaller steps (3.12->3.15->3.18->3.21)? I would probably do a similar series of steps for the software, although they may not exactly rise in step with each other.

As noted previously, if your running 3.15 binaries, you should be running at least 3.15 MPF. It's no surprise that it works OK for you but running binaries newer than the policy is not something we test. We maintain compatibility with the versions that are supported at the time of release. So, when there is a PR to the MPF on the 3.21.x branch the MPF is tested against the last release of 3.18 and 3.15.

Your already in untested waters running older MPF with newer binaries, I would just work on getting to 3.21.0 or 3.21.x MPF.

– Nick Anderson | Doer of Things | (+1) 785-550-1767 | https://northern.tech

[Bas van der Vlies]

>
> Your already in untested waters running older MPF with newer binaries, I
> would just work on getting to 3.21.0 or 3.21.x MPF.
>

I agree with Nick and maybe add switch to a `cfbs` setup so you can
easily upgrade masterfiles or other modules/promise_types/bundles. It
saves a lo tof time. Just start a new server with this setup and
gradually connect your environment to this server.

I just upgrade all our hosts to 3.21.0




--
--
Bas van der Vlies
| High Performance Computing & Visualization | SURF| Science Park 140 |
1098 XG Amsterdam
| T +31 (0) 20 800 1300 | bas.van...@surf.nl | www.surf.nl |

[t.d...@servicemusic.org.uk]

Thanks, Nick!  Most useful.

Thanks for confirming that masterfiles version should be ahead of binary.   It sounds like I can be bold in aiming for an across-the-board giant leap of masterfiles from current 3.12 straight to 3.21, regardless of binary version (mostly 3.15.3 at present).  Naturally I''ll test!

Thanks for the heads up about "rxdirs".  I had actually done a little "toe in the water" testing yesterday and "cf-agent" had shown me that problem looming on the road ahead!

I observe that the community download page https://cfengine.com/downloads/cfengine-community/ offers RHEL9/CentOS9 for 3.21.0  LTS (good) but not for 3.18.3 LTS.  If 3.18.3 now supports RHEL9, could that page be investigated and rectified, please?  (Because of "rxdirs" I'd rather avoid 3.21 for the moment... there's plenty of other catch-up for me to do prior to that!  So if a 3.18/RHEL9 RPM s available that would be great.)

Meanwhile: a possible "semodule" bug report for 3.21 on RHEL9.  Latest OS (yum upgrade); I had installed before Christmas.  I hit the problem just now; then  did a yum upgrade (lots of packages upgraded), then rebooted, then tried again.  Problem still there.  This is the latest::

--------------------------------------------------------------------------

$ cat /etc/redhat-release
Red Hat Enterprise Linux release 9.0 (Plow)
$
$ sudo yum upgrade
Updating Subscription Management repositories.
Red Hat Enterprise Linux 9 for x86_64 - AppStre  41 kB/s | 2.4 kB     00:00    
Red Hat Enterprise Linux 9 for x86_64 - BaseOS   43 kB/s | 2.4 kB     00:00    
Dependencies resolved.
Nothing to do.
Complete!
$
$ sudo yum install Downloads/cfengine-community-3.21.0-1.el9.x86_64.rpm
Updating Subscription Management repositories.
Red Hat Enterprise Linux 9 for x86_64 - AppStre  38 kB/s | 2.4 kB     00:00    
Red Hat Enterprise Linux 9 for x86_64 - BaseOS   42 kB/s | 2.4 kB     00:00    
Dependencies resolved.
================================================================================
 Package                 Arch        Version            Repository         Size
================================================================================
Installing:
 cfengine-community      x86_64      3.21.0-1.el9       @commandline      2.3 M

Transaction Summary
================================================================================
Install  1 Package

Total size: 2.3 M
Installed size: 8.0 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1
  Running scriptlet: cfengine-community-3.21.0-1.el9.x86_64                 1/1
  Installing       : cfengine-community-3.21.0-1.el9.x86_64                 1/1
  Running scriptlet: cfengine-community-3.21.0-1.el9.x86_64                 1/1
libsemanage.semanage_pipe_data: Child process /usr/libexec/selinux/hll/pp failed with code: 255. (No such file or directory).
cfengine-enterprise: libsepol.policydb_read: policydb module version 21 does not match my version range 4-20
cfengine-enterprise: libsepol.sepol_module_package_read: invalid module in module package (at section 0)
cfengine-enterprise: Failed to read policy package
libsemanage.semanage_direct_commit: Failed to compile hll files into cil files.
 (No such file or directory).
semodule:  Failed!
warning! semodule failed

  Verifying        : cfengine-community-3.21.0-1.el9.x86_64                 1/1
Installed products updated.

Installed:
  cfengine-community-3.21.0-1.el9.x86_64                                        

Complete!
$

--------------------------------------------------------------------------

That mismatch "module version 21 does not match my version range 4-20" doesn't look right.

(As discussed earlier, I actually want to stick with 3.18 for the initial RHEL9 work, because of "rxdirs", but there isn't a 3.18/RHEL9 RPM.)

Thanks again.

-- David Lee

[Nick Anderson]

"t.d…@servicemusic.org.uk" <t.d...@servicemusic.org.uk> writes:

I observe that the community download page https://cfengine.com/downloads/cfengine-community/ offers RHEL9/CentOS9 for 3.21.0 LTS (good) but not for 3.18.3 LTS. If 3.18.3 now supports RHEL9, could that page be investigated and rectified, please? (Because of "rxdirs" I'd rather avoid 3.21 for the moment… there's plenty of other catch-up for me to do prior to that! So if a 3.18/RHEL9 RPM s available that would be great.)

That's because we have not released a 3.18 package for el9. The 3.18.x output I showed you was from nightly builds, so 3.18.4 (hopefully).

Meanwhile: a possible "semodule" bug report for 3.21 on RHEL9. Latest OS (yum upgrade); I had installed before Christmas. I hit the problem just now; then did a yum upgrade (lots of packages upgraded), then rebooted, then tried again. Problem still there. This is the latest::

Yeah, we have noticed some issues with el9 and needing to be updated in order for the package to install. We are still looking at how to best address it.

[craig.c...@northern.tech]

Hi David,

I am working on this right now and have worked up a solution.

Essentially our compiled cfengine-enterprise.pp selinux policy is compiled to a certain policydb version.

Probably the best work-around for you right now is to re-compile the policy yourself from sources in the cfengine core repository.

$ git clone https://github.com/cfengine/core --depth 1 # or use what you have already checked out

$ sudo yum install selinux-policy-devel

$ cd core/misc/selinux

$ make -f /usr/share/selinux/devel/Makefile -j1

$ sudo semodule -n -i cfengine-enterprise.pp

See this pair of PRs for core and buildscripts for my pretty close to done work in progress for solving this in the installer package:

https://github.com/cfengine/core/pull/5146

https://github.com/cfengine/buildscripts/pull/1187

Where I have added a Requires for the version of selinux-policy package that we use when we build the package and similar work-around notes as above if the module fails to import.

Cheers,

Craig

[t.d...@servicemusic.org.uk]

Craig,

Many thanks.  In this context, we are simply end-users.  We'd like to get RHEL9 prepared, but we can probably wait until you have an RPM (assuming the next few weeks).  That said, when you believe your RPM is in a reasonable beta (or release-candidate or pre-release) state let me know and I can give it a try and report back.  This would apply both to 3.21 and to 3.18 backport.

Thanks.

-- David Lee

[craig.c...@northern.tech]

Hi David,

We have just fixed up our nightly community packages for rhel8 and rhel9 and these builds should ierrornclude the fix for selinux policy module import during install.

If the selinux policy included in our package is newer than the version in the OS you will get an error with instructions on what to do. The source for the selinux policy is included in the RPM now so you can build it custom for your system if need be.

For example, on an rhel 9 system that is not upgraded I get:

Error:

  Problem: conflicting requests

    - nothing provides selinux-policy >= 34.1.43 needed by cfengine-community-3.21.1a.921398ce3-25452.el9.x86_64

The best option is to upgrade your system which should ensure that your selinux and kernel are the same or newer than the policy included in our package.

cf-remote can show you the package URL:

$ cf-remote --version 3.21.x list --edition community el9 Available releases: master, 3.21.x, 3.21.0, 3.18.x, 3.18.3, 3.18.2, 3.18.1, 3.18.0 Using 3.21.x LTS: http://buildcache.cfengine.com/packages/testing-pr/jenkins-community-nightly-3.21.x-42/PACKAGES_x86_64_linux_redhat_9/cfengine-community-3.21.1a.921398ce3-25452.el9.x86_64.rpm

or can install if the host has nopasswd sudo access:

$ cf-remote --version 3.21.x install --edition community --clients vagrant@rhel-9

Also available on the website: https://cfengine.com/downloads/cfengine-community-nightly/

Let us know how it goes!

-Craig

[t.d...@servicemusic.org.uk]

Craig,

Many thanks for the update.  Apologies for the delay in acknowledging (I do a three day week, usually Wed-Fri).

I've successfully grabbed the 3.18 and 3.21 RPMs from the "nightly" URL at the end of your email.  Thanks.

With both versions, my "yum install ...." from that file fails quickly at the "selinux" point that you mention.  Yet a subsequent "yum upgrade" says "Nothing to do. Complete!"  A "yum list" reports that our Satellite channel is at version "34.1.29-1.el9_0.2".

Is there a particular reason why your RPMs requires the very, very latest version of that RH "selinux-policy" RPM?  Is there some unavoidably essential feature of it that is absent at 34.1.29?

I know that developers love working on the very latest things.  But could I, as an end-user (not developer) of the product, suggest that the release environment (as perhaps distinct from the development environment) that is packaging these RPMs be accepting of somewhat earlier versions?  End-user sites, such as us, often have very good reasons (local stability, etc.) for deliberately being a little behind latest releases.  And if we're having that trouble, doubtless other sites, too, will also encounter it.

Happy to continue that particular conversation off-list if you wish.  Use my Diamond email address for that.

Anyway, if you could package with a more flexible/generous dependency, that would be useful.

All the best.

-- David Lee

[craig.c...@northern.tech]

Much gratitude to you David for testing things out and getting back.

Yes, I hear you about latest versus stable and agree.

I will do some research to see if we can make our packages a bit more adaptable. SELinux is the real culprit here I think. Either that or our understanding of how to work with it and packaging.

Be well, and hope to get back to you soon,

Craig

[t.d...@servicemusic.org.uk]

Craig,

Update: good news and bad news.

Our relatively conservative Satellite is now offering version 34.1.43 of selinux-policy.  So I have upgraded my test RHEL9 VM to that version (in fact a general "yum update" to include other RHEL9 updates on offer).  I also downloaded the latest nightly builds of both 3.18 and 3.21 community edition.

The good news is that these nightly-build RPMs now install cleanly.

The bad news is that "cf-agent --bootstrap=<local-policyhost-cname>" segfaults very early.  I tried this with four RPMs:  3.18 and 3.21 from latest nightly build and also from a nightly build that I happened to have to hand from Feb 22/23 (i.e. about a month ago).  In all cases it segfaults.   Here is the end of a typical "--bootstrap" run (with added "-d" and "-v"):

-----------------------------------------

   debug: Current umask is 22^M
 verbose: Setting abort classes from ...^M
 verbose: Setting ifelapsed to 0^M
 verbose: ----------------------------------------------------------------^M
 verbose:  Begin policy/promise evaluation ^M
 verbose: ----------------------------------------------------------------^M
 verbose: Using bundlesequence =>  {"main"}^M
 verbose: B: *****************************************************************^M
 verbose: B: BEGIN bundle main^M
 verbose: B: *****************************************************************^M
   debug: DeRefCopyPromise(): promiser:'description'^M
   debug: DeRefCopyPromise():     copying constraint: 'string'^M
 verbose: V:     Computing value of 'description'^M
   debug: Evaluating vars promise: description^M
   debug: V: 'description' => 'NONE'^M
 verbose: A: Promise was KEPT^M
 verbose: P: END meta promise (description)^M
   debug: Evaluating vars promise: description^M
   debug: DeRefCopyPromise(): promiser:'Check Keys'^M
   debug: DeRefCopyPromise():     copying bundle: 'failsafe_cfe_internal_checkkeys'^M
   debug: DeRefCopyPromise():     copying constraint: 'comment'^M
Segmentation fault^M

-----------------------------------------

I also tried (speculatively!) with SELinux permissive (rather than our usual enforcing), but that doesn't help the segfault.

Any known issues?

-- David Lee

[craig.c...@northern.tech]

No known issues that we are aware of. Can you try enabling core dumps and running the cf-support tool to collect a backtrace of the fault?

For your system I would recommend installing systemd-coredump.

I will give it a try in a VM and see if I get similar results.

-Craig

[craig.c...@northern.tech]

I did try with a rhel-9 vm and had no issues with 3.21.1 via cf-remote. Which other packages did you use? Can you send URLs?

outside the vm, on the host:

cf-remote install --edition community --clients vagrant@rhel-9

then on the vm:

sudo su

cf-agent -IB localhost

cf-agent -KIf update.cf

cf-agent -KI

No segfaults sadly. I did notice I have the same version of selinux-policy package you mentioned: 34.1.43.1.el9.

Hopefully you can install systemd-coredump and get some backtraces for us to examine.

-Craig

[t.d...@servicemusic.org.uk]

Craig,

Many thanks.

I've tweaked the machine to produce coredumps.

And because this machine was basically "@minimal_environment" I also added loads of extra RPMs, including even "@development", on the speculative guess that it might have been the absence of something.  But it still coredumps (which is at least consistent).

I  have a core dump (and can produce more), and "cf-support" has produced what looks like a decent tar.gz.  It says "Please send <long-tarfile_anme> to CFEngine support staff."  How can I do that, please?

Meanwhile I had a quick look at the gdb backtrace.  It is:

-----------------------------------------------------------

(gdb) backtrace
#0  0x0000000000000000 in ?? ()
#1  0x00007f4852579b75 in PromiseRuntimeHash ()
   from /var/cfengine/lib/libpromises.so.3
#2  0x00007f4852579eb0 in AcquireLock ()
   from /var/cfengine/lib/libpromises.so.3
#3  0x000000000042097c in VerifyMethod ()
#4  0x0000000000420ee7 in VerifyMethodsPromise ()
#5  0x000000000040e254 in KeepAgentPromise ()
#6  0x00007f4852566663 in ExpandPromise ()
   from /var/cfengine/lib/libpromises.so.3
#7  0x000000000040f3d9 in ScheduleAgentOperations ()
#8  0x0000000000410594 in main ()

-----------------------------------------------------------

-- David Lee

[Vratislav Podzimek]

Hi David, see a comment below.

On Wed, 2023-03-29 at 08:32 -0700, t.d...@servicemusic.org.uk wrote:
> Craig,
>
> Many thanks.
>

> I've tweaked the machine to produce coredumps.
>
> And because this machine was basically "@minimal_environment" I also added loads of extra RPMs,
> including even "@development", on the speculative guess that it might have been the absence of
> something.  But it still coredumps (which is at least consistent).
>
> I  have a core dump (and can produce more), and "cf-support" has produced what looks like a decent
> tar.gz.  It says "Please send <long-tarfile_anme> to CFEngine support staff."  How can I do that,
> please?
>
> Meanwhile I had a quick look at the gdb backtrace.  It is:
>
> -----------------------------------------------------------
> (gdb) backtrace
> #0  0x0000000000000000 in ?? ()
> #1  0x00007f4852579b75 in PromiseRuntimeHash ()

This function uses OpenSSL to calculate hash of a given promise. Can you check `ldd
/var/cfengine/bin/cf-agent` and if all those libraries are OK and in expected locations? On RHEL 9,
CFEngine should use the system-provided OpenSSL libraries.

--
Vratislav

[t.d...@servicemusic.org.uk]

Vratislav,

About a week ago I sent Craig Comstock a core dump off-list.  You might want to contact him.

Meanwhile here's some output:

---------------------------------------------------------------------

$ sudo ldd /var/cfengine/bin/cf-agent
linux-vdso.so.1 (0x00007ffcdc38e000)
libpromises.so.3 => /var/cfengine/lib/libpromises.so.3 (0x00007f1f6ec47000)
liblmdb.so => /var/cfengine/lib/liblmdb.so (0x00007f1f6ec30000)
libacl.so.1 => /var/cfengine/lib/libacl.so.1 (0x00007f1f6ec25000)
libyaml-0.so.2 => /var/cfengine/lib/libyaml-0.so.2 (0x00007f1f6ec03000)
libcurl.so.4 => /var/cfengine/lib/libcurl.so.4 (0x00007f1f6eb78000)
libssl.so.3 => /lib64/libssl.so.3 (0x00007f1f6eac4000)
libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007f1f6e697000)
libpcre.so.1 => /var/cfengine/lib/libpcre.so.1 (0x00007f1f6e653000)
libxml2.so.2 => /var/cfengine/lib/libxml2.so.2 (0x00007f1f6e4e8000)
libpam.so.0 => /lib64/libpam.so.0 (0x00007f1f6e4d6000)
libm.so.6 => /lib64/libm.so.6 (0x00007f1f6e3fb000)
libc.so.6 => /lib64/libc.so.6 (0x00007f1f6e1f0000)
libattr.so.1 => /var/cfengine/lib/libattr.so.1 (0x00007f1f6e1e8000)
libzstd.so.1 => /lib64/libzstd.so.1 (0x00007f1f6e111000)
libz.so.1 => /lib64/libz.so.1 (0x00007f1f6e0f7000)
liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f1f6e0cb000)
libaudit.so.1 => /lib64/libaudit.so.1 (0x00007f1f6e09b000)
libeconf.so.0 => /lib64/libeconf.so.0 (0x00007f1f6e090000)
/lib64/ld-linux-x86-64.so.2 (0x00007f1f6edb7000)
libcap-ng.so.0 => /lib64/libcap-ng.so.0 (0x00007f1f6e087000)
$
$ rpm -qf /lib64/libssl.so.3
openssl-libs-3.0.1-47.el9_1.x86_64
$ sudo yum info openssl-libs-3.0.1-47.el9_1.x86_64
Updating Subscription Management repositories.
Red Hat Enterprise Linux 9 for x86_64 - AppStre 9.6 kB/s | 2.8 kB     00:00    
Red Hat Enterprise Linux 9 for x86_64 - BaseOS  8.9 kB/s | 2.4 kB     00:00    
Installed Packages
Name         : openssl-libs
Epoch        : 1
Version      : 3.0.1
Release      : 47.el9_1
Architecture : x86_64
Size         : 6.4 M
Source       : openssl-3.0.1-47.el9_1.src.rpm
Repository   : @System
From repo    : rhel-9-for-x86_64-baseos-rpms
Summary      : A general purpose cryptography library with TLS implementation
URL          : http://www.openssl.org/
License      : ASL 2.0
Description  : OpenSSL is a toolkit for supporting cryptography. The
             : openssl-libs package contains the libraries that are used by
             : various applications which support cryptographic algorithms and
             : protocols.

$ sudo rpm -qa --last | grep openssl-libs
openssl-libs-3.0.1-47.el9_1.x86_64            Fri 24 Mar 2023 10:47:38 GMT
$

---------------------------------------------------------------------

-- David Lee

[t.d...@servicemusic.org.uk]

Any progress on this yet?  It is still coredumping.

At last, I've been able to build CFE from source on RHEL9.   ("git pull" yesterday, so decently up-to-date.)   On "./configure" I used "--enable-debug-yes" to try to get more information from gdb, and did a "make install".

Running under gdb gives some information additional to what I got above.

(gdb) run -KI --bootstrap=cph-mpf

...

(gdb) bt

#0  0x0000000000000000 in ?? ()

#1  0x00007ffff7eb3785 in PromiseRuntimeHash (pp=0x564800,
    salt=0x7fffffffc800 "method_Check Keys_", digest=0x7fffffff8f40 "",
    type=HASH_METHOD_MD5) at locks.c:675
#2  0x00007ffff7eb3c7a in AcquireLock (ctx=0x46cb50,
    operand=0x7fffffffc800 "method_Check Keys_",
    host=0x7ffff7fa2340 <VUQNAME> "cs04r-sc-vserv-223", now=1685026629,
    ifelapsed=0, expireafter=120, pp=0x564800, ignoreProcesses=false)
    at locks.c:811
#3  0x0000000000425ed7 in VerifyMethod (ctx=0x46cb50, call=...,
    a=0x7fffffffd8a0, pp=0x564800) at verify_methods.c:122
#4  0x0000000000425cb8 in VerifyMethodsPromise (ctx=0x46cb50, pp=0x564800)
    at verify_methods.c:75
#5  0x000000000040fe5e in KeepAgentPromise (ctx=0x46cb50, pp=0x564800,
    param=0x0) at cf-agent.c:1909
#6  0x00007ffff7e98e8e in ExpandPromiseAndDo (ctx=0x46cb50, iterctx=0x57f720,
    act_on_promise=0x40fa13 <KeepAgentPromise>, param=0x0,
    actuate_ifelse=false) at expand.c:224
#7  0x00007ffff7e991af in ExpandPromise (ctx=0x46cb50, pp=0x552cd0,
    act_on_promise=0x40fa13 <KeepAgentPromise>, param=0x0) at expand.c:307
#8  0x000000000040f1ca in ScheduleAgentOperations (ctx=0x46cb50, bp=0x552030)
    at cf-agent.c:1564
#9  0x000000000040ee5c in KeepPromiseBundles (ctx=0x46cb50, policy=0x54c3a0,
    config=0x46c2a0) at cf-agent.c:1474
#10 0x000000000040d8ee in KeepPromises (ctx=0x46cb50, policy=0x54c3a0,
    config=0x46c2a0) at cf-agent.c:939
#11 0x000000000040c54e in main (argc=3, argv=0x7fffffffe618) at cf-agent.c:301

(gdb) frame 1
#1  0x00007ffff7eb3785 in PromiseRuntimeHash (pp=0x564800,
    salt=0x7fffffffc800 "method_Check Keys_", digest=0x7fffffff8f40 "",
    type=HASH_METHOD_MD5) at locks.c:675
675         EVP_DigestUpdate(context, pp->promiser, strlen(pp->promiser));
(gdb) p *pp
$1 = {parent_section = 0x552c30, classes = 0x560c40 "any",
  comment = 0x564df0 "Without a valid keypair we aren't going to be able\n", ' ' <repeats 20 times>, "to establish trust", promiser = 0x5698c0 "Check Keys",
  promisee = {item = 0x0, type = RVAL_TYPE_NOPROMISEE}, conlist = 0x56a8d0,
  org_pp = 0x552cd0, offset = {start = 0, end = 0, line = 71, context = 0}}

(gdb) p (int)strlen(pp->promiser)
$2 = 10
(gdb)

To my totally untrained eye it looks reasonable.

Line 675 is "EVP_DigestUpdate(context, pp->promiser, strlen(pp->promiser));" and "pp->promiser" is a neat string of ten characters.

Thoughts?

-- David Lee

[vratislav...@northern.tech]

On Thu, 2023-05-25 at 08:15 -0700, t.d...@servicemusic.org.uk wrote:
> Any progress on this yet?  It is still coredumping.
>
> At last, I've been able to build CFE from source on RHEL9.   ("git
> pull" yesterday, so decently up-to-date.)   On "./configure" I used
> "--enable-debug-yes" to try to get more information from gdb, and did
> a "make install".
>
> Running under gdb gives some information additional to what I got
> above.
>
> (gdb) run -KI --bootstrap=cph-mpf
> ...
> (gdb) bt
> #0  0x0000000000000000 in ?? ()
> #1  0x00007ffff7eb3785 in PromiseRuntimeHash (pp=0x564800,
>     salt=0x7fffffffc800 "method_Check Keys_", digest=0x7fffffff8f40
> "",
>     type=HASH_METHOD_MD5) at locks.c:675

Oh! If your RHEL 9 machine runs OpenSSL 3, MD5 may not be supported at
all. Or, by any chance, is your machine running in FIPS mode?

CFEngine Enterprise uses SHA256 instead.

--
Vratislav

[t.d...@servicemusic.org.uk]

You mention FIPS.  Yes, this trial RH9 installation is using FIPS.  Nevertheless with RH8 (both with and without FIPS) we have been OK.

I'll try it without FIPS soon.

So it sounds as though CFE has some sort of RH9+FIPS bug (absent from RH8+FIPS) causing a segfault.

Is there a ticket for this bug?

-- David Lee

[t.d...@servicemusic.org.uk]

Apologies for the delay.  At last, I've got our local non-FIPS RHEL9.OS-deployment baseline to an equivalent level to the FIPS-based one I had been previously had to use.

Success!  The bootstrap no longer segfaults, but runs to completion. Then "update.cf" and a couple of runs of the agent policy ("promises.cf") also run satisfactorily.  (A few local policy issues, but that's OK; it's our problem, and to be expected with a new OS release.)

But there still remain two issues that I believe Northern.Tech should regard as bugs:

• the segfault-failure in FIPS mode really shouldn't happen
• I wasn't able to use 3.18.5; rather I had to use an earlier 3.18.4 overnight build

The latter is because it required a very recent "selinux-policy".  We discussed this principle earlier in the thread; see above.   The distributed application RPM should be conservative in what pre-reqs it needs.  Try to ensure your build/distribution are specifically down-level (e.g. six months or even more) from RH "bleeding edge".  Customers need stable OS platforms, even if the applications on it are a little newer, so those applications should be distributed with those stable (older) platforms in mind.

Can someone open the bug reports?  Or do you want me to do so?

-- David Lee

-- Diamond Light Source

[Vratislav Podzimek]

On Fri, 2023-07-07 at 02:17 -0700, t.d...@servicemusic.org.uk wrote:
> Apologies for the delay.  At last, I've got our local non-FIPS RHEL9.OS-deployment baseline to an
> equivalent level to the FIPS-based one I had been previously had to use.

No need to apologize, thanks for the useful feedback/info!

> Success!  The bootstrap no longer segfaults, but runs to completion. Then "update.cf" and a couple
> of runs of the agent policy ("promises.cf") also run satisfactorily.  (A few local policy issues,
> but that's OK; it's our problem, and to be expected with a new OS release.)
>
> But there still remain two issues that I believe Northern.Tech should regard as bugs:

>  * the segfault-failure in FIPS mode really shouldn't happen
Yes, it should be handled in a nicer way with a CRITICAL error saying what's going on. I've just
created a ticket [1] for this.


>  * I wasn't able to use 3.18.5; rather I had to use an earlier 3.18.4 overnight build

> The latter is because it required a very recent "selinux-policy".  We discussed this principle
> earlier in the thread; see above.   The distributed application RPM should be conservative in what
> pre-reqs it needs.  Try to ensure your build/distribution are specifically down-level (e.g. six
> months or even more) from RH "bleeding edge".  Customers need stable OS platforms, even if the
> applications on it are a little newer, so those applications should be distributed with those
> stable (older) platforms in mind.

This is actually quite tricky. We have seen in the past that things didn't work with our SELinux
policy built on systems with an older version of selinux-policy when a newer version was installed.
In other cases we have seen our policy built with new version of selinux-policy not working on
systems that had older version of it. And while we can hardly ask people to downgrade selinux-policy
and re-introducing CVEs that were fixed in the newer version if they want to use CFEngine requiring
an update of the package is less problematic.

To me, this is really a problem of how selinux-policy is packaged and how compatibility, both
forward and backwards, is handled there. But that's a Red Hat's/IBM's problem not ours. We could
probably provide CFEngine packages with all CFEngine processes being unconfined and thus not
affected by SELinux, but that's also non-trivial and completely beats the purpose of using SELinux
on the particular machines.

[1] https://northerntech.atlassian.net/browse/CFE-4226

--
Vratislav

> --
> You received this message because you are subscribed to the Google Groups "help-cfengine" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to help-

> cfengine+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/help-cfengine/c9f8f33e-
> c8fd-48e5-a777-5dad9e7c60b6n%40googlegroups.com.

[t.d...@servicemusic.org.uk]

Vratislav ,

Many thanks.

• segfault:  Thanks for regarding this is a bug and opening a ticket.  Can you make me a "watcher" (or equivalent) there, so I get notified of progress?
  
• "selinux-policy" etc.  OK, fair enough.  I had just never seen this before.  On reflection, this particular development exercise is on more than the usual number of "bleeding edges".   I've managed to make it work by ensuring that my test Satellite channel is also suitable "bleeding edge" which naturally provides an appropriate version of that RPM.  Things will probably settle down, and at least we are aware of it.

All the best.

-- David Lee

[Vratislav Podzimek]

Hi!

On Fri, 2023-07-14 at 03:03 -0700, t.d...@servicemusic.org.uk wrote:
> Vratislav ,
>
> Many thanks.
>  * segfault:  Thanks for regarding this is a bug and opening a ticket.  Can you make me a "watcher" (or equivalent) there, so I get notified of progress?
I couldn't find your user in the JIRA. I'm afraid you'll have to create a user account there and
then use the "watch" icon/menu on the ticket.

>  * "selinux-policy" etc.  OK, fair enough.  I had just never seen this before.  On reflection,

> this particular development exercise is on more than the usual number of "bleeding edges".   I've
> managed to make it work by ensuring that my test Satellite channel is also suitable "bleeding
> edge" which naturally provides an appropriate version of that RPM.  Things will probably settle
> down, and at least we are aware of it.

Yeah, it's been quite painful for us as well. Let's hope it all settles down, like you wrote.

--
Vratislav